Communication Theory, 6:2 (May 1996), 167-187.
Klaus Bruhn Jensen made a provocative argument for locating meaning in a theory of social semiotics in a 1991 Communication Yearbook article in which he attempted to move beyond essentialistic notions of meaning to focus on audiences as cultural agents acting as members of interpretive communities. In a criticism of Jensen's social semiotics theory, Newcomb (1991) observed that various theories (transmission, cognitive, sociocultural) have only provided partial approaches to locating meaning by
"tightroping on whatever line divides structurally determined, socioideological effects from free, multiple, solipsistic, exciting, and potentially useless individual interpretation. A model for resolving tension between the perspectives or for studying the relationship has not emerged."
One reason, perhaps, why such a model has not emerged might be that communication is defined primarily as language based. While language is a powerful model for understanding communication, it has its limits as an explanatory schema because it ignores other forms of nonverbal communication such as visual communication.
This essay will focus on visual communication from two perspectives--one, it will position visual communication as an important and largely ignored platform for a more complex approach to understanding meaning and, two, it will propose that visual communication processes are different from language based communication processes because of the way observation impacts upon thinking. The objective is to apply semiotic theory to visual comunication in order to better explain, not only visual interpretation, but also communication in general.
The model for visual interpretation proposed in this essay sees
visual communication as grounded in perception, extended internally
through cognition and language, and modified externally through
social and cultural frames. Furthermore, it will suggest that
language-based models such as semiology are inadequate to explain
the complexity of visual communication because they lack a theory
to describe perceptual interpretation. Such a theory can be found
in semiotics, particularly in the work of C.S. Peirce.
Semiotics and Semiology
The framework for this discussion of meaning production is signs and how they work. Semiotics, at its most basic, is the study of sign systems--what signs mean and how they relate to one another. The relationship between signs and meaning is one of the most important questions in communication theory and, as Anderson (1993) reminds us, it is "the fundamental relationship upon which all theory and research findings must rest."
The word semiotics is often used to refer to both traditions--the European and the American--but there are differences. The American approach to semiotics as developed by philosopher Charles Sanders Peirce focuses on the logic of meaning and the philosophy of knowledge. The European approach, referred to as semiology, was developed by the Swiss linguist Ferdinand de Saussure and focuses on language as a sign system.
Peircean Semiotics A sign, as defined by Peirce, is "something that stands to somebody for something in some respect or capacity" (1955, p. 99); and again, "A sign is an object which stands for another to some mind" (1991, p. 141). In other words, his triadic model of a sign identifies 1.) the word or image sign that 2.) stands for (in my mind or yours) 3.) some other object or concept. In line with his philosophical focus, the most basic premise of his semiotic theory is that all that we can know is mediated by signs. (Peirce, 1991, p. 141-143)
This very important "stands for" process is the point where meaning is generated both through encoding by the source (production) and decoding by the receiver (reception)--and this relationship is the focus of the process of interpretation. The importance of interpretation is apparent in the definition given by Italian semiotician, Umberto Eco, who further defines a sign as "something that is interpreted." (Eco 1986, p. 15)
Peircean semiotics allows us to talk of all signs, whether language or non-language, as components of all forms of meaning. He explains that, "All the universe is perfused with signs, if it is not composed exclusively of signs." (1991, p. 258) To Peirce signs are not just words and meaning is not necessarily a product of convention or language. (1991, p. 181-183, 251-252; Rosensohn p. 54-56) He developed a tripartite system to classify the complex world of signs as icons, indexes, and symbols.
Peirce defines an icon as similar to its subject; in other words, iconic signs carry some quality of the thing they stand for as a portrait stands for a person. Most often an iconic sign is a representation such as a drawing or photograph where likeness or resemblance is a determining characteristic; early picture writing or pictographs, however, also had iconic elements. An index is physically connected with its object, an indication that something exists or has occurred--a footprint means someone just walked by or smoke that means there is a fire. Symbolic signs, however, arbitrarily stand for something through a process of consensus as a word stands for a concept. A symbol, such as a leaf on a flag, is linked by convention with its object. We learn that a maple leaf stands for the country of Canada. Symbols, therefore, are conventional like most spoken and written words and subject to a more closed than open interpretation process. (Peirce, 1991, p. 239)
Although symbols are arbitrary, icons and indexes are "motivated," that is they are more likely to resemble their object in some way rather than being created by convention. As Fiske and Hartley explain in their book on Reading Television, "the greater the motivation, the smaller the role played by socially based convention; and the weaker the motivation, the more constraining is the convention. (1978, p. 39) Icons and indexes are also more open to interpretation since they are not based on arbitrarily determined meaning relationships but rather on personal experience. Symbols may be complex, but once the meanings are learned, they are less subject to idiosyncratic interpretations. A stop sign is a stop sign, but a rose is not always a rose.
The distinctions between and among icon, index, and symbol are not firm boundaries. In fact, a rose can be an icon (a picture), an index (sign of summer), and a symbol (the War of Roses). Photographs are indexical as well as iconic because they are reality grounded. In communication production and reception, the meaning may shift from one to another as the communication act progresses. For example, a study of Sinead O'Conner's iconoclastic tearing up of the picture of Pope Paul (Shaw, 1994) illustrates how conflict can be generated when an iconic representation is turned into a symbol.
Saussure explained a sign and the process by which it is interpreted as a two-part construct which he described as containing first a sound or image, called a signifier, and the concept for which it stands, called the signified. (Culler, 1976, p. 9) This is similar to Peirce's sign and object. For linguistic-oriented theorists who work in the Saussurian model, the "stands for" relationship is often seen as arbitrary, in other words, the link between the sign, or expression, and what it stands for is understood by convention. (Culler, 1976, p. 10-15) This arbitrariness, which leads to a more closed model of communication, is true in most spoken and written language. Obviously, with language, there is room for interpretation, but the space for idiosyncratic interpretation is more tightly drawn through the process of learned meanings and dictionary definitions.
The concept of open and closed texts originated with linguistic texts but can be applied to visuals as well. A murder mystery, whether a book or a movie, for example, is a closed text where the code is known and cause and effect chains are not open. All the reader does is supply a sense of the rules of the game. "Open" texts, on the other hand, such as most poetry and Finnegan's Wake make complex demands upon the reader. Such texts are seen differently by different readers at different times. The metaphor holds for visual communication as well. The stop sign is a relatively closed text, however, a news photograph, advertisement, a Renaissance painting, or our perceptions of the natural environment are much more open to individual interpretation.
The idea that visuals are primarily open texts and subject to
idiosyncratic interpretation is an important point. As Robert
Rutherford Smith (1980) pointed out in a review of works by Eco
and Sebeok, "It is in the field of open texts, from the work
of Joyce and Brecht to the music of Berio and Stockhausen, that
the important work of the twentieth century is being accomplished."
He pointed out that Eco's notion that the reader plays an important
role in the interpretation of "modern," open texts,
corrects the historic imbalance in communication theory that has
focused on the sender. This essay would add that it corrects another
historic imbalance and that is in the lack of recognition of visual
communication as generating a complex and highly sophisticated
process of interpretation.
Approaches to Meaning
Peircean semiotics is much broader than theories of linguistic processing. Eco's definition of the boundaries of semiotics, for example, includes a variety of natural communication systems from zoosemiotics (animal communication) to biosemiotics (all biological signaling systems). Deely's book, Basics of Semiotics, (1990) follows Peirce's suggestions that semiosis should be divided between cognitive--zoosemiotics (animal), anthroposemiotics (human), and non-cognitive--physiosemios (the broad physical universe), and phytosemiosis (plant life).
In Fiske's critique of Jensen's work (1991), he makes the point that he is a Saussurean scholar and suggests that semiology is a richer arena than Peircean semiotics because it is more "generative." Peirce's theory, however, opens up a much broader interpretation of sign systems in contrast to much of the work of traditional communication theory which is grounded in language. Semioticians are interested in any system that uses signs to create meaning and most argue that communication should includes sign systems other than language. As Bopry reminds us (1994) only a percentage of human communication is verbal, there being a vast amount of communication on the non-verbal level. This is why Newcomb's point about partial approaches to meaning is so insightful.
One of the more important sign systems used to communicate non-verbally is that employed in visual communication where signs are used to stand for experiential knowledge as well as ideas. We perceive information visually from nature and the reality around us; we perceive visual information, as well, from mediated forms of communication. Both forms of visual perception are grounded in observation, a neglected area in most language-based theories of meaning.
Some scholars even argue that movies, television, and advertising are dominated by visuals rather than language. The reality-grounded perceptual process used in understanding these media are largely untutored and mastered through experience rather than education, a fact that apparently makes the process seem suspect and less sophisticated than language-based meaning systems . Almost a hundred years ago, Peirce made a plea that scholars "not be in haste to deride a kind of thinking that is evidently founded upon observation." (1991, p. 258)
Visual communication scholars, however, would argue that driving on a freeway, watching an MTV program, or playing Nintendo involves a very complex set of interpretive skills, even if they are grounded in observation, rather than language. In other words, visual communication is complex and the information processing facility is sophisticated, even if much of it is hard wired or self taught.
The point of this essay is not to argue the superiority of visual
codes systems vis a vis language-based codes. However, the notion
that the visual code is complex and different from language, suggests
we need a different model to understand the various facets of
sign interpretation. The interpretation of visual information,
because it uses more open communication codes, is highly subjective
and that puts more demands on a viewer/observer than on a receiver
of language-based communication which operates within the more
restricted space of convention. It also demands an understanding
of how perceptual and cognitive processes interact.
Perception, Cognition and Convention
Any discussion of visual communication automatically involves some discussion of perception, cognition and the way they relate. In a recent survey of the foundation disciplines used by visual communication scholars, psychology, particularly perception and cognition, was identified as the most important areas of study. (Moriarty, 1995)
Peircian based semiotic theory includes language-based signs as well as non-language--whether auditory, visual, tactile or proprioceptive stimuli (internalized feelings and sensations)--which provides a theoretical platform for understanding a broad range of communication situations. Such an approach parallels perceptual theory which, by its very nature, includes stimuli from all the senses. Figure 1 is a model of a broad-based perceptual process that attempts to illustrate the relationships between perception, cognition, and convention. This is a model that is devloped from one used by Gerald Murch in his Visual and Auditory Perception book (1973). It has been adapted to include the concept of context and convention which was not a part of his original model.
The model indentifies the initial stages in the perceptual process that begin with stimuli which are either effective or ineffective in making impact on the sensory register. The primary activities that comprise sensory registration are attention and selction. The second major step is cognition. The activities of recognition, organization, classification, and discrimination make the bridge from perception, or sensory registration, to cognition, or making sense of incoming data. The next major step in the process is encoding the information into memory, either short term or long term; and the last step is the generation of some sort of response, if response is needed.
What the new version of this model adds is the idea of cognition as the internal process and subject to internal influences, which is represented by the left side of the model; at the same time that convention--as expressed through the social and cultural environment-- operates as an external influence on the cognitive process, which is represented in the model by the activities on the right side of the diagram. In other words, cognitive processing does not happen in a vacuum. It is encapsulated in an environment that includes both internal and external influences.
Perception At the most basic level, the meaning of a sign is internalized by the process of perception--the intersection of our senses with reality-based data as information from the perceived world is "registered." Two important factors are personal observation and individual experience: i.e. understanding shadows means you have experienced sunny spots and shade. We understand a great many things--falling leaves, a track on a sandy beach, a smile or frown--from experiences with reality.
Uexkull's (1981) Umwelt theory, the notion that there is a buffer zone, the "Umwelt," between the internal life of the individual and the outside world, is a perceptual approach that hypothesizes that what we know of the outside world is what we are told by our senses. In Uexkull's philosophy, this "telling" consists of signs communicated to us by our senses--a perceptual parallel to Peirce's interpretant. These signs, then, are the only reality we know and "the rules and laws under which these signs and sign process communicate themselves to our mind are the only true laws of nature."
It is fashionable to dismiss this element in communication dubbing it the "brute sense data," physicalist, or essentialist position. The point, however, is not that the natural sign carries its own essential meaning, but that interpretation can happen without language. As in the model in Figure 1, internalized perception is the ground; cognition and convention are the superstructure.
Larry Gross (1985) proposes categories of perception as sensory-based knowledge that are immediately processed to some point of meaning without being translated into words. Gross sorts signs into those that are natural and signs that have communicative intent. He describes nonsign events as events that we ignore or code "transparently," i.e. we know what they mean without any conscious awareness of interpretive activity, similar to what happens when we drive on a busy highway.
The point is that language follows sensory information. Danish scholar Torben Kragh Grodal (1994) explains that, "We learn about the world and our bodies by 'analog' sensation, and natural languages work on top of this information." Furthermore, language learning is not a good model for how visual perception operates. Grodal insists that recent research in visual perception "does not substantiate a portrayal of visual information by means of 'language'-like learning of an arbitrary visual language." In other words, perceptual research supports the view that visual information is processed "by means of some ecological conventions, not by means of totally cultural conventions."
As Arnold hypothesizes in his preface to the Cherwitz and Hikins book (1986, p. ix), the human perceptual and cognitive apparatus is "programmed to see the world directly." This view reflects that of Messaris who develops this point in his book, Visual Literacy, (1994) but stands in opposition to the work of such theorists as Nelson Goodman (1968) who believe that all visuals are interpreted through convention. In other words, Arnold and Messaris believe that the human perceptual system is largely "hard wired" although it can be developed more or less depending upon one's environment as Bill McKibbon's book, The Age of Missing Information , (1993) points out. Some scholars also speculate that this is the kind of information that is being lost as we approach Baudrillard's simulacra (1988) where everything is presumed to exist in hyper-reality or virtual reality.
The idea of non-verbal perception of meaning also contrasts with the work of language-based theorists who believe that converting perception into knowledge happens only through language. (Cherwitz and Hikins, 1986) Visual communication scholars, however, argue that it is possible to apprehend signs and make sense of them without the mediation of language. Their training in perception leads them to believe that we naturally make sense of visual information without language. Language is used for interpreting some kinds of information--particularly abstract and theoretical concepts--but a great deal of what we process visually is managed without stopping to find a word for it.
In both semiotics and perceptual theory, the individual actively detects or constructs the surrounding world using all senses. Gibson's discussion (1966, 1979) of perceptual systems illuminates the notion of sensory redundancy--we experience reality through as many senses as possible. This perceptual cross-over is another factor that helps to explain the openness of the interpretation process, as well as the ability of people with damaged sensory systems (blind, deaf, etc) to compensate for the missing sense. Gibson's notion of selection is that the decision to see or not see is made by the perceptual system, not the brain.
Cognition Because perception is active, the individual selects the information and modifies it depending upon the individual's previous experiences. It takes repeated observations for us to make sense of the patterns around us and that is where perception interacts with cognition through the processes of recongition, organization, and discrimination. In other words, visual interpretation involves both the eyes and the brain: what we understand is moderated by what we know or have experienced in the past and how we have made sense of these experiences and recorded them in memory. .
Gibson's (1966, 1979) theory of the active or mediated perceptual process posits an interpretive step that intervenes between sensation and cognition. The signs sent to the brain by the various senses during the investigation of a stimulus will be synthesized by the brain and a new sign will be created, a far more active position than that voiced by behaviorists, and one more in keeping with the Peircean concept of an interpretant.
The interpretation of a sunset or a shadow, however, is more than just sensation or recognition because it also involves the manipulation of mental models. We have a model in our mind for "sunny" and "shade" and we make sense of things by comparing them with these mental abstractions or schemas. This happens largely independent of language, although language can be used to anchor these perceptions with a label and associate them with other concepts that come to mind as a word rather than a situation.
For example, Biederman (1990, p. 41-41) recommends the following experiment in visual cognition: close your eyes and change channels on your television set with the remote control. Every time you hear a click open your eyes and then close them immediately. He has found from experiments that most people will have little trouble identifying and interpeting the image and notes, "In a 100-millisecond exposure of a novel scene, people can usually interpret its meaning...and recognize a pattern in a single glance." This is also a good illustration of the gestalt nature of visual interpretation. You don't have to think the word /rose/ to interpret meaning from a picture or a real flower in nature. Our minds process information too fast to stop and put verbal labels on everything, so interpretation must be happening independent of language processing.
Convention Meaning can be internalized as is much of what we learn through visual processing of reality-based information, but it can also be socially or culturally driven which creates an externalized dimension in interpretation. In other words, much of what we know that is language or code based including most visual symbols, is derived from social learning. For example, most of us are taught that the little squiggles on a piece of paper are letters and they can be combined to make words and sentences; likewise the way we dress is derived from a socially determined code of fashion. This is where Peirce's notion of interpretive communities or communities of inquirers is useful. [1955, p. 255-257; Delaney, p. 14, 77-81] This suggests a continuum of interpretation factors moving from internal to external. So not only do we understand things using both the eyes and the brain, we understand things using internally derived information learned from experiences combined with externally based conventions.
The Matrix When the factors of perception and cognition are seen on a continuum, along with individualized and socialized sense-making on another continuum, a matrix can be constructed that identifies the key factors in the visual interpretation process. This makes it possible to locate the type of interpretative strategies being used to make sense of something we see.
In the upper left of the Matrix depicted in Figure 2 are activities that represent internalized perceptions such as the fear that arises when walking on a dark street on a dark night. The fight or flight response is deeply embedded in our psyches and most people don't have to be taught to respond with anxiety to such a situation. In the upper right are perceptual activities that represent individualized understanding, such as the appreciation of a dramatic sunset. This perception is based on experience with sunsets, combined with notions based on personal experience that this one is unusual and worthy of a moment's pause. In the lower left are socially driven perceptions such as the identification of a hairstyle or piece of clothing as attractive. In the lower right is the understanding and appreciation of complex social events such as religious services or football games.
Since the interpretative logic is built on a set of continua, specific activities can be located at different relative positions on the map. In line with reception and postmodern theories, different people's response to the same stimulii can also be located at different points on the map.
Representational Communication Mediated communication combines both perceptual, cognitive, and conventional responses in the interpretive process. There is less agreement, however, on how that mix is managed. Fiske and Hartley (1978, p. 16), for example, claim that television is an extension of spoken language and "is itself subject to many of the rules that have been shown to apply to language." Most visual communication scholars, as well as instructional media experts, would probably find television communication to be equally as grounded by visual communication processes, if not more so.
This difference in viewpoint comes from Fiske and Hartley's admitted dependence on language-based theories of communication. They explain, "Both language and television mediate reality: there is no pristine experience which social man can apprehend with the culturally determined structures, rituals and concepts supplied to him via his language. Language is the means by which men enter into society to produce reality." (1978, p. 17)
A similar disagreement shows up in the discussion of television and filmic codes. Understanding shadows in a film also means that you can equate the representation with your lived experience--again reflecting the indexical element of sign interpretation. This point is carefully developed by Messaris in his book, Visual Literacy, (1994) in which he deconstructs various aspects of the filmic code to determine the reality base behind what are often considered to be conventions. In other words, his view of the filmic code is that it is less a conventionally determined code than an experiential one. Fiske and Hartley, however, say television conventions such as a dissolve are more accurately described as arbitrary; they claim the arbitrary nature is "disguised" by the apparent natural iconic motivation. (1978, p. 40) Messaris would say the opposite is true and the convention disguises the true reality-grounded nature of the image.
The need for reality based perceptual experiences is also true
for figure/ground perception, which is the foundation for selection.
Likewise, depth perception also draws on lived experiences with
things like horizons and shadows in order to understand perspective,
but it is further affected in illustrations by such conventions
as the technique of representing distance by the convergence of
lines at the assumed horizon or vanishing point.
So far we have discussed the basics of semiotics and how meaning of signs is constructed through a complex interaction of perception and cognition as well as individual and conventional factors. Now let's consider the concept of interpretation, particularly as it relates to visual communication.
Peirce (1991, p. 7, 28-19) developed a model of meaning interpretation that provides more breadth in analyzing the "stands for" relationship than a strictly arbitrary approach and is more in line with the notions of experiential and perceptual knowledge. Because Peirce is first of all interested in cognitive philosophy, the focus of his epistemological theory is on how signs are logically or semiotically linked to their objects. The logical relationship between the sign and "the something" it stands for--what we will call here the "concept"--is identified as the interpretant.
It was mentioned earlier that the sign/object/stands for relationship is the basis of Peircean semiotics. Peirce's concept of interpretant explains how this "stands for" relationship creates meaning. Interpretation is an area that is largely ignored in more general communication theories including qualitative research where it would provide a useful conceptual grounding.
As Peirce explains, the interpretant "creates in the mind of that person an equivalent sign, or perhaps a more developed sign." (1955, p. 99) By interpretant he means the idea ("something it stands for") as it is decoded or a subsequent thought to which the sign gives rise. (1991, p. 34) In other words, a thought is a sign requiring interpretation by a subsequent thought in order to achieve meaning.
For example, the word /dog/ and a picture of the animal both stand for some idea or concept of "dogness." In addition, there are interpretations imposed on this concept of dogness based on our personal experiences and also on additional information and description that accompanies the sign. Other verbal interpretants for the word /dog/ could be puppy, bitch, hound, or faithful companion or even frankfurter. Each further interpretation elaborates on the concept and make it richer--i.e. it extends the interpretation by introducing referents that we carry within our memories. (Sebeok, 1991, p. 18-20)
Interpretation means that every interpretant, besides translating the immediate object or the content of the sign, also increases our understanding of it in new ways. Peirce calls this the criterion of interpretability and explains it as a two-step function that involves both translation and extension, a process that leads to an infinite chain of signification. He explained that the meaning process--finding the signified--is an infinite process of interpretation. The idea of an endless chain of signification is what makes Peircean semiotics such an open system of meaning construction. The phrase "unlimited semiosis," another term to describe this form of open interpretation, is credited to Umberto Eco (1979).
In peirce's triad of term/proposition/argument, he maintained that any term, word or phrase is a rudimentary proposition. Eco uses the example of the word /father/ which, at the point of utterance or visual presentation produces a two-argument predicate: if father, then someone who is a child of this father--i.e. the proposition 'every father has or has had a child' opens up extensional meanings. "In truth," Eco concludes, "the sign always opens up something new. No interpretant, in adjusting the sign interpreted, fails to change its borders to some degree." (Eco, 1986 p. 43-44)
This process works because our memories, which provide the foundation for the meaning process, are, as Sebeok puts it, "reservoirs of interpretants." (1991, p. 131) In fact, Sebeok and Eco have both arrived at the idea of interpretation as being similar to using an encyclopedia. In contrast to a dictionary which just gives formal definitions, an encyclopedia extends a concept with historical meaning as well as a richer description. Opening an encyclopedia is the semiotician's metaphor for how the mind works in the process of interpretation. Fry and Fry explain that in decoding a text, the audience member must draw selectively from an encyclopedia of socially stored constructs that help to make sense of the text (1983, p. 453)
Embedded in the concept of the interpretant, then, is the idea of a base of knowledge and experience that contributes to this process of creating rich description and imagery, which is basic to visual communication processes. As Burgin (1983) explains in his discussion of how we interpret photographic images, "We cannot choose what we know, and neither can we choose what part of our dormant knowledge will be awakened by the stimulus of an image, reciprocally reactivated and reinforced by it."
Burgin doesn't refer specifically to Peirce's notion of the interpretant but he describes a similar concept, the "pre-text," that comes into play at the point of visual interpretation, "when we look at an image it is instantly and irreversibly integrated and collated with the intricate psychic network of our knowledge." In other words, responses such of 'racist' or 'sexist' are not in the photographs themselves, but rather they are "a complex of texts, rhetorics, and codes woven into the fabric of the popular pre-conscious. It is these which are the pre-text.."
Prior knowledge as a base for sense making points to another important
aspect of interpretation--the role of the audience. As Burgin
suggested and in keeping with reader reception and poststructuralist
theories, meaning lies with the audience rather than with the
text, whether visual or language based. Fry and Fry explain, "Texts
are made meaningful through a process of audience signification."
In other words, "the interpretant resides in the mind of
the person; it is not embedded in the sign itself." (1983,
p. 448) Because of the way the interpretant functions in sign
interpretation, the responsibility for signification lies with
the audience rather than with the text and that is true for visual
texts as well as written or spoken texts, but particularly important
for visual texts. That doesn't mean, of course, that the source
has no role in encoding symbols. However, the important notion
here is that the critical interpretation step happens solely in
the mind of the receiver and with visual information that process
of interpretation is far more open than in the interpretation
of verbal texts. In other words, there is far more opportunity
for aberrant interpretation that is decidely different from the
communication intended by the source. This is a phenomenon that
has been investigated at length by Rune Pettersson (1993, 1994,
1995) , among others.
The Logic of Interpretation
Peirce's primary contribution to cognition--and only incidentally to communication--was his proposition that "all thinking is the inferential interpretation of signs." (1991, p. 11) Peirce's statement, as well as Gross and Worth's notion of symbolic sign interpretation as a process of inference, introduces the idea that thinking based on inference is critical to interpretation. So we have at the base of Peircean semiotics a thinking process based on inference that results in the interpretation of signs. This is how semioticians approach semiotic interpretation of texts. They deconstruct the signs as cues and infer the different types and levels of meanings contained in the links between the cues. This semiotic process of interpretation is similar to that used by individuals to make sense of visuals and the visual environment.
The complexity of this type of inferential interpretation is illustrated in the use of an upscale car in an advertisement to stand for the flaunting of achievement at a class reunion. In order to make sense of such a message, a semiotician would have to recognize the car and the fact that it is associated with accomplishment, as well as the school and the fact that it is cueing a class reunion, as well as all the emotional/aspirational contexts surrounding the reunion situation. Those meanings are not stated explicitly anywhere in the ad; all the interpretation happens through inference from visual cues.
The process of interpretation involves more than simple inference,
however. Eco (1979, p. 158) suggests that audience members do
not merely "decode" a text, but they also go through
a process of "synthetic inference" which involves both
denotative and connotative processes. In denotation the primary
content is indicated (realism, representation) and in connotation
the audience's extensions and elaborations (associations, attitudes,
emotions) are engaged. In other words, as Fry and Fry explain,
(1983, p. 447- 448) in this complex inferential process where
information is being actively synthesized, the involved audience
extends as well as decodes the meaning as two parts of the interpretation
Peirce's Abductive Reasoning
One way to understand how interpretation works is to analyze the logical processes by which create inferences and make sense of things. It is important to realize that Peirce was a logician and he was more concerned with an analysis of thinking than language. Necessarily, then, his understanding of how signs operate would be reduced to a logical form, one which he calls abduction (also hypothesis and retroduction).
Peirce made a provocative statement that "logic, in its general sense, is...only another name for semiotic, the quasi-necessary, or formal, doctrine of signs." (1955, p. 98) Clearly he believes that the way people make sense of signs through logical, quasi-formal reasoning, in the sense that it follows a predictable form. Abduction is Peirce's addition to the traditional analysis of logic which, since the time of Aristotle, has been based on inductive and deductive forms.
Peirce's abduction is an inferential process of fashioning an hypothesis to account for a state of affairs; the more "clues" that are available or conditions that are known, the more likely the hypothesis will lead to a truthful conclusion. (1955, p. 150-156; 1957, p. 126-143, 235-255; Davis, 1972, p. 22-49) In order to accumulate the clues, however, the abductive process must begin with observation and that is why abductive reasoning relates so well to the visual communication process. Becker's (1978, p. 42) description of making sense of television is similar to Peirce's abduction: We don't tend to turn it on and watch through a program instead we watch it in fits and starts and we get bits and pieces of all sorts of messages that we put together to create our own messages, our own pictures of reality.
Peirce described the formation of an abductive hypothesis as 'an act of insight,' the idea coming to us "like a flash." (5.181) Abductive reasoning then assembles the observations and attributes a variety of characteristics or conditions to a subject until a match is made and an hypothesis or conclusion can be stated.
The student on my class record is white, 24 years old, female, and goes by the name of Mary Morris
The person I am talking to has all those characteristics
Therefore the person I am talking to is Mary Morris
(A is x, y, and z; B is x, y, and z; therefore A is B)
Peirce described this thinking process as a syllogism based on hypothesis (1991, p. 61, 181). He explains how"the whole series of mental performances between the notice of the wonderful phenomenon and the acceptance of the hypothesis" happens.
...a conjecture arises that furnishes a possible Explanation, by which I mean a syllogism exhibiting the surprising fact as necessarily consequent upon the circumstances of its occurrence together with the truth of the credible conjecture, as premisses. (1991, p. 269)
Hoopes refers to this thinking process in both abduction and induction as "statistical inference," which relies more on hypothesis testing or educated guessing than logically derived proof statements as in deduction. (1991, p. 85) In semiotics, Eco and Sebeok refer to a similar thinking process which they call the conjectural paradigm.
This notion of the senses as active detectors in reasoning is similar to the metaphor of the semiotician as detective, one that Sebeok and others have written about, a useful metaphor in understanding how abduction operates. (Sebeok and Sebeok, 1983, You Know My Method.) Nowhere is the power of abductive thinking made clearer than in Eco and Sebeok's book, The Sign of Three, (1983) which compares Peirce's approach to thinking with the reasoning of two great detectives Conan Doyle's Sherlock Holmes and Poe's Dupin. The reasoning approach used by detectives in crime fiction is very similar to the interpretive process of abduction which has been described as "educated guessing toward a hypothesis." In both cases the process begins with observations and then proceeds in a back-and-forth process of developing hypotheses and comparing the observations with information known and filed in memory. This has been described as an exercise in massive parallel processing which is not very language like (Grodal, 1994).
In order to better understand the difference in logical operations, let's compare abduction with induction and deduction. In Aristotelian logic deduction moves from a general rule to specific cases and induction moves from specific cases to a generality. Modern logicians believe this is an inexact way of distinguishing the two and instead base the difference on the degree of confidence contained in the conclusion.
In deduction the truth of the premises ensures that necessarily the conclusion must also be true; in other words, knowing that the premises are true gives us a sufficient reason for believing the conclusion, i.e. we infer a conclusion.
If it's snowing , the streets are slippery
If the streets are slippery, then there will be accidents
If it's snowing, there will be accidents
(If A then B, If B then C; therefore, if A then C).
With induction, however, the conclusion makes some prediction on the bases of several cases each of which make certain claims about the same subject
An apple is a fruit and it grows in the summer
An orange is a fruit and it grows in the summer
A melon is a fruit and it grows in the summer
Therefore fruits grow in the summer
(X is B and C, Y is B and C, Z is B and C, therefore all B are C ).
With induction we can eventually determine the conclusion with a high degree of certainty by further observations. The phrases "probably" or "most likely" cue an inductive form of reasoning. (Barker, 1974) Inductive reasoning is used heavily in scientific research where experiments are used to test a theory.
Peirce theorized that semiotic interpretation (abduction) takes a different form of logic than induction and deduction, although induction and abduction are closely related. (CP 1.65-68; 2.96-97; 5.145, 181; 7.97, 202-207; 1972, p. 22-49.) Abduction and induction are similar in that both seek a theory--i.e. an hypothesis. In contrast, deduction starts with a theory so it relies more on formal reasoning than hypothesizing. Peirce explains the difference between the two,
"Abduction makes its start from the facts, without, at the outset, having any particular theory in view, though it knows a theory is needed to explain the facts. Induction makes its start from a hypothesis without at the outset having any particular facts in view though it feels the need of facts to support the theory. Abduction seeks a theory. Induction seeks for facts." (Sebeok and Sebeok, 1983, p. 24-25)
Technically speaking only deduction uses a formal syllogistic logic based on interlocked premises that lead to a conclusion. ( Gensler, 1989). Induction and abduction both operate with cases or characteristics that suggest a generalization or an identification. For that reason, some logicians suggest that formal logical reasoning, or syllogistic reasoning, can only be deductive. Peirce in his explanation of abduction, however, used a "stripped down" formula that eliminated the multiple cases and characteristics of induction and abduction in order to illustrate their operations in terms of the more familiar deductive syllogism. (1957, p. 126-130) His classic analysis of the differences in the inferential structures is as follows:
Deduction: Rule: All the beans from this bag are white
Case: These beans are from this bag
Result: These beans are white
Induction: Case: These beans are from this bag
Result: These beans are white
Rule: All the beans from this bag are white
Abduction: Rule: All the beans from this bag are white
Result: These beans are white
Case: These beans are from this bag
This formulaic reduction has had the unfortunate consequence of obscuring the relationships between the abductive premises and the conclusion. The confusion is created by calling the second line a "result" when the objective of the form is to arrive at a conclusion, rather than a case, based on some result (i.e. a murder in a detective story). The consequence of this confusion is apparent in various replications of this schema by other scholars which manipulate Peirce's abduction pattern in order to try to get a better "fit" with the underlying logic as they understand it. Mick (1986), for example, developed a model of abductive thinking that rearranged the first two premises in Peirce's schema:
Result: This grad is successful
Rule: since all successful grads....
.:. Case: therefore the car must be expensive
Mick's syllogism starts with the result then moves to the rule rather than following Peirce's model which starts with the rule and moves to the result; however, both end with the case. Mick's confusion may probably stems from his analysis of how the deductive model works. Peirce describes how this reversal can happen:
"the first premise is not actually thought, though it is in the mind habitually. This, of itself, would not make the inference unconscious. But it is so because it is not recognized as an inference; the conclusion is accepted without our knowing how. (1991, p. 208)
Peirce also saw abduction as a more "creative" form
of reasoning than deduction and, to a lesser extent, induction
and referred to the abductive conclusion as a "flash of insight."
(CP 5.181) His phrasing is similar to the characterization of
ideation by modern day creativity scholars as leading to an "aha"
response, a link that ties abductive thinking to the gestalt cognitive
approach. In discussing this aspect of abduction, Davis (1972,
p. 48) referred to it as "a creative leap of the mind"
and noted how people who have arrived at a conclusion through
abduction will often exclaim, "I see!" Creative people
in advertising, for example, refer to the process they use to
come up with ideas as visualization--i.e. making an idea come
to life by expressing it in a visual form, which can, of course,
include words as well as graphics.
To reinforce the notion that abduction provides a logical explanation for visual interpretation consider that a detective "detects" things and an inspector is someone who "inspects," or looks. Abduction begins with observation--and the observations are usually visual. That is the same process used by viewers to make sense of a picture, illustration, or film.
Consider how a semiotician develops an expert reading of something like the previously discussed car ad and compare that with the interpretation process you employ personally in making sense of such a visual. If Eco, Sebeok, and Burgin are right, then you will observe different cues which bring forth information and associations from your own mental filing cabinet (car is expensive, building is a high school, class reunion is the time to show off your success) and you will develop hypotheses about how these various details relate until the ideas come together to mean something. In other words, the interpretative method of semiotics based on abduction, is very similar to how we process visual information. And that is why semiotics and the notion of abduction can be such a useful tool in visual communication research and theory building.
Nature can also be seen as a system of coded signs and Eco makes the argument that the roots of abduction (semiotic interpretation) lie far back in time with hunters and trackers who could read the signs of nature, much as Sherlock does. Medicine, in its procedures for detecting symptoms is another foundational area of abductive thinking. It is significant that Conan Doyle was trained in medicine and used one his medical professors as the model for Sherlock Holmes.
Christian Metz in his important work "The Perceived and the Named" (1980) described how visual interpretation takes place in film through two processes he described as montage and decoupage. Montage describes how we make sense of relationships between objects, as well as internal composition and sequencing. Decoupage is the assembly of pertinent traits needed for perception and classification; these pertinent traits of code recognition were also described by Eco. He noted that visual recognition is based on certain perceptible traits of the object and its images. Metz's point was that we don't need the whole image or the totality of all the details to recognize something because these key traits retain enough of the schema for perception and cognition. As we work through the processes of montage and decoupage, we are using cues--bits and pieces of visual information--to construct hypotheses which eventually lead to a sense-making experience; i.e. we arrive at some meaning interpretation.
Whether it be interpreting clues to solve a crime, analyzing symptoms to diagnose an illness, tracking an animal through the forest, or making sense of an MTV video, signs have to be interpreted and many of those signs are visual.
Because of the resemblance factor for icon interpretation and the experience factor for index interpretation, the formal training needed may be less than that needed for language interpretation--although the life experience may be even more demanding. In other words, as Paul Messaris book on Visual Literacy (1994) so clearly articulates, the visual communication process operates relatively untutored in our society, at least in comparison to language, and that suggests that visual interpretation may call for a different set of interpretative skills that are more intuitive than conventional, the type of skills used in abductive thinking.
Given the lack of clearly defined arbitrary codes in visual communication and the critical role of observation in making sense of indexical and iconic signs, we may speculate that the visual interpretation process may demand even more of the individual than the verbal interpretation process. That's also true of symbolic visual signs--even though they are arbitrary, their meaning is rarely taught in any formal way.
Visual and verbal communication differ in their interpretive processes since language interpretation is more involved with manipulating a conventionally learned code and visual communication involves observations which lead to hypotheses about meanings. Visual interpretation, like the interpretation of semioticians, is more appropriately described asabductive begining with observing clues in the visual (perception) and moving to a conclusion by hypothesizing relationships and patterns (cognition, convention) through massive parallel processing to arrive at a unified gestalt.
Semiotics provides tools in its concepts of an interpretant and
abductive thinking to better understand how the open systems of
meaning employed in visual communication are communicated and
interpreted. And a better understanding of visual communication
can contribute to a more complete understanding of how communication
in general operates.
The focus of this essay is on how viewers understand a visual
and interpret its meaning. The central focus of this discussion
is interpretation--how meaning is arrived at--and this paper will
make the argument that Peirce's notion of abductive reasoning
provides a useful theoretical frame in which to analyze visual
interpretation. It is the thesis of this paper that a different
kind of interpretive logic operates for visual communication processes
than for language-based communication processes and this logic
is best articulated in the semiotic literature where the concept
of interpretation is more carefully conceptualized.
Sandra Moriarty is a professor in the School of Journalism and
Mass Communication at the University of Colorado, Campus Box 287,
Boulder CO 80309.
Anderson, James A. (1993). "The Role of Interpretation in Communication Theory," inVerbo-Visual Literacy: Understanding and Applying New Educational Communication Media Technologies. Nikos Metallinos, ed. 1993 International Symposium of International Visual Literacy Association, Delphi, Greece.
Arnold, Caroll C. (1986). Preface to Communication and Knowledge: An Investigation in Rhetorical Epistemology, by Riachrad A. Cherwitz and James W. Hikins. Columbia SC: Univ. of South Carolina Press.
Barker, Stephen F. (1974). Elements of Logic,. NY: McGraw-Hill.
Baudrillard, Jean. (1988). The Ecstacy of Communication. New York: Semiotext.
Becker, Samuel. (1978). "Visual Stimuli and the Construction of Meaning." In Visual Learning, Thinking, and Communication. Bikkar Randhawa and William Coffman, eds. New York: Academic Press.
Biederman, Irving. (1990). "Higher-Level Vision." In Visual Cognition and Action, Daniel Osherson, Stephen Kosslyn, and John Hollerback, eds. Cambridge: The MIT Press.
Bopry, Jeanette. (1994). "Visual Literacy in Education: A Semiotic Perspective," Journal of Visual Literacy, 14:1, 35-49
Burgin, Victor. (1983). "Seeing sense," in Howard Davis & Paul Walton, eds. Language: Image and Media, New York: St. Martin's Press.
Cherwitz and Hikins. (1986) Communication and Knowledge: An Investigation in Rhetorical Epistemology, by Richard A. Cherwitz and James W. Hikins. Columbia SC: Univ. of South Carolina Press.
Culler, Jonathan. (1976). Ferdinand de Saussure. England: Penguin Books.
Davis, William H. (1972). Peirce's Epistemology. Martinus Nijhoff:The Hague.
Deely, John. (1990). Basics of Semotics. Bloomington IN: Indiana University Press.
Delaney, C. F. (1993). Science, Knowledge, and Mind: A Study in the Philosophy of C.S. Peirce. Notre Dame IN: University of Notre Dame Press.
Eco, Umberto. (1979). A Theory of Semiotics. Bloomington: Indiana University Press, Midland Book Edition.
Eco, Umberto. (1986). Semiotics and the Philosophy of Language. Bloomington: Indiana University Press, Midland Book Edition.
Eco, Umberto and Thomas A Sebeok, ed. (1983). The sign of Three. Bloomington: Indiana University Press.
Fiske, John. (1990). Introduction to Communication Studies. London: Routledge, 2n ed.
Fiske, John. (1991). "Semiological Struggles," Communication Yearbook 14, pp. 33-39
Fiske, John and John Hartley. (1978). Reading Television, London: Methuen & Co.,
Fry, Donald L. and Virgina H. (1983). "A Semiotic Model for The Study of Mass Communication." in M. McLaughlin, ed. Communication Yearbook 9: p. 443-462.
Gensler, Harry J. (1989). Logic: Analyzing and Appraising Arguments, Englewood Cliffs NJ: Prentice Hall.
Gibson, J.J. (1979). The Ecological Approach to Visual Perception. Boston: Houghton Mifflin; (1966) The Senses considered as Perceptual Systems. Boston: Houghton Mifflin.
Goodman, Nelson. (1968). Languages of Art. Indianapolis, IN: Bobbs-Merill.
Grodal, Torben. (1994). "Visual Communication as Textsigns and as Embodied Mental Processes," Nordic Visual Communication Conference, Oslo.
Gross, Larry. (1985)."Life vs. art: The Interpretation of Visual Narratives," Studies in Visual Communication, 4, 2-11.
Hoopes, James. (1991). Peirce on Signs. Chapel Hill NC: The University of North Carolina Press.
Jensen, Klaus Bruhn. (1991). "When Is Meaning? Communication Theory, Pragmatism, and Mass Media Reception." Comunication Yearbook 14, pp. 3-32
Messaris, Paul. (1994). Visual Literacy: Image, Mind, & Reality. Boulder CO: Westview Press.
Metz, Christian. (1980). "The Perceived and the Named." Studies inVisual Communication 3, 56-68.
McKibbon, Bill. (1993). The Age of Missing Information. New York: Dutton.
Mick, David G. (1986). "Consumer Research and Semiotics: Exploring the Morphology of Signs, Symbols, and Significance." Journal of Consumer Research. September, 196-212.
Moriarty, Sandra. (1995). "Visual Communication Theory: A Search for Roots." Visual Communication Conference 9, Flagstaff AA, June, 1995.
Murch, Gerald. (1973). Visual and Auditory Perception. New York: Bobbs and Merrill.
Newcomb, Horace M. (1991). "The Search for Media Meaning," Comunication Yearbook 14, pp. 40-47.
Peirce, C.S. (1931-35). Collected Papers, Vols. 1-6, Charles Hartshorne and Paul Weiss, eds. Cambridge: Harvard University Press.
Peirce, C.S. (1958). Collected Papers, Vols. 7-8, Arthur W. Burks ed. Cambridge: Harvard University Press.
Peirce, C.S. (1955). Philosophical Writings of Peirce. Justus Buchler, ed. New York: Dover Publications.
Peirce, C.S. (1957). Essays in the Philosophy of Science. Vincent Tomas, ed. New York: The Liberal Arts Press.
Peirce, C.S. (1991). Peirce on Signs. James Hoopes, ed. Chapel Hill NC: University of N. Carolina Press.
Pettersson, Rune. (1993) Visual Information. 2nd Ed. Englewood Cliffs, NJ: Educational Technology Publications; "Associations From Pictures." (1995) In Darrell G. Beauchamp, Roberts A. Braden, Robert E. Griffin. (Eds.) Imagery and Visual Literacy. Annual Conference Proceedings; "Comprehensibility." (1994) In Nikos Metallinos, (ed.) Verbo-Visual Literacy. 1993 International IVLA Symposium, Delphi Greece.
Sebeok, Thomas A. (1991). A sign is just a sign. Bloomington: Indiana University Press.
Sebeok, Thomas A. (1981). The Play of Musement. Bloomington: Indiana University Press.
Sebeok, Thomas A. (1983). "One, two, three spells U B E R T Y," in Umberto Eco and Thomas A. Sebeok, eds. The sign of three. Bloomington: Indiana University Press.
Sebeok, Thomas A. and Jean Umiker-Sebeok. (1983). "You know my method." in Umberto Eco and Thomas A. Sebeok, eds. The sign of three. Bloomington IN: Indiana University Press.
Sebeok, Thomas A. (1991). A Sign is Just a Sign. Bloomington IN: Indiana University Press.
Shaw, David. (1994). "Getting to the Pulp of It: Sinead O'Connor and the Iconoclastic Pulpit." Speech Communication Association, New Orleans.
Robert Rutherford Smith, 1980), "Review of Umberto Eco's The Role of the Reader and Thomas A. Sebeok's The Sign and Its Masters," Journal of Comunication, 30 (1, Winter,) 205-210.
Uexkull, T. von. (1981). "The Sign Theory of Jakob von Uexkull."
In M. Krampen, et. all eds, Classics of Semiotics, NY:
clickhere to return to Professor Moriarty's home page
click here to return to the visual communication