Visual Communication 94, Feather River Inn, Blairsden, California

In a review of the contributions of various philosophers to semiotics, Thomas Sebeok, founder and director of the Center for Language and Smiotic Studies at Indiana University observed that Bacon, for one, "did not commit the vulgar error of identifying language with communication." (1991, p. 71) But if language is not identifiable with communication, then what is? It is the primary thesis of this essay that visuals are equally as important as verbal language in communication, if not more so, and that Sebeok is right in asserting that privileging verbal language is narrow minded, if not vulgar.

The five questions driving this investigation focus on the relationship between semiotics and visual communication, the relative importance of visual and verbal communication systems, and the place visual communication fits within the larger world of sign systems.

1. What is semiotics and what elements/aspects of it are most useful in analyses of visual communication?

Semiotics is the science of signs and semiotic analysis is used in studies of sign processes in such fields as communication, cognition, linguistics, anthropology, marketing, medicine and cellular biology. The contemporary focus that most concerns us at this conference is on the communication aspects of semiotics, particularly the elements relevant to visual communication.

The communication focus is reflected in the definition of semiotics given by Jakobson: "the exchange of any messages whatever and of the system of signs which underlie them." (Sebeok, 1991, p. 60) Fiske adds another dimension to this definition--the generation of meaning. (1990, p. 42) In other words, messages are made of signs and conveyed through sign systems called codes; the more we share the same codes in a communication exchange, the closer our meanings will be. Presumably that is true for visual as well as verbal communication and a comparison of those two communication repertoires will be an important thread throughout this paper.

But what is a sign? A sign is anything that stands for something else --i.e. a sign stands for an object or concept. (Hopes, 1991, p. 141; Eco, 1986, p. 15) The "stands for" process is the point where meaning is created both through encoding (by the source) and decoding (by the receiver--or "reader" in semiotic analysis) as in these stylized representations of people. The Swiss linguist Ferdinand de Saussure developed a similar construct which he described as a sound or image-- called a signifier--and the concept for which it stands--called the signified. (Fiske, 1990, p. 43-44) Because of his background as a linguist, Saussure saw the relationship between these two as arbitrary, in other words, the link between the sign, or expression, and what it stands for is understood by convention. This arbitrariness is true in most spoken and written language--such as the names for these individuals depicted in these sketches--however, that may not be so for other types of signs such as visuals that provide stylized cues to stimulate recognition through resemblance.

The American C. S. Peirce, who is characterized as the founder of American semiotics, disagrees with Saussure on the arbitrariness condition. Because Peirce is first of all a cognitive philosopher, he defines signs in a broader way than language and focuses on how they are logically or semiotically linked to their objects. Peirce's foundational contributions to semiotics have shaped the field, as Berger explains, to largely focus on meaning and modes of cognition (the codes needed to understand a text). (1982, p. 39.)

To Peirce signs are of three types--icons, indexes, and symbols. (Hoopes, 1991, p. 239) Symbols are arbitrary, but icons and indexes are, in semiotician language, "motivated," that is they are more likely to resemble their object in some way, rather than being arbitrary. Peirce defines an icon as similar to its subject--a representation such as a drawing or photograph where likeness or resemblance is a determining characteristic. An index is physically connected with its object--as an indication that something exists or has occurred, such as a footprint that means someone just walked by or smoke that means there is a fire. A symbol such as a leaf on a flag is arbitrarily linked with its object--something that has acquired meaning through convention. Symbols, like most spoken and written words, are arbitrary.

Peirce's categorization of signs provides a richer context for understanding visuals and how they convey meaning than Saussure's signified and signifier--although Saussure's dyadic relationship is included in Peirce's categories and it is at the base of all semiotic analysis.

Another important element, particuarly in a comparison of visual and verbal communication, is the notion of code or sign system. A code functions as a system of rules. Language is a code and linguistic competence, i.e. understanding the elements and structure of language, is foremost on the agenda of most modern schools. The concept of semiotic code, however, is broader than just language and includes such sign systems as kinesics or body language, braille, sign language, and algebra and chemistry. In this advertisement, for example, the complex message about gender and religion is being communicated through a manipulation of color codes. Nature can even be seen as a system of coded signs and Eco makes the argument that the roots of semiotic interpretation lie far back in time with hunters and trackers who could read the signs of nature.

Visual communication has various types of codes that help determine the message. The argument can be made that visual communication operates, to some degree, similar to a language sign system in that the basic characteristics of language are similar--elements, syntax, and grammar. For example, the visuals used in graphic design, photography, video and film contain discrete sign elements (shots, typography, drawings, etc.) which are variously called semes or signemes; syntax which structures the links between the elements (composition in a still form and transitions in film); and rules about their use which form a type of grammar --visual or aesthetic codes that carry their own meaning. (i.e. estabishing shots, reaction shots, etc.).

Also important is the notion of the interpretant, another concept which is important to Perice's semiotics and particularly relevant to an understanding of how visual communication operates. It was mentioned earlier that the sign/object relationship is the basis of semiotics. Peirce adds the concept of interpretant to that dyad to create a triadic construct of sign/object/interpretant. By interpretant he means the idea contained in the concept as it is decoded or a subsequent thought to which the sign gives rise. (Hoopes, 1991, p. 34) He explains that a thought is a sign requiring interpretation by a subsequent thought in order to achieve meaning. Our memories, which provide the foundation for the meaning process, are, as Sebeok puts it, "reservoirs of interpretants." (1991, p. 131)

For example the word /dog/ and a picture of the animal both stand for some concept of "dogness." In addition, there are interpretations imposed on this concept of horseness based on our personal experiences and also on additional information and description that accompanies the sign. Other verbal interpretants for the word /dog/ could be puppy, bitch, hound, or faithful companion-- as well as equus, cheval...or even frankfurter or ugly woman. Each further interepretation tends to elaborate on the concept and make it richer. (Sebeok, 1991, p. 18-10)

This review of the basic concepts of semiotics should make it clear that many of these concepts are linked inextricably to visual communication. Even though some semiotics studies are primarily focused on literature and language, and, as Deely notes language has a prileged role in semiotics (1990, p. 27), it is inconceivable to try to explain semiotics without referring to visual communication factors and functions and it makes a lot of sense to use semiotics to explain how meaning is produced visually.

Finally to conclude this discussion of the basics of semiotics, it should be noted that what we are talking about is an action process. As Deely explained in his book on semiotics, Peirce's primary contribution was the full development of semiotics as a distinct body of knowledge and this discipline rests on the notion of "a dynamic view of signification as a process." (1990, p. 23)

2. Which is primary--verbal or visual communication?

It is important to realize that the semiotic process, in Peirce's view, was more centrally concerned with thinking than with communicating. Peirce was a student of philosophy of knowledge and he was most concerned with an analysis of thought rather than language.

Inference Peirce's primary contribution to cognition--and only incidentally to communication--was his proposition that "all thinking is the inferential interpretation of signs" (Hoopes, 1991, p. 11). In other words, a thought is a sign interpretation, an idea which provides the link between cognition and communication. He explained that the meaning process--finding the signified--is an infinite process of interpretation and, furthermore, that to interpret means to define a relationship, i.e. what is involved in the something-stands-for-something-else construct. (Eco, 1986, p. 2, 44). So we have at the base of Peircian semiotics a thinking process based on inference which results in interpretation.

The reason Peirce's concept of interpretant is so important to us in visual communication is that visual meaning is more open for interpretation than verbal. Contrary to conventional wisdom and the views of some leading semioticians and most linguists, the interpretation of a visual message may be more complex and more demanding of the decoder because of the inferential dimension than is the interpretation of a verbal message. The complexity of this type of inferential interpretation is illustrated in this use of an upscale car to stand for the flaunting of achievement. In order to make sense of this ad, you have to recognize the car and the fact that it stands for accomplishment, the school and the fact that it is cueing a class reunion, and all the emotional context associated with the reunion situation.

The "Primary" Debate Peirce, and particularly Sebeok, present us with a theory of semiotics that is not only broader than a language-based sign system, it also treats both nonverbal and visual communication with the respect due to these seminal communication systems.

Saussure, however, is a linguist and although he admits that signs can be other than words, his work privileges language as the most important sign system. Likewise, the Soviet school of semiotics (Ivanov, Lotman, and Toporov among others) calls language a "primary modeling system," because it uses natural language as its base, and all other sign systems as "secondary." Natural language, in other words, is the primary substructure for all other sign systems. (Sebeok, 1991, p. 50)

Other semioticians disagree and point to development learning as a better source for the primary designation: as infants we develop visual skills before language skills; in this sense visual communication should be considered primary rather than secondary.

Sebeok also disagrees with the language-as-primary theory and points out that language evolved first as an adaptive function principally to enhance imitative signaling--the evolutionary focus was more on language-as-modeling-system than on speech-as-communication. Furthermore, at the time language evolved, homo habilis, the earliest known species, had very sophisticated visual and nonverbal sign system repertoires but a primitive verbal system limited to gutturals rather than articulate, linear speech. Sebeok argues that "properly speaking, language itself is a secondary modeling system." He is comparing language based sign systems with other visual and nonverbal systems which are equally as complex, as well as antecedent to language. He concludes that "the general belief that language replaced the cruder systems is totally wrong." (Sebeok, 1991, p. 57, 71)

I would like to suggest that visual interpretation is more complex than verbal interpretation and therefore more appropriately designated as a primary sign system. The reason is because the learner has to manage more of the interpretative function with visuals than with language because of their different learning protocols. Verbal signs are arbitrary and verbal messages are highly conventionalized, so we are taught to speak and read these signs through an extensive educational program. Visual signs are more often iconographic or indexical and those forms of interpretation are, as Sebeok noted in a discussion of nonverbal communication, largely "wired in" rather than arbitrary and conventional. (1991, p. 65) Even visual signs that are arbitrary, such as this plateful of symbols, are not formally taught to us.

We acquire visual competence through development and experience rather than training--in other words, for our basic nonverbal and visual communication skills we are largely self-taught and that includes such skills as making sense of MTV, watching multiple channels simultaneously, and negotiating freeways as we puzzle out a map. Even though we may be self taught, our skill levels in certain areas where development is important can be extremely high.

However, we have lost visual skills, too. Bill McKibbon in his book The Age of Missing Information makes the point that our ancestors were much more tuned in to the land, to weather, to animal behavior, and to nature than we are. Their additional sophistication in understanding natural signs was largely a product of experience--with some tutoring from wiser adults. (McKibbon, 1992)

Today we learn a basic survival set of visual and nonverbal skills with only minimal help from mom and dad--we learn more on our own than we do with language whose initial learning is directed by parents and later by schools. We do need training if we want to develop more complex or specialized visual skills in order to track something in the woods, to puzzle out the images of the cells we see in a microscope, to read a negative or X-ray, or to understand a Picasso painting.

Furthermore the lack of a system for assigning conventional meanings for visual communication suggests that visual information is subject to more active personal interpretation--more so than with language. The interpretation of visual information, like semiotic approaches to meaning interpretation, are highly subjective and highly projective, which puts more demands on the receiver. Because of the resemblance factor for icon interpretation and the experience factor for index interpretation, the formal training needed may be less than for language--although the life experience may be more demanding-- but regardless, the visual and nonverbal systems operate relatively untutored in our society, at least in comparison to language. With visuals we are much more on our own, both in learning and in interpreting, and that's why I believe visual learning in our contemporary society is equally as challenging an accomplishment as verbal learning.

3. Why is semiotics, with its roots in philosophy and linguistics, particularly useful in analyzing visual communication?

Peirce's statement quoted earlier that semiotics is the study of an "inferential interpretation process" introduces another concept that is important in understanding how visual communication works and that is inference. Visual and verbal comunication differ in their interpretive logic. Most language-based models of logic focus on induction and deduction as platforms for reasoning. Visual thinking, however, is more easily analyzed by abduction, a process which Peirce referred to as hypothesis building and Eco and Sebeok refer to as "the conjectural paradigm." Deduction and induction are syllogistic; abduction reasons from what Hoopes calls "statistical inference" which relies more on hypothesis testing or educated guessing than rules of logic. (1991, p. 85)

Abduction operates largely from a nonverbal platform although words can be and are often used in support. In medicine, for example, doctors use words to probe for subjective symptons but rely on their own senses to probe for objective signs which are compared with the doctor's knowledge of past cases. Medical diagnosis is a good example of abductive reasoning supported by induction.

Nowhere is the power of abductive thinking made clearer than in Eco and Sebeok's book, The Sign of Three, which compares Peirce's approach to thinking with the reasoning of two great detectives Conan Doyle's Sherlock Holmes and Poe's Dupin (1983). A detective, of course, detects things or discovers their existence; an inspector is someone who inspects, or looks. The reasoning approach used by dectectives in crime fiction is very similar to the interpretive process of abduction which has been described as "educated guessing toward a hypothesis." In both cases the process begins with observations and then proceeds in a back-and-forth process of developing hypotheses and comparing the observations with information known and filed in memory. Whether it be solving a crime or tracking an animal through the forest, it is the same sign interpretation process described by Peirce and it begins with observation--observations which are usually visual and often indexical.

And this type of search for meaning is infinite, in Peirce's view, because every signifier can be translated into other signifiers and interpretants through an endless process of inference chaining.

Now ask yourself, how does a semiotician develop an expert reading of something? Some critics look upon this process as something more closely aligned to tea leaf reading because they don't understand how abduction works. Here's an example of visual semiotics: Ask yourself how it works. What are its visemes? How does the visual syntax structure your process of creating meaning? What do the visual codes or grammar contribute to your interpretation? And how did the chaining of interpretants work?

Now analyze your interpretation process in making sense of visuals such as this. Did you use syllogisms or observation testing? If Eco and Sebeok are right, then you will observe different cues which bring forth information from your own mental filing cabinet and you will develop hypotheses about various aspects of the meaning until the picture comes together. In other words, the interpretative method of semiotics based on its use of abduction, is more closely aligned with the way we process information visually which, to answer the second question, is why semiotics is such a useful tool in visual communication research and theory building.

4. Where does viscomm fit in the panoply of communication sign systems?

One objective of this paper is to defend the notion that visual communication is a primary sign system and it deserves the respect of scholars and philosophers. As part of that argument, it might be helpful to sort out the various categories of sign systems and locate visual communication relative to other systems.

The first premise of this model, and it is a heretical idea--at least to linguists--is that human language is only one of many sign systems and it is certainly not "primary." In other words, there is a large world of communication sciences disciplines and language-based sign systems are only a part of that universe. As indicated in a quote from Sebeok, we shouldn't think of communication as based solely or primarily on language: "Our habit of thinking of communication as consisting exclusively of language has delayed the study of communication." (Sebeok, 1991, p. 57). So we will begin with a model of communication sign systems that does not begin with language.

Several writers have surveyed the scope of semiotic studies and this analysis is based on Eco's review in A Theory of Semiotics (1979) and Sebeok's discussion in A Sign is Just a Sign (1991). At the first level are a number of different types of semiotic systems that Sebeok has identified as functioning in a communicative way. The variety of the communication systems that are not language based should illustrate convincingly Sebeok's point about the "vulgar error."

For example, protosemiotics has recently been suggested as a term to use in describing the sign systems discussed in analyzing the solar system. Zoosemiosis includes animal communication--all those puzzling questions we ask about how bats fly, how flocks of birds turn synchronously, the meaning of dolphin and whale sounds, and the mapping communicated by the honeybee's dance, as suggested in this visual.

Under the general area of biosemiosis, endosemiotics studies communication at the molecular level. One might ask how cells can be described as communicating organisms however molecular and cellular biologists and chemists are involved in puzzling out signaling systems used both inside a cell as proteins and acids are moved around, and between cells as they create complex molecular and cellular structures. Likewise, the neural, genetic, and metabolic codes all operate through semiosis, as well as the processes studied in chemical, thermal, structural, and electrical engineering. Other biosemiotic areas include phytosemiotics (vegetable) with its photosynthesis processes and mycosemiotics (fungi).

Anthroposemiosis includes human communication and this, of course, is the area we are most concerned with. Interestingly, human communication seems far less well developed as a theoretical field than the scientific areas on the left of the model.

In trying to model human communication there are several points of puzzlement. For example, is language only verbal or are there other types of sign systems which could be called languages--visual, kinesics, sign language, braille, etc? The whole nonverbal area is huge and grossly underestimated in studies of human communication. The problem with nonverbal communication, however, is a semantic one; there is no way to refer to this area other than in opposition to language which inherently privileges verbal communication. Sebeok observes that "the human's rich repertoire of nonverbal messages--by shared contrast with language--never constituted a unified field of study and therefore lacks a positive integrative label." In another place he laments that there is "as yet no universally agreed upon global designation for studies of nonverbal signs." (Sebeok, 1991, p. 23, 60)

The model shown here is a preliminary attempt at depicting the important divisions in the world of communication sign systems and locating them relative to one another. It is not inclusive and, at this point, it is only a starting point for discussion. I welcome your suggestions.

5. Visemics: Is it time for a discipline of our own?

The model, plus the previous discussion of the ties between semiotics and visual communication argues for greater respect for visual communication; and this leads to a discussion of the nature of the visual communication field. Is it time to declare visual communication a discipline?

What exactly is included in a study of visual communication? One of the biggest reasons this area gets less respect than language is because it is so loosely articulated. If we build on a language-based model summarized earlier, then we would include 1. visual elements or "visemes," 2. syntax (composition, transitions) and 3. grammar (the visual aesthetic codes). As Berger points out, "The notion that film, television programs and other cultural phenomena are like languages in many respects is central to semiological analysis." (Berger, 1984, p. 190) However well developed the linguistic model may be, is it adequate as an analytical frame for visual communication? If not, what's missing?

Furthermore, is the name right? While visual communication does not have the semantic limitations that plague the area of nonverbal communication, it still may not be an effective designator because of the implied parallelism. /Visual communication/ is a common phrase but the expression /verbal communication/ is less often used--which suggests that the idea of communication may contain verbal as a root concept, thus there's no need to designate it. What the phrase /visual communication/ does is separate out visuals as a subset of the broader language-based field of communication and we've already determined that visual communication is broader than that.

Perhaps what we need is a name like kinesics or linguistics--a term that summarizes, clarifies, and claims an area of study. Is there anything parallel to linguistics that carries the semantic power of ownership--vististics, visive, visivics, visus, or visemics, for example. The last phrase is my preference because it acknowledges the fundamental importance of semiotics and the concept of sign systems. Or is such phrase making even necessary?

Conclusion

This paper suggests that semiotics is a very useful methodological tool for analyzing visual communication because the thinking processes it proposes are parallel to the interpretive processes used in creating and understanding visuals. Furthermore, it argues that it is more logical to designate visual communication as a primary system than language based upon what we know about developmental communication and the complex demands of visual interpretation. Finally, since the two areas of visual communication and semiotics are so closely linked, the time might be right for announcing a new discipline with a name that joins the two--such as visemics.

References

Berger, Arthur Asa. 1982. Media Analysis Techniques. Newbury Park CA: Sage.

Berger, Arthur Asa. 1984. Signs in Contemproary Culture. New York: Longman.

Deely, John. 1990. Basics of Semiotics. Bloomington: Indiana University Press.

Eco, Umberto. 1979. A Theory of Semiotics. Bloomington: Indiana University Press, Midland Book Edition.

Eco, Umberto. 1986. Semiotics and the Philosophy of Language. Bloomington: Indiana University Press, Midland Book Edition.

Eco, Umberto and Thomas A Sebeok, ed. 1983. The Sign of Three. Bloomington: Indiana University Press.

Hoopes, James. Peirce on Signs. 1991. Chapel Hill NC: The Unviersity of North Carolina Press.

Sebeok, Thomas A. 1991. A Sign is Just a Sign. Bloomington: Indiana University Press.

clickhere to return to Professor Moriarty's home page

click here to return to the visual communication research page