This section of the class notes considers two very basic issues: how to describe the abstract organizational structure of a design, and how to represent that structure in a concrete design.
Semiotics is the study of signs. Signs mediate meaning, and are not just simple "tokens" or physical marks; they can be complex combinations of other lower level signs, such as whole sentences, spoken or written, newspaper advertisements, whole books, etc. The American logician Charles Sanders Peirce (pronounced "purse") introduced the term "semiotics," and several of its basic ideas. In particular, he emphasized that meanings are not directly attached to words; instead, there are events (or processes or activities) of semiosis - i.e. constructions of meaning - each involving:
This three-fold approach to meaning may sound straightforward and intuitive, but it differs greatly from more common and naive theories, such as denotational semantics for programming languages, which assume a function from programs (which are signs) to their denotations (which are meanings), typically defined by higher order functions on abstract mathematical domains, or more simply, by a compiler to a particular machine.
Peirce's approach is also more general than assuming just a relation between meanings, as in the second main source of semiotics, the Swiss linguist Ferdinand de Saussure. In both traditions, signs can be anything that mediates meaning, including words, images, sounds, gestures, and objects, but in the tradition of Saussure, signs have only:
Signifier: The letters 't-r-e-e'.Note that a "sign" is a particular combination of a signifier and a signified. The same concept could be indicated by other signifiers, and the same signifier could refer to other things; in each case, we would have a different sign. Thus meaning for Saussure is relational, not just functional. (This explanation augments that in the "Signs" chapter of Semiotics for Beginners, by Daniel Chandler.)
Signified: The concept of tree.
Peirce's definition of sign is better, though more complex, because it includes the creation of the link between signifier and signified as an explict component; this can include the interpreter, the context of interpretation, and the process of interpretation. Peirce's notion is triadic, whereas Saussure's is only dyadic (i.e., binary). Although Peirce's notion, like Saussure's, is relational, in contrast to Saussure, it is open rather than determined in advance, as well as triadic. In our application area of user interface design, the interpreter is usually the user or the designer. More detailed discussion appears in On Notation.
Yet another important notion from semiotics is Peirce's three way classification of signs into symbols, icons, and indices. These concepts have many applications to user interface design; again see On Notation for details. You should carefully note that each of these three terms has a technical meaning that is not the same as its ordinary everyday meaning!
Saussure's most important insights are probably that signs come in systems, not just one by one, and that they can have internal structure. Another insight of Saussure that is not often sufficiently emphasized, is that sign systems are organized by systematic differences among signs; we can relate this to a famous saying of Gregory Bateson, that "information is a difference that makes a difference." Saussure's idea that signs come in systems is illustrated by examples like the vowel systems of various accents of the same language, and the tense systems for verbs in various languages. The vowel system example shows that the same sign system can be realized in different ways; we call these different models. The vowel system example also shows that two different models of the same sign system can have the same elements but use them in a different way; so it's how elements are used that makes the models different, not the elements themselves. Models of sign systems are not just sets, they are sets with some kind of structure; we will learn more about this later. Alphabets also provide good examples where the sets overlap; for example, the Greek, Roman and Cyrillic alphabets each have some tokens in common; this motivates the need for "signs" as tokens that come in systems and have an interpretation; they cannot be just tokens as such. We can also motivate the need for systems of signs by noting that a sign system with just one element cannot convey any information (more technically, this is because its Shannon information content is zero).
Later we will see that it is useful to view sign systems as abstract data types, because the same information can be represented in a variety of different ways; for example, dates, times, and sports scores, each have multiple representations. This leads naturally to the idea that representations are mappings between abstract data types, as illustrated in an informal way by the examples in the UCSD Semiotic Zoo, which show how the failure of a representation to properly preserve some structure results in its being a suboptimal representation.
Algebraic semiotics attempts to combine the major insights of Peirce and Saussure (among others) into a precise formalism that can be applied to the practical engineering of sign systems, e.g., in user interface design. This involves a fundamental change in perspective, from the analytic perspective of traditional semiotics, to a synthetic perspective, which is concerned with construction rather than just analysis. One important insight that this formalism incorporates is that signs need not be the simple little things that we usually call "signs," but instead can be very complex, such as a book, or a series of books, or even a whole library; or a movie or series of movies, or the GUI to an operating system. It is the job of user interface designers to build (parts of) such systems. Another insight that algebraic semiotics pursues is the importance of studying errors, that is, badly designed sign systems, in order to better understand what it means for a sign system to be well designed. This is interest is illustrated in the exhibits of the UC San Diego Semiotic Zoo.
Traditional semiotics seems to have a Platonistic bias, in assuming that signs (which, please recall, we take to include token, meaning and their relation) have an existence which is independent of human beings. But here we take a quite different approach, in basing signs on human social activity. We claim that signs do not have meanings in themselves, but only have whatever meaning we give to them. Moreover, insofar as these meanings are shared, they are necessarily social, and Plato's ideal entities play no role in this.
Later in this course we will go much deeper into the structure of signs, and the representation and interpretation of signs. Peirce and Saussure are only the beginning of the story.
Perhaps the most important issue in interface design is how to represent the content and structure of the designed object so that it is as easy to use as possible (note that interface design, by definition, is concerned with designing the interface, assuming that the functionality of the object is known). This implies that interface design requires a good theory of representation. It is easy to see that the literature on interface design gives many heuristic guidelines for obtaining good representations, but fails to provide a general theory of representation, with systematic principles at a high level of generality. The purpose of algebraic semiotics is to fill this gap. Although he theory is still at a relatively early stage of development, enough has already been done so that many practical applications are possible, and the outlines of the general theory can certainly be seen.
The two most fundamental concepts of algebraic semiotics are semiotic theories and semiotic morphisms. Semiotic theories describe the structure and content both of interfaces and of the functionality that they are supposed to represent, while semiotic morphisms are the actual representations, conceived as mappings from a "source space" of structure and content, to a target space of representations.
Here are some simple examples: The original Windows interface was a mapping from DOS to the GUI primitives of window, menu, icon, etc. In fact, most modern operating systems have the same approach, providing a graphical interface for an underlying functionality, usually based on the "desk top" metaphor. Clocks are maps from an abstract space of time to some physical display. We will see many other examples in this class; the present discussion just provides a hint of what is to come. We will also see that what makes one such representation better than another always boils down to the preservation of structure, reflecting values of users which have a social basis.
We will also see that semiotic theories generalize the so-called conceptual spaces introduced by Gilles Fauconnier in the discipline of cognitive linguistics. Whereas conceptual spaces include schematic representations of human concepts and relations, semiotic theories also support more complex "constructors" that build structures for windows, scrollbars, icons, etc.. Similarly, semiotic morphisms generalize the conceptual maps of Fauconnier, in that they preserve the additional structure of semiotic theories. A major application of conceptual spaces is the study of metaphor, as maps of conceptual spaces, or more generally, as blends of conceptual spaces. Some cognitive linguistics will be covered near the end of the class.
There is a very basic and important duality between designing signs and interpreting or understanding signs: in the first, we know about signs in the source system, and we seek to map them to a target system in a way that best preserves the information of greatest interest, while in the second, we know about one (or more) target sign, and we seek to infer signs in the source system from which it (they) might have come. Note that interpretation may require us to infer properties of the mapping as well as of the source space, i.e., it requires a rather sophisticated abstraction from the given target sign(s); it is amazing that this process often appears to be completely effortless. On the other hand, this appearance of effortlessness also seems to make it difficult for us to realize that the underlying processes are actually very sophisticated. They appear effortless only because they are unconscious.
Examples of design beyond the usual use of some mixture of media to present some given content, include giving a good explanation for something, finding a good "icon" (in the usual informal sense of computer graphics), choosing a good file name, making a good analogy, and writing a poem that captures a certain mood. Examples of the dual mode include understanding graphics, multimedia texts, verbal explanations, poems, films, metaphors, equations, or indeed, understanding anything at all. (But it is interesting to notice that people often in fact do not fully understand signs, e.g., films, even though they may spend a lot of time with them, and enjoy them. And of course there are signs that no one understands, such as texts in the Etruscan language.)
Another important class of examples involves the evaluation of signs, as requested in several of the homework problems for this class. Here you must not only infer properties of the source sign system and the semiotic morphism, but also consider some alternative semiotic morphisms, possibly even involving different target sign systems. Good designs are exactly those that are easy to interpret.
Both design and understanding involve a transformation (movement, translation, interpretation or representation) of signs from a source system to representing signs in a target system, but the examples of understanding require inference in the opposite direction from that of the mapping. Algebraic semiotics not only helps us see the dual relation between design and understanding, but also suggests ways that each process contributes to the other. In general, and perhaps surprisingly, it is easier to investigate understanding than to investigate design, but on the other hand, design is requires starting from more abstract signs, and hence is inherently more creative. Once we understand something about understanding some sign system, we can apply this knowledge to the corresponding design problems; of course the dual trajectory also works, but is perhaps less common. There are many interesting applications of this approach, one of which is designing visualizations for scientific information, as discussed earlier in the course. Another, which to which we now turn, is developing a better understanding of humor.
SHOULD HAVE A PICTURE HERE.
Despite the formal mathematical character of the definitions of sign system and semiotic morphism, these concepts can be used very informally in practice, just as simple arithmetic is used in everyday life. For example, to see if we have enough gas left to drive from San Diego to Los Angeles, we make some assumptions, use some approximations, and only do the divisions and multiplications roughly. It would not make much sense to first work out an exact formula taking account of all contingencies, then do a careful analysis of the likelihoods, and finally calculate the mean and variance of the resulting probability distribution (though this is the sort of thing that NASA does for space shuttle missions). In user interface design, our goal is often just to get a rough understanding of why some design options may be better than others, and for this purpose, assumptions, approximations, and rough calculations are sufficient, especially when there is time pressure.
On the other hand, a precise formalism can be very valuable for developing a thorough understanding of difficult concepts, for proving general results about them, and for (perhaps only partially) automating aspects of the design process.