CSE 171: User Interface Design: Social and Technical Issues
10. More Notes on Algebraic Semiotics, and Some Other Topics

There is a very basic and important duality between designing signs and interpreting or understanding signs: in the first, we know about signs in the source system, and we seek to map them to a target system in a way that best preserves the information of greatest interest, while in the second, we know about one (or more) target sign, and we seek to infer signs in the source system from which it (they) might have come. Note that interpretation may require us to infer properties of the mapping as well as of the source space, i.e., it requires a rather sophisticated abstraction from the given target sign(s); it is amazing that this process often appears to be completely effortless. On the other hand, this appearance of effortlessness also seems to make it difficult for us to realize that the underlying processes are actually very sophisticated. They appear effortless only because they are unconscious.

Examples of design beyond the usual use of some mixture of media to present some given content, include giving a good explanation for something, finding a good "icon" (in the usual informal sense of computer graphics), choosing a good file name, making a good analogy, and writing a poem that captures a certain mood. Examples of the dual mode include understanding graphics, multimedia texts, verbal explanations, poems, films, metaphors, equations, or indeed, understanding anything at all. (But it is interesting to notice that people often in fact do not fully understand signs, e.g., films, even though they may spend a lot of time with them, and enjoy them. And of course there are signs that no one understands, such as texts in the Etruscan language.)

Another important class of examples involves the evaluation of signs, as requested in several of the homework problems for this class. Here you must not only infer properties of the source sign system and the semiotic morphism, but also consider some alternative semiotic morphisms, possibly even involving different target sign systems. Good designs are exactly those that are easy to interpret.

Both design and understanding involve a transformation (movement, translation, interpretation or representation) of signs from a source system to representing signs in a target system, but the examples of understanding require inference in the opposite direction from that of the mapping. Algebraic semiotics not only helps us see the dual relation between design and understanding, but also suggests ways that each process contributes to the other. In general, and perhaps surprisingly, it is easier to investigate understanding than to investigate design, but on the other hand, design is requires starting from more abstract signs, and hence is inherently more creative. Once we understand something about understanding some sign system, we can apply this knowledge to the corresponding design problems; of course the dual trajectory also works, but is perhaps less common. There are many interesting applications of this approach, one of which is designing visualizations for scientific information, as discussed earlier in the course. Another, which to which we now turn, is developing a better understanding of humor.

10.1 Humor

An important preliminary observation is that blends are not unique, as is well illustrated by blended words like "boathouse" and "houseboat," as discussed on pages 18-22 of An Introduction to Algebraic Semiotics, with Applications to User Interface Design; see also the discussion in Formal Notation for Conceptual Blending; this means that even the combination of the two words can be highly ambiguous. Oxymorons are discussed on the top of page 22 of An Introduction to Algebraic Semiotics, with Applications to User Interface Design; see also the oxymoron page. The ambiguity of blending plays an important role here, since an oxymoron has two different blends of two given words, one of which has a standard meaning, and the other of which has some kind of conflict or incongruity in it. Often the second meaning only arises because the word "oxymoron" has been introduced, and this deliberate creation of a surprising ambiguity is what makes these a form of humor. For example, in "military intelligence" the standard meaning is an agency that gathers intelligence (i.e., information, especially secret information) for military purposes, while the second, conflictual meaning is something like "stupid smartness," playing off the common (but incorrect) prejudice that the military are stupid, plus the more usual meaning of intelligence.

Many newspaper cartoons, consisting of 1 to 4 small scenes (i.e., panels), achieve their effect by setting up some situation, and then recontextualizing it, i.e., introducing new elements and relations into the conceptual space, which have the effect of forcing a new organization for some parts of the conceptual space that was originally set up. Let's call this reblending. The first space is in general itself some kind of blend, and its reconceptualization is also a blend, of the old space with the new material; this often has a humorous effect. An informal survey of cartoons in the local newspaper found that more than half of the intendedly humorous cartoons achieved their effect by reblending a given conceptual space with some new material, to give some parts of the old one surprising new meanings. A lot of humor seems to have a similar character. Here is a link to a simple visual humor example that involves reblending but is not an oxymoron. Oxymorons can also be seen as involving a "cross space" mapping which imports at least some of the (generally less conventional) contradictory meaning into the (generally more conventional) blend; but the "two blend" approach developed above is more fundamental.

Similar phenomena can be found in puns, as well as in music, poetry, and probably in every art form, where the effect is by no means always comic. It seems clear that evolution has provided us with positive feedback for improvements in understanding in the form of mental pleasure, since this has an obvious survival value. One familiar example is the so called "Eureka" experience, when we suddenly see the solution to some problem that we have been pondering for a long time.

In light of our success with oxymorons, cartoons, etc, it seems reasonable to believe that some fairly large areas of humor can be characterized in terms of reblending. Once we have this understanding, we are in a position to apply it to design. For example, we might want to make the use of certain difficult interfaces a lighter and more pleasant task by introducing some humor. But it is important to notice that, because the psychological impact of recontextualization depends on its novelty, repeating the same joke again and again will not be effective interface design, and in fact, will prove irritating to users; this implies that humor must be introduced very carefully and selectively. Many designs have had to be redone for this or closely related reasons. For example, overly cute icons or avatars can quickly become irritating (or slowly, if they are a bit less cute). Two instances from Microsoft include the barking dog in PowerPoint and the obsequious paper clip in Word (this has been made a non-default option in recent releases). Note that cuteness as a semiotic phenomenon is closely related to humor, in that it involves blending partially conflicting systems, such as child-like facial features, with a serious message.

This discussion of humor differs from the current cognitive linguistics literature in its emphasis on re-blending. For example, Seana Coulson's Extenporaneous blending: Conceptual integration in humorous discourse from talk radio discusses humor in terms of blending material from "apparently unrelated" domains, which is called "double scope blending" in The Way We Think, by Fauconnier and Turner.

10.2 More Terminology and Dualities

It is important to distinguish between conceptual spaces (also called mental spaces) and semiotic systems. The first concept, originally due to Gilles Fauconnier, is used in cognitive linguistics, while the second has been developed for applications to user interface design. Conceptual spaces have only constants (called elements) and basic relations among them, whereas semiotic systems have multi-argument constructors, as well as sorts, levels, priorities, and axioms. Similarly, we should distinguish between conceptual blending, also called conceptual integration, and structural blending, which we may as well also call structural integration, where the former is blending of conceptual spaces and the latter is blending of semiotic systems, which may involve non-trivial constructors. While conceptual spaces seem adequate for the meaning of language, they are not adequate for user interface design, or for other applications where structure is important, such as music. For example, conceptual spaces and conceptual blending can help us understand concepts about music, but semiotic spaces and structural blending will be needed for dealing with the structure of music, e.g., with how a melody can be combined with a sequence of chords. Conceptual spaces are good for concepts about (i.e., how we talk about) things, but are more awkward for talking about the structure of things. It also seems that greater cultural variation can be found in conceptual blending than in structural blending, because the former deals with concepts about something, whereas they latter deals with the structure of its representations and/or instances.

A potentially confusing point is that it is convenient to speak of the class of models of a theory (e.g., all the books that have the structure given by a theory T of books) as a "space," or even as "the semiotic space of T," although this terminology conflicts with that of cognitive linguistics, in which conceptual spaces are generally considered to describe single models, rather than sets of models.

Another important duality is that between semiotic theories (which we also call semiotic systems) and spaces of models. The former involve constructors, axioms, levels, etc., while the latter are sets of (all possible) models of these theories, where each model is a concrete realization, generally described using set theory. This course has not paid much attention to the models, but instead has worked with theories. One reason to prefer theories over models is that they provide axioms to constrain the allowable representations. For example, in considering the semiotic morphisms that produce indices from books, we may well want to impose axioms on the target space of indices, requiring that indexed items must be phrases of 1, 2 or 3 words, and that the page total for indexed phrases must occupy not more than 2% of a book's total pages.

We have taken the direction of a semiotic morphism to be the direction that models or instances of the theories are mapped. For example, if B is is a semiotic system for books and T is a semiotic system for tables of contents, then the books (which are models of B) are mapped to their tables of contents, which are models of T. But a subtle point is that the way we describe this mapping of models is via an inclusion T -> B of theories, because of structure of tables of content is a substructure of the structure of books. It seems to be rather counter-intuitive that, from a technical point of view, the mapping on models goes in the opposite direction from that between the theories, but this is an essential part of the duality between theories and models (technically, this kind of duality is called a "Galois connection"). Note that this is also consistent with the fact that blending diagrams are drawn "upside down" in algebraic semiotics, compared with cognitive linguistics.

Dynamics is another feature that is not easily handled by conceptual spaces, but is clearly necessary for user interface design. Unfortunately, there has not been much time to discuss the somewhat complex details in this class, but there is a natural extension of semiotic systems to dynamic semiotic systems using the theory of hidden algebra, which extends algebraic specification to dynamical systems.

10.3 Applications Involving Narrative

We can understand (some aspects of) narrative structure in terms of semiotic morphisms. For example, books on screen writing by Syd Field prescribe a precise (but naive) dramatic structure for the plots of Hollywood movies: they should have three acts, for setup, conflict, and resolution, with "plot points" that move action from one act to the next. We can describe this "Syd Field structure" using a very simple sign system with a main constructor that builds a thing of sort "plot" from three things of sort "act", and we can then check whether a given film has this structure by seeing whether there is a semiotic morphism from the (structure of the) film to this sign system.

It is clear that as internet bandwidth grows, video will be used more and more; moreover, complex interactive websites are aready common, many with story lines. Techniques for drama apply directly to such cases, but are also much more broadly applicable, in suggesting ways to make displays (i.e., "texts" in the very general sense of signs that are to be interpreted by users) more interesting. My own favorite examples of this come from mathematics. Probably we have all had the experience of trying to read a proof and being frustrated, at least initially, by not being able to see where it was going, or why it was structured as it was. This is often because the proof author did not describe the difficulties that arose in constructing the proof, but instead just described the machinery that was constructed to overcome the difficulties. For example, one often comes upon an assertion in the middle of a proof whose relation to the main result to be proved is far from clear. It will help if the proof author has broken this assertion out of the main text and labelled it a "lemma," but even then, it is often far from clear why it will be needed later on. This occurs because the values of professional mathematicians are quite different from those of students, and of most users of computer-based theorem provers: professional mathematicians like to demonstrate their technical dexterity with proofs that surprise readers (see Reality and Human Values in Mathematics).

The "Syd Field" structure might suggest first showing how a proof fails without a lemma (in dramatic terms, this gives rise to a conflict), and then showing how it goes through with the lemma - the "first act" will of course set up the proof, including what is to be proved and what is assumed. We used this structure in proving a simple property of flags on our Tatami project website (you may ignore the proof details, and look at the explanation for why the first attempt at the main result fails, motivating the second attempt, which succeeds with a lemma).

It is also interesting to consider the problem of translating from one language to another. This has been much studied in computer science, and with today's increased processing and memory power, is finally bearing some fruit (though sometimes that fruit may seem a bit raw - e.g. the translation option on websearch engines often gives results that are amusing and/or depressing). Semiotic morphisms illuminate some of the difficulties. It will help bring out the issues if we focus on the particularly difficult case of poetry, noting that less severe versions of the same difficulties can arise in any form of language. First, notice that poems may have, or fail to have, many different kinds of structure; for example, they may or may not be divided into stanzas, have a fixed meter, or rhyme; they are usually divided into lines, but even this does not always hold. Moreover, some poems have a geometrical form that is of interest (e.g., poems by e e cummings), and many different conventions are used for punctuation. In general, poems simultaneously exhibit more than one structure. All of this is easily expressed using sign systems, with various kinds of constructors and relations. For example, iambic pentameter is a particular sign system for syllabification. Sonata form plays a similar role in music from the classical period.

Secondly, notice that structures are often more natural for one language than for another. For example, it is much easier to rhyme in Romance languages like Spanish and Latin than in English. From this, it follows that it may be desirable to fail to preserve certain features, such as rhyme, when translating across languages; and it may even be desirable to add some feature to the translation that was not present in the original, e.g., adding rhyme when translating from Chinese to Spanish. Of course, every feature may be preserved to a certain extent, rather than being either fully preserved or not preserved.

Many people agree that mathematical proofs are an area where better design could have a big impact, e.g., in K12 education. The above ideas provide a way to explore how computer technology might help with this. Of course, there are many other design areas where making the content more dramatic could help, such as textbooks, courses, and manuals; private sector media already make extensive use of drama in their work, e.g., look at TV newscasts, or "real crime" documentaries, or other "reality" programming. But it should not be thought that drama is the only issue in narrative; the extensive literatures on creative writing and the novel raise many other issues. Also, there is the structure of oral narratives of personal experience that we studied in The Structure of Narrative, due to the linguist William Labov and others, and there is a famous discussion of the semiotic structure of Russian fairy tales due to Vladimir Propp. The main point of the present discussion is to claim that any instance of such a structure can be seen as arising through a semiotic morphism to a sign system that embodies the given structure. A text can then be seen as a blend of all the structures that are involved. I also claim that being aware of this viewpoint can help designers in their practical work, if they are also aware of the variety of narrative structures that are potentially applicable.

It is interesting to see how our theory of semiotic morphisms solves a problem that has been noticed by many theorists of narrative structure, which is that texts often fail to include some of the features that are supposed to be part of the generic structure; even important features are sometimes left out. This corresponds to the fact, with which we are already familiar, that semiotic morphisms can be partial, i.e., they can fail to preserve some aspects of source signs. Moreover, the fact the target is a sign system, not just some fixed given structure, means that many different structures, perhaps with a variety of substructures, etc., can be possible, not just some single fixed structure. This allows a great deal of flexibility. For example, the Labov structure permits arbitrary sequences of narrative clauses, and allows many different kinds of evaluative material. Moreover, its opening and closing sections are optional. Artists often play with structure to create interesting effects; for example, false endings in classical symphonies.

The Labov structure can be applied in many different ways. For example, user manuals for computer systems often describe sequences of steps, and these may be less tedious to read if they are given a more narrative-like structure, although this can of course be overdone, with unpleasant results, as we previously saw with humor and cuteness. The approach here is to consider semiotic morphisms from the Labov narrative structure sign system.

10.4 A Flame on the Information Scandal

It is said that we live in an "Age of Information," but it is an open scandal that there is no theory, not even a definition, of information that is both broad enough and precise enough to give such an assertion much meaning. An appropriate theory would help us to understand and to design information systems, in a wide sense that includes computer-based systems as well as systems that are based on more traditional media such as paper. However, a major motivating example is Information Systems in the narrow sense of computer-based systems for storing and retrieving information, e.g., database systems. User interface design also provides valuable insights into the kinds of problem that are important.

The need for a good theory of information is pressing. Society is demanding ever larger and more complex information systems. Billions, perhaps trillions, are spent each year on software, but many systems that are built are never used, and at least one third of systems begun are abandoned before completion. Moreover, many systems once thought adequate no longer are, while many others were never adequate. Among many sobering examples are the disastrous baggage handling system at the Denver International Airport, an IBM default on an 8 billion dollar contract to build the next generation U.S. air traffic control system, and a major IBM public relations disaster with its computer feed of real time sports data to journalists at the Atlanta Olympic games. Our knowledge of how to build effective information systems is very far from meeting the needs of society. Errors in requirements, that is, in understanding what kind of system is needed, have been identified as the most important problem, and it is also widely agreed that social factors are the most important source of difficulty in writing good requirements for large and complex systems. Thus it is very dangerous to ignore the social dimension of information!

This implies that an adequate theory of information would have to take account of social context, including how information is produced and used, rather than merely how it is represented; that is, we need a social theory of information, not merely a theory of representation. On the other hand, formal aspects of information are inherent to technical systems: computers are engines for storing, processing and retrieving formal representations. Thus the essence of designing such systems successfully is to reconcile their social and technical aspects. In addition to these practical problems, another important issue is the intellectual coherence of offerings within departments devoted to computer science, information science, etc. The lack of an adequate notion of information may be even more of a scandal here, due to the historic emphasis on adequate theoretical foundations in the academic world. The very widespread ignoring of the social aspects of computation and information is also highly problematic in this context.

But perhaps it is impossible to find an adequate theory of information. Bowker has discussed mythologies that support the notion of information, Haraway has given a daring modern cyborg myth, and Agre has argued that the notion of information is itself a myth, mobilized to support certain institutions, such as libraries. Nevertheless, in the paper Towards a Social, Ethical Theory of Information, I make an attempt to show how a notion of information can be grounded in the dual aspects of the social and the structural, and in addition, argue that information has an inherent ethical dimension. This theory is a social semiotics, and it could be taken as a theoretical foundation for this course, although I have chosen not to emphasize it; however it does seem that the practical considerations developed in this course lend much support to such a theory of information.

Undoubtedly the best known and most popular information theory today is that of Claude Shannon. Perhaps its most basic concept that of the bit, which of course is very fundamental in computer science. The number of bits associated with some information is really just a measure of its size, and is useful for determining the amount of memory needed for storing it, or how long it would take to download it. However, the number of bits in a file tells us nothing about its content. So, although this is a valuable theory in its proper domain, it is of no use for the broader challenges that we must face in this course, such as preserving the most important information when designing an overview of some information source (e.g, on the homepage of a large website).

Shannon's theory of information is a reductionist, scientific theory, with many real engineering applications, and so many attempts have been made to extend it to cover not just the size, but also the content, of information. None of these attempts have been successful, and I would claim that no reductionist theory can possibly succeed, because no such approach can take account of the concepts, methods, and values of the members of social groups, which we know from our study of ethnomethodology, are esential to understanding how real information/social systems actually work.

In fact, semiotics is an information theory of exactly the kind we need, provided it is considered to be grounded in social reality, rather than in Plato's abstract mathematical heaven. Thus, we have been studying information theory all quarter! Social foundations for semiotics are covered in some detail in the paper Towards a Social, Ethical Theory of Information, and are also briefly reviewed in the webpaper The Ethics of Databases, which is actually mainly concerned with the reverse process, of inferring the values of the designers from a structural analysis of an interface. This is a very interesting kind of exercise, with many potential applications; I use the name natural ethics for the broad project of extracting values from artificial objects, regarded as signs. The first key to this project is that the levels and priorities of sign systems, and the preservation properties of semiotic morphisms, reflect the importance that a designer has assigned to sign parts, and these should of course correspond to the values of the intended community of users.

10.5 A Note on Andersen's Dynamic Logic

This is an extremely interesting piece, although parts are rather difficult, so I suggest that if you read it, you can skip anything that doesn't make sense to you. We may go over some of the material in class. There are many interesting ideas here that could be explored in a project for the class. Among other things, this paper discusses the logic of multiple contradictions in narrative (building on Greimas), the relation between static and dynamic analyses of narrative, modal logic, fuzzy logic, and more, all in the dynamical systems paradigm of the paper we read earlier, Multimedia Phase-spaces. If you get interested in this, you should also look at my Notes on Gradient Logic.

To CSE 171 homepage
To the next section of the class notes
To the previous section of the class notes
Maintained by Joseph Goguen
© 2000 - 2005 Joseph Goguen, all rights reserved.
Last modified: Tue May 31 11:30:15 PDT 2005