CSE 271: User Interface Design: Social and Technical Issues
10. More Notes on Algebraic Semiotics, and Other Topics

There is an important basic duality between designing signs and interpreting signs: in the first, we know about signs in the source system, and we map them to a target system that can best preserve the information of greatest interest, whereas in the second, we know about one or more target sign, and we seek to infer signs in the source system from which it might have come, which also requires us to do some inference about the mapping. Examples of design that go beyond the usual computer science applications of using some mixture of media to present some given content, include giving a good explanation for something, finding a good "icon" (in the usual informal sense of computer graphics), choosing a good file name, and making a good analogy. Examples of the dual situation include understanding graphics, multimedia texts, verbal explanations, poems, films, metaphors, equations, or indeed, anything at all. (It is interesting to notice that people often in fact do not fully understand signs, e.g., films, even though they may spend a lot of time with them, and enjoy them.) Another very important class of examples concerns the evaluation of signs, as requested in several of the homework problems for this class: here you must not only figure out what the source sign system and semiotic morphism might be, but also what some alternative semiotic morphisms, to potentially different target sign systems, might be.

In each case, there is a transformation (movement, translation, interpretation or representation) of signs in one system to signs in another system, and the examples of design go in the opposite direction from the examples of understanding. Thus algebraic semiotics helps us see the dual relation between design and understanding, and also suggests ways that each process can contribute to the other:- In general, it is easier to investigate understanding than it is to investigate design, which after all is inherently more creative. But once we understand something about a mode of understanding, we can apply it to the corresponding design problems; of course the reverse trajectory also works, but it will be less common. There are many interesting applications of this approach, one of which is designing visualizations for scientific information, as discussed earlier in the course. Another is to developing a better understanding of humor.

Many newspaper cartoons, consisting of 1 to 4 small scenes (i.e., panels), achieve their effect by setting up some situation, and then recontextualizing it, i.e., introducing new elements and relations into the conceptual space, which have the effect of forcing a new organization for some parts of the conceptual space that was originally set up. The first space is in general itself some kind of blend, and its reconceptualization is also a blend, of the old space with the new material; this often has a humorous effect. Similar phenomena can be found in music, poetry, and probably in every art form, though the effect is by no means always comic. It seems to me clear that evolution has provided us with positive feedback for improvements in understanding in the form of mental pleasure, since this has an obvious survival value. One familiar example is the so called "Eureka" experience, when we suddenly see the solution to some problem that we have been pondering for a long time.

In light of our success with viewing an oxymoron as an inconsistent pair of blends, often with a "cross space" mapping which imports at least some of the (generally less conventional) contradictory meaning into the (generally more conventional) blend, it seems reasonable to believe that some fairly large areas of humor can be characterized in terms of reblending. Once we have this understanding, we are in a position to apply it to design. For example, we might want to make the use of certain difficult interfaces a lighter and more pleasant task by (carefully and selectively!) introducing some humor. It is important to notice that, because the psychological impact of recontextualization depends on its novelty, repeating the same joke again and again will not be effective interface design, and in fact, will prove irritating to users. Many designs have had to be redone for this or closely related reasons. For example, overly cute icons or avatars can quickly become irritating (or slowly, if they are a bit less cute), and cuteness as a semiotic phenomenon is closely related to humor, in that it involves partially conflicting interpretations (e.g., child-like facial features but a serious message). Two current instances from MicroSoft include the barking dog in PowerPoint and the obsequious paper clip in Word (although we are told that this is soon to become a non-default option).

We can also understand (some aspects of) narrative structure in terms of semiotic morphisms. For example, books on screen writing by Syd Field prescribe a precise (but also naive) dramatic structure for the plots of Hollywood movies: they should have three acts, for setup, conflict, and resolution, with "plot points" that move action from one act to the next. We can describe this "Syd Field structure" using a very simple sign system with a main constructor that builds a thing of sort "plot" from three things of sort "act", and we can then check whether a given film has this structure by seeing whether there is a semiotic morphism from the (structure of the) film to this sign system.

It is clear that as internet bandwidth grows, video will be used more and more; moreover, complex interactive websites are aready common, many with story lines. Techniques for drama apply directly to such cases, but are also much more broadly applicable, in suggesting ways to make displays (i.e., "texts" in the very general sense of signs that are to be interpreted by users) more interesting. My own favorite examples of this come from mathematics. Probably we have all had the experience of trying to read a proof and being frustrated, at least initially, by not being able to see where it was going, or why it was structured as it was. This is often because the proof author did not describe the difficulties that arose in constructing the proof, but instead just described the machinery that was constructed to overcome the difficulties. For example, one often comes upon an assertion in the middle of a proof whose relation to the main result to be proved is far from clear. It will help if the proof author has broken this assertion out of the main text and labelled it a "lemma," but even then, it is often far from clear why it will be needed later on.

The "Syd Field" structure might suggest first showing how a proof fails without a lemma (in dramatic terms, this gives rise to a conflict), and then showing how it goes through with the lemma - the "first act" will of course setup the proof, including what is to be proved and what is assumed. We used this structure in proving a simple property of flags on our Tatami project website (ignore the proof details, and look at the explanation for why the first attempt at the main result fails, motivating the second attempt, which succeeds with a lemma).

It is also interesting to consider the problem of translating from one language to another. This has been much studied in computer science, and with today's increased processing and memory power, is finally bearing some fruit (though sometimes that fruit may seem a bit raw - e.g. if you try the translation option on some websearch engines, the results may be amusing and/or depressing). Semiotic morphisms illuminate some of the difficulties. It will help bring out the issues if we focus on the particularly difficult case of poetry, noting that less severe versions of the same difficulties can arise in any form of language. First, notice that poems may have, or fail to have, many different kinds of structure; for example, they may or may not be divided into stanzas, have a fixed meter, or rhyme; they are usually divided into lines, but even this does not always hold. Moreover, some poems have a geometrical form that is of interest (e.g., poems by e e cummings), and many different conventions are used for punctuation. All of this is easily expressed using sign systems, with various kinds of constructors and relations. For example, iambic pentameter is a particular sign system for syllabification. Sonnata form plays a similar role in music from the classical period.

Secondly, notice that structures are often more natural for one language than for another. For example, it is much easier to rhyme in Romance languages like Spanish and Latin than in English. From this, it follows that it may be desirable to fail to preserve certain features, such as rhyme, when translating across languages; and it may even be desirable to add some feature to the translation that was not present in the original, e.g., adding rhyme when translating from Chinese to Spanish. Of course, every feature may be preserved to a certain extent, rather than being either fully preserved or not preserved.

Many people agree that mathematical proofs are an area where better design could have a big impact, e.g., in K12 education. The above ideas provide a way to explore how computer technology might help with this. Of course, there are many other design areas where making the content more dramatic could help, such as textbooks, courses, and manuals; private sector media already make extensive use of drama in their work, e.g., look at TV newscasts, or "real crime" documentaries, or other "reality" programming. But it should not be thought that drama is the only issue in narrative; the extensive literatures on creative writing and the novel raise many other issues. Also, there is the the structure of oral narratives of personal experience that we studied at The Structure of Narrative, due to the linguist William Labov and others, and there is a famous discussion of the semiotic structure of Russian fairy tales due to Vladimir Propp. The main point of the present discussion is to claim that any instance of such a structure can be seen as arising through a semiotic morphism to a sign system that embodies the given structure. I also claim that being aware of this viewpoint can help designers in their practical work, if they are also aware of the variety of narrative structures that are potentially applicable.

It is interesting to see how our theory of semiotic morphisms solves a problem that has been noticed by many theorists of narrative structure, which is that texts often fail to include some of the features that are supposed to be part of the generic structure; even important features are sometimes left out. This corresponds to the fact, with which we are already familiar, that semiotic morphisms can be partial, i.e., they can fail to preserve some aspects of source signs. Moreover, the fact the target is a sign system, not just some fixed given structure, means that many different structures, perhaps with a variety of substructures, etc., can be possible, not just some single fixed structure. This allows a great deal of flexibility. For example, the Labov structure permits arbitrary sequences of narrative clauses, and allows many different kinds of evaluative material. Moreover, its opening and closing sections are optional. Artists often play with structure to create interesting effects; for example, false endings in classical symphonies. See The Structure of Narrative for the details of narrative structure.

The Labov structure can be applied in many different ways. For example, user manuals for computer systems often describe sequences of steps, and these may be less tedious to read if they are given a more narrative-like structure, although this can of course be overdone, with unpleasant results, as we previously saw with humor and cuteness. The point here is to consider semiotic morphisms to the Labov narrative structure sign system.

A Flame on the Information Scandal

It is said that we live in an "Age of Information," but it is an open scandal that there is no theory, not even a definition, of information that is both broad enough and precise enough to give such an assertion much meaning. An appropriate theory would help us to understand and to design information systems, in a wide sense that includes computer-based systems as well as systems that are based on more traditional media such as paper. However, a major motivating example is Information Systems in the narrow sense of computer-based systems for storing and retrieving information, e.g., database systems. User interface design also provides valuable insights into the kinds of problem that are important.

The need for a good theory of information is pressing. Society is demanding ever larger and more complex information systems. Billions, perhaps trillions, are spent each year on software, but many systems that are built are never used, and at least one third of systems begun are abandoned before completion. Moreover, many systems once thought adequate no longer are, while many others were never adequate. Among many sobering examples are the disastrous baggage handling system at the Denver International Airport, an IBM default on an 8 billion dollar contract to build the next generation U.S. air traffic control system, and a major IBM public relations disaster with its computer feed of real time sports data to journalists at the Atlanta Olympic games. Our knowledge of how to build effective information systems is very far from meeting the needs of society. Errors in requirements, that is, in understanding what kind of system is needed, have been identified as the most important problem, and it is also widely agreed that social factors are the most important source of difficulty in writing good requirements for large and complex systems. Thus it is very dangerous to ignore the social dimension of information!

This implies that an adequate theory of information would have to take account of social context, including how information is produced and used, rather than merely how it is represented; that is, we need a social theory of information, not merely a theory of representation. On the other hand, formal aspects of information are inherent to technical systems: computers are engines for storing, processing and retrieving formal representations. Thus the essence of designing such systems successfully is to reconcile their social and technical aspects. In addition to these practical problems, another important issue is the intellectual coherence of offerings within departments devoted to computer science, information science, etc. The lack of an adequate notion of information may be even more of a scandal here, due to the historic emphasis on adequate theoretical foundations in the academic world.

But perhaps it is impossible to find an adequate theory of information. Bowker has discussed mythologies that support the notion of information, Haraway has given a daring modern cyborg myth, and Agre has argued that the notion of information is itself a myth, mobilized to support certain institutions, such as libraries. Nevertheless, in the paper Towards a Social, Ethical Theory of Information, I make an attempt to show how a notion of information can be grounded in the dual aspects of the social and the structural, and in addition, argue that information has an inherent ethical dimension. This theory is a social semiotics, and it could be taken as a theoretical foundation for this course, although I have chosen not to emphasize it here; however it does seem that the practical considerations developed in this course lend support to such a theory of information.

Undoubtedly the best known and most popular information theory today is that of Claude Shannon. Perhaps its most basic concept that of the bit, which of course is very fundamental in computer science. The number of bits associated with some information is really just a measure of its size, and is useful for determining the amount of memory needed for storing it, or how long it would take to download it. However, the number of bits in a file tells us nothing about its content. So, although this is a valuable theory in its proper domain, it is of no use for the broader challenges that we have faced in this course, such as preserving the most important information when designing an overview of some information source (e.g, the homepage of a large website).

Shannon's theory of information is a reductionist, scientific theory, with many real engineering applications, and so many attempts have been made to extend it to cover not just the volume, but also the content, of information. None of these attempts have been successful, and I would claim that no reductionist theory can possibly succeed, because no such approach can take account of the concepts, methods, and values of the members of social groups, which we know from our study of ethnomethodology, are esential to understanding how real information/social systems actually work.

In fact, semiotics is an information theory of exactly the kind we seek, provided it is considered to be grounded in social reality, rather than in Plato's abstract mathematical heaven. Thus, we have been studying information theory all quarter! Social foundations for semiotics are covered in some detail in the paper Towards a Social, Ethical Theory of Information, and are also briefly reviewed in the webpaper The Ethics of Databases, which is actually mainly concerned with the reverse process, of inferring the values of the designers from a structural analysis of an interface. This is a very interesting kind of exercise, with many potential applications; I use the name natural ethics for the broad project of extracting values from artificial objects, regarded as signs. The first key to this project is that the priorities and levels of sign systems, and the preservation properties of semiotic morphisms, reflect the importance that a designer has assigned to sign parts.

A Note on Andersen's Dynamic Logic

Although I think this is a terrifically interesting piece, I am afraid that parts may be too difficult for this class. Hence I suggest you read it over lightly, and pause whenever you find something that interests you. We will not follow up on this material.

To CSE 271 homepage
To the next section of the class notes
To the previous section of the class notes
Maintained by Joseph Goguen