There is an important basic duality between designing signs and interpreting signs: in the first, we know about signs in the source system, and we map them to a target system that can best preserve the information of greatest interest, whereas in the second, we know about one or more target sign, and we seek to infer signs in the source system from which it might have come, which also requires us to do some inference about the mapping. Examples of design that go beyond the usual computer science applications of using some mixture of media to present some given content, include giving a good explanation for something, finding a good "icon" (in the usual informal sense of computer graphics), choosing a good file name, and making a good analogy. Examples of the converse include understanding graphics, multimedia texts, verbal explanations, poems, films, metaphors, equations, or indeed, anything at all. (It is interesting to notice that people often in fact do not fully understand signs, e.g., films, even though they may spend a lot of time with them, and enjoy them.)
In each case, there is a transformation (movement, translation, interpretation or representation) of signs in one system to signs in another system, and the examples of design go in the opposite direction from the examples of understanding. Thus algebraic semiotics helps us see the dual relation between design and understanding, and also suggests ways that each process can contribute to the other. In general, it is easier to investigate understanding than it is to investigate design, which after all is inherently more creative. But once we understand something about a mode of understanding, we can apply it to the corresponding design problems; of course the reverse trajectory also works, but it will be less common. There are many interesting applications of this approach, one of which is designing scientific information visualizations, as discussed earlier in the course. Another is to better understanding humor.
Many newspaper cartoons, consisting of 1 to 4 small scenes (i.e., panels), achieve their effect by setting up some situation, and then recontextualizing it, i.e., introducing new elements and relations into the conceptual space, which have the effect of forcing a new organization for some parts of the concpetual space that was originally set up. The first space is in general itself some kind of blend, and its reconceptualization is also a blend, of the old space with the new material; this often has a humorous effect. Similar phenomena can be found in music, poetry, and probably in every art form, though the effect is by no means always comic.
In light of our success with viewing an oxymoron as an inconsistent pair of blends, often with a "cross space" mapping which imports at least some of the (generally less conventional) contradictory meaning into the (generally more conventional) blend, it seems reasonable to believe that some fairly large areas of humor can be characterized in terms of inconsistent blends. Once we have this understanding, we are in a position to apply it to design. For example, we might want to make the use of certain difficult interfaces a lighter and more pleasant task by (carefully and selectively!) introducing some humor. It is important to notice that, because the psychological impact of recontextualization depends on its novelty, repeating the same joke again and again will not be effective interface design, and in fact, will prove irritating to users. Many designs have had to be redone for this or closely related reasons. For example, overly cute icons or avatars can quickly become irritating (or slowly, if they are a bit less cute), and cuteness as a semiotic phenomenon is closely related to humor, in that it involves partially conflicting interpretations (e.g., child-like facial features but a serious message). Two current instances from MicroSoft include a barking dog in PowerPoint and an obsequious paper clip in Word.
We can also understand (some aspects of) narrative structure in terms of semiotic morphisms. For example, books on screen writing by Syd Field prescribe a precise (but also naive) dramatic structure for the plots of Hollywood movies: they should have three acts, for setup, conflict, and resolution, with "plot points" that move action from one act to the next. We can describe this "Syd Field structure" using a very simple sign system with a main constructor that builds a thing of sort "plot" from three things of sort "act", and we can then check whether a given film has this structure by seeing whether there is a semiotic morphism from the (structure of the) film to this sign system.
It is clear that as internet bandwidth grows, video will be used more and more; moreover, complex interactive websites are aready common, many with story lines. Techniques for drama apply directly to such cases, but are also much more broadly applicable, in suggesting ways to make displays (i.e., "texts" in the very general sense of signs that are to be interpreted by users) more interesting. My own favorite examples of this come from mathematics. Probably we have all had the experience of trying to read a proof and being frustrated, at least initially, by not being able to see where it was going, or why it was structured as it was. This is often because the proof author did not describe the difficulties that arose in constructing the proof, but instead just described the machinery that was constructed to overcome the difficulties. For example, one often comes upon an assertion in the middle of a proof whose relation to the main result to be proved is far from clear. It will help if the proof author has broken this assertion out of the main text and labelled it a "lemma," but even then, it is often far from clear why it will be needed later on. (See What is a Proof? for further background on this topic.)
The "Syd Field" structure might suggest first showing how a proof fails without a lemma (in dramatic terms, this gives rise to a conflict), and then showing how it goes through with the lemma - the "first act" will of course setup the proof, including what is to be proved and what is assumed. We used this structure in proving a simple property of flags on our Tatami project website (ignore the proof details, and look at the explanation for why the first attempt at the main result fails, motivating the second attempt, which succeeds with a lemma).
It is also interesting to consider the problem of translating from one language to another. This has been studied much studied in computer science, and with today's increased processing and memory power, is finally bearing some fruit (though sometimes that fruit may seem a bit raw - e.g. if you try the translation option on some websearch engines, the results may be amusing and/or depressing). Semiotic morphisms illuminate some of the difficulties. It will help bring out the issues if we focus on the particularly difficult case of poetry, noting that less severe versions of the same difficulties can arise in any form of language. First, notice that poems may have, or fail to have, many different kinds of structure; for example, they may or may not be divided into stanzas, have a fixed meter, or rhyme; they are usually divided into lines, but even this does not always hold. Moreover, some poems have a geometrical form that is of interest (e.g., poems by e e cummings), and many different conventions are used for punctuation. All of this is easily expressed using sign systems, with various kinds of constructors and relations. For example, iambic pentameter is a particular certain sign system for syllabification. Sonnata form plays a similar role in music from the classical period.
Secondly, notice that structures are often more natural for one language than for another. For example, it is much easier to rhyme in Romance languages like Spanish and Latin than in English. From this, it follows that it may be desirable to fail to preserve certain features, such as rhyme, when translating across languages; and it may even be desirable to add some feature to the translation that was not present in the original, e.g., adding thyme when translating from Chinese to Spanish. Of course, every feature may be preserved to a certain extent, rather than being either fully preserved or not preserved.
Many people agree that mathematical proofs are an area where better design could have a big impact, e.g., in K12 education. The above ideas provide a way to explore how computer technology might help with this. Of course, there are many other design areas where making the content more dramatic could help, such as textbooks, courses, and manuals; private sector media already make extensive use of drama in their work, e.g., look at TV newscasts. But it should not be thought that narrative structure is only about drama; many other structures can be found in the extensive literature on writing, among other places. For example, the linguist William Labov has studied the structure of oral narratives of personal experience, and based on extensive empirical research, has found that they consist of an (optional) abstract, an (optional but usual) orientation, a series of narrative clauses with accompanying evaluative material, and an optional closing or summary section. For a fairly simple (but entirely real) example annotated according to Labov's theories is online at story.html. Also Vladimir Propp has given a famous discussion of the semiotic structure of Russian fairy tales. The main point of the present discussion is to claim that any instance of such a structure can be seen as arising through a semiotic morphism to a sign system that embodies the given structure. I also claim that being aware of this viewpoint can help designers in their practical work, if they are also aware of a variety of narrative structures that are potentially applicable.
It is interesting to see how our theory of semiotic morphisms solves a problem that has been noticed by many theorists of narrative structure, which is that texts often fail to include some of the features that are supposed to be part of the generic structure; even important features are sometimes left out. This corresponds to the fact, with which we are already familiar, that semiotic morphisms can be partial, i.e., can fail to preserve some aspects of source signs. Moreover, the fact the target is a sign system, not just some fixed given structure, means that many different structures, perhaps with a variety of substructures, etc., can be possible, not just some single fixed structure. This allows a great deal of flexibility. For example, the Labov structure permits arbitrary sequences of narrative clauses, and allows many different kinds of evaluative material. Moreover, its opening and closing sections are optional. Artists often play with structure to create interesting effects; for example, false endings in classical symphonies.
It is not hard to formalize the Labov narrative structure as a sign system; it looks much the same as the sign system for the fragment of English grammar given in An Introduction to Algebraic Semiotics, with Applications to User Interface Design. The top level sort is of course for narratives, lets denote it Narr, and the second level sorts include Cls, for narrative clauses, which we take as potentially including evaluative material, Eval for a stand-alone evaluative clause, Open for the opening section, which may include an orientation and/or an abstract, and Coda for a closing section. There are several constructors in C1 for building narratives from other material, corresponding to various options for leaving out some material; the priority on these corresponds to how common or ordinary they are. The easiest way to summarize the situation is to give a syntactic expression, the instances of which correspond to the structures of narratives built according to the rules of this sign system. The following uses an "extended BNF" notation:
The Labov structure can be applied in many different ways. For example, user manuals for computer systems often describe sequences of steps, and these may be less tedious to read if they are given a more narrative structure, although this can of course be overdone, with unpleasant results, as we previously saw with humor and cuteness. The point here is to consider semiotic morphisms to the Labov narrative structure sign system.
This course has taken a non-traditional point of view that emphasizes the importance of social issues in user interface design, and even goes so far as to claim that good design must take account of the social context in which the system will be used, not just its technical aspects, and not even just its users as individuals. Unlike most computer science courses, this course is not focussed on implementation issues, in part because they are changing so fast that it seems more valuable to look at some of the more fundamental issues that determine the success of technical solutions. Of course technical issues cannot be neglected, since they generally determine what it is possible to build, which often is not what would be be optimal from a social point of view. The title of the course, "User Interface Design: Social and Technical Issues," is intended to convey this spirit, which has informed the design of the course from its beginning to its end, and it is also emphasized on the class's homepage, in its first lecture, and in this Afterward.
I believe that a good working knowledge of topics like adjacency pair, noticeable absence, mitigated speech act, semiotic morphism, narrative clause, and collaborative work can be tremendously helpful to user interface designers, enabling them to do a better job than others who are less aware of social issues, and who lack the tools to deal with such issues. This is espeically true for those who will go on to manage projects, which I expect most UCSD students will (UCSD does not usually admit students who will stay in terminal programming jobs).
Although somewhat radical from the viewpoint of traditional user interface design - and even traditional semiotics - sign systems and semiotic morphisms formalize and codify intuitions that are familiar to practitioners in many areas of communication and design, including journalism, architecture, cinema, sculpture, graphical design, and of course user interface design. These ideas include having a clear idea of the goals of a design, having a good grasp of the medium (i.e., sign system) to be used, knowing which operations and structures are most important, and of course understanding the social context. This in particular includes the importance of involving users in the design process, through techniques such as participatory design, evolutionary prototyping, and usability trials. More detailed principles include trading off content for structure in an overview, and preserving constructors and priorities in generated displays. These can be applied to justify many common design conventions, for example placing a slide bar on the right of a window using the same coloring scheme as the window.
The most important message of this course is that successful design depends on achieving a good match between technical capabilities and social context (e.g., "the market"), and that certain specific social theories can be useful in this regard. I cannot resist concluding this discussion with the slogan, "Nothing is more useful than a good theory."
It is said that we live in an "Age of Information," but it is an open scandal that there is no theory, not even a definition, of information that is both broad enough and precise enough to give such an assertion much meaning. An appropriate theory would help us to understand and to design information systems, in a wide sense that includes computer-based systems as well as systems that are based on more traditional media such as paper. However, a major motivating example is Information Systems in the narrow sense of computer-based systems for storing and retrieving information, e.g., database systems. User interface design also provides valuable insights into the kinds of problem that are important.
The need for a good theory of information is pressing. Society is demanding ever larger and more complex information systems. Billions, perhaps trillions, are spent each year on software, but many systems that are built are never used, and at least one third of systems begun are abandoned before completion. Moreover, many systems once thought adequate no longer are. Among many sobering examples are the disastrous baggage handling system at the new Denver International Airport, an IBM default on an 8 billion dollar contract to build the next generation U.S. air traffic control system, and a major public relations disaster with its computer feed of real time sports data to journalists at the Atlanta Olympic games. Our knowledge of how to build effective information systems is very far from meeting the needs of society. Errors in requirements, that is, in understanding what kind of system is needed, have been identified as the most important problem, and it is also widely agreed that social factors are the most important source of difficulty in writing good requirements for large and complex systems.
This implies that an adequate theory of information would have to take account of social context, including how information is produced and used, rather than merely how it is represented; that is, we need a social theory of information, not merely a theory of representation. On the other hand, formal aspects of information are inherent to technical systems: computers are engines for storing, processing and retrieving formal representations. Thus the essence of designing such systems successfully is to reconcile their social and technical aspects. In addition to these practical problems, another important issue is the intellectual coherence of offerings within departments devoted to computer science, information science, etc. The lack of an adequate notion of information may be even more of a scandal here, due to the historic emphasis on adequate theoretical foundation in the academic world.
But perhaps it is impossible to find an adequate theory of information. Bowker has discussed mythologies that support the notion of information, Haraway has given a daring modern cyborg myth, and Agre has argued that the notion of information is itself a myth, mobilized to support certain institutions, such as libararies. Nevertheless, in the paper Towards a Social, Ethical Theory of Information, I make an attempt to show how a notion of information can be grounded in the dual aspects of the social and the structural, and in addition, argue that information has an inherent ethical dimension. This theory is a social semiotics, and can be taken as a theoretical foundation for this course, although I have chosen not to emphasize it in this course; however it does seem that the parctical applications developed in the course lend some support to such a theory of information.