We distinguish between conceptual spaces and semiotic theories (also called semiotic systems and semiotic spaces). Conceptual spaces, originally called "mental spaces," are due to Gilles Fauconnier, amd are used in cognitive linguistics, while semiotic theories have been developed for applications to user interface design. Conceptual spaces have only constants (called elements) and simple axioms which assert that certain relations hold among certain elements, whereas semiotic theories allow multi-argument constructors, as well as sorts, levels, priorities, and complex axioms with variables. Similarly, we should distinguish between conceptual blending, also called conceptual integration, and structural blending, which we may as well also call structural integration, where the former is blending of conceptual spaces and the latter is blending of semiotic theories, which may involve non-trivial constructors. While conceptual spaces seem adequate for the meaning of language, they are not adequate for user interface design, or for other applications where structure is important, such as music. For example, conceptual spaces and conceptual blending can help us understand concepts about music, but semiotic spaces and structural blending will be needed for dealing with the structure of music, e.g., with how a melody can be combined with a sequence of chords. Conceptual spaces are good for concepts about (i.e., how we talk about) things, but are more awkward for talking about the structure of things. It also seems that greater cultural variation can be found in conceptual blending than in structural blending, because the former deals with concepts about something, whereas they latter deals with the structure of its representations and/or instances.
There is an important duality between semiotic theories and their models. The former involve constructors, axioms, levels, etc., while the latter are concrete realizations, generally described using set theory. This course has not paid much attention to the models, but instead has worked with theories. One reason to prefer theories over models is that they provide axioms to constrain the allowable representations. For example, in considering the semiotic morphisms that produce indices from books, we may well want to impose axioms on the target space of indices, requiring that indexed items must be phrases of 1, 2 or 3 words, and that the page total for indexed phrases must occupy not more than 2% of a book's total pages. A confusing point is that it is convenient to speak of the class of models of a theory (e.g., all the books that have the structure given by a theory T of books) as a "space," or even as "the semiotic space of T"; this terminology unfortunately conflicts with the use of "space" in the phrase "semiotic space", as well as with its use in the "conceptual spaces" of cognitive linguistics, which are are generally considered to describe just one model, rather than a set of models.
We have taken the direction of a semiotic morphism to be the direction that models or instances of the theories are mapped. For example, if B is is a semiotic theory for books and T is a semiotic theory for tables of content, then books (which are the models of B) are mapped to their tables of contents, which are models of T. But a potentially confusing point is that the way we describe this mapping of models is via an inclusion T -> B of theories, because of structure of tables of contents is a substructure of the structure of books. It may seem rather counter-intuitive that, from a technical point of view, the mapping on models goes in the opposite direction from that between the theories, but this is an essential part of the duality between theories and models (technically, this kind of duality is called a "Galois connection"). Note that this is also consistent with the fact that blending diagrams are drawn "upside down" in algebraic semiotics, compared with cognitive linguistics.
Dynamics is clearly a necessary aspect of user interface design, but is difficult to handle in conceptual spaces, and is hardly treated at all by classical semiotics. Some algebraic semiotic approaches to dynamics are discussed in Section 6.3.
The following semiotic method for synthesis is recommended:
Let's first consider books. If you look at the physical structure of a book, you will see that it has the most important information printed on its front cover and spine, and that inside it has pages; basic information on the cover and spine include title, author, edition number, and publisher; this information also appears on the title page inside, along with the date of publication (or possibly this is on the back of the title page - publishers may want to make the date harder to find, with the hope that users may not realize that a book has become old). Looking at the contents, you will see that chapters are (usually) the main structuring device, and that their main selectors give a chapter number, a title, and a page number; chapters are (usually) divided into sections, and possibly subsections, each of which also has a number, title, and page. I think these are the things that anyone would eventually discover, even if they were not already familiar with books. Taking them as constructors and selectors, with target medium a small number of printed pages, yields exactly the very familiar form known as an outline. Notice that the entire content of the book has been lost (unless you count titles as content).
Another example is source code. Surely the most important structuring device is the division into files (if there is more than one file). To see what attributes (i.e., selectors) are important, we can check what the unix ls command displays under various options. For example, ls -lat will display the files in a directory, with their name, owner, size, and date of last modification. If we preserve these and display them in a natural way in a color graphics window, taking account of human perceptual capabilities, we can get something much like the code browser built at Bell Labs , which displays file structure in blocks, file age using color, and file size using block size. Notice that the entire content of the program, i.e., all of its code, has been lost (however, it can be viewed with the zoom feature that this system also provides).
It is easy to find many other examples, such as the detailed analysis of scrollbars given in Semiotic Morphisms, Representations, and Blending for User Interface Design and briefly summarized in section 6 of the class notes. The conclusion is that algebraic semiotics is a powerful tool for designing and evaluating user interfaces in general, and interfaces to collections of data in particular (this phrasing is intended to include not just databases, but also file systems, digital libraries, etc.).
The following four principles summarize some significant contributions that algebraic semiotics can make to the design process:
Although Principle F/C is probably the most important, and at this time is certainly the most thoroughly studied and supported, there are other principles that deserve attention, although the range of their applicability has not yet been carefully examined: Principle L/P says it is more important to preserve levels than to preserve priorities; Principle HL/LL says it is more important to preserve higher levels than lower levels; Principle HL/C says it is more important to preserve high levels than content; Principle P/C says it is more important to preserve priorities than content.
The following outlines a recommended semiotic method for analysis of an interface; it resembles what is done in contemporary semiotic analyses in the humanities, but is both more limited and much more precise. Note that essentially anything can be regarded as an "interface" for the purposes of this method.
What the McCracken & Wolfe call "organizational structure" is important because it determines the structure of the displays that can be used; website navigation structure is a prime example. There are many ways to represent such structures. For the moment, we focus on just two. The first is a direct reflection of a single hierarchical structure, such as that for an online clothing store. The second defines a more comprehensive semiotic theory of all possible finite hierarchical structures; this is a system of signs, consistent with Saussure's advocacy of considering signs in systems rather than individually in isolation. In the first case, constructors correspond to the menus for each sub-hierarchy, while in the second, the constructors build arbitrary menus and sub-hierarchies. In the first case, semiotic morphisms represent a single structure, whereas in the second case, they describe a particular uniform way to represent all such structures. We will see that the notions of level and priority also differ in the two cases.
The organizational structure of a website is usually a finite tree, or perhaps a finite acyclic graph if some substructure can be accessed in more than one way. We formalize the tree case: there are finitely many sorts, denoted s, s/n, s/n/m, s/n/m/k, ... where n,m,k,... are positive integers, and where the number of slashes indicates the level; there is just one level 0 or top sort, namely s, there are r first level sorts, s/1,...,s/r, and so on. Each sort is a node in the hierarchy tree, and the parent of a node s/n/....m/k is s/n/.../m, obtained by deleting its last integer.
Each non-leaf node N has exactly one constructor, which we shall call a menu, though it could be implemented as a navigation bar or some other way. A node N with k child nodes has a constructor of the form c(x1,...,xk) of sort N, where x1,...,xk are k arguments of sorts N/1,...,N/k, and where c is the title or label of the node; a leaf node N has no children, but still has a constant constructor of sort N. Priorities can be considered implicit in these constructors, with x1 having the highest priority, x2 the next, ..., and xk the lowest priority. An alternative is to assume that the ordering x1,...,xk represents the order of choices in the menu, which in general differs from their priority (e.g., the last element in a long list is more prominent than the fifth, and therefore presumably has a higher priority); in this case, explicit priority orderings can be given to the arguments of the constructors.
The following describes parts of the first three levels of a simple example: let the top level constructor be DansClothingStore(x1,x2,x3) of sort s, where x1,x2,x3 have sorts s/1,s/2,s/3 respectively; DansClothingStore is the label for the top level, i.e., the site name. Let Departments(y1,y2,y3), Specials(y1), ContactUs(y1,y2) be constructors of sorts s/1, s/2, s/3 respectively, with y1,y2,y3 of appropriate sorts; these are the second level constructors. Let Women(z1,z2), Men(z1,z2), Kids(z1,z2) be constructors of sort s/1/1; these are some of the third level constructors. Then we can construct the site hierarchy as the term
t = DansClothingStore(Departments(Women(t1,t2), Men(t3,t4), Kids(t5,t6)), Specials(t7), ContactUs(t8,t9))where t1,...,t9 are terms that describe the deeper structure of the site, which we omit for simplicity. Note that t is the only constant of sort s that can be constructed in this particular theory. The following tree shows the (partial) organization of this website:
In the second approach, each constant term of top sort in the semiotic theory describes a hierarchical structure like that of the single structure constructed in the first approach. This larger semiotic space has the following sorts: Nat (for natural numbers), NatList (for lists of Nats), String (for strings of characters), and Node (for nodes). The constructors for NatList are null (the empty list) and _._ (for append). Let Nat and Node be subsorts of NatList, so that every natural number and every node is a NatList. Here nodes are described by the paths that reach them, and we let t be an "attribute" function Node -> String that assigns titles to nodes. The level of a node is thus its length, and the priority of each child of a node is given by its final integer (or alternatively, by a separate priority ordering).
In the second approach to the same clothing store, null, 1, 2, 3 are nodes, as well as n/m for appropriate pairs n,m, and n/m/r for appropriate n,m,r; and t(null) = "Dan's Clothing Store", t(1) = "Departments", t(2) = "Specials", t(3) = "Contact Us", and similarly for all the t(n/m) and t(n/m/r). Fully formalized versions of these two approaches are given in Section 5.5 below, using the BOBJ algebraic specification language.
Note that in both approaches, nodes are represented by the paths that reach them from the root, although this is only a notational convention for the first approach. Moreover, whereas the first approach must explicitly construct each node it uses, the second constructs all possible nodes (as paths) and then selects the ones it needs.
The slogan "Content organization drives visual organization" on page 82 (Chapter 5) of McCracken & Wolfe is an obvious corollary of our principle that good designs are semiotic morphisms, from a source space to a display space, that optimize our preservation principles. Preserving the website structure in a visual display has exactly the meaning of making it easy to navigate the site, including knowing where you are, and where you can go from there.
The four principles on page 83 of McCracken & Wolfe are taken from Geltalt psychology; each has to do with similarity and difference. Although the principles are good, they are very vague, and little guidance is given on how to apply them in practice. In fact, something very essential has been left out, namely that ordering relations in the content organization space should be reflected (or "preserved", in our language of morphisms) by the display. These relations include the hierarchical level of nodes, and the relative importance of the child nodes of a given node at a given level. What needs to be said (and is not said in the text) is that the display should reflect the similarities of the child nodes of a given node, as well as the differences in their importance; moreover, the display of nodes that are distant should reflect that difference, while still preserving an overall similarity of style, due to being in the same hierarchy.
It is easy to see that the positive examples in McCracken & Wolfe do exactly this; e.g., the six groups of links in Figure 5-2 are arranged by the size of the card-sorted groups (used as a measure of their importance), and there is even an (implicit) ordering within groups by the importance of items, whereas Figure 5-1 fails to indicate either levels or priorities.
Algebraic semiotics can explain and improve the four principles in McCracken & Wolfe, making them both more general and more precise. We first observe that the Alignment Principle just describes one way to satisfy the Consistency Principle, which can also be satisfied, for example, with color or with size; although alignment is very basic to the way that most browsers display many HTML commands, it is not necessarily basic for more creative graphical layouts, or even for all the natural ways to present HTML; for example, consider speech generation for blind users. Thus Alignment is a secondary principle. Exactly the same can be said for the Proximity Principle. Thus, Consistency and Contrast are the most basic of these four principles. However, we prefer to use the terms similarity and difference for the fundamental intuitions involved, and will do so hereafter.
Thus, our task is to explain how resources for displaying similarity and difference should be used, i.e., which items should be made to look similar or different, and to what degree. To preserve levels in the source semiotic space means to display items at the same level similarly, and to display items at different levels differently, so that the more different the levels are, the more different their displays are. Moreover, the difference in the display should reflect the nature of the underlying difference in level. This is exactly what is accomplished by hierarchical menus, nested navigation bars, indented menus, etc., but not by a random listing of all nodes.
Preserving the priorities of the child nodes of a given node in the source space means that, although they should look similar (by the principle of level preservation), the differences in their priorities should also be clearly displayed; this is usually done by careful allocation of their linear position in a list. Finally, our Principle L/P says it is more important to preserve (the similarities and differences of) levels than (the similarities and differences of) priorities. This means that the display should reflect levels more clearly than priorities, which again is nicely accomplished by hierarchical menus, nested navigation bars, etc., but not a random listing of all nodes.
In summary, our principles of Level and Priority Preservation, with the Optimality Principle L/P, provide precise guidance for determining similarity and difference when designing a display for a hierarchy. Moreover, these principles apply much more generally than just to organizational hierarchies. The four Gestalt principles can be considered corollaries that apply to the special case of certain visual displays.
Nested navigation bars preserve levels, priorities, and the structure of paths in a particularly nice way. Each level appears as a "bar," i.e., a menu for the relevant choices at that level, with the item chosen highlighted, and with a thick line indicating the connection (i.e., path) from that item to the next bar (menu); menus not on the chosen path are not shown, and the path itself appears as a sequence of thick lines. This is superior to the more common nexted (indented) linear display in its more direct representation of the underlying path.
Below we give precise algebraic specifications for the two approaches to describing structural hierarchies given in Section 5.3. The notation used in described in class, but you may also consult Formal Notation for Conceptual blending, although its exposition is more technical.