A converse situation is very often encountered, in which we know about the
target sign system, and seek to infer properties of signs in the source system
from their images in the target system. This occurs, for example, when we try
to understand a poem, an equation, a sentence, or indeed, anything at all.
Let us call this the **inverse problem**, as opposed to the **direct
problem** of design.

We address questions about the nature of translation between sign systems,
and the reasons for preferring one translation to another, by studying maps
from signs in one system to "represent" signs in another system. These maps
are called **semiotic morphisms**, and are made precise in Definition 2 of
the paper [B]. Examples include
metaphors, analogies, etc., as well as representations in the more familiar
user interface design sense. Just as we defined sign systems as theories
rather than models, so their mappings should translate from the
*language* of one sign system to the language of another, instead of
just translating the concrete signs in a model. This may seem indirect, but
it has important advantages over a model based approach to representations,
as we discussed in Section 2.

A good semiotic morphism should preserve as much of the structure of its
source sign system as feasible. Certainly it should map sorts to sorts,
subsorts to subsorts, data sorts to data sorts, constants to constants,
constructors to constructors, etc. But it turns out that in many real world
examples, not everything *can* be preserved. So we must allow all
these maps to be *partial*. Axioms should also be preserved - but
again in practice, sometimes not all axioms are preserved. The extent to
which things are preserved provides a way of comparing the **quality** of
semiotic morphisms (this is the point of Definition 3 in [B]).

As a first example of a semiotic morphism, suppose we want to represent
time-of-day (**TOD**) in the little screen
**W**. Clearly there are many ways to do this;
all of them must map the sort time to the sort screen, map the constructor 0
to some string of (less than 25) strings of (less than 81) characters, and
map the constructor s to a function sending each such string of strings to
some other string of strings. There isn't anything else to preserve in this
very simple example except the axiom, which however is very important.

Recall that the items of abstract syntax in **TOD** are strings of up to 1439 s's followed by a single 0.
One simple representation just maps these strings directly to strings of
strings of s's such that the total number of s's is the same. Let N(t) be the
number of s's in some t from **TOD**. Let Q(t)
and R(t) be the quotient and remainder after dividing N(t) by 80. Then there
will be Q(t) lines of 80 s's followed by one line of R(t) s's. This is
guaranteed to fit on our screen because Q(1439) = 17 < 25, and R(t) < 80
by definition. Although this is a digital representation, it is so detailed
for humans that it is essentially analog, in the sense that after becoming
familiar with it, a user gets a "feel" for the approximate number of (these
strange 80 minute) hours, and the number of minutes that remain. (The maximum
number of items that most humans can "subitize", i.e. immediately recognize
without counting, is 4 or 5.)

One obvious representation just displays N(t) in decimal notation, giving a string of 1 to 4 decimal digits. This is very different from our usual representations; but we could imagine some strange culture that divides its days into 18 "hours" each having 100 "minutes", except the last, which has 40. (Actually, this is less strange than what our culture actually does with months!) Here N(0) is 0, and s just adds 1, except that s(1439) = 0.

A more familiar representation is constructed as follows: Let N1 and N2 be
the quotient and remainder of N divided by 60, both in base 10, with 0's
added in front if necessary so that each has exactly 2 digits. Now form the
string of characters "N1 : N2". This is the so-called "military"
representation of time; let's denote it by M. Then M(0) = 00:00, and of
course you know how s goes. The spoken variant of military time has the form
"N1 hundred N2 hours" (unless N2 = 00, in which case it is omitted). The use
of "hundred" and "hours" may seem odd here, because it isn't hundreds and it
isn't hours! - but at least it's clear - and that's the point - the word
"hundred" functions like the colon in the written form, and it has a
distinctive sound. Notice that this representation has been defined as a
*composition* of N with a *re*-representation of **TOD** to itself.

I think you can now see for yourself how to construct other representations of time as semiotic morphisms, including even the "analog" representation of a clock face. Here 0 has both hands up, satisfaction of the axiom follows because something even stronger is true, namely s(719) = 0, which is built into the circular nature of this geometrical representation. You might think that satisfying this stronger axiom means that the representation is poor - but is it really? Think about why representation modulo 720 works, but (say) modulo 120 or modulo 300 do not work; what about modulo 360? Why? Hint: Think about the context of use of the representation.

Now let's imagine someone who has to design a text editor based on the
screen **W**; part of their job is to
construct a semiotic morphism *E* from
**TXT** to **W**. One issue to be addressed is the correct placement of
spaces, so that words are always separated, and sentences end with a period
followed by two spaces; another issue is the initial capital letter. The
designer will also have to decide what to do about words that want to go past
the end of a line, e.g., to wrap around, hyphenate, or fill in with extra
spaces. The limit of 24 lines will also pose problems.

If instead of **TXT** we have **PS** available for input to **W**, then a designer can consider some more sophisticated
issues, such as automatic insertion of commas. It may also be interesting to
think about representations from **PS** to
**TXT**. The most obvious and familiar one
just gives the unparsed form of the parsed sentence; but again we can imagine
adding some punctuation.

An interesting semiotic morphism from the experimental work of [9] is the correspondance between the physical order of lights on the experimental device, and the order of clauses describing those lights (it turned out to be narrative order).

Finally, a somewhat spectucular example of a semiotic morphism is the LaTeX2HTML program used to produce the original version of the document that you are now reading. LaTeX2HTML takes a LaTeX source file and produces an HTML website that preserves the structure of the source file. However, lIt is far from perfect, and a fair amount of editing was required to get a good quality first version of the current website; for example, subsections were always given separate pages, even when they were only a few lines long. A similar program for Windows, called TTH, has somewhat better quality.

To begin to formalize and extend the intuition that Peirce's 3-fold classification of signs reflects the "naturalness" or "immediacy" of their signification, notice that semiotic morphisms may preserve aspects of a given sign system to a greater or lesser degree, and that this degree affects the quality of the representations which it provides. The structure of the entire system of related signs must be considered in determining what makes one representation better than another, because the structure, priority and level of signs must be considered in constructing their representations.

The fuller exposition of this material given in [B] (further developing ideas from [9]) includes detailed definitions of what it
means for a semiotic morphism *M* to be **level preserving**,
**constructor preserving**, **priority preserving**, **attribute
preserving**, and **structure preserving**. It is important to notice
that semiotic morphisms need not be totally defined; that is, each of the
functions denoted *M* can be undefined on some of what is in *S1*.
For example, there need not be any representation in *S2* for some signs
in *S1*; in particular, some components of *M* could even be
totally undefined, i.e., it could be an empty function.

It might at first seem obvious that a good structure preserving semiotic
morphism *M : S1 -> S2* should faithfully represent *all* of the
semiotic structure of *S1* in terms of that available in *S2*. But
if the resulting structures in *S1* are too complex, then they may be
hard for human beings to understand. For example, if *S1* consists of
parse trees for English sentences and *S2* consists of the usual "printed
page" text format, then it is possible to translate all the syntactic
information that is available in *S1* into structures in *S2* with
so-called "phrase structure" notation, which uses brackets to delimit phrases
and uses subscripts on brackets to indicate the class of phrase is involved.
For example, the representation

may be useful for some purposes, but it is clearly not optimal for others. There is a trade-off between the degree of structure preservation and the complexity of the resulting representation.

It may be that neither of the morphisms *M*, *M'* preserves
strictly more structure than the other, or that one preserves more structure
but produces more complex representations. For example, *M* might
preserve more levels while *M'* preserves more attributes. The
experiments in [9] show that
preserving levels is more important than preserving priorities, which is more
important than preserving attributes. They also show a strong tendency to
preserve higher levels at the expense of lower levels. This was unexpected,
because of work by Rosch and her students explicating the notion of "basic
level" for lexical concepts (e.g., "bird") [14, 15]. For
natural language, the sentential level was long supposed basic, but much
recent research emphasizes the discourse level. In short, structure is more
important than content; it is also easier to describe precisely.

Note that when we consider preserving various structural aspects of signs, we are dealing with various forms of Pierce's notion of iconicity, i.e., with properties of the sign itself. It is more complex to characterize Peirce's notion of indexicality in our framework, because it involves relations that hold between a sign and its representation, which takes us into a larger semiotic space that includes both; this can be done, but we will not pursue it further here.

When we see a complex sign, such as a diagram in a technical report, or an
abstract for a lecture, we assume that it is intended to convey some
information, and we then try to infer what it is that the author intended.
In the language of this paper, we seek to infer a source sign from the given
target sign. The difficulty of course is that we know little or nothing
about the source sign system, or about the semiotic morphism (indeed, it is
doubtful that such a source sign system really exists in the usual sense).
Hence we must try to infer a systematic mapping from an unknown space of
intentions to another space of displays, such that the observed display is
the result of applying that mapping to the inferred source sign, under the
assumption that the author carefully chose the given sign as an appropriate
way to represent certain information. An important aid to this inference
process is the assumption that the author has used a *systematic*
mapping which will apply to other signs from the same space, not just the one
that we happen to have seen; this assumption allows us to eliminate many
irrelevant possibilities when inferring the author's intent. That is, as
readers, or more generally, as interpreters, we *assume* existence of
a semiotic morphism (or more precisely, of the intent that a semiotic
morphism formalizes).

Inferring a sign in the source system from which a given sign in the target system arises is an instance of what we called an inverse problem; it is the essence of semiosis. We called the converse, to design a sign in the target system, a direct problem. It might seem that inverse problems are more difficult than direct problems, because we have to infer both the semiotic morphisms and the source space, whereas a direct problem often only requires constructing a morphism (and perhaps fine tuning the target space). However, the opposite seems to be true in practice! Ordinary humans infer the meaning of typical signs (e.g., complex sentences) almost effortlessly and instantly, whereas design (e.g., user interface design, technical writing, automobile body design, etc.) is a demanding task for professionals. In my opinion, it is not so hard to do bad design, but good design is inherently demanding, difficult, and creative, because the result must be easily used by the target audience, i.e., the designer must design the morphism so that the inverse problem is as easy as possible for the intended users. (Actually, things may be a little more complex, because users may use the assumption that an interface was designed to be easily understood as part of their process of understanding, so designers may need to consider what users will think that they have been thinking!)

Notice that we are not addressing art in this note, but rather design, in the sense of craft. Art is even more inherently demanding, difficult, and creative than design; for example, it often involves deliberate transgression of preservation properties (among other things). Whereas design is typically utilitarian, i.e., done for a practical purpose, contemporary art is typically not utilitarian, even though traditional art tends to be utilitarian (more precisely, this distinction may not even exist for traditional art forms.) Also the distinction between art and craft is not so rigid as this discussion suggests.

I have noticed in my own design practice that I often have creative leaps, i.e., design ideas just occur, which I then try to understand in more formal ways, e.g. via semiotic morphisms, although if I am stuck, it may help to think more formally about the structure of the source space, preservation properties, etc. The creative process is to some extent unpredictable and uncontrollable; this is more true of artistic creation than of design, but it holds for both. The best designs often seem both surprising and obvious, and they also often seem to come suddenly out of nowhere, but usually after a lot of hard work.

19 October 1996

Revised 27 February 2000, and further edited in May 2000 and May 2001. Additional minor edits in April 2004.