Dear Batya,

Thank you for your reply to my email; I think we have many common interests, and in particular, I have been much inspired by your work on value sensitive design, and have adopted many of its ideas. Your website is very impressive (and much easier to navigate and understand than mine!).

My interest in interface design began in the mid-1980s at SRI, where I obtained a grant from ONR to do some experimental psychology on multimedia instruction focused on the coordination of the media involved, and the effects of dis-coordination; we developed a kind of semiotic framework to describe the experimental results. Unfortunately this work was too far ahead of its and/or too obscurely written, to be easily publishable, despite its novel experimental results. Around the same time, we also did an large experimental study of airline pilot verbal interaction for NASA, using a 747 simulator with many crews of three experienced commercial airline pilots. This resulted in training methods for structuring operational discourse in the cockpit. The NASA work was run through a small company that I ran, called Structural Semantics; altogether there were about ten NASA contracts, and a number of consulting jobs for various companies.

In the early 1990s, I founded and ran (until 1996) the Centre for Requirements and Foundations at Oxford University; this centre studied requirements capture and analysis, mainly using video ethnography and ethnomethodology, and did several detailed case studies for British Telecom, including the use of PDAs by stock traders.

My interest in interface design was revived in 1998 when I received a grant from Japan to develop a distributed cooperative theorem proving system, because I noticed that the usual interfaces to theorem provers are very primitive (command line, usually with many obscure abbreviations) and really difficult to learn and to use. So a major subproject became to develop some general principles for the design of theorem prover interfaces. This led to further development of the earlier semiotics, and to combining it with ideas from algebraic semantics to make it more applicable to engineering problems.

Semiotic morphisms are a basic concept that models representations in one sign system (the target system) for signs from another (the source). Semiotic morphisms can be partial, i.e., they do not necessarily have to preserve all of the signs or all the structure of the source system. The degree to which semiotic morphisms preserve various features provides a basis for comparing the quality of representations, and leads to an interesting study of trade-off principles. Our most recent work places more emphasis on the direct integration of values with the entire design process; we are teaching graduate (CSE 271) and undergraduate (CSE 171) courses on interface design using these ideas, and hope to write a textbook. Your brilliant encyclopaedia article will be required reading in these courses next quarter.

For a theorem prover, proofs (in some fixed logical system) are considered as forming a sign system; an extension of this system includes additional information to help with understanding proofs, such as motivation, background tutorials, and examples. Another sign system is given by website technology (XHTML, JavaScript, XML, etc.). Then representations of proofs as websites are morphisms from the first system (or its extension) to the second, and orderings on semiotic morphisms compare aspects of the quality of such representations. Some website design principles, called the "Tatami conventions," were extracted from this, and embodied in our Tatami system, and its Kumo interface tool, which combines proof assistant and website generation capabilities; it generates so-called "proofweb" data structures that use XHTML, JavaScript, etc., which can then be viewed with any browser. Much of the internal computations are done using the BOBJ algebraic specification and verification engine. Details appear in the various publications.

The graduate course is project based, and students have done websites for local community services using the methods of the course. One of these was an ambitious website for all the animal shelters in the San Diego area (the public face emphasizes adoption, but there are rather extensive database functions for the shelters to use internally). Another was a simpler website for a local jazz festival that donates profits to local charities. Yet another was a website for condominium owners that has been adapted by a number of condominium owner associations, to help them deal with problems with the condominium manangement (which apparently is quite common). One of the most eye-opening projects was for a county agency that arranges temporary placement with forster parents for children of parents (mostly single mothers) who are undergoing treatment for drug addition; the value conflicts among stakeholders were shocking to most students (sometimes mothers attack agency employees with knives!). See the CSE 271 website.

There are also two rather large current projects that I am working on with graduate students; they are not exclusively interface projects, but both have a large and important interface component. The first is a database schema mapping tool, called SCIA, that helps users construct translations from data over one kind of schema to data over another. Such tools are used in data mining, heterogeneous database integration, scientific workflow construction, and many other areas, including ontology translation and integration for the semantic web. Algebraic semiotics and value driven design were systematically used throughout the whole design, build, test lifecycle of this tool. Controlled experiments were conducted to evaluate a key new feature of the system called "critical points." SCIA is supported by SEEK, a large NSF project to provide computational infrastructure for an NSF network of 26 long term ecological reserch sites in the US. The main researcher is Jenny Wang.

The second current project, called Griot, is a general engine for the generation and presentation of interactive multimedia productions, which may be educational or artistic in nature. Griot integrates graphics, narrative structure, text, and audio (including music); what is most novel is that these structures are generated on the fly from underlying ontologies and media libraries, and are different every time, especially in response to user input, but even without. The heart of the novelty is an algorithm for blending conceptual spaces, called Alloy, based on recent ideas from cognitive semantics and cognitive linguistics, combined with algebraic semiotics to make them implementable. For example, Griot can use Alloy to create novel metaphors on the fly, for use in poetry. The system has been successfully used in a number of performances for large audiences; e.g., see the website for our CalIT2 opening celebration performance. This project has no formal funding, but the student (Fox Harrell) is supported by a number of genereous fellowships.

Unfortunately all this work is little known in the CHI community, because we tend to publish in the application area of the tools we build, rather than the traditional CHI venues.

Sorry for taking up so much of your time with such a long email, but i hope you found some bits of it interesting, and i hope we can keep in touch in the future.

With all best regards,

      joseph