Neural Networks and their applications Garrison W. Cottrell Computer Science & Engineering Department and Institute for Neural Computation University of California, San Diego Outline of the talk I. Motivation: Why neural nets? II. Human-style computation: What's it like? III. Neural nets: What are they like? IV. Example applications A. NETTalk B. Face and emotion recognition V. Conclusions and some things to ponder CSE 90 2 Introduction o Why are people smarter than machines? o Our take on the problem: o Basic differences in architecture o Our research program: Explore brain-like computational models CSE 90 3 Mutual Constraints People are able to combine lots of different kinds of knowledge quickly in understanding English - For example, in understanding the relationships given in a sentence: Syntax (structure) gives us some information: The boy kissed the girl. But usually we need semantics (meaning) too: I saw the grand canyon flying to New York. I saw the sheep grazing in the field. Ditto for pronoun reference: The city council refused the demonstrators a permit because they were communists. [Russia or China?] CSE 90 4 Again, things that people do well involve integrating information from multiple sources. E.g., word sense disambiguation: 1. Discourse Context: "I'm taking the car" 2. Grammar: "The carpenter saw the wood" 3. Meaning frequency: "Bob threw a ball" 4. Semantics: o Associations between word senses "dog's bark" "deep pit" o The fit between roles and role fillers: "Bob threw the fight" 5. Pragmatic: "The man walked on the deck" "Nadia swung the hammer at the nail and the head flew off"(Hirst 83) CSE 90 5 Computers are different ... The boy the girl the dog bit liked cried. U CN REED THIIS CANDT YU? CSE 90 6 Constraints on a brain model o The 100 step program constraint (Feldman, 1985) o Neurons are slow: respond in a few milliseconds o Yet mental events only take half a second This implies: o parallelism o simple steps CSE 90 7 Constraints on a brain model Computer Brain _________________________________________________________________ speed fast slow order serial parallel component good poor reliablity fault poor no degradation tolerance signals precise imprecise, symbolic terse programming needs it does it CSE 90 8 Parallel Distributed Processing Models o Networks of simple processing "units" o connected by positive and negative links o spread activation and inhibition to other units o The "solution" is a stable state - when nothing changes CSE 90 9 . An Example: The Necker Cube o Units represent hypotheses about the inputs o Connections encode constraints between hypotheses o The network relaxes to a stable state: the "interpretation" CSE 90 10 Learning A bit of history: Rosenblatt (1962) discovered the perceptron convergence procedure o Guaranteed to learn anything computable (by a two-layer perceptron) o Unfortunately, not everything was computable (Minsky & Papert, 1969) CSE 90 11 Learning Rumelhart, Hinton and Williams (1985) have (re-)discovered a learning technique for multiple layered nets, called back- propagation. Here's how it works: o present an input, let activation propagate through the network o give a teaching signal o propagate the error back through the network (hence the name back propagation) o change the connections according to the error CSE 90 12 Learning Networks using back-propagation have learned: o exclusive-or o to compress images o to retrieve properties of animals and plants from partial descriptions o to read aloud from text o to recognize faces o to recognize emotion in faces o to drive a Chevy Van at 55 MPH o to predict financial markets o to detect credit card fraud CSE 90 13 NETtalk (Terrence Sejnowski & Charles Rosenberg, 1986) o 7 groups of input units for seven letters o Job is to produce phoneme corresponding to middle letter o output is fed to DECTALK speech synthesizer CSE 90 14 NETtalk Training corpus CSE 90 15 Applications: Face and Emotion Recognition Recognizing faces is: hard! (And recalling names is worse...) Recognizing emotions is harder. seems easy for us but: easy to make mistakes a lot of cortex is dedicated to it (~10% of temporal cortex) a subsystem: Prosopagnosia Useful: Identifying criminals, political dissidents... Computer Interfaces: feedback to adaptive systems Nuclear plant operator monitoring - awake? CSE 90 16 Applications: Face Recognition The Network: The left side network is an auto-encoder: It is trained to simply reproduce its input. The right hand network does classification using the features extracted by the left hand network. CSE 90 17 Conclusions o Neural nets are doing some amazing things! o Especially good at pattern recognition o Good at prediction o Neural nets feature: o Uniform representation of multiple knowledge sources o "Automatic" generalization o Adaptability o Learning o Development of its own representations CSE 90 18