Sharon Oviatt
Oregon Health & Science University
What Are Multimodal Systems, and Why Are We Building Them?
What Types of Multimodal Interfaces Exist, and What Is Their History and Current Status?
What Are the Goals and Advantages of Multimodal Interface Design?
What Methods and Information Have Been Used to Design Novel Multimodal Interfaces?
What Are the Cognitive Science Underpinnings of Multimodal Interface Design?
When Do Users Interact Multimodally?
What Are the Integration and Synchronization Characteristics of Users ’Multimodal Input?
What Individual Differences Exist in Multimodal Interaction and What Are the Implications for Designing Systems for Universal Access?
Is Complementarity or Redundancy the Main Organizational Theme That Guides Multimodal Integration?
What Are the Primary Features of Multimodal Language?
What Are the Basic Ways in Which Multimodal Interfaces Differ from Graphical User Interfaces?
What Basic Architectures and Processing Techniques Have Been Used to Design Multimodal Systems?
What Are the Main Future Directions for Multimodal Interface Design?
Acknowledgments
References
Figure 14.1: Multimodal interface terminology.
Figure 14.2: Multimodal command to “pan ”the map, which illustrates mutual disambiguation occurring between incoming speech and gesture information, such that lexical hypotheses were pulled up on both n-best lists to produce a correct final multimodal interpretation.
Figure 14.3: Ten myths of multimodal interaction: separating myth from empirical reality.
Figure 14.4: Typical information processing flow in a multimodal architecture designed for speech and gesture. VR = virtual reality; TTS =text-to-speech; App =?.