A multimodal communication with a haptic glove: on the fusion of speech and deixis over a raised line drawing
Conference Paper
Overview
Research
Identity
Additional Document Info
Other
View All
Overview
abstract
Mathematics instruction and discourse typically involve two modes of communication: speech and graphical presentation. For the communication to remain situated, dynamic synchrony must be maintained between the speech and dynamic focus in the graphics. In normals, vision is used for two purposes: access to graphical material and awareness of embodied behavior. This embodiment awareness keeps communication situated with visual material and speech. Our goal is to assist blind students in the access to such instruction/ communication. We employ the typical approach of sensory replacement for the missing visual sense. Haptic fingertip reading can replace visual material. For the embodied portion of the communication, we want to make the blind student aware of the deictic gestures performed by the teacher over the graphic in conjunction with speech. We propose the use of haptic gloves paired with computer vision based tracking to help blind students maintain reading focus on a raised line representation of a graphical presentation to which the instructor points while speaking. In this initial phase of our research, we conducted three experiments that show that: 1) The gloves convey sense of direction; 2) The gloves do not interfere in fingertip reading; 3) A person can navigate with the help of this system while listening to a story; 4) It is possible to fuse the information received from both modes. We discuss these findings in this paper. Copyright 2008 ACM.
name of conference
Proceedings of the 1st international conference on PErvasive Technologies Related to Assistive Environments
