Gesture cues for conversational interaction in monocular video
Conference Paper
Overview
Identity
Additional Document Info
Other
View All
Overview
abstract
We present our work on the determination of cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation. The basis for this integrating between gesticulation and speech discourse is the psycholinguistic concept of the co-equal generation of gesture and speech from the same semantic intent. We use the psycholinguistic device known as the 'catchment' as the locus around which this integration proceeds. We videotape gesture and speech elicitation experiments in which a subject describes her living space to an interlocutor. We extract the gestural motion of both hands using the Vector Coherence Mapping algorithm that combines spatial, momentum and skin color constraints in parallel using a fuzzy image processing approach. We extract the voiced units in the discourse as F0 units are correlate these with transcribed speech. Psycholinguistics researchers perceptually microanalyze the same video tape to produce a transcript that is annotated with the video timestamp and perceived gesturespeech entities. These serve to direct our high level analysis of the gesture trace and F0 data. We report the results of our analysis that show that the feature of 'handedness' and the kind of symmetry in two-handed gestures provide effective cues for discourse segmentation. We also present observations on how the gesture traces provide cues to segment hand use, high level discourse repair, and super-segmental cues for discourse grouping.
name of conference
Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378)
Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378)
author list (cited authors)
Quek, F., McNeill, D., Ansari, R., Ma, X., Bryll, R., Duncan, S., & McCullough, K. E.
citation count
16
complete list of authors
Quek, F||McNeill, D||Ansari, R||Ma, Xin-Feng||Bryll, R||Duncan, S||McCullough, KE