Gestural Hand Motion Oscillation and Symmetries for Multimodal Discourse: Detection and Analysis**This research has been supported by the U.S. NSF STIMULATE program (IRI-9618887), Gesture, Speech, and Gaze in Discourse Segmentation, the NSF KDI program (BCS-9980054), Cross-Modal Analysis of Signal Sense: Multimedia Corpora and Tools for Gesture, Speech, and Gaze Research, and the NSF ITR program (ITR-0219875), Beyond the Talking Head and Animated Icon: Behaviorally Situated Avatars for Tutoring. The Authors also thank Travis Rose for editing help
Conference Paper
Overview
Identity
Additional Document Info
Other
View All
Overview
abstract
2003 IEEE. We address the detection and analysis of gestural hand motion oscillation and symmetries in natural speech. First, we extract hand motion trajectory signals from video dataset. Second, we present our windowed correlation coefficient approach for gestural symmetry extraction. The signs and magnitudes of the correlation coefficients in the cardinal directions of the subject's torso characterize the symmetries. Third, we present a wavelet-based approach that extracts the time-frequency properties of hand motion oscillation. By analyzing these frequency properties durations of homogeneous gestural oscillations are detected. Finally, we apply our approach to a real video dataset captured in candid conversation. We relate the hand motion oscillatory gestures and symmetric gestures to the phases of speech and multimodal discourse analysis. We demonstrate the ability of our algorithm to extract gestural symmetries and oscillations and show how symmetric gestures and oscillatory gestures correspond to natural discourse structure.
name of conference
2003 Conference on Computer Vision and Pattern Recognition Workshop