Koh, Jung In (2017-08). Developing a Hand Gesture Recognition System for Mapping Symbolic Hand Gestures to Analogous Emoji in Computer-mediated Communication. Master's Thesis. Thesis uri icon

abstract

  • Recent trends in computer-mediated communications (CMC) have not only led to expanded instant messaging through the use of images and videos, but have also expanded traditional text messaging with richer content, so-called visual communication markers (VCM) such as emoticons, emojis, and stickers. VCMs could prevent a potential loss of subtle emotional conversation in CMC, which is delivered by nonverbal cues that convey affective and emotional information. However, as the number of VCMs grows in the selection set, the problem of VCM entry needs to be addressed. Additionally, conventional ways for accessing VCMs continues to rely on input entry methods that are not directly and intimately tied to expressive nonverbal cues. One such form of expressive nonverbal that does exist and is well-studied comes in the form of hand gestures. In this work, I propose a user-defined hand gesture set that is highly representative to VCMs and a two-stage hand gesture recognition system (trajectory-based, shape-based) that distinguishes the user-defined hand gestures. While the trajectory-based recognizer distinguishes gestures based on the movements of hands, the shape-based recognizer classifies gestures based on the shapes of hands. The goal of this research is to allow users to be more immersed, natural, and quick in generating VCMs through gestures. The idea is for users to maintain the lower-bandwidth online communication of text messaging to largely retain its convenient and discreet properties, while also incorporating the advantages of higher-bandwidth online communication of video messaging by having users naturally gesture their emotions that are then closely mapped to VCMs. Results show that the accuracy of user-dependent is approximately 86% and the accuracy of user-independent is about 82%.
  • Recent trends in computer-mediated communications (CMC) have not only led to expanded
    instant messaging through the use of images and videos, but have also expanded
    traditional text messaging with richer content, so-called visual communication markers
    (VCM) such as emoticons, emojis, and stickers. VCMs could prevent a potential loss of
    subtle emotional conversation in CMC, which is delivered by nonverbal cues that convey
    affective and emotional information. However, as the number of VCMs grows in the selection
    set, the problem of VCM entry needs to be addressed. Additionally, conventional
    ways for accessing VCMs continues to rely on input entry methods that are not directly
    and intimately tied to expressive nonverbal cues. One such form of expressive nonverbal
    that does exist and is well-studied comes in the form of hand gestures. In this work,
    I propose a user-defined hand gesture set that is highly representative to VCMs and a
    two-stage hand gesture recognition system (trajectory-based, shape-based) that
    distinguishes the user-defined hand gestures. While the trajectory-based recognizer
    distinguishes gestures based on the movements of hands, the shape-based recognizer
    classifies gestures based on the shapes of hands. The goal of this research is to allow
    users to be more immersed, natural, and quick in generating VCMs through gestures.
    The idea is for users to maintain the lower-bandwidth online communication of text
    messaging to largely retain its convenient and discreet properties, while also
    incorporating the advantages of higher-bandwidth online communication of video
    messaging by having users naturally gesture their emotions that are then closely mapped
    to VCMs. Results show that the accuracy of user-dependent is approximately 86% and
    the accuracy of user-independent is about 82%.

publication date

  • August 2017