Bohari, Umema Hakimuddin (2018-08). To Draw or Not to Draw: Recognizing Stroke-Hover Intent in Gesture-Free Bare-Hand Mid-Air Drawing Tasks. Master's Thesis. Thesis uri icon


  • Over the past several decades, technological advancements have introduced new modes of communication
    with the computers, introducing a shift from traditional mouse and keyboard interfaces.
    While touch based interactions are abundantly being used today, latest developments in computer
    vision, body tracking stereo cameras, and augmented and virtual reality have now enabled communicating
    with the computers using spatial input in the physical 3D space. These techniques are now
    being integrated into several design critical tasks like sketching, modeling, etc. through sophisticated
    methodologies and use of specialized instrumented devices. One of the prime challenges in
    design research is to make this spatial interaction with the computer as intuitive as possible for the
    Drawing curves in mid-air with fingers, is a fundamental task with applications to 3D sketching,
    geometric modeling, handwriting recognition, and authentication. Sketching in general, is a
    crucial mode for effective idea communication between designers. Mid-air curve input is typically
    accomplished through instrumented controllers, specific hand postures, or pre-defined hand gestures,
    in presence of depth and motion sensing cameras. The user may use any of these modalities
    to express the intention to start or stop sketching. However, apart from suffering with issues like
    lack of robustness, the use of such gestures, specific postures, or the necessity of instrumented
    controllers for design specific tasks further result in an additional cognitive load on the user.
    To address the problems associated with different mid-air curve input modalities, the presented
    research discusses the design, development, and evaluation of data driven models for intent recognition
    in non-instrumented, gesture-free, bare-hand mid-air drawing tasks.
    The research is motivated by a behavioral study that demonstrates the need for such an approach
    due to the lack of robustness and intuitiveness while using hand postures and instrumented
    devices. The main objective is to study how users move during mid-air sketching, develop qualitative
    insights regarding such movements, and consequently implement a computational approach to
    determine when the user intends to draw in mid-air without the use of an explicit mechanism (such
    as an instrumented controller or a specified hand-posture). By recording the user's hand trajectory,
    the idea is to simply classify this point as either hover or stroke. The resulting model allows for
    the classification of points on the user's spatial trajectory.
    Drawing inspiration from the way users sketch in mid-air, this research first specifies the necessity
    for an alternate approach for processing bare hand mid-air curves in a continuous fashion.
    Further, this research presents a novel drawing intent recognition work flow for every recorded
    drawing point, using three different approaches. We begin with recording mid-air drawing data
    and developing a classification model based on the extracted geometric properties of the recorded
    data. The main goal behind developing this model is to identify drawing intent from critical geometric
    and temporal features. In the second approach, we explore the variations in prediction
    quality of the model by improving the dimensionality of data used as mid-air curve input. Finally,
    in the third approach, we seek to understand the drawing intention from mid-air curves using
    sophisticated dimensionality reduction neural networks such as autoencoders. Finally, the broad
    level implications of this research are discussed, with potential development areas in the design
    and research of mid-air interactions.

publication date

  • August 2018