Paulson, Brandon C. (2010-05). Rethinking Pen Input Interaction: Enabling Freehand Sketching Through Improved Primitive Recognition. Doctoral Dissertation. Thesis uri icon


  • Online sketch recognition uses machine learning and artificial intelligence techniques

    to interpret markings made by users via an electronic stylus or pen. The

    goal of sketch recognition is to understand the intention and meaning of a particular

    user's drawing. Diagramming applications have been the primary beneficiaries

    of sketch recognition technology, as it is commonplace for the users of these tools to

    rst create a rough sketch of a diagram on paper before translating it into a machine

    understandable model, using computer-aided design tools, which can then be used to

    perform simulations or other meaningful tasks.

    Traditional methods for performing sketch recognition can be broken down into

    three distinct categories: appearance-based, gesture-based, and geometric-based. Although

    each approach has its advantages and disadvantages, geometric-based methods

    have proven to be the most generalizable for multi-domain recognition. Tools, such as

    the LADDER symbol description language, have shown to be capable of recognizing

    sketches from over 30 different domains using generalizable, geometric techniques.

    The LADDER system is limited, however, in the fact that it uses a low-level recognizer

    that supports only a few primitive shapes, the building blocks for describing

    higher-level symbols. Systems which support a larger number of primitive shapes have

    been shown to have questionable accuracies as the number of primitives increase, or

    they place constraints on how users must input shapes (e.g. circles can only be drawn

    in a clockwise motion; rectangles must be drawn starting at the top-left corner).

    This dissertation allows for a significant growth in the possibility of free-sketch

    recognition systems, those which place little to no drawing constraints on users. In

    this dissertation, we describe multiple techniques to recognize upwards of 18 primitive

    shapes while maintaining high accuracy. We also provide methods for producing

    confidence values and generating multiple interpretations, and explore the difficulties

    of recognizing multi-stroke primitives. In addition, we show the need for a standardized

    data repository for sketch recognition algorithm testing and propose SOUSA

    (sketch-based online user study application), our online system for performing and

    sharing user study sketch data. Finally, we will show how the principles we have

    learned through our work extend to other domains, including activity recognition

    using trained hand posture cues.

publication date

  • May 2010