Beyond Recognition: Gesture Variation for Continuous Interaction


Gestural interaction has become commonplace in consumer electronics. Finger gestures captured on touch screens provide intuitive ways to interface with complex tasks, and some gestures such as pinch-zoom have become iconic. Most techniques for coding gesture are based on forms of activity detection that involve recognition of gesture in unitary form. Meanwhile users’ casual perception of the potential of gestural input goes beyond simple recognition. People imagine gestural interaction to be intuitive and continuous, where aspects of a gesture organically map onto the response of an interactive system. What are the ways in which we might capture gesture quality to enable forms of continuous interaction?


Since we aimed at showing that continuous gesture interaction is suitable for open-ended creative tasks,

… we proposed to study how people are able to control stroke gesture variation (such as size, speed, orientation). We found that this is possible if the gesture is performed slowly. In this case, characteristics are fairly independent (non-ballistic movements).

… we had an machine learning based algorithm to estimate gesture variation as well as performing real-time gesture recognition (see the related post about the algorithm:

… we finally proposed to study how people rate the attractiveness of a real-world application that makes use of gesture-based continuous interaction. The application emulates a PhotoBooth-like App. We used our algorithm to recognize the gesture and estimate the size and the time-progression within the gesture. These two characteristics were used to change image FX processing parameters. In other words, the gesture is used to select the effect, the variation is used to modulate the effect.


Download the article Beyond_Recognition_AltCHI2013.pdf