EFFICIENT GESTURE PROCESSING

• US 20120016641A1
• Filed: 07/13/2010
• Published: 01/19/2012
• Est. Priority Date: 07/13/2010
• Status: Abandoned Application

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Inventors: Raffa, Giuseppe, Nachman, Lama, Lee, Jinwon
• Application Number:
  US12/835,079
• Publication Number:
  US 20120016641A1
• Time in Patent Office: Days
• US Class Current: 703/2
• CPC Class Codes: G01C 19/00 Gyroscopes; Turn-sensitive devices using vibrating masses; Turn-sensitive devices without moving masses; Measuring angular rate using gyroscopic effects
• CPC Class Codes: G01P 15/18 in two or more dimensions
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10

Abstract

Embodiments of the invention describe a system to efficiently execute gesture recognition algorithms. Embodiments of the invention describe a power efficient staged gesture recognition pipeline including multimodal interaction detection, context based optimized recognition, and context based optimized training and continuous learning. Embodiments of the invention further describe a system to accommodate many types of algorithms depending on the type of gesture that is needed in any particular situation. Examples of recognition algorithms include but are not limited to, HMM for complex dynamic gestures (e.g. write a number in the air), Decision Trees (DT) for static poses, peak detection for coarse shake/whack gestures or inertial methods (INS) for pitch/roll detection.

Claims

1. An article of manufacture comprising a machine-readable storage medium that provides instructions that, if executed by a machine, will cause the machine to perform operations comprising:

receiving data from a sensor indicating a motion, the sensor having an accelerometer;

determining, via a first set of one or more algorithms, whether the motion is a gestural motion based on at least one of time duration of the data and an energy level of the data; and

determining, via a second set of one or more algorithms, a candidate gesture based on the data in response to determining the motion is a gestural motion, the second set of algorithm(s) to include a gesture recognition algorithm.

2. The article of manufacture of claim 1, the operations further comprising:

discarding the data in response to determining the motion is not a gestural motion.

3. The article of manufacture of claim 1, wherein the first set of algorithm(s) includes one or more low-complexity algorithms and the machine includes a low-power processor and a main processing unit, the first set of algorithm(s) to be executed via the low-power processor and the second set of algorithm(s) to be executed via the main processing unit.

4. The article of manufacture of claim 1, wherein the gesture recognition algorithm is based on a Hidden Markov Model (HMM).

5. The article of manufacture of claim 4, wherein determining a candidate gesture comprises:

using context of a system user to select a subset of one or more allowed gestures; and

restricting gesture models loaded by the HMM algorithm to the subset of allowed gesture(s).

6. The article of manufacture of claim 4, wherein determining a candidate gesture comprises:

using context of a system user to reject a subset of one or more disallowed gestures; and

selecting an HMM Filler model that discards the subset of disallowed gesture(s).

7. The article of manufacture of claim 4, wherein an HMM training set and one or more gesture models is based on physical activity of a user of the machine.

8. The article of manufacture of claim 4, the gesture rejection algorithm to validate a gesture recognized by the HMM algorithm by comparing duration and energy of the gestural motion with one or more of a minimum and a maximum value of duration and energy obtained from a database of training data.

9. The article of manufacture of claim 1, wherein the sensor is included in the machine, the machine comprises a mobile device, and determining, via the first algorithm(s), whether the motion is a gestural motion is further based on at least one of a user context of the mobile device, and an explicit action from the user to indicate a period of gesture commands.

10. The article of manufacture of claim 1, wherein determining a candidate gesture based on the data comprises

accessing a database of one or more example gesture inputs, the example gesture input(s) to include a minimum and a maximum time duration;

and

verifying the time duration of the gestural motion is within the minimum and maximum time durations of an example gesture input.

11. The article of manufacture of claim 1, wherein the data from the sensor is included in a series of data segments, one or more segments to indicate a motion defined by an energy threshold, and receiving the data from the sensor is in response to the data exceeding the energy threshold.

12. An article of manufacture comprising a machine-readable storage medium that provides instructions that, if executed by a machine, will cause the machine to perform operations comprising:

receiving data from a sensor indicating a motion, the sensor having an accelerometer;

determining a subset of one or more gesture recognition algorithms from a plurality of gesture recognition algorithms based, at least in part, on one or more signal characteristics of the data; and

determining a gesture from the data from the sensor based, at least in part, on applying the subset of gesture recognition algorithm(s) to the data.

13. The article of manufacture of claim 12, wherein the signal characteristic(s) of the data comprise an energy magnitude of the data.

14. The article of manufacture of claim 13, wherein determining the subset of gesture recognition algorithms is based, at least in part, on comparing the energy magnitude of the data with one or more magnitude values associated with one of the plurality of gesture algorithms.

15. The article of manufacture of claim 12, wherein the signal characteristic(s) of the data comprise a time duration of the data.

16. The article of manufacture of claim 15, wherein determining the subset of gesture recognition algorithms is based, at least in part, on comparing the time duration of the data with one or more time values associated with one of the plurality of gesture algorithms.

17. The article of manufacture of claim 12, wherein the signal characteristic(s) of the data comprise a frequency spectrum of the data.

18. The article of manufacture of claim 17, wherein determining the subset of gesture recognition algorithms is based, at least in part, on comparing the frequency spectrum of the data with one or more spectrum patterns stored associated with one of the plurality of gesture algorithms.

19. A method comprising:

receiving data from a sensor indicating a motion, the sensor having an accelerometer;

determining, via a first set of one or more algorithms, whether the motion is a gestural motion based on at least one of time duration of the data and an energy level of the data; and

determining, via a second set of one or more algorithms, a candidate gesture based on the data in response to determining the motion is a gestural motion, the second set of algorithm(s) to include a gesture recognition algorithm.

20. The method of claim 19, the first set of algorithm(s) to comprise one or more low-complexity algorithms to be executed via a low-power processor and the second set of algorithm(s) to be executed via a main processing unit.

21. The method of claim 19, wherein the second set of algorithm(s) includes a Hidden Markov Model (HMM) gesture rejection algorithm and determining a candidate gesture comprises:

using context of a system user to select a subset of one or more allowed gestures; and

restricting gestures used by the HMM algorithm to the subset of allowed gesture(s).

22. A method comprising:

receiving data from a sensor indicating a motion, the sensor having an accelerometer;

determining a subset of one or more gesture recognition algorithms from a plurality of gesture recognition algorithms based, at least in part, on one or more signal characteristics of the data; and

determining a gesture from the data from the sensor based, at least in part, on applying the subset of gesture recognition algorithms to the data.

23. The method of claim 22, wherein the one or more signal characteristic(s) of the data comprise at least one of

an energy magnitude of the data,

a time duration of the data, and

a frequency spectrum of the data.

24. The method of claim 23, wherein determining the subset of gesture recognition algorithms is based, at least in part, on at least one of

comparing the energy magnitude of the data with one or more magnitude values associated with one of the plurality of gesture algorithms if the signal characteristic(s) of the data comprise an energy magnitude of the data,

comparing the time duration of the data with time values associated with one of the plurality of gesture algorithms if the signal characteristic(s) of the data comprise a time duration of the data, and

comparing the frequency spectrum of the data with spectrum patterns associated with one of the plurality of gesture algorithms if the signal characteristic(s) of the data comprise a frequency spectrum of the data.