Efficient gesture processing

• US 10,353,476 B2
• Filed: 01/03/2017
• Issued: 07/16/2019
• Est. Priority Date: 07/13/2010
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Raffa, Giuseppe, Nachman, Lama, Lee, Jinwon
• Primary Examiner(s): Vincent, David R
• Application Number:
  US15/397,511
• Publication Number:
  US 20170220122A1
• Time in Patent Office: 924 Days
• Field of Search: 706 15, 706 45
• CPC Class Codes: G01C 19/00 Gyroscopes; Turn-sensitive devices using vibrating masses; Turn-sensitive devices without moving masses; Measuring angular rate using gyroscopic effects
• CPC Class Codes: G01P 15/18 in two or more dimensions
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10

Abstract

Embodiments of the invention describe a system to efficiently execute gesture recognition algorithms. Embodiments of the invention describe a power efficient staged gesture recognition pipeline including multimodal interaction detection, context based optimized recognition, and context based optimized training and continuous learning. Embodiments of the invention further describe a system to accommodate many types of algorithms depending on the type of gesture that is needed in any particular situation. Examples of recognition algorithms include but are not limited to, HMM for complex dynamic gestures (e.g. write a number in the air), Decision Trees (DT) for static poses, peak detection for coarse shake/whack gestures or inertial methods (INS) for pitch/roll detection.

Claims

1. At least one tangible computer readable storage medium having instructions stored thereon that, when executed on a machine, cause the machine to:

receive data from a motion sensor;

select a subset of one or more gesture recognition algorithms from a plurality of gesture recognition algorithms, wherein selecting the subset comprises;

comparing characteristics of a partial data segment against a gesture-matching database, the partial data segment being a subset of a gesture'"'"'s complete motion data segment received from the motion sensor,

enabling one or more gesture-recognition algorithms with training gesture motion characteristics in the gesture-matching database that match the data characteristics of the partial data, and

identifying a candidate gesture by using the one or more enabled gesture-recognition algorithms to analyze the gesture'"'"'s complete data segment and comparing the analysis of each gesture-recognition algorithm to a template-matching database;

determine a gesture based on the best match obtained from comparison of the candidate gestures with the template-matching database; and

trigger an event on the machine, wherein the event corresponds to the determined gesture.

2. The at least one tangible computer readable storage medium of claim 1, wherein the machine is to further:

determine an energy magnitude of the data based, at least in part, on the amplitude of the data.

4. The at least one tangible computer readable storage medium of claim 2, wherein the machine is to select the subset of gesture recognition algorithm(s) based, at least in part, on a comparison of minimum/maximum energy magnitude values of the data with minimum/maximum energy magnitude values associated with each of the plurality of gesture algorithms.

5. The at least one tangible computer readable storage medium of claim 1, the machine to further:

determine a frequency spectrum of the data based, at least in part, on the amplitude of the data and a phase of the data.

6. The at least one tangible computer readable storage medium of claim 5, wherein the machine is to select the subset of gesture recognition algorithm(s) based, at least in part, on a comparison of the frequency spectrum of the data with one or more spectrum patterns associated with each of the plurality of gesture algorithms.

7. The at least one tangible computer readable storage medium of claim 1, wherein the motion sensor comprises at least one of an accelerometer or a gyroscope.

21. The tangible computer readable medium of claim 1 wherein the one or more gesture recognition algorithms include Hidden Markov Model (HMM), Decision Trees, Peak Detection, and Pitch/Roll Inertial.

3. The at least one tangible computer readable storage medium 2, wherein the machine is to select the subset of gesture recognition algorithm(s) based, at least in part, on a comparison of a total energy magnitude of the data with a total energy magnitude value associated with each of the plurality of gesture algorithms.

8. A mobile computing device comprising:

a motion sensor;

a memory;

at least one processor;

an algorithm selection module, stored in the memory and executed via the at least one processor, to select a subset of one or more gesture recognition algorithms from a plurality of gesture recognition algorithms wherein selecting the subset comprises;

comparing characteristics of a partial data segment against a gesture-matching database, the partial data segment being a subset of a gesture'"'"'s complete motion data segment received from the motion sensor,

enabling one or more gesture-recognition algorithms with training gesture motion characteristics in the gesture-matching database that match the data characteristics of the partial data, and

identifying a candidate gesture by using the one or more enabled gesture-recognition algorithms to analyze the gesture'"'"'s complete data segment and comparing the analysis of each gesture-recognition algorithm to a template-matching database; and

a gesture recognition module, stored in the memory and executed via the at least one processor, to determine a gesture based on the best match obtained from comparison of the candidate gestures with the template-matching database;

wherein the determined gesture triggers an event on the processor, the event corresponding to the determined gesture.

9. The mobile computing device of claim 8, wherein the at least one processor comprises a low power processing unit to execute the algorithm selection module, and a main processing unit to execute the gesture recognition module.

10. The mobile computing device of claim 8, wherein the algorithm selection module is to further:

determine an energy magnitude of the data from the motion sensor based, at least in part, on the amplitude of the data from the motion sensor.

11. The mobile computing device of claim 10, wherein the algorithm selection module is to select the subset of gesture recognition algorithm(s) based, at least in part, on a comparison of a total energy magnitude of the data from the motion sensor with a total energy magnitude value associated with each of the plurality of gesture algorithms.

12. The mobile computing device of claim 10, wherein the algorithm selection module is to select the subset of gesture recognition algorithm(s) based, at least in part, on a comparison of minimum/maximum energy magnitude values of the data from the motion sensor with minimum/maximum energy magnitude values associated with each of the plurality of gesture algorithms.

13. The mobile computing device of claim 8, wherein the algorithm selection module is to further:

determine a frequency spectrum of the data from the motion sensor based, at least in part, on the amplitude of the data from the motion sensor and a phase of the data from the motion sensor.

14. The mobile computing device of claim 13, wherein the algorithm selection module is to select the subset of gesture recognition algorithm(s) is based, at least in part, on a comparison of the frequency spectrum of the data from the motion sensor with one or more spectrum patterns associated with each of the plurality of gesture algorithms.

15. The mobile computing device of claim 8, wherein the motion sensor comprises an accelerometer.

16. The mobile computing device of claim 8, wherein the motion sensor comprises a gyroscope.

17. The mobile computing device of claim 8, further comprising:

a wearable housing including the motion sensor, the at least one processor, and the memory.

22. The mobile computing device of claim 8 wherein the one or more gesture recognition algorithms include Hidden Markov Model (HMM), Decision Trees, Peak Detection, and Pitch/Roll Inertial.

18. A machine-implemented method comprising:

receiving data from a motion sensor;

selecting a subset of one or more gesture recognition algorithms from a plurality of gesture recognition algorithms, wherein selecting the subset comprises;

comparing characteristics of a partial data segment against a gesture-matching database, the partial data segment being a subset of a gesture'"'"'s complete motion data segment received from the motion sensor,

enabling one or more gesture-recognition algorithms with training gesture motion characteristics in the gesture-matching database that match the data characteristics of the partial data, and

identifying a candidate gesture by using the one or more enabled gesture-recognition algorithms to analyze the gesture'"'"'s complete data segment and comparing the analysis of each gesture-recognition algorithm to a template-matching database;

determining a gesture based on the best match obtained from comparison of the candidate gestures with the template-matching database; and

triggering an event that corresponds to the determined gesture.

19. The machine-implemented method of claim 18, further comprising:

determining an energy magnitude of the data based, at least in part, on an amplitude of the data;

wherein selecting the subset of gesture recognition algorithm(s) is based, at least in part, on at least one of;

comparing a total energy magnitude of the data with a total energy magnitude value associated with each of the plurality of gesture algorithms;

or

comparing minimum/maximum energy magnitude values of the data with minimum/maximum energy magnitude values associated with each of the plurality of gesture algorithms.

20. The machine-implemented method of claim 18, further comprising:

determining a frequency spectrum of the data based, at least in part, on an amplitude of the data and a phase of the data;

wherein selecting the subset of gesture recognition algorithm(s) is based, at least in part, on comparing the frequency spectrum of the data with one or more spectrum patterns associated with each of the plurality of gesture algorithms.

23. The machine-implemented method of claim 18 wherein the one or more gesture recognition algorithms include Hidden Markov Model (HMM), Decision Trees, Peak Detection, and Pitch/Roll Inertial.