MULTIPLE SENSOR GESTURE RECOGNITION

• US 20130328763A1
• Filed: 11/29/2012
• Published: 12/12/2013
• Est. Priority Date: 10/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Latta, Stephen G., Mount, Brian J., Poulos, Adam G., Kohler, Jeffrey A., Tomlin, Arthur C., Steed, Jonathan T.
• Granted Patent: US 9,030,408 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Methods for recognizing gestures using adaptive multi-sensor gesture recognition are described. In some embodiments, a gesture recognition system receives a plurality of sensor inputs from a plurality of sensor devices and a plurality of confidence thresholds associated with the plurality of sensor inputs. A confidence threshold specifies a minimum confidence value for which it is deemed that a particular gesture has occurred. Upon detection of a compensating event, such as excessive motion involving one of the plurality of sensor devices, the gesture recognition system may modify the plurality of confidence thresholds based on the compensating event. Subsequently, the gesture recognition system generates a multi-sensor confidence value based on whether at least a subset of the plurality of confidence thresholds has been satisfied. The gesture recognition system may also modify the plurality of confidence thresholds based on the plugging and unplugging of sensor inputs from the gesture recognition system.

Claims

1. A method for recognizing a particular gesture, comprising:

receiving a plurality of sensor inputs;

acquiring one or more multi-sensor gesture filters associated with the particular gesture, the one or more multi-sensor gesture filters include a particular filter;

detecting that a new sensor input is available for recognizing the particular gesture;

adding the new sensor input to the plurality of sensor inputs;

updating the one or more multi-sensor gesture filters in response to the new sensor input;

generating a plurality of single-sensor gesture recognition results based on the plurality of sensor inputs, the generating a plurality of single-sensor gesture recognition results is performed subsequent to the adding the new sensor input to the plurality of sensor inputs;

determining that the particular filter is satisfied based on the plurality of single-sensor gesture recognition results; and

executing a command on a computing system in response to the determining that the particular filter is satisfied.

2. The method of claim 1, further comprising:

detecting a compensating event; and

updating the particular filter in response to the detecting a compensating event, the particular filter includes a first confidence threshold and a second confidence threshold, the updating the particular filter includes updating the first confidence threshold based on the compensating event, the updating the particular filter is performed prior to the determining that the particular filter is satisfied.

3. The method of claim 2, wherein:

the updating the first confidence threshold includes increasing the first confidence threshold based on the compensating event.

4. The method of claim 2, wherein:

the plurality of sensor inputs includes a first input derived from a depth sensor; and

the compensating event includes detecting a motion associated with the depth sensor above a particular threshold for more than a particular period of time.

5. The method of claim 2, wherein:

the plurality of sensor inputs includes a first input derived from an image sensor; and

the compensating event includes detecting a lighting condition associated with the image sensor below a particular threshold for more than a particular period of time.

6. The method of claim 2, wherein:

the compensating event includes detecting a temperature above a particular threshold for more than a particular period of time.

7. The method of claim 1, further comprising:

synchronizing the plurality of sensor inputs, the plurality of sensor inputs includes a first input derived from an IMU and a second input derived from an inward facing eye tracking system, the plurality of sensor inputs includes a third input derived from a microphone, the determining that the particular filter is satisfied includes recognizing three conditions within a particular period of time, a first condition of the three conditions includes recognizing a nodding motion based on the first input, a second condition of the three conditions includes recognizing that a particular object is being looked at based on the second input, a third condition of the three conditions includes recognizing a particular verbal command.

8. The method of claim 1, further comprising:

receiving a request for an identification of the plurality of sensor inputs from a requesting application;

identifying the plurality of sensor inputs in response to the receiving a request; and

outputting the identification of the plurality of sensor inputs prior to the acquiring one or more multi-sensor gesture filters.

9. The method of claim 8, wherein:

the identification of the plurality of sensor inputs includes an identification of each sensor input of the plurality of sensor inputs.

10. The method of claim 1, wherein:

the plurality of sensor inputs includes a first input derived from a depth sensor, the depth sensor senses motion associated with one or more objects;

the particular gesture is associated with selecting a particular virtual object; and

the generating a plurality of single-sensor gesture recognition results includes performing body tracking based on the first input.

11. An electronic device for recognizing a particular gesture, comprising:

a first sensor, the first sensor generates a first input;

a second sensor, the second sensor generates a second input;

a memory, the memory stores one or more multi-sensor gesture filters, a particular filter of the one or more multi-sensor gesture filters includes a first confidence threshold and a second confidence threshold; and

one or more processors, the one or more processors detect a compensating event, the one or more processors update the first confidence threshold based on the compensating event, the one or more processors determine a first confidence value that the particular gesture has occurred based on the first input, the one or more processors determine a second confidence value that the particular gesture has occurred based on the second input, the one or more processors generate a multi-sensor confidence value based on whether the first confidence value satisfies the first confidence threshold and whether the second confidence value satisfies the second confidence threshold.

12. The electronic device of claim 11, wherein:

the one or more processors update the first confidence threshold by increasing the first confidence threshold based on the compensating event.

13. The electronic device of claim 11, wherein:

the first sensor includes a depth sensor; and

the compensating event includes detecting a motion associated with the depth sensor above a particular threshold for more than a particular period of time.

14. The electronic device of claim 11, wherein:

the compensating event includes detecting a temperature above a particular threshold for more than a particular period of time.

15. The electronic device of claim 11, wherein:

the one or more processors synchronize the plurality of sensor inputs.

16. The electronic device of claim 11, wherein:

the first sensor includes a depth sensor;

the second sensor includes a microphone; and

the particular gesture includes selecting a particular virtual object.

17. One or more storage devices containing processor readable code for programming one or more processors to perform a method for operating a multi-sensor gesture recognition system comprising the steps of:

receiving a plurality of sensor inputs, the plurality of sensor inputs includes a first input and a second input;

synchronizing the first input and the second input;

acquiring one or more multi-sensor gesture filters associated with a particular gesture, a particular filter of the one or more multi-sensor gesture filters includes a first confidence threshold and a second confidence threshold;

detecting a compensating event;

adjusting the first confidence threshold based on the compensating event;

determining a first confidence value that the particular gesture has occurred based on the first input;

determining a second confidence value that the particular gesture has occurred based on the second input;

generating a multi-sensor confidence value based on whether the first confidence value satisfies the first confidence threshold and whether the second confidence value satisfies the second confidence threshold;

determining a first period of time associated with the multi-sensor confidence value; and

outputting the multi-sensor confidence value and the first period of time.

18. The one or more storage devices of claim 17, wherein:

the first input is derived from a depth sensor; and

the compensating event includes detecting a motion associated with the depth sensor above a particular threshold for more than a particular period of time.

19. The one or more storage devices of claim 18, further comprising:

detecting that a new sensor input is available for recognizing the particular gesture;

adding the new sensor input to the plurality of sensor inputs; and

updating the one or more multi-sensor gesture filters in response to the new sensor input.

20. The one or more storage devices of claim 19, wherein:

the second input is derived from an image sensor;

the particular gesture includes selecting a particular virtual object; and

the determining a first confidence value that the particular gesture has occurred based on the first input includes performing skeletal tracking based on the first input.