Gesture recognizer system architecture

• US 9,280,203 B2
• Filed: 08/02/2011
• Issued: 03/08/2016
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Latta, Stephen G., Markovic, Relja, Tomlin, Arthur Charles, Snook, Gregory N.
• Primary Examiner(s): HAILU, TADESSE
• Application Number:
  US13/196,618
• Publication Number:
  US 20110285626A1
• Time in Patent Office: 1,680 Days
• Field of Search: 715/863, 345/158, 463 1- 4
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Systems, methods and computer readable media are disclosed for a gesture recognizer system architecture. A recognizer engine is provided, which receives user motion data and provides that data to a plurality of filters. A filter corresponds to a gesture, that may then be tuned by an application receiving information from the gesture recognizer so that the specific parameters of the gesture—such as an arm acceleration for a throwing gesture—may be set on a per-application level, or multiple times within a single application. Each filter may output to an application using it a confidence level that the corresponding gesture occurred, as well as further details about the user motion data.

Claims

1. A method for providing recognition of gestures made by a user using a gesture filter representing a gesture, comprising:

receiving data indicative of a user motion or pose, the data being captured by a camera;

determining an output of the gesture filter based on the data;

determining, based on the data, to send the output to a first application of a plurality of applications in response to determining that the data corresponds to the first application and does not correspond to a system application of the plurality of applications; and

sending the output to the first application of the plurality of applications.

2. The method of claim 1, further comprising:

receiving second data indicative of a user motion or pose, the second data being captured by the camera;

determining a second output of the gesture filter based on the second data; and

sending the second application the second output when the second data corresponds to a second application of the plurality of applications, the first and second applications executing concurrently.

3. The method of claim 2, wherein the gesture filter comprises a parameter, and further comprising:

receiving from the first application a first value for the parameter;

wherein determining the output of the gesture filter based on the data comprises;

determining the output based on the first value for the parameter;

receiving from the second application a second value for the parameter; and

wherein determining the second output of the gesture filter based on the second data comprises;

determining the second output based on the second value for the parameter.

4. The method of claim 1, further comprising:

receiving second data indicative of a user motion or pose, the second data being captured by the camera;

determining a second output of the gesture filter based on the data; and

sending the system application the output when the second data corresponds to the system application of the plurality of applications, the data being indicative of a command issued to the system application, the system application and the first application executing concurrently.

5. The method of claim 4, wherein sending the first application the output when the data is indicative of a command issued to a system-software application comprises:

sending the first application the output when the data is indicative of initializing execution of a user application, navigating up or down in a hierarchical list, opening a file, closing a file, saving a file, ending execution of a user software, pausing the user software, saving a state of the user software, or controlling an aspect of an operating system.

6. The method of claim 1, wherein sending the first application the output when the data corresponds to a first application of a plurality of applications comprises:

sending the first application the output when the data is indicative of a command issued to user software.

7. The method of claim 6, wherein sending the first application the output when the data is indicative of a command issued to user software comprises:

sending the first application the output when the data is indicative of manipulating an item or avatar in the first application.

8. The method of claim 1, wherein, sending the first application the output when the data corresponds to a first application of a plurality of applications comprises:

sending the first application the output when the data is captured by the camera; and

further comprising;

receiving second data indicative of sound captured by a microphone; and

sending an indication of the second data to a second application of the plurality of applications.

9. The method of claim 1, wherein the gesture filter comprises a parameter, and further comprising:

receiving from the first application a value for the parameter; and

wherein, determining an output of the gesture filter based on the data comprises;

determining the output of the gesture filter based on the value of the parameter.

10. The method of claim 9, wherein a parameter comprises an indication of a body part, a volume of space, a velocity, a direction of movement, an angle, or a place where a movement occurs.

11. A system for providing recognition of gestures made by a user using a gesture filter representing a gesture comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed by the processor, cause the system at least to;

receive data indicative of a user motion or pose, the data being captured by a camera;

determine an output of the gesture filter based on the data;

determine, based on the data, to send the output to a first application of a plurality of applications in response to determining that the data corresponds to the first application and does not correspond to a system application of the plurality of applications; and

send the output to the first application of the plurality of applications.

12. The system of claim 11, wherein the gesture filter has a plurality of contexts, and a parameter of each context of the base information about the gesture is unique.

13. The system of claim 11, wherein the memory further bears processor-executable instructions that, when executed by the processor, cause the system at least to:

receive the gesture filter from the first application.

14. The system of claim 11, wherein the gesture filter comprises a parameter, and wherein the memory further bears processor-executable instructions that, when executed by the processor, cause the system at least to:

set a value for the parameter in response to receiving data captured by the depth camera indicative of a change in the user'"'"'s fatigue, a change in the user'"'"'s competence, or a second user replacing the user; and

wherein the processor-executable instructions that, when executed by the system, cause the system to at least determine an output of the gesture filter based on the data further cause the system to at least;

determine an output of the gesture filter based on the value for the parameter.

15. The system of claim 11, wherein the gesture filter comprises a parameter, and wherein the memory further bears processor-executable instructions that, when executed by the processor, cause the system at least to:

in response to receiving second data indicative of a user motion or pose, the data being captured by the camera, receive from the first application a value for the parameter; and

wherein the processor-executable instructions that, when executed by the system, cause the system to at least determine an output of the gesture filter based on the data further cause the system to at least;

determine an output of the gesture filter based on the value for the parameter.

16. The system of claim 11, wherein processor-executable instructions that, when executed by the processor, cause the system to at least determine the output of the gesture filter based on the data further causes the system at least to:

determine a confidence level that the data is indicative of the gesture being performed.

17. The system of claim 11, wherein the gesture comprises:

a crouch, a jump, a lean, an arm throw, a toss, a swing, a dodge, a kick, or a block.

18. The system of claim 11, wherein a gesture filter comprises a type of input and a type of output, and wherein the memory further bears processor-executable instructions that, when executed by the processor, cause the system at least to:

replace the gesture filter with a second filter, the second filter having the same type of input and type of output as the gesture filter,

receive second data captured by a camera;

determine a second output of the second filter based on the second data; and

send the first application the second output when the data corresponds to a first application of a plurality of applications.

19. The system of claim 11, wherein the memory further bears processor-executable instructions that, when executed by the processor, cause the system at least to:

receive sound data; and

wherein the processor-executable instructions that, when executed by the system, cause the system to at least determine an output of the gesture filter based on the data further cause the system to at least;

determine the output of the gesture filter based on the sound data.

20. A computer-readable storage device that is not a propagating signal for providing recognition of gestures made by a user using a gesture filter representing a gesture, the computer-readable storage device bearing computer-readable instructions that, upon execution on a computer, cause the computer to perform operations comprising:

receiving data indicative of a user motion or pose, the data being captured by a camera;

determining an output of the gesture filter based on the data;

determining, based on the data, to send the output to a first application of a plurality of applications in response to determining that the data corresponds to the first application and does not correspond to a system application of the plurality of applications; and

sending the output to the first application of the plurality of applications.