Gesture recognizer system architecture

• US 7,996,793 B2
• Filed: 04/13/2009
• Issued: 08/09/2011
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Markovic, Relja, Latta, Stephen G., Snook, Gregory N., Tomlin, Arthur Charles
• Primary Examiner(s): HAILU, TADESSE
• Application Number:
  US12/422,661
• Publication Number:
  US 20100199230A1
• Time in Patent Office: 848 Days
• Field of Search: 715/863, 715/864, 715/859, 345/157, 345/158, 345/633, 348/333.02
• US Class Current: 715/864
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Systems, methods and computer readable media are disclosed for a gesture recognizer system architecture. A recognizer engine is provided, which receives user motion data and provides that data to a plurality of filters. A filter corresponds to a gesture, that may then be tuned by an application receiving information from the gesture recognizer so that the specific parameters of the gesture—such as an arm acceleration for a throwing gesture—may be set on a per-application level, or multiple times within a single application. Each filter may output to an application using it a confidence level that the corresponding gesture occurred, as well as further details about the user motion data.

Claims

1. A method for providing recognition of gestures made by a user using a filter representing a gesture, the filter comprising base information about the gesture, comprising:

receiving data captured by a camera, the data corresponding to a first application;

applying the filter to the data and determining an output from the base information about the gesture;

sending the first application the output;

receiving second data captured by the camera, the data corresponding to a second application;

applying the filter to the second data and determining a second output from the base information about the gesture; and

sending the second application the second output.

2. The method of claim 1, wherein a filter comprises a parameter about the gesture, further comprising:

receiving from the first application a value for a parameter of the filter; and

wherein determining the output from the base information about the gesture comprises;

determining the output from the value of the parameter.

3. The method of claim 2, further comprising:

receiving from the second application a second value for the parameter of the filter, the second value differing from the value; and

wherein applying the filter to the second data and determining a second output from the base information about the gesture comprises;

determining the second output from the second value of the parameter.

4. The method of claim 2, wherein receiving from the first application the value for a parameter comprises setting the parameter with the value, and receiving data captured by a camera comprises receiving data from a first user, further comprising:

setting the parameter with a second value in response to receiving data captured by the depth camera indicative of a change in the first user'"'"'s fatigue, an change in the first user'"'"'s competence, or a second user replacing the first user.

5. The method of claim 2, wherein receiving from the first application a value for the parameter of the filter comprises:

receiving third data captured by the camera, the third data corresponding to a user of the first application making a body motion or a body pose.

6. The method of claim 2, wherein a parameter comprises an indication of a body part, a volume of space, a velocity, a direction of movement, an angle, or a place where a movement occurs.

7. The method of claim 1, wherein the output comprises a confidence level that the data is indicative of the gesture being performed.

8. The method of claim 1, wherein the data captured by a camera is indicative of a user'"'"'s first gesture being performed, the first application has a context, and the output comprises a confidence level that the gesture is being performed, further comprising:

changing the context;

receiving third data captured by the camera indicative of the user'"'"'s first gesture being performed;

determining from the third data that a confidence level that a second gesture represented by a second filter is being performed is greater than the confidence level from the second data that the represented by the filter is being performed.

9. The method of claim 1, wherein the gesture comprises:

a crouch, a jump, a lean, an arm throw, a toss, a swing, a dodge, a kick, or a block.

10. The method of claim 1, wherein the filter is a member of a genre package comprising a plurality of filters representing gestures, the genre package comprising a first-person shooter, action, driving, or sports genre package.

11. The method of claim 1, wherein the base information about the gesture has a plurality of contexts, and a parameter of each context of the base information about the gesture is unique.

12. The method of claim 1, wherein the filter is received from the first application.

13. The method of claim 1, wherein a filter comprises a number of inputs, a collective type of inputs, a number of outputs, and a collective type of outputs, comprising:

replacing the filter with a second filter, the second filter comprising base information about the gesture, the number of inputs of the second filter equaling the number of inputs of the filter, the collective type of inputs of the second filter equaling the collective type of inputs of the filter, the number of outputs of the second filter equaling the number of outputs of the filter, the collective type of outputs of the second filter equaling the collective type of outputs of the filter,

receiving third data captured by a camera;

applying the third filter to the data and determining a third output from the base information of the second filter; and

sending the first application the third output.

14. A system for providing recognition of gestures made by a user using a filter representing a gesture, the filter comprising base information about the gesture, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause operations comprising;

receiving data captured by a camera, the data corresponding to a first application;

applying the filter to the data and determining an output from the base information about the gesture;

sending the first application the output;

receiving second data captured by the camera, the data corresponding to a second application;

applying the filter to the second data and determining a second output from the base information about the gesture; and

sending the second application the second output.

15. The system of claim 14, wherein a filter comprises a parameter, wherein the memory further bears processor-executable instructions that, when executed on the processor cause the performance of operations comprising:

receiving from the first application a value for a parameter of the filter; and

wherein determining the output from the base information about the gesture comprises;

determining the output from the value of the parameter.

16. The system of claim 15, wherein the filter is one of a plurality of complimentary filters, and wherein the memory further bears processor-executable instructions that, when executed on the processor cause the performance of operations comprising:

setting a parameter with the value; and

setting a complimentary parameter of a complimentary filter based on the value.

17. The system of claim 14, wherein the memory further bears processor-executable instructions that, when executed on the processor cause the performance of operations comprising:

receiving voice data; and

determining the first output based on the voice data.

18. The system of claim 14, wherein

the output comprises an indication of a velocity of movement, a release point, a distance, or a body part that made the gesture.

19. The system of claim 14, wherein the memory further bears processor-executable instructions that, when executed on the processor cause the performance of operations comprising:

applying a second filter to the data, the second filter representing a second gesture, the second filter comprising base information about the second gesture, and determining a third output from the base information about the second gesture; and

sending the first application the third output.

20. A computer readable storage medium for providing recognition of gestures made by a user using a filter representing a gesture, the filter comprising base information about the gesture, the computer readable storage medium comprising computer readable instructions that when executed on a processor, cause the performance of operations comprising:

receiving data captured by a camera, the data corresponding to a first application; and

applying the filter to the data and determining from the base information about the gesture;

sending the first application the output;

receiving second data captured by the camera, the data corresponding to a second application;

applying the filter to the second data and determining a second output from the base information about the gesture; and

sending the second application the second output.