Gesture recognizer system architecture

• US 8,869,072 B2
• Filed: 08/02/2011
• Issued: 10/21/2014
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Latta, Stephen G., Markovic, Relja, Tomlin, Arthur Charles, Snook, Gregory N.
• Primary Examiner(s): HAILU, TADESSE
• Application Number:
  US13/196,588
• Publication Number:
  US 20110285620A1
• Time in Patent Office: 1,176 Days
• Field of Search: 715/863, 345/157, 345/158
• US Class Current: 715/863
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Systems, methods and computer readable media are disclosed for a gesture recognizer system architecture. A recognizer engine is provided, which receives user motion data and provides that data to a plurality of filters. A filter corresponds to a gesture, that may then be tuned by an application receiving information from the gesture recognizer so that the specific parameters of the gesture—such as an arm acceleration for a throwing gesture—may be set on a per-application level, or multiple times within a single application. Each filter may output to an application using it a confidence level that the corresponding gesture occurred, as well as further details about the user motion data.

Claims

1. A method for providing gesture input to a plurality of applications, comprising:

receiving data indicative of a user motion or pose, the data being captured by a camera;

determining a result of processing the data, the result comprising a three-dimensional model of at least part of the user;

sending the result to a first gesture filter of a first application of the plurality of applications, the first application being configured to process the result with the first gesture filter to determine a first output indicative of whether the result is indicative of the user performing a gesture represented by the first gesture filter; and

sending the result to a second gesture filter of a second application of the plurality of applications, the second application being configured to process the result with the second gesture filter to determine a second output indicative of whether the result is indicative of the user performing a gesture represented by the second gesture filter.

2. The method of claim 1, wherein determining the result of processing the data comprises:

determining at least a partial skeletal model of the user from the data.

3. The method of claim 1, wherein the first gesture filter and the second gesture filter represent the same gesture, and wherein the first application is configured to determine the first output to be indicative of the user not performing the gesture, and the second application is configured to determine the second output to be indicative of the user performing the gesture.

4. The method of claim 1, wherein the first gesture filter and the second gesture filter represent the same gesture, and wherein the first application is configured to determine a value of the first output that is different from a value of the second output determined by the second application.

5. The method of claim 1, further comprising:

before receiving the data indicative of the user motion or pose, receiving second data indicative of a second user motion or pose; and

sending the second data to the first application of the plurality of applications, the first application being configured to modify how the first gesture filter recognizes the gesture represented by the first gesture filter based on the second data.

6. The method of claim 5, wherein sending the second data to the first application of the plurality of applications, the first application being configured to modify how the first gesture filter recognizes the gesture represented by the first gesture filter based on the second data, comprises:

sending the second data to the first application of the plurality of applications, the first application being configured to modify a value of a parameter of the first gesture filter, the parameter comprising;

an indication of a body part, a volume of space, a velocity, a direction of movement, an angle, or a place where a movement occurs.

7. The method of claim 1, wherein the first output comprises:

a confidence level that the data is indicative of the gesture being performed.

8. The method of claim 1, wherein the data is indicative of a user'"'"'s first gesture being performed, the first application has a context, and the first output comprises a confidence level that the gesture is being performed, further comprising:

changing the context;

receiving second data captured by the camera indicative of the user'"'"'s first gesture being performed;

determining a second result of processing the second data; and

sending the second result to the first application, the first application being configured to process the second result with the first gesture filter to determine a new first output, the first application being configured to process the second result with a third gesture filter to determine a third output, the third output indicating a higher confidence level that a third gesture represented by the third gesture filter has been performed than a confidence level that the first gesture has been performed is indicated by the first output.

9. The method of claim 1, wherein the gesture comprises:

a crouch, a jump, a lean, an arm throw, a toss, a swing, a dodge, a kick, or a block.

10. The method of claim 1, wherein the first gesture filter is a member of a genre package comprising a plurality of gesture filters representing gestures, the genre package comprising a first-person shooter, action, driving, or sports genre package.

11. The method of claim 1, wherein the first gesture filter has a first context and a second context, and a parameter, a value of the parameter for the first context differing from a value of the parameter for the second context.

12. The method of claim 1, wherein the first gesture filter comprises an input having a type, and an output having a type, further comprising:

replacing the first gesture filter with a new first gesture filter, the type of the input of the first gesture filter being the same as a type of an input of the new first gesture filter, the type of the output of the first gesture filter being the same as a type of an output of the new first gesture filter.

13. A system for providing gesture input to a plurality of applications, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system to at least;

receive data indicative of a user motion or pose, the data being captured by a camera;

determine a result of processing the data, the result comprising a three-dimensional model of at least part of the user;

send the result to a first gesture filter of a first application of the plurality of applications, the first application being configured to process the result with the first gesture filter to determine a first output indicative of whether the result is indicative of the user performing a gesture represented by the first gesture filter; and

send the result to a second gesture filter of a second application of the plurality of applications, the second application being configured to process the result with the second gesture filter to determine a second output indicative of whether the result is indicative of the user performing a gesture represented by the second gesture filter.

14. The system of claim 13, wherein the instructions that cause the system to at least determine the result of processing the data further cause the system to at least:

determine at least a partial skeletal model of the user from the data.

15. The system of claim 13, wherein the first gesture filter and the second gesture filter represent the same gesture, and wherein the first application is configured to determine the first output to be indicative of the user not performing the gesture, and the second application is configured to determine the second output to be indicative of the user performing the gesture.

16. The system of claim 13, wherein the first gesture filter and the second gesture filter represent the same gesture, and wherein the first application is configured to determine a value of the first output that is different from a value of the second output determined by the second application.

17. The system of claim 13, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

before receiving the data indicative of the user motion or pose, receive second data indicative of a second user motion or pose; and

send the second data to the first application of the plurality of applications, the first application being configured to modify how the first gesture filter recognizes the gesture represented by the first gesture filter based on the second data.

18. The system of claim 17, wherein the instructions that cause the system to at least send the second data to the first application of the plurality of applications, the first application being configured to modify how the first gesture filter recognizes the gesture represented by the first gesture filter based on the second data further cause the system to at least:

send the second data to the first application of the plurality of applications, the first application being configured to modify a value of a parameter of the first gesture filter, the parameter comprising;

an indication of a body part, a volume of space, a velocity, a direction of movement, an angle, or a place where a movement occurs.

19. The system of claim 13, wherein the data is indicative of a user'"'"'s first gesture being performed, the first application has a context, and the first output comprises a confidence level that the gesture is being performed, and wherein the memory further bears processor-executable instructions that, when executed upon the processor, cause the processor to at least:

change the context;

receive second data captured by the camera indicative of the user'"'"'s first gesture being performed;

determine a second result of processing the second data; and

send the second result to the first application, the first application being configured to process the second result with the first gesture filter to determine a new first output, the first application being configured to process the second result with a third gesture filter to determine a third output, the third output indicating a higher confidence level that a third gesture represented by the third gesture filter has been performed than a confidence level that the first gesture has been performed is indicated by the first output.

20. A computer-readable storage medium for providing gesture input to a plurality of applications bearing computer-readable instructions, that when executed on a computer, cause operations comprising:

receiving data indicative of a user motion or pose, the data being captured by a camera;

determining a result of processing the data, the result comprising a three-dimensional model of at least part of the user;

sending the result to a first gesture filter of a first application of the plurality of applications, the first application being configured to process the result with the first gesture filter to determine a first output indicative of whether the result is indicative of the user performing a gesture represented by the first gesture filter; and

sending the result to a second gesture filter of a second application of the plurality of applications, the second application being configured to process the result with the second gesture filter to determine a second output indicative of whether the result is indicative of the user performing a gesture represented by the second gesture filter.