GESTURE RECOGNIZER SYSTEM ARCHITICTURE

• US 20100199230A1
• Filed: 04/13/2009
• Published: 08/05/2010
• Est. Priority Date: 01/30/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Markovic, Relja, Latta, Stephen G., Snook, Gregory N., Tomlin, Arthur Charles
• Granted Patent: US 7,996,793 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Systems, methods and computer readable media are disclosed for a gesture recognizer system architecture. A recognizer engine is provided, which receives user motion data and provides that data to a plurality of filters. A filter corresponds to a gesture, that may then be tuned by an application receiving information from the gesture recognizer so that the specific parameters of the gesture—such as an arm acceleration for a throwing gesture—may be set on a per-application level, or multiple times within a single application. Each filter may output to an application using it a confidence level that the corresponding gesture occurred, as well as further details about the user motion data.

Claims

1. A method for providing recognition of gestures made by a user to a first application, comprising:

providing a filter representing a gesture to the first application, the filter comprising base information about the gesture;

receiving data captured by a camera, the data corresponding to the first application;

applying the filter to the data and determining an output from the base information about the gesture; and

sending the first application the output.

2. The method of claim 1, wherein a filter comprises at least one parameter about the gesture, further comprising:

receiving from the first application a value for at least one parameter; and

determining the output from the base information about the gesture and each parameter includes;

determining the output from the value of the parameter.

3. The method of claim 2, further comprising:

receiving from a second application a second value for at least one parameter of a second filter representing the gesture, the second filter comprising the base information about the gesture, the second value differing from the value received from the first application;

receiving second data captured by a camera;

applying the second filter to the second data and determining an output from the base information about the gesture and each parameter of the second filter; and

sending the second application the output.

4. The method of claim 2, wherein receiving from the first application the value for at least one parameter includes setting each parameter with the value, and receiving data captured by a camera includes receiving data from a first user, further comprising:

setting each parameter with a second value in response to receiving data captured by the depth camera indicative of one selected from the group consisting of a change in the first user'"'"'s fatigue, an change in the first user'"'"'s competence, and a second user replacing the first user.

5. The method of claim 2, wherein the value of each parameter is determined by an end user of the application through making a body motion or making a body pose.

6. The method of claim 2, wherein a parameter is one selected from the group consisting of a body part, a volume of space, a velocity, a direction of movement, an angle, and a place where a movement occurs.

7. The method of claim 1, wherein the output comprises a confidence level that the data is indicative of the gesture being performed.

8. The method of claim 1, wherein the data captured by a camera is indicative of a user'"'"'s first gesture being performed, the first application has a context, and the output comprises a confidence level that a gesture is being performed, further comprising:

changing the context;

receiving a second data captured by the camera indicative of the user'"'"'s first gesture being performed;

determining from the second data that a confidence level that a second gesture represented by a second filter is being performed is greater than the confidence level from the second data that the represented by the filter is being performed.

9. The method of claim 1, wherein the gesture is at least one selected from the group consisting of:

a crouch, a jump, a lean, an arm throw, a toss, a swing, a dodge, a kick, and a block.

10. The method of claim 1, wherein the filter is a member of a genre package comprising a plurality of filters representing gestures, the genre package selected from the group consisting of first-person shooter, action, driving, and sports.

11. The method of claim 1, wherein the base information about the gesture has a plurality of contexts, and the parameter of each context of the base information about the gesture is unique.

12. The method of claim 1, wherein the filter is received from the first application.

13. The method of claim 1, wherein a filter has a number of inputs, a collective type of inputs, a number of outputs, and a collective type of outputs, comprising:

replacing the filter with a second filter, the second filter comprising base information about the gesture, the number of inputs of the second filter equaling the number of inputs of the filter, the collective type of inputs of the second filter equaling the collective type of inputs of the filter, the number of outputs of the second filter equaling the number of outputs of the filter, the collective type of outputs of the second filter equaling the collective type of outputs of the filter, receiving second data captured by a camera;

applying the second filter to the data and determining a second output from the base information of the second filter; and

sending the application the second output.

14. A system for providing a library of gestures to an application, comprising:

a processor;

a gesture library comprising at least one filter representing a gesture, the filter comprising base information about the gesture, wherein a filter may be added to or removed from the gesture library; and

a gesture recognizer engine that receives data captured by a camera corresponding to the application, determines an output from the base information about the gesture, and sends the application the output.

15. The system of claim 14, wherein a filter comprises at least one parameter, and a value of the at least one parameter is determined from the application, and the gesture recognizer engine further:

determines the output from the value of the parameter.

16. The system of claim 15, wherein a filter comprises at least one parameter, and the filter is one of a plurality of complimentary filters, and the gesture library further:

sets at least one parameter with the value; and

sets at least one complimentary parameter of a complimentary filter based on the value.

17. The system of claim 14, further comprising:

a voice receiver that receives voice data; and

the gesture recognizer engine further determines the output based on the voice data.

18. The system of claim 14, wherein the gesture recognizer engine further:

sends the application at least one selected from the group consisting of a velocity of movement, a release point, a distance, and a body part that made the gesture.

19. The system of claim 14, wherein the gesture recognizer engine further:

sends the application a plurality of outputs, each output corresponding to a filter of a plurality of filters.

20. A computer readable storage medium, comprising computer readable instructions that when executed on a processor, cause the processor to perform the operations of:

providing a filter representing a gesture to the plurality of applications, the filter comprising base information about the gesture and at least one parameter about the gesture;

receiving from the first application a value for at least one parameter; and

receiving data captured by a depth camera, the data corresponding to a first application of the plurality of applications; and

applying the filter to the data and determining from the base information about the gesture and the value of the parameter a confidence level that the data is indicative of the gesture being performed.