SKELETAL JOINT RECOGNITION AND TRACKING SYSTEM

• US 20120162065A1
• Filed: 03/02/2012
• Published: 06/28/2012
• Est. Priority Date: 06/29/2010
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tossell, Philip, Wilson, Andrew, Kipman, Alex Aben-Athar, Lee, Johnny Chung, Balan, Alex, Shotton, Jamie, Moore, Richard, Williams, Oliver, Geiss, Ryan, Finocchio, Mark, Perez, Kathryn Stone, Kornblum, Aaron, Clavin, John
• Application Number:
  US13/410,681
• Publication Number:
  US 20120162065A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A system and method are disclosed for recognizing and tracking a user'"'"'s skeletal joints with a NUI system and further, for recognizing and tracking only some skeletal joints, such as for example a user'"'"'s upper body. The system may include a limb identification engine which may use various methods to evaluate, identify and track positions of body parts of one or more users in a scene. In examples, further processing efficiency may be achieved by segmenting the field of view in smaller zones, and focusing on one zone at a time. Moreover, each zone may have its own set of predefined gestures which are recognized.

Claims

1. A method of gesture recognition, comprising:

a) segmenting a field of view of a scene into a plurality of zones;

b) defining one or more recognizable gestures for a specified zone of the plurality of zones;

c) receiving position information from a user in the scene, the user having a first body part and second body part;

d) recognizing a gesture from the first body part where the first body part has performed a recognizable gesture in the specified zone;

e) ignoring a gesture performed by the second body part where the second body part has not performed a recognizable gesture in the specified zone; and

f) performing an action associated with the gesture from the first body part recognized in said step d).

2. The method of claim 1, said step e) of ignoring a gesture performed by the second body part comprising the step of having a definition of body parts from which gestures are recognized in the specified zone, said second body part not being included in the definition.

3. The method of claim 2, said step of having a definition of body parts from which gestures are accepted comprising the step of a user indicating that the second body part is not included in the definition of body parts from which gestures are accepted.

4. The method of claim 1, said step e) of ignoring a gesture performed by the second body part comprising the step of not receiving position information from the second body part.

5. The method of claim 4, said step e) of not receiving position information from the second body part comprising the step of identifying and tracking body parts other than the second body part.

6. The method of claim 4, said step e) of not receiving position information from the second body part comprising the step of not tracking the second body part because the second body part in not in the specified zone.

7. The method of claim 1, the second body part being in a zone of the plurality of zones other than the specified zone when the gesture made with the second body part is ignored, and further comprising the step of recognizing and acting on the same gesture from the second body part when made in the specified zone of the plurality of zones.

8. The method of claim 1, further comprising the step of displaying an avatar on a screen associated with the computing environment, the avatar having a body part which is a virtual copy of the second body part, the user controlling movement of the virtual body part with movement of the user'"'"'s first body part.

9. A method of recognizing and tracking body parts of a target, comprising:

a) obtaining body part proposals from a stateless body part proposal system receiving position information from a scene, wherein the stateless body part proposal system comprises body part locations without reference to prior frames of the scene;

b) obtaining body part proposals from a stateful body part proposal system, wherein the stateful body part proposal system comprises body part locations with reference to prior frames of the scene;

c) reconciling the candidate body parts into whole or partial targets by a resolution system;

d) segmenting a field of view of the scene into at least one zone smaller than the field of view of the scene; and

e) determining whether a body part reconciled into a whole or partial target in said step c) has performed a recognized gesture in the at least one zone.

10. The method of claim 9, said step a) of obtaining body part proposals from a stateless machine-learning body part proposal system comprising the step of obtaining body part proposals for a head and shoulders of the user by centroid probabilities.

11. The method of claim 9, said step b) of obtaining body part proposals from a stateful body part proposal system comprising the step of obtaining body part proposals for a head and shoulders of the user by at least one of magnetism and persistence from a past frame.

12. The method of claim 9, said step of reconciling the candidate body parts into whole or partial skeletons comprising running one or more scored tests which allow identification of the hypothesis that has the greatest support.

13. The method of claim 12, said step of performing one or more tests comprising the step of performing a test checking for pixel motion near the hand proposals to detect how fast the pixels in the vicinity of a hand proposal are moving.

14. The method of claim 9, said step b) comprising the step of identifying a first group of joints by the steps of:

f) identifying candidate head and shoulder proposals that correspond to real players;

g) evaluating hand proposals which potentially belong to each shoulder of each candidate in said step f); and

h) evaluating elbow proposals which connects hand proposals in said step g) with shoulder proposals in said step f).

15. The method of claim 14, said step h) comprising the step of trying a plurality of possible arm hypotheses, performing one or more tests to score the arm hypotheses, and using an arm hypothesis having a highest score as the identified positions of joints in the first group of joints.

16. The method of claim 12, wherein the step of performing one or more tests includes the step of performing a trace and saliency test where depth map samples inside a possible arm hypothesis and outside a possible arm hypothesis are evaluated against expected depth map values for the possible arm hypothesis and a score is produced.

17. A computer-readable storage medium capable of programming a processor to perform a method of recognizing and tracking body parts of a user having at least limited use of at least one immobilized body part, the method comprising:

a) receiving an indication from the user of the identity of the at least one immobilized body part;

b) identifying a first group of joints of the user, the joints not included within the at least one immobilized body part;

c) identifying positions of joints in the first group of joints; and

d) performing an action based on positions of the joints identified in said step c).

18. The computer-readable storage medium of claim 17, said step a) further comprising the step of receiving an indication from the user of whether the at least one immobilized body part is permanently or temporarily immobilized.

19. The computer-readable storage medium of claim 17, further comprising the steps of displaying an avatar on a screen associated with the computing environment, the avatar having a virtual body part which corresponds to an immobilized body part of the at least one immobilized body parts, and receiving an indication from the user of a substitute body part other than the immobilized body part to control the virtual body part of the onscreen avatar.

20. The computer-readable storage medium of claim 17, said step a) of receiving an indication from the user of the identity of the at least one immobilized body part comprising one of:

a1) receiving an indication that the user'"'"'s legs are immobilized,

a2) receiving an indication that the user'"'"'s arms are immobilized,

a3) receiving an indication that the user'"'"'s right arm and right leg are immobilized, and

a4) receiving an indication that the user'"'"'s left arm and left leg are immobilized.