Problem states for pose tracking pipeline

• US 8,630,457 B2
• Filed: 12/15/2011
• Issued: 01/14/2014
• Est. Priority Date: 12/15/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Craig, Robert
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US13/327,098
• Publication Number:
  US 20130156260A1
• Time in Patent Office: 761 Days
• Field of Search: 382/103, 382/154, 348/169, 345/419
• US Class Current: 382/103
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A human subject is tracked within a scene of an observed depth image supplied to a pose tracking pipeline. An indication of a problem state is received from the pose tracking pipeline, and an identification of the problem state is supplied to the pose tracking pipeline. A virtual skeleton is received from the pose tracking pipeline that includes a plurality of skeletal points defined in three-dimensions. The pose tracking pipeline selects a three-dimensional position of at least one of the plurality of skeletal points in accordance with the identification of the problem state supplied to the pose-tracking pipeline.

Claims

1. A method of tracking a human subject within a scene, the method comprising:

receiving from a source an observed depth image of the scene including the human subject;

supplying the observed depth image to a pose tracking pipeline;

receiving an indication of a problem state from the pose tracking pipeline;

supplying an identification of the problem state to the pose tracking pipeline; and

receiving from the pose tracking pipeline a virtual skeleton including a plurality of skeletal points defined in three-dimensions, the pose tracking pipeline selecting a three-dimensional position of at least one of the plurality of skeletal points in accordance with the identification of the problem state supplied to the pose-tracking pipeline.

2. The method of claim 1, wherein the indication of the problem state is received from the pose tracking pipeline as information derived from the observed depth image by the pose tracking pipeline;

and

wherein the method further comprises identifying the problem state from a set of available problem states based on the information derived from the observed depth image.

3. The method of claim 2, wherein identifying the problem state from the set of available problem states includes applying classifier analysis to the set of available problem states to classify each problem state of the set as either an applicable problem state or an inapplicable problem state based on the information derived from the observed depth image.

4. The method of claim 2, wherein identifying the problem state from the set of available problem states includes applying regressor analysis to the set of available problem states to assign a measurement and/or probability to each problem state of the set.

5. The method of claim 1, wherein receiving the indication of the problem state from the pose tracking pipeline includes receiving the identification of the problem state from an intermediate process of the pose tracking pipeline;

and

wherein supplying the identification of the problem state to the pose tracking pipeline includes supplying the identification of the problem state to another intermediate process of the pose tracking pipeline.

6. The method of claim 1, wherein the pose tracking pipeline includes a segmentation module to receive the observed depth image supplied to the pose tracking pipeline, and to output pixel classification information classifying each pixel of the observed depth image as either a foreground pixel belonging to the human subject or a background pixel not belonging to the human subject;

and

wherein receiving the indication of the problem state from the pose tracking pipeline includes receiving one or more of a message indicating the problem state and/or the pixel classification information indicating the problem state.

7. The method of claim 1, wherein the pose tracking pipeline includes a pixel assignment module to output pixel assignment information labeling each pixel of the observed depth image classified as a foreground pixel belonging to the human subject with body part information indicating a likelihood that that the foreground pixel belongs to one or more body parts of the human subject;

and

wherein receiving the indication of the problem state from the pose tracking pipeline includes receiving one or more of a message indicating the problem state and/or the pixel assignment information indicating the problem state.

8. The method of claim 1, wherein the pose tracking pipeline includes a model fitting module to output a set of proposed virtual skeletons for the human subject;

and

wherein receiving the indication of the problem state from the pose tracking pipeline includes receiving one or more of a message indicating the problem state and/or the set of proposed virtual skeletons indicating the problem state.

9. The method of claim 1, wherein the pose tracking pipeline includes a model resolution module to receive a set of proposed virtual skeletons, and output the virtual skeleton from the skeleton tracking pipeline as a selected virtual skeleton;

wherein receiving the indication of the problem state from the pose tracking pipeline includes receiving one or more of a message indicating the problem state and/or the selected virtual skeleton indicating the problem state.

10. The method of claim 1, wherein the problem state indicated by the pose tracking pipeline is one of a plurality of problem states indicated by the pose tracking pipeline;

and

wherein the method further includes supplying an identification of the plurality of problem states to the pose tracking pipeline.

11. The method of claim 1, further comprising:

identifying an intermediate process of the pose tracking pipeline to supply the identification of the problem state based on an operating parameter, the operating parameter including one or more of;

the problem state indicated by the pose tracking pipeline; and

/or

an intermediate process of the pose tracking pipeline from which the indication of the problem state was received.

12. The method of claim 1, wherein the indication of the problem state received from the pose tracking pipeline corresponds to a problem state of the observed depth supplied to the pose tracking pipeline;

and

wherein the virtual skeleton received from the pose tracking pipeline corresponds to the observed depth image supplied to the pose tracking pipeline.

13. The method of claim 1, further comprising:

supplying a subsequently observed depth image of the scene including the human subject to the pose tracking pipeline;

wherein the indication of the problem state received from the pose tracking pipeline corresponds to a problem state of the observed depth supplied to the pose tracking pipeline; and

wherein the virtual skeleton received from the pose tracking pipeline corresponds to the subsequently observed depth image supplied to the pose tracking pipeline.

14. The method of claim 1, wherein the problem state includes an occluded state corresponding to a state of the observed depth image where at least a portion of the human subject is blocked from view within the scene by an object.

15. The method of claim 1, wherein the problem state includes a cropped state corresponding to a state of the observed depth image where at least a portion of the human subject resides outside a boundary of the scene.

16. The method of claim 1, wherein the problem state includes a proximate state corresponding to a state of the observed depth image where at least a portion of the human subject resides within a threshold depth proximity to an object within the scene.

17. The method of claim 1, wherein the problem state includes a crossed state corresponding to a state of the observed depth image where a body part of the human subject crosses a virtual boundary dividing the virtual skeleton into at least two regions from a native region of the body part to a non-native region of the body part.

18. The method of claim 1, wherein the problem state includes a velocity limited state corresponding to a state of the observed depth image where the human subject has a velocity in the scene that exceeds an upper velocity threshold and/or a lower velocity threshold.

19. A computing device, comprising:

a logic subsystem; and

a data-holding subsystem holding instructions executable by the logic subsystem of the computing device to;

receive from a source an observed depth image of a scene including a human subject;

supply the observed depth image to a pose tracking pipeline;

receive an indication of a problem state from the pose tracking pipeline;

supply an identification of the problem state to the pose tracking pipeline; and

receive from the pose tracking pipeline a virtual skeleton including a plurality of skeletal points defined in three-dimensions, the pose tracking pipeline selecting a three-dimensional position of at least one of the plurality of skeletal points in accordance with the identification of the problem state supplied to the pose-tracking pipeline.

20. A computing device, comprising:

a logic subsystem; and

a data-holding subsystem holding instructions executable by the logic subsystem of the computing device, the instructions including;

a pose tracking pipeline configured to output a virtual skeleton for an observed depth image of a human subject and an identification of a problem state supplied to the pose tracking pipeline for each body part of a plurality of body parts of the human subject;

a problem state module configured to;

receive information derived from the observed depth image from an intermediate process of the pose tracking pipeline;

identify the problem state from a set of available problem states for each body part of the plurality of body parts based on the information received from the intermediate process; and

supply an identification of the problem state for each body part identified by the problem state module to another intermediate process of the pose tracking pipeline, the pose tracking pipeline selecting a three-dimensional position of at least one of the plurality of skeletal points in accordance with the identification of the problem state supplied to the pose-tracking pipeline for at least one body part.