System for fast, probabilistic skeletal tracking

• US 8,437,506 B2
• Filed: 09/07/2010
• Issued: 05/07/2013
• Est. Priority Date: 09/07/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Williams, Oliver, Geiss, Ryan
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US12/876,418
• Publication Number:
  US 20120056800A1
• Time in Patent Office: 973 Days
• Field of Search: 382/100, 382/103, 382106-107, 382/123, 382/155, 382/162, 382/168, 382/180, 382/181, 382189-199, 382/219, 382/232, 382/254, 382/274, 382/276, 382/291, 382/312, 382/305, 382/154, 463/36, 348/14.16, 348/46, 356/5.01, 359/9
• US Class Current: 382/103
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/6615 using models with different levels of detail [LOD]
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A system and method are disclosed for recognizing and tracking a user'"'"'s skeletal joints with a NUI system. The system includes one or more experts for proposing one or more skeletal hypotheses each representing a user pose within a given frame. Each expert is generally computationally inexpensive. The system further includes an arbiter for resolving the skeletal hypotheses from the experts into a best state estimate for a given frame. The arbiter may score the various skeletal hypotheses based on different methodologies. The one or more skeletal hypotheses resulting in the highest score may be returned as the state estimate for a given frame. It may happen that the experts and arbiter are unable to resolve a single state estimate with a high degree of confidence for a given frame. It is a further goal of the present system to capture any such uncertainty as a factor in how a state estimate is to be used.

Claims

1. In a system including a computing environment coupled to a capture device for capturing state information from a field of view of the capture device, the state information representing a position of a user, a method of estimating state information comprising:

(a) receiving image data from the field of view;

(b) producing one or more computer models estimating state information by one or more experts, said step of producing one or more computer models based on computing body part proposals from the image data and generating one or more skeletal hypotheses from the body part proposals by a known configuration of how body parts are connected in humans; and

(c) analyzing the one or more computer models produced in said step (b) by one or more methodologies of an arbiter to choose one or more computer models estimated to be the best representation of the state information.

2. The method of claim 1, further comprising the step (d) of generating a confidence level in the one or more computer models estimated to be the best representation of the state information.

3. The method of claim 1, said step (b) of producing one or more computer models comprising the step of producing one or more computer models based on the image data from the field of view captured in a current frame.

4. The method of claim 3, said of producing one or more computer models based on the image data from the field of view captured in a current frame comprising the step of producing one or more computer models based on body part proposals computed from the image data from the current field of view.

5. The method of claim 1, said step (b) of producing one or more computer models comprising the step of producing one or more computer models based on the image data from the field of view captured in a prior frame.

6. The method of claim 1, said step of generating one or more skeletal hypotheses from the body part proposals by a known configuration of how body parts are connected in humans comprising the step of generating skeletal hypotheses from at least one of a tree structure of a human body including a torso and limbs as branches, a head triangle including a triangle formed by a head and shoulders and a torso volume including a torso.

7. The method of claim 1, said step (b) of producing one or more computer models comprising the step of identifying a body part in a past frame that is proximate to a body part indicated in the depth image in a current frame, identifying whether the body part from the past and current frame may be the same body part, and snapping the position of the body part from the last frame to the depth image in the current frame.

8. The method of claim 1, said step (b) of producing one or more computer models comprising the step of producing one or more skeletal hypotheses including an estimated position of joints of a user captured for the frame by the capture device.

9. The method of claim 8, said step of producing one or more skeletal hypotheses omitting joint information for one or more joints, the missing joint information accounted for the one or more skeletal hypotheses by assigning a null candidate to a missing joint, growing a missing joint based on positions of adjacent joints, and using a position of the missing joint identified in a past frame as a basis for the position of the missing joint in the current frame.

10. A software pipeline for generating a state estimate for a given frame of captured image data, the state estimate representing an estimate of a position of a user within a field of view captured within the image data, comprising:

a preprocessing routine for receiving the image data, removing a background from the image data, and processing a foreground into one or more body part proposals;

one or more experts for receiving information including the one or more body part proposals and generating a plurality of computer models, each computer model representing an estimation of the position of the user in the given frame of captured image data; and

an arbiter for receiving the plurality of computer models, scoring the computer models by one or more methodologies which compare the plurality of computer models against depth data from the given frame and/or state estimate data from a prior frame, and outputting at least one computer model estimated by the arbiter to best approximate the position of the user in the frame.

11. A software pipeline as recited in claim 10, the arbiter further including a depth score methodology for scoring each of the plurality of computer models by examining the computer model against the depth data for the given frame.

12. A software pipeline as recited in claim 11, wherein the depth score methodology examines trace and saliency samples in scoring each of the plurality of computer models.

13. A software pipeline as recited in claim 10, the arbiter further including an explained space methodology for scoring each of the plurality of computer models by examining how well the computer model explains all of the depth data in the foreground.

14. A software pipeline as recited in claim 10, the arbiter further including a static joint configuration scoring methodology for scoring each of the plurality of computer models by examining whether the computer models have lengths and angles between joints which conform to lengths and angles that are possible for body parts of a human body.

15. A software pipeline as recited in claim 10, the arbiter further including a motion scoring methodology for scoring each of the plurality of computer models by examining image data and/or a state estimation for a prior frame and determining the likelihood that a body part indicated by a computer model could have moved to the position indicated in the given frame from the position indicated in the prior frame within the time elapsed between the given frame and prior frame.

16. A computer-readable storage medium capable of programming a processor to perform a method tracking body parts of a user captured with a capture device as a user moves within a field of view of the capture device to determine a state estimate of the user'"'"'s position in a current frame of image data captured by the capture device, comprising:

(a) receiving image data from the capture device;

(b) processing the image data received in said step (a) to remove a background and generate body part proposals for images in a foreground;

(c) generating a plurality of skeletal hypotheses using at least one of the body part proposals generated in said step (b) and image data from an earlier time, the plurality of skeletal hypotheses defining a probability distribution;

(d) selecting one or more skeletal hypotheses as being the most probable state estimates based on the probability distribution, where the probability distribution indicates one or more skeletal hypotheses as the probable state estimate; and

(e) indicating that no state estimate is determined for the frame of image data where the probability distribution does not indicate one or more skeletal hypotheses as being probable state estimates.

17. A computer-readable storage medium as recited in claim 16, said step (d) of selecting one or more skeletal hypotheses as being the most probable state estimates comprising the steps of scoring the skeletal models based on an evaluation of each skeletal model relative to at least one of the image data for the current frame and a state estimate from a frame earlier than the current frame of image data.

18. A computer-readable storage medium as recited in claim 16, further comprising the step of re-sampling the skeletal hypotheses to arrive at a final set of skeletal hypotheses, with a skeletal hypothesis having a higher probability having a greater chance of being selected into the final set of skeletal hypotheses.

19. A computer-readable storage medium as recited in claim 16, said step (e) comprising the step of scoring the skeletal hypotheses and indicating that no state estimate is determined for the frame of image data where no skeletal hypothesis scores above a predetermined threshold value.