Predicting Joint Positions

• US 20120239174A1
• Filed: 03/17/2011
• Published: 09/20/2012
• Est. Priority Date: 03/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shotton, Jamie Daniel Joseph, Kohli, Pushmeet, Girshick, Ross Brook, Fitzgibbon, Andrew, Criminisi, Antonio
• Granted Patent: US 8,571,263 B2
• Time in Patent Office: Days
• US Class Current: 700/93
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Predicting joint positions is described, for example, to find joint positions of humans or animals (or parts thereof) in an image to control a computer game or for other applications. In an embodiment image elements of a depth image make joint position votes so that for example, an image element depicting part of a torso may vote for a position of a neck joint, a left knee joint and a right knee joint. A random decision forest may be trained to enable image elements to vote for the positions of one or more joints and the training process may use training images of bodies with specified joint positions. In an example a joint position vote is expressed as a vector representing a distance and a direction of a joint position from an image element making the vote. The random decision forest may be trained using a mixture of objectives.

Claims

1. A computer-implemented method of predicting joint positions comprising:

receiving an input image of a scene comprising at least part of a human or animal body;

for each of a plurality of image elements of the input image, making a plurality of votes, each vote being for a position in the input image corresponding to a joint of the human or animal body;

the votes being made by comparing each image element with test image elements displaced therefrom by learnt spatial offsets; and

aggregating the votes to obtain at least one predicted joint position.

2. A method as claimed in claim 1 comprising assigning a confidence to each predicted joint position.

3. A method as claimed in claim 1 comprising aggregating the votes by taking into account weights that have been learnt during a training process the weights expressing information about uncertainty of the votes.

4. A method as claimed in claim 3 comprising for each vote, adapting the weight according to a depth of an image element which made the vote.

5. A method as claimed in claim 1 comprising expressing each vote using a vector related to the direction and distance from an image element of the input image making the vote to a position in the input image where the joint is voted to be.

6. A method as claimed in claim 1 wherein making the plurality of votes comprises applying each image element of the input image to a random decision forest which has been trained using one or more decision tree node splitting objectives and a set of training images having labeled joint positions or labeled body parts.

7. A method as claimed in claim 6 which comprises using at least two decision tree node splitting objectives which are different from one another.

8. A method as claimed in claim 6 wherein the one or more decision tree node splitting objectives are selected from any of:

optimizing information gain of a body part classification task, optimizing reduction in variance of voted joint positions, optimizing reduction in variance of distance to voted joint positions, optimizing reduction in variance of angle to voted joint positions, optimizing a sum of squared distances from a mean voted joint position, optimizing a balance of a random regression tree.

9. A method as claimed in claim 1 wherein aggregating the votes comprises any of:

forming a discrete voting space of votes, using a Parzen window density estimator, using a Parzen window density estimator with a mean shift mode detection process, using expectation maximization, using k-means clustering, using agglomerative clustering, calculating a mean vote.

10. A method of training a random decision forest to produce votes for positions of joints of a human or animal body in an image comprising:

receiving a plurality of training images having labeled joint positions;

receiving at least one decision tree node splitting objective;

selecting parameters for use at nodes of trees in the random decision forest by using the training images and the at least one objective;

at each leaf node of each tree in the random decision forest obtaining a plurality of votes by applying the training images to the random decision forest with the selected parameters;

each vote being for a relative position in a training image predicted to correspond to a joint of the human or animal body;

aggregating the votes at each leaf node by any of;

listing the votes, forming a histogram of votes, calculating a mean of the votes, and fitting a multi-modal distribution to the votes by any of expectation maximization, mean shift mode detection, k-means clustering and agglomerative clustering.

11. A method as claimed in claim 10 comprising filtering the votes using a threshold prior to aggregating the votes.

12. A method as claimed in claim 11 wherein the threshold is learnt on a per joint basis using a validation set of images.

13. A method as claimed in claim 11 wherein the threshold is a distance of a joint position from an image element voting for that joint position.

14. A method as claimed in claim 10 comprising aggregating the votes by using a probability distribution density estimator to find one or more modes and determining a weight for each mode the weight being related to a number of votes that reached that mode.

15. A method as claimed in claim 14 comprising using a mean shift mode detection process to assess the number of votes that reached each mode.

16. A method as claimed in claim 14 comprising setting a parameter of the density estimator on a per joint basis.

17. A computer-implemented joint position prediction system comprising:

an input arranged to receive an input image of a scene comprising at least part of a human or animal body;

a processor arranged, for each of a plurality of image elements of the input image, to make a plurality of votes, each vote being for a position in the input image corresponding to a joint of the human or animal body;

the processor being arranged to aggregate the votes to obtain at least one predicted joint position;

the processor being arranged to store each vote using a vector related to the direction and distance from an image element of the input image making the vote to a position in the input image where the joint is voted to be.

18. A system as claimed in claim 17 wherein the processor is arranged to aggregate the votes by taking into account weights expressing an uncertainty associated with each vote.

19. A system as claimed in claim 18 wherein the processor is arranged to adapt the weights according to depth values associated with image elements.

20. A computer game system comprising a joint position prediction system as claimed in claim 17.