TRACKING HAND/BODY POSE

• US 20170116471A1
• Filed: 01/04/2017
• Published: 04/27/2017
• Est. Priority Date: 09/23/2014
• Status: Active Grant

• Current Assignee: Microsoft Corporation
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shotton, Jamie Daniel Joseph, Sharp, Toby, Fitzgibbon, Andrew William, Kohli, Pushmeet, Robertson, Duncan Paul, Keskin, Cem, Taylor, Jonathan, Izadi, Shahram
• Granted Patent: US 9,911,032 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 30/19167 Active pattern learning
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Tracking hand or body pose from image data is described, for example, to control a game system, natural user interface or for augmented reality. In various examples a prediction engine takes a single frame of image data and predicts a distribution over a pose of a hand or body depicted in the image data. In examples, a stochastic optimizer has a pool of candidate poses of the hand or body which it iteratively refines, and samples from the predicted distribution are used to replace some candidate poses in the pool. In some examples a best candidate pose from the pool is selected as the current tracked pose and the selection processes uses a 3D model of the hand or body.

Claims

1. A method of tracking pose of a hand or body comprising:

receiving a stream of images depicting the hand or body;

inputting at least part of one of the images to a plurality of layers of a prediction engine to obtain a distribution over poses of the hand or body;

accessing a pool of candidate poses of the hand or body; and

periodically replacing a sub-set of the candidate poses in the pool with samples from the distribution over poses of the hand or body.

2. A method as recited in claim 1, further comprising initializing the pool of candidate poses with samples from the distribution over poses of the hand or body.

3. A method as recited in claim 2, further comprising computing the samples by using a pre-specified distribution over pose for each of a plurality of pose classes.

4. A method as recited in claim 1, wherein the plurality of layers of the prediction engine include a first layer comprising a classifier arranged to calculate a distribution over global rotation classes of the pose.

5. A method as recited in claim 1, wherein the plurality of layers of the prediction engine include a first layer comprising a discriminative fern ensemble.

6. A method as claimed in claim 1, wherein the plurality of layers of the prediction engine include:

a first layer comprising a classifier; and

a second layer comprising a plurality of decision jungles.

7. The method as claimed in claim 6 wherein the plurality of decision jungles comprises rotation regressors, translation regressors and pose classifiers.

8. A method as recited in claim 1, further comprising using a stochastic optimizer to iteratively refine the candidate poses.

9. A method as recited in claim 8, wherein using the stochastic optimizer to iteratively refine the candidate poses comprises randomly adjusting the pose of a sub-set of the candidate poses by making local random perturbations.

10. A method as recited in claim 8, wherein using the stochastic optimizer to iteratively refine the candidate poses comprises randomly adjusting the pose of a sub-set of the candidate poses by replacing a subset of parameters of a pose candidate with parameters from another pose candidate.

11. A method as recited in claim 8, wherein using the stochastic optimizer to iteratively refine the candidate poses comprises:

assigning an age to each candidate pose;

incrementing the age if the candidate pose is randomly adjusted; and

treating candidate poses with different ages as being in different candidate pose pools.

12. A method as recited in claim 8, wherein using the stochastic optimizer to iteratively refine the candidate poses comprises computing scores of candidate poses of the pool by rendering synthetic images from a 3D hand or body model in a manner which takes into account occlusions.

13. A method as recited in claim 1 wherein the stream of images depicting the hand or body is received from a moving capture device.

14. A method as recited in claim 1 wherein the stream of images depicting the hand or body is received from a capture device located over the shoulder of a user.

15. A system comprising:

a processor;

a memory;

an input interface arranged to receive an image depicting the hand or body;

a prediction engine stored in the memory and executed on the processor, the prediction engine configured to;

receive at least part of the image; and

determine a distribution over poses of the hand or body;

a stochastic optimizer stored in the memory and executed on the processor, the stochastic optimizer configured to iteratively refine candidate poses in a pool of candidate poses of the hand or body; and

a hand or body pose tracker stored in the memory and executed on the processor, the hand or body pose tracker configured to periodically replace a sub-set of the candidate poses in the pool of candidate poses with samples from the distribution over poses of the hand or body.

16. A system as recited in claim 15, further comprising a renderer implemented at least in part on a parallel processing unit, wherein the stochastic optimizer is further configured to use the renderer as part of a scoring process.

17. A system as recited in claim 15, wherein the prediction engine has a layered architecture.

18. A system as recited in claim 17, wherein the prediction engine includes a first layer comprising a global rotation classifier.

19. A system as recited in claim 15, wherein the prediction engine includes a plurality of trained experts being decision jungles.

20. One or more computer storage media storing instructions that, when executed by a computing device, control the computing device to:

receive a stream of images depicting a hand of a user;

input at least part of one of the images to a plurality of layers of a prediction engine to obtain a distribution over poses of the hand;

access a pool of candidate poses of the hand or body; and

periodically replace a sub-set of the candidate poses in the pool with samples from the distribution over poses of the hand or body.