Computing pose and/or shape of modifiable entities

• US 8,724,906 B2
• Filed: 11/18/2011
• Issued: 05/13/2014
• Est. Priority Date: 11/18/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shotton, Jamie Daniel Joseph, Fitzgibbon, Andrew William, Taylor, Jonathan James, Cook, Matthew Darius
• Primary Examiner(s): Mariam, Daniel
• Application Number:
  US13/300,542
• Publication Number:
  US 20130129230A1
• Time in Patent Office: 907 Days
• Field of Search: 382/103, 382/154, 382/209, 382/215, 382/216, 382/218
• US Class Current: 382/218
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

Computing pose and/or shape of a modifiable entity is described. In various embodiments a model of an entity (such as a human hand, a golf player holding a golf club, an animal, a body organ) is fitted to an image depicting an example of the entity in a particular pose and shape. In examples, an optimization process finds values of pose and/or shape parameters that when applied to the model explain the image data well. In examples the optimization process is influenced by correspondences between image elements and model points obtained from a regression engine where the regression engine may be a random decision forest. For example, the random decision forest may take elements of the image and calculate candidate correspondences between those image elements and model points. In examples the model, pose and correspondences may be used for control of various applications including computer games, medical systems, augmented reality.

Claims

1. A computer-implemented method of calculating pose or shape of an articulated or deformable entity comprising:

receiving at least one image of the entity;

accessing a model of a class of articulated or deformable entities of which the imaged entity is a member, the model comprising a plurality of parameters specifying the pose or shape of the model;

using the image to access a plurality of candidate correspondences between image elements of the received image and model points which are locations on or in the model;

carrying out an optimization process to find values of the parameters specifying the pose or shape of the model which agrees with the received image; and

where the optimization is influenced by at least some of the candidate correspondences; and

at least one of

accessing the candidate correspondences from a random decision forest arranged to take image elements of the received image and, for a plurality of image elements of the received image, calculate a probability distribution over candidate correspondences using information associated with its leaves, or

carrying out the optimization process by optimizing an energy function, the energy function being over agreement between the model and the received image, where the energy function omits model points which are not visible from a view point of an image capture device which captured the received image by taking into account the direction of surface normals of the model.

2. A method as claimed in claim 1 comprising carrying out the optimization process by optimizing an energy function, the energy function being over agreement between the model and the received image, where the energy function comprises a term arranged to favor correspondences between image elements and model points which are similar where similarity is assessed using the candidate correspondences.

3. A method as claimed in claim 1 comprising accessing the candidate correspondences from a random decision forest arranged to take image elements of the received image and, for each image element, calculate a probability distribution over candidate correspondences using information associated with its leaves.

4. A method as claimed in claim 1 comprising carrying out the optimization process by iteratively fixing and optimizing different terms of an energy function;

the energy function being over agreement between the model and the received image.

5. A method as claimed in claim 4 comprising iteratively fixing and optimizing the correspondences and the parameters specifying the pose or shape of the model and where initial correspondences are selected from the candidate correspondences.

6. A method as claimed in claim 1 comprising receiving a sequence of images of the entity over time;

using the sequence to find at least one temporally static model parameter; and

fixing that parameter in the optimization process.

7. A method as claimed in claim 1 comprising carrying out the optimization process by optimizing an energy function over agreement between the model and the received image, where the energy function comprises a term arranged to encourage nearby image points to map to nearby model points.

8. A method as claimed in claim 1 comprising carrying out the optimization process by optimizing an energy function, the energy function being over agreement between the model and the received image, where the energy function comprises a term arranged to omit model points which are not visible from a view point of an image capture device which captured the received image by taking into account the direction of surface normals of the model.

9. A method as claimed in claim 1 comprising receiving a plurality of images of the entity taken from different view points at the same time and using the plurality of images to access the candidate correspondences.

10. A method as claimed in claim 1 comprising carrying out the optimization process by optimizing an energy function, the energy function being over agreement between the model and the received image, where the energy function comprises a term arranged to penalize values of the pose parameters which are known to be unlikely, or where the method comprises receiving a sequence of images of the entity over time, which do not follow a specified motion model.

11. A method as claimed in claim 1 comprising carrying out the optimization by accessing, for each image element of the received image, a plurality of candidate correspondences, and searching for model points only amongst the candidate correspondences.

12. A method as claimed in claim 1 comprising carrying out the optimization by accessing, for each image element of the received image, at least one candidate correspondence and using that candidate correspondence to guide search of the model points.

13. A computer-implemented method of calculating pose or shape of an articulated or deformable entity comprising:

receiving at least one depth image of the entity;

accessing a model of a class of articulated or deformable entities of which the imaged entity is a member, the model comprising a plurality of parameters specifying the pose of the model;

using the image to access, from a random decision forest, a plurality of candidate correspondences between image elements of the received image and model points which are locations on or in the model;

the random decision forest being arranged to take image elements of the received image and, for each image element, calculate a distribution over candidate correspondences using information associated with its leaves; and

carrying out an optimization process to find values of the parameters specifying the pose or shape of the model which agrees with the received image and also to find optimal correspondences between the image elements and model points; and

where the optimization is influenced by at least some of the candidate correspondences;

the optimization process comprising summing over image elements of the depth image a measure based on a distance between an image element and its corresponding model point.

14. A method as claimed in claim 13 wherein the depth image is a silhouette image comprising a binary image identifying foreground and background regions of a depth image where the depth is flattened to a fixed depth.

15. A method as claimed in claim 14 comprising iteratively fixing and optimizing the correspondences and the parameters specifying the pose or shape of the model and where initial correspondences are selected from the candidate correspondences.

16. A method as claimed in claim 13 comprising carrying out the optimization process by iteratively fixing and optimizing different terms of an energy function;

the energy function being over agreement between the model and the received image.

17. A method as claimed in claim 13 comprising carrying out the optimization process by optimizing an energy function, the energy function being over agreement between the model and the received image, where the energy function comprises a term arranged to omit model points which are not visible from a view point of an image capture device which captured the received image by taking into account the direction of surface normals of the model.

18. An apparatus for calculating pose or shape of an articulated or deformable entity comprising:

an input arranged to receive at least one image of the entity;

a memory storing a model of a class of articulated or deformable entities of which the imaged entity is a member, the model comprising a plurality of parameters specifying the pose or shape of the model;

a regression engine arranged to calculate a plurality of candidate correspondences between image elements of the received image and model points which are locations on or in the model; and

an optimizer arranged to find values of the parameters specifying the pose or shape of the model which agrees with the received image;

the optimizer being influenced by at least some of the candidate correspondences,

wherein

the regression engine comprises a random decision forest arranged to take image elements of the received image and, for a plurality of image elements of the received image, calculate a probability distribution over candidate correspondences using information associated with its leaves, or

the optimizer being further arranged to optimize an energy function, the energy function being over agreement between the model and the received image, where the energy function omits model points which are not visible from a view point of an image capture device which captured the received image based at least partly on the direction of surface normals of the model.

19. An apparatus as claimed in claim 18 the optimizer being arranged to optimize an energy function, the energy function being over agreement between the model and the received image, where the energy function comprises a term arranged to favor correspondences between image elements and model points which are similar where similarity is assessed using the regression engine.

20. An apparatus as claimed in claim 18 wherein the regression engine comprises a random decision forest arranged to take image elements of the received image and, for each image element, calculate a probability distribution over candidate correspondences using information associated with its leaves.