Systems and methods for human body pose estimation

• US 7,925,081 B2
• Filed: 12/12/2007
• Issued: 04/12/2011
• Est. Priority Date: 12/12/2007
• Status: Expired due to Fees

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Fujifilm Holdings Corporation)
• Inventors: Gupta, Abhinav, Chen, Francine R., Chen, Trista Pei-chun, Kimber, Donald G.
• Primary Examiner(s): RAHMJOO, MANUCHEHR
• Application Number:
  US11/955,270
• Publication Number:
  US 20090154796A1
• Time in Patent Office: 1,217 Days
• Field of Search: 382/159, 382/106, 382/115, 382/116, 382/117, 382/118, 382/103, 382/285, 382/154, 382/168, 382/181, 382/184, 382/190, 700/29, 345/473, 345/474
• US Class Current: 382/159
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition

Abstract

Systems and computer-implemented methods for use in body pose estimation are provided. Training data is obtained, where the training data includes observation vector data and corresponding pose vector data for a plurality of images. The observation vector data is representative of the images in observation space. The pose vector data is representative of the same images in pose space. Based on the training data, a model is computed that includes parameters of mapping from the observation space to latent space, parameters of mapping from the latent space to the pose space, and parameters of the latent space. The latent space has a lower dimensionality than the observation space and the pose space.

Claims

1. A computer-implemented method for use in body pose estimation, comprising:

(a) obtaining training data that includes observation vector data and corresponding pose vector data for a plurality of images,

wherein the observation vector data is representative of the images in observation space, and the pose vector data is representative of the same images in pose space; and

(b) computing, based on the training data, a model that includes parameters of mapping from the observation space to latent space, parameters of mapping from the latent space to the pose space, and parameters of the latent space;

wherein the latent space has a lower dimensionality than the observation space, and

wherein the latent space has a lower dimensionality than the pose space.

2. The method of claim 1, wherein the parameters of the latent space, that are computed at step (b), include points in latent space that are embedded in the model.

3. The method of claim 1, wherein the parameters of the latent space, that are computed at step (b), correspond to the training data.

4. The method of claim 1, wherein step (b) includes:

(b.1) performing principal component analysis, based on the pose vector data of the training data, to obtain an initial projection of points in latent space; and

(b.2) performing optimization of the parameters of latent space, the parameters of mapping from observation space to latent space, and the parameters of mapping from latent space to pose space.

5. The method of claim 4, wherein step (b) also includes:

(b.3) repeating step (b.2) until a specified criterion is satisfied.

6. The method of claim 1, further comprising:

(c) obtaining further observation vector data corresponding to a further image for which corresponding pose vector data is unknown; and

(d) determining a point in pose space based on the further observation vector data by using the model to perform mapping of the further observation data from observation space to latent space, and from latent space to pose space.

7. The method of claim 6, further comprising:

(e) estimating a pose, that corresponds to the further observation vector data, based on the point in pose space determined at step (d).

8. The method of claim 6, wherein step (d) comprises:

(d.1) determining a point in latent space based on the further observation vector data by using the model to perform mapping from observation space to latent space; and

(d.2) determining the point in pose space by using the model to perform mapping from the point in latent space to pose space.

9. The method of claim 8, wherein:

step (d.1) includes

(d.1.i) computing a conditional probability distribution in latent space based on the further observation vector data and the model; and

(d.1.ii) determining a maximum a posterior estimation of the conditional probability distribution computed at (d.1.i); and

step (d.2) includes determining the point in pose space by using the model to perform mapping from the maximum a posterior estimation, determined at (d.1.ii), to pose space.

10. The method of claim 8, wherein:

step (d.1) includes

(d.1.i) computing a conditional probability distribution in latent space based on the further observation vector data and the model;

(d.1.ii) obtaining a plurality of samples from the computed conditional probability distribution; and

(d.1.iii) determining which of the plurality of samples has a greatest posterior probability; and

step (d.2) includes determining the point in pose space by using the model to perform mapping from the sample, determined at (d.1.iii) to have the greatest posterior probability, to pose space.

11. The method of claim 1, wherein:

the plurality of images includes a sequence of time ordered frames of video, each of which is associated with a point in time; and

at step (b) the computing of the model takes into account

an image associated with a point in time, and

one or more point in latent space corresponding one or more image associated with one or more previous point in time.

12. The method of claim 11, wherein at step (b) the computing of the model takes into account

an image associated with a point in time, and

a point in latent space corresponding an image that is associated with an immediately preceding point in time.

13. The method of claim 11, further comprising:

(c) obtaining further observation vector data corresponding to a further image for which corresponding pose vector data is unknown; and

(d) determining a point in pose space, based on the further observation vector data and one or more point in latent space corresponding to one or more image associated with one or more point in time prior to the further image, by using the model to perform mapping of the further observation data from observation space to latent space, and from latent space to pose space.

14. The method of claim 13, wherein step (d) comprises:

(d.1) determining a point in latent space based on the further observation vector data and the one or more point in latent space corresponding to one or more image associated with one or more point in time prior to the further image, by using the model to perform mapping from observation space to latent space; and

(d.2) determining the point in pose space by using the model to perform mapping from latent space to pose space.

15. The method of claim 14, wherein:

step (d.1) includes

(d.1.i) computing a conditional probability distribution in latent space based on the further observation vector data, the one or more point in latent space corresponding to the one or more image associated with one or more point in time prior to the further image, and the model; and

(d.1.ii) determining a maximum a posterior estimation of the conditional probability distribution computed at (d.1.i); and

step (d.2) includes determining the point in pose space by using the model to perform mapping from the maximum a posterior estimation, determined at (d.1.ii), to pose space.

16. The method of claim 14, wherein:

step (d.1) includes

(d.1.i) computing a conditional probability distribution in latent space based on the further observation vector data, the one or more point in latent space corresponding to the one or more image associated with one or more point in time prior to the further image, and the model;

(d.1.ii) obtaining a plurality of samples of the computed conditional probability distribution; and

(d.1.iii) determining which of the plurality of samples has a greatest posterior probability; and

step (d.2) includes determining the point in pose space by using the model to perform mapping from the sample, determined at (d.1.iii) to have the greatest posterior probability, to pose space.

17. A system for use in body pose estimation, comprising:

an input module configured to obtain training data that includes observation vector data and corresponding pose vector data for a plurality of images,

wherein the observation vector data is representative of the images in observation space, and the pose vector data is representative of the same images in pose space; and

one or more processor configured to compute, based on the training data, a model that includes parameters of mapping from the observation space to latent space, parameters of mapping from the latent space to the pose space, and parameters of the latent space;

wherein the latent space has a lower dimensionality than the observation space, and

wherein the latent space has a lower dimensionality than the pose space.

18. The system of claim 17, wherein:

the input module is also configured to obtain further observation vector data corresponding to a further image for which corresponding pose vector data is unknown; and

the one or more processor is also configured to

determine a point in pose space based on the further observation vector data by using the model to perform mapping of the further observation data from observation space to latent space, and from latent space to pose space, and

estimate a pose, that corresponds to the further observation vector data, based on the determined point in pose space.

19. The system of claim 17, wherein:

the plurality of images includes a sequence of time ordered frames of video, each of which is associated with a point in time; and

the one or more processor is configured to compute the model taking into account an image associated with a point in time, and one or more point in latent space corresponding one or more image associated with one or more previous point in time.

20. A computer program product, comprising a non-transitory computer-readable medium having computer program instructions embodied thereon to cause one or more processor to implement a method for use in body pose estimation, the method comprising:

obtaining training data that includes observation vector data and corresponding pose vector data for a plurality of images,

wherein the observation vector data is representative of the images in observation space, and the pose vector data is representative of the same images in pose space; and

computing, based on the training data, a model that includes parameters of mapping from the observation space to latent space, parameters of mapping from the latent space to the pose space, and parameters of the latent space;

wherein the latent space has a lower dimensionality than the observation space, and

wherein the latent space has a lower dimensionality than the pose space.

21. The computer program product of claim 20, wherein the method further comprises:

obtaining further observation vector data corresponding to a further image for which corresponding pose vector data is unknown;

determining a point in pose space based on the further observation vector data by using the model to perform mapping of the further observation data from observation space to latent space, and from latent space to pose space; and

estimating a pose, that corresponds to the further observation vector data, based on the determined point in pose space.

22. The computer program product of claim 20, wherein:

the plurality of images includes a sequence of time ordered frames of video, each of which is associated with a point in time; and

wherein the computing of the model takes into account an image associated with a point in time, and one or more point in latent space corresponding one or more image associated with one or more previous point in time.