Method for estimating a pose of an articulated object model

• US 8,830,236 B2
• Filed: 04/28/2011
• Issued: 09/09/2014
• Est. Priority Date: 04/30/2010
• Status: Active Grant

• Current Assignee: Vizrt AG (Vizrt Group Holding AS)
• Original Assignee: LiberoVision AG (Vizrt Group Holding AS)
• Inventors: Germann, Marcel, Wuermlin Stadler, Stephan, Keiser, Richard, Ziegler, Remo, Niederberger, Christoph, Hornung, Alexander, Gross, Marcus
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US13/096,488
• Publication Number:
  US 20110267344A1
• Time in Patent Office: 1,230 Days
• Field of Search: None
• US Class Current: 345/420
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A computer-implemented method for estimating a pose of an articulated object model that is a computer based 3D model of a real world object observed by one or more source cameras, including the steps of obtaining a source image from a video stream; processing the source image to extract a source image segment maintaining, in a database, a set of reference silhouettes, each being associated with an articulated object model and a corresponding reference pose; comparing the source image segment to the reference silhouettes and selecting reference silhouettes by taking into account, for each reference silhouette, a matching error that indicates how closely the reference silhouette matches the source image segment retrieving the corresponding reference poses of the articulated object models; and computing an estimate of the pose of the articulated object model from the reference poses of the selected reference silhouettes.

Claims

1. A computer-implemented method for estimating a pose of an articulated object model, wherein the articulated object model is a computer based 3D model that is executed on a digital computer or a computer system comprising a computer memory and a processing unit coupled to the computer memory, wherein the computer based 3D model is of a real world object observed by one or more source cameras, and the articulated object model represents a plurality of joints and of links that link the joints, and wherein the pose of the articulated object model is defined by the spatial location of the joints, the method comprising the steps of:

obtaining at least one source image from a video stream comprising a view of the real world object recorded by a source camera;

processing by the processing unit the at least one source image to extract a corresponding source image segment comprising the view of the real world object separated from the image background;

maintaining, in a database in computer readable form, a set of reference silhouettes, each reference silhouette being associated with an articulated object model and with a particular reference pose of this articulated object model;

comparing by the processing unit the at least one source image segment to the reference silhouettes and selecting a predetermined number of reference silhouettes by taking into account, for each reference silhouette,

wherein the processing unit determines a matching error that indicates how closely the reference silhouette matches the source image segment and/or

wherein the processing unit determines a coherence error that indicates how much the reference pose is consistent with the pose of the same real world object as estimated from at least one of preceding and following source images of the video stream;

retrieving by the processing unit the reference poses of the articulated object models associated with the selected of reference silhouettes; and

computing by the processing unit an estimate of the pose of the articulated object model from the reference poses of the selected reference silhouettes.

2. The computer-implemented method of claim 1, wherein the step of comparing the at least one source image segment to the reference silhouettes comprises the steps of, for each reference silhouette that the source image segment is compared to:

determining a projective transform that maps the source image segment onto the reference silhouette, in particular by scaling the source image segment to be of the same size as the reference silhouette; and

computing the matching error as either being proportional to the relative size of the image area in which the mapped source image segment and the reference silhouette do not overlap, or as being a measure of a distance between the outlines of the scaled source image segment and of the reference silhouette, with the matching error optionally also being dependent on parameters of the projective transform;

and using this matching error as a measure for how closely the source image segment and the reference silhouette match.

3. The computer-implemented method of claim 2, wherein

the step of scaling is accomplished by re-sampling either the source image segment or the reference silhouettes or both to have bounding boxes of the same pixel size, and

both the source image segment and the reference silhouette are represented by binary images having the same pixel dimension, and computing the error value by counting the number of corresponding pixels from the source image segment and the reference silhouette that differ in value.

4. The computer-implemented method according to claim 1, wherein the step of comparing the at least one source image segment to the reference silhouettes comprises the steps of, for each reference silhouette that the source image segment is compared to:

retrieving the pose of the same real world object estimated from a preceding source image of the video stream;

computing the coherence error as being proportional to the difference between this preceding pose estimate and the reference pose of the reference silhouette, and using this coherence error as a measure of consistency with the preceding source image.

5. The computer-implemented method according to claim 1, wherein the step of computing an estimate of the pose of the articulated object model from the reference poses of the selected reference silhouettes further comprises:

repeating the preceding steps for one or more further source images from one or more further video streams from further source cameras, each further source image comprising a view of the same real world object having been recorded at the same time but from a different viewpoint, thereby obtaining for each further source image a predetermined number of selected reference silhouettes and associated selected reference poses;

performing an optimization to select for each source image a most plausible reference pose, by computing for each combination of selected reference poses for the different source images a total joint consistency measure by,

projecting the joints of the selected reference poses of this combination into 3D space, estimating a 3D position of the joints and computing, for each joint, a joint consistency measure that expresses how well the estimated 3D joint position matches the projection of the joint from the selected reference poses;

combining the joint consistency measures of all joints to obtain the total joint consistency measure;

selecting the combination of reference poses for the different source images that optimizes the total joint consistency measure.

6. The computer-implemented method of claim 5, wherein the step of performing an optimization further comprises the step of varying and optimizing a 2D offset of each silhouette in the plane of its associated source image in order to correct for source camera calibration errors.

7. The computer-implemented method of claim 1, comprising the further step of displaying, on a display device, at least one source image with estimated joint positions superimposed over the source image and accepting a user input for interactively modifying one or more joint positions.

8. A computer-implemented method according to claim 1, wherein, in order to determine a 3D pose matching a given 2D pose associated with a source image, the following steps are performed:

computing, from the given 2D pose an approximate 3D pose comprising approximate joint positions whose projection approximately matches the positions of the corresponding joints of the 2D pose when projected into the image plane of the source image associated with the 2D pose;

modifying the approximate 3D pose to exactly match the 2D pose by, for each joint, moving the position of the joint from the approximate 3D joint position to a position defined by the intersection of a ray passing from the camera through the joint position in the source image with a plane parallel to the image plane of the source image and passing through the approximate 3D joint position.

9. A computer-implemented method for estimating a pose of an articulated object model, wherein the articulated object model is a computer based 3D model that is executed on a digital computer or a computer system comprising a computer memory and a processing unit coupled to the computer memory, wherein the computer based 3D model is of a real world object observed by two or more source cameras, and the articulated object model represents a plurality of joints and of links that link the joints, and wherein the pose of the articulated object model is defined by the spatial location of the joints, called 3D joint positions, the method comprising the steps of

determining by the processing unit an initial estimate of the 3D pose, that is, the 3D joint positions of the articulated object model;

associating by the processing unit each link with one or more projection surfaces, wherein the projection surfaces are surfaces defined in the 3D model, and the position and orientation of each projection surface is determined by the position and orientation of the associated link;

iteratively adapting the 3D joint positions by the processing unit, for each joint,

wherein the processing unit determines a position score assigned to its 3D joint position, the position score being a measure of the degree to which image segments from the different source cameras, when projected onto the projection surfaces of links adjacent to the joint, are consistent which each other;

wherein the processing unit varies the 3D joint position of the joint until an optimal position score is achieved;

repeating the step of iteratively adapting the 3D joint positions for all joints for a predetermined number of times or until the position scores converge.

10. The computer-implemented method of claim 9, wherein the step of varying the 3D joint position of the joints varies the 3D joint positions subject to anthropometric constraints, the anthropometric constraints being at least one of:

the joint is on or above the ground;

lengths of topologically symmetric links do not differ more than 10%;

the lengths of links are within anthropometric standards;

distances between joints not connected by a link are within anthropometric standards.

11. The computer-implemented method of claim 9, wherein the projection surfaces, for each link, comprise a fan of billboards, each billboard being associated with a source camera, and each billboard being a planar surface spanned by its associated link and a vector that is normal to both this link and to a line connecting a point of the link to the source camera.

12. Currently Amended) The computer-implemented method of claim 9, wherein the position score of a 3D joint position of a joint is computed by the steps of, for each link adjacent to the joint,

projecting the images from the different source cameras onto the associated projection surfaces of the link and from there into a virtual image as seen by a virtual camera;

for an area that correspond to the projection of these projection surfaces into the virtual image, computing a partial position score for this link according to the degree to which the image segments from the different source cameras overlap and have a similar colour;

combining the partial position scores to obtain the position score.

13. The computer-implemented method of claim 12, wherein computing and combining the partial position score comprises the steps of:

computing the partial position score for each pair of source cameras contributing to the virtual image;

combining these partial position scores by adding them, weighing each partial position score according to the angle between the viewing directions of the associated pair of source cameras.

14. A computer-implemented method for rendering a virtual image as seen from a virtual camera, given an articulated object model, wherein the articulated object model is a computer based 3D model that is executed on a digital computer or a computer system comprising a computer memory and a processing unit coupled to the computer memory, wherein the computer based 3D model is of a real world object observed by two or more source cameras, and the articulated object model represents a plurality of joints and of links that link the joints, and wherein the pose of the articulated object model is defined by the spatial location of the joints, the method comprising the steps of:

determining by the processing unit an estimate of the 3D pose, that is, the 3D joint positions of the articulated object model;

associating by the processing unit each link with one or more projection surfaces, wherein the projection surfaces are surfaces defined in the 3D model, and the position and orientation of each projection surface is determined by the position and orientation of the associated link;

wherein the projection surfaces, for each link, comprise a fan of billboards, each billboard being associated with a source camera, and each billboard being a planar surface spanned by its associated link and a vector that is normal to both this link and to a line connecting a point of the link to the source camera;

for each source camera, projecting segments of the associated source image onto the associated billboard, creating billboard images;

for each link, projecting the billboard images into the virtual image and blending the billboard images to form a corresponding part of the virtual image.

15. A computer-implemented method for determining a segmentation of a source image segment, the method comprising the steps of:

obtaining at least one source image from a video stream comprising a view of a real world object recorded by a source camera;

processing by a processing unit the at least one source image to extract a corresponding source image segment comprising the view of the real world object separated from the image background;

maintaining, in a database in computer readable form, a set of reference silhouettes, each reference silhouette being associated with a reference segmentation, the reference segmentation defining sub-segments of the reference silhouette, each sub-segment being assigned a unique label;

determining by the processing unit a matching reference silhouette which most closely resembles the source image segment and retrieving the reference segmentation of the reference silhouette;

for each sub-segment, overlaying both a thickened and a thinned version of the sub-segment over the source image segment and labelling the source image pixels which lie within both the thickened and the thinned version with the label of the sub-segment;

labelling all remaining pixels of the source image segment as unconfident pixels;

for each sub-segment, determining by the processing unit a colour model that is representative of the colour of the pixels labelled with the sub-segment'"'"'s label;

labelling the unconfident pixels according to the colour model, by assigning each of the unconfident pixel to a sub-segment whose colour model most closely fits the colour of each of the unconfident pixels.