ESTIMATING 3D SHAPE AND TEXTURE OF A 3D OBJECT BASED ON A 2D IMAGE OF THE 3D OBJECT

• US 20070031028A1
• Filed: 06/20/2006
• Published: 02/08/2007
• Est. Priority Date: 06/20/2005
• Status: Active Grant

• Current Assignee: University of Basel
• Original Assignee: University of Basel
• Inventors: Vetter, Thomas, Romdhani, Sami, Pierrard, Jean-Sebastian
• Granted Patent: US 7,756,325 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: B33Y 50/00 Data acquisition or data processing for additive manufacturing
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole

Abstract

Disclosed is an improved algorithm for estimating the 3D shape of a 3-dimensional object, such as a human face, based on information retrieved from a single photograph by recovering parameters of a 3-dimensional model and methods and systems using the same. Beside the pixel intensity, the invention uses various image features in a multi-features fitting algorithm (MFF) that has a wider radius of convergence and a higher level of precision and provides thereby better results.

Claims

1. A method for estimation of object properties of a 3-dimensional object from a 2-dimensional image by recovering model parameters to adjust a 3-dimensional model of the 3-dimensional object from the 2-dimensional image by a Multi-Feature Fitting Algorithm comprising the following steps:

a) retrieving several image related informations from the 2-dimensional image;

b) processing retrieved informations separately by specific cost functions;

c) superimposing results of specific cost functions in the Multi-Feature Fitting Algorithm;

d) approaching a global extreme of Multi-Feature Cost Function by adjusting model parameters related to the Multi-Feature Fitting Algorithm;

e) using the model parameters in the 3-dimensional model to display a 3-dimensional estimation of a 3-dimensional shape.

2. The method according to claim 1 wherein the 3-dimensional model is a 3-dimensional Generalized Flexible 3D Object Model (GF3DOM).

3. The method according to claim 1 or 2 wherein the Multi-Feature Fitting Algorithm comprises a linear combination of cost functions.

4. The method according to claim 3 wherein the Multi-Feature Fitting Algorithm complies with the following formula:

$\underset{\mathrm{\theta <br><br>}}{\min }<br><br>{\mathrm{\tau <br><br>}}^c<br><br>C^c+{\mathrm{\tau <br><br>}}^e<br><br>C^e+{\mathrm{\tau <br><br>}}^s<br><br>C^s+{\mathrm{\tau <br><br>}}^p<br><br>C^p+{\mathrm{\tau <br><br>}}^t<br><br>C^t.$

5. The method according to claim 1 or 2 wherein information about pixel intensity is used in a pixel intensity cost function.

6. The method according to claim 5 wherein the pixel intensity cost function complies with following formula:

$C^c=\underset{i\in <br><br>\mathrm{Inlier}}{\sum <br><br>}<br><br>{\left(I<br><br>\left(x,y\right)<br><br>o<br><br>\text{\hspace{1em}<br><br>}<br><br>p<br><br>\left(u_i,v_i;<br><br>\mathrm{\theta <br><br>}\right)-t^c<br><br>\left(u_i,v_i;<br><br>\mathrm{\theta <br><br>}\right)\right)}^2$

7. The method according to claim 1 or 2 wherein a generation of an outlier mask comprises the following steps:

a) performing a coarse fitting without outlier mask, but with a large weight on a prior probability by at least one iteration of the fitting algorithm to provide a rough estimation of pose and shape of the 3D object;

b) selecting of at least one image patches in an input image area, whereby the image patches are located at constant positions in a/the GF3DOM, and using correspondences determined by the coarse fitting result to determine position of these patches in the input image;

c) executing a Grab Cut Algorithm to segment relevant and non-relevant regions, thereby producing the outlier mask.

8. The method according to claim 1 or 2 wherein information about edge features is used in a edge feature cost function.

9. The method according to claim 8 wherein the edge feature cost function complies with following formula:

$C^e=\frac{1}{2}<br><br>\underset{i\in <br><br>\mathrm{Edge}}{\sum <br><br>}<br><br>{\left(q_{k<br><br>\left(i\right)}^e-p_i<br><br>\left(\mathrm{\theta <br><br>}\right)\right)}^2$

10. The method according to claim 1.) or 2.) wherein information about the texture constraint is used in a texture constraint cost function.

11. The method according to claim 10 wherein the texture constraint cost function complies with following formula:

$C^t=\frac{1}{2}<br><br>{\mathrm{e<br><br>}}^{\mathrm{tT}}·<br><br>{\mathrm{e<br><br>}}^t.$

12. The method according to claim 1 or 2 wherein information about specular highlight is used in a specular highlight cost function.

13. The method according to claim 12 wherein the specular highlight cost function complies with following formula:

C^S=e^sT.e^s.

14. A method in a data processing system for the estimation of object properties of a 3-dimensional object, wherein said data processing system includes a CCD-array, a display and an operating system, said method comprising the steps according to one of the claims 1 or 2.

15. A computer program element comprising a computer program code means to make the computer execute the method according to one of the claims 1 or 2.

16. A data processing system for estimation of object properties of a 3-dimensional object, wherein said data processing system includes a display and an operating system, said data processing system comprising means to execute the method according to one of the claims 1 or 2.