Image processing system, 3-dimensional shape estimation system, object position/posture estimation system and image generation system

• US 8,320,660 B2
• Filed: 06/02/2006
• Issued: 11/27/2012
• Est. Priority Date: 06/03/2005
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Ishiyama, Rui
• Primary Examiner(s): Bhatnagar, Anand
• Assistant Examiner(s): Park, Soo
• Application Number:
  US11/921,315
• Publication Number:
  US 20090129665A1
• Time in Patent Office: 2,370 Days
• Field of Search: 382/154, 345/419
• US Class Current: 382/154
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole

Abstract

An object of the present invention is to process an image without a need to previously find out the initial value of a parameter representing an illumination condition and without a need for a user to manually input the illumination parameter. An image processing system includes a generalized illumination basis model generation means 300, a position/posture initial value input means 102, a face model generation means 103, an illumination basis model calculation means 301, a perspective transformation means 202, an illumination correction means 203, and a parameter update means 204. The generalized illumination basis model generation means 300 previously calculates a generalized illumination basis model. The face model generation means 103 generates an individual 3-dimensional shape and texture from the generalized 3-dimensional face model. The illumination basis model calculation means 301 generates an individual illumination basis from the generalized illumination basis model using the same parameter. The parameter update means 204 searches for a parameter of a shape, texture, and position/posture to minimize the error when the input image is reproduced.

Claims

1. An image processing system comprising:

an individual data generation unit generating individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using, a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

a generalized illumination basis model generation unit generating a generalized illumination basis model which corresponds to the generalized 3-dimensional object model and is used for generating illumination basis representing a basic variation in a reflection state caused due to a difference in the direction or intensity of illumination projected onto an individual object; and

a second illumination basis calculation unit calculating an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions using the generalized illumination basis model generated by the generalized illumination basis model generation unit.

2. The image processing system according to claim 1, further comprising an output unit outputting the individual 3-dimensional shape data generated by the individual data generation unit and individual illumination basis generated by the second illumination basis calculation unit.

3. The image processing system according to claim 1, further comprising:

a reproduced image generation unit generating a reproduced image which is an image reproducing the same illumination condition as an input image using the individual illumination basis group calculated by the second illumination basis calculation unit; and

a parameter update unit repeatedly updating the 3-dimensional shape parameter and texture parameter to be given to the generalized 3-dimensional object model such that the reproduced image generated by the reproduced image generation unit is close to the input image until a predetermined convergence condition is satisfied.

4. The image processing system according to claim 3, wherein

the second illumination basis calculation unit calculates the individual illumination basis corresponding to the individual 3-dimensional shape data that the generalized illumination basis model generation unit generates using the generalized 3-dimensional object model by using the same parameter as the 3-dimensional shape parameter to be given to the generalized 3-dimensional object model.

5. The image processing system according to claim 4, wherein

the generalized illumination basis model generation unit includes:

a parameter setting unit setting a random 3-dimensional shape parameter to be given to the generalized 3-dimensional object model;

a third illumination basis calculation unit calculating the illumination basis groups corresponding to the respective parameters based on the sample 3-dimensional shape group which is generated by giving the parameters set by the parameter setting unit to the generalized 3-dimensional object model; and

a basis calculation unit calculating the basis vector of the generalized illumination basis model using the illumination basis group calculated by the third illumination basis calculation unit and the basis vector and eigenvalue of the generalized 3-dimensional object model.

6. The image processing system according to claim 5, wherein

the third illumination basis calculation unit calculates the individual illumination basis using spherical harmonics based on the individual 3-dimensional shape data.

7. The image processing system according to claim 5, wherein

the third illumination basis calculation unit includes:

an illumination condition setting unit setting a plurality of different illumination conditions;

a luminance calculation unit calculating the luminance value of each portion of the surface of the object under various illumination conditions set by the illumination condition setting unit; and

a calculation unit calculating the individual illumination basis group based on the luminance value calculated by the luminance calculation unit by main component analysis.

8. An image processing method comprising:

an individual data generation step of generating individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

a generalized illumination basis model generation step of generating a generalized illumination basis model which corresponds to the generalized 3-dimensional object model and is used for generating illumination basis representing a basic variation in a reflection state caused due to a difference in the direction or intensity of illumination projected onto an individual object; and

a second illumination basis calculation step of calculating an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions using the generated generalized illumination basis model.

9. The image processing method according to claim 8, further comprising an output step of outputting the individual 3-dimensional shape data generated in the individual data generation step and individual illumination basis generated in the second illumination basis calculation step.

10. The image processing method according to claim 8, further comprising:

a reproduced image generation step of generating a reproduced image which is an image reproducing the same illumination condition as an input image using the individual illumination basis group calculated in the second illumination basis calculation step; and

a parameter update step of repeatedly updating the 3-dimensional shape parameter and texture parameter to be given to the generalized 3-dimensional object model such that the generated reproduced image is close to the input image until a predetermined convergence condition is satisfied.

11. The image processing method according to claim 10, wherein

the second illumination basis calculation step calculates the individual illumination basis corresponding to the individual 3-dimensional shape data that the generalized illumination basis model generation step generates using the generalized 3-dimensional object model by using the same parameter as the 3-dimensional shape parameter to be given to the generalized 3-dimensional object model.

12. The image processing method according to claim 11, further comprising:

a parameter setting step of setting a random 3-dimensional shape parameter to be given to the generalized 3-dimensional object model;

a third illumination basis calculation step of calculating the illumination basis groups corresponding to the respective parameters based on the sample 3-dimensional shape group which is generated by giving the set parameters to the generalized 3-dimensional object model; and

a basis calculation step of calculating the basis vector of the generalized illumination basis model using the calculated illumination basis group and the basis vector and eigenvalue of the generalized 3-dimensional object model.

13. The image processing method according to claim 12, wherein

the third illumination basis calculation step calculates the individual illumination basis using spherical harmonics based on the individual 3-dimensional shape data.

14. The image processing method according to claim 12, wherein

the third illumination basis calculation step sets a plurality of different illumination conditions, calculates the luminance value of each portion of the surface of the object under the set various illumination conditions, and calculates the individual illumination basis group based on the calculated luminance value by main component analysis.

15. A non-transitory computer-readable storage medium storing an image processing program, the image processing program allowing a computer to execute:

individual data generation processing that generates individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

generalized illumination basis model generation processing that generates a generalized illumination basis model which corresponds to the generalized 3-dimensional object model and is used for generating illumination basis representing a basic variation in a reflection state caused due to a difference in the direction or intensity of illumination projected onto an individual object; and

second illumination basis calculation processing that calculates an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions using the generated generalized illumination basis model.