Image processing system, 3-dimensional shape estimation system, object position/posture estimation system and image generation system

• US 20090129665A1
• Filed: 06/02/2006
• Published: 05/21/2009
• Est. Priority Date: 06/03/2005
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Ishiyama, Rui
• Granted Patent: US 8,320,660 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole

Abstract

An object of the present invention is to process an image without a need to previously find out the initial value of a parameter representing an illumination condition and without a need for a user to manually input the illumination parameter. An image processing system includes a generalized illumination basis model generation means 300, a position/posture initial value input means 102, a face model generation means 103, an illumination basis model calculation means 301, a perspective transformation means 202, an illumination correction means 203, and a parameter update means 204. The generalized illumination basis model generation means 300 previously calculates a generalized illumination basis model. The face model generation means 103 generates an individual 3-dimensional shape and texture from the generalized 3-dimensional face model. The illumination basis model calculation means 301 generates an individual illumination basis from the generalized illumination basis model using the same parameter. The parameter update means 204 searches for a parameter of a shape, texture, and position/posture to minimize the error when the input image is reproduced.

Claims

1. An image processing system comprising:

individual data generation means for generating individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

first illumination basis calculation means for calculating an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions based on the individual 3-dimensional shape data and individual texture data generated by the individual data generation means;

reproduced image generation means for generating a reproduced image which is an image reproducing the same illumination condition as an input image using the individual illumination basis group calculated by the first illumination basis calculation means; and

parameter update means for repeatedly updating the 3-dimensional shape parameter and texture parameter to be given to the generalized 3-dimensional object model such that the reproduced image generated by the reproduced image generation means is close to the input image until a predetermined convergence condition is satisfied.

2. The image processing system according to claim 1, wherein

the first illumination basis calculation means calculates the individual illumination basis using spherical harmonics based on the individual 3-dimensional shape data generated by the individual data generation means.

3. The image processing system according to claim 1, wherein

the first illumination basis calculation means includes:

illumination condition setting means for setting a plurality of different illumination conditions;

luminance calculation means for calculating the luminance value of each portion of the surface of the object under various illumination conditions set by the illumination condition setting means; and

calculation means for calculating the individual illumination basis group based on the luminance value calculated by the luminance calculation means.

11. A 3-dimensional shape estimation system that uses the image processing system as claimed in claim 1 and estimates the 3-dimensional shape of an object, comprising

shape output means for outputting, when a predetermined convergence condition is satisfied, a 3-dimensional shape generated by the generalized 3-dimensional object model based on the 3-dimensional shape parameter and texture parameter obtained at their convergence time.

12. An object position/posture estimation system that uses the image processing system as claimed in claim 1 and estimates the position and posture of an object, comprising

position/posture output means for outputting, when a predetermined convergence condition is satisfied, a position/posture parameter representing the position and posture of an object obtained at its convergence time.

13. An image generation system that uses the image processing system as claimed in claim 1 and generates a reproduced image which is an image reproducing the same illumination condition as an input image, comprising

image output means for outputting, when a predetermined convergence condition is satisfied, a reproduced image obtained at its convergence time.

4. An image processing system comprising:

individual data generation means for generating individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

generalized illumination basis model generation means for generating a generalized illumination basis model which corresponds to the generalized 3-dimensional object model and is used for generating illumination basis representing a basic variation in a reflection state caused due to a difference in the direction or intensity of illumination projected onto an individual object; and

second illumination basis calculation means for calculating an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions using the generalized illumination basis model generated by the generalized illumination basis model generation means.

5. The image processing system according to claim 4, further comprising output means for outputting the individual 3-dimensional shape data generated by the individual data generation means and individual illumination basis generated by the second illumination basis calculation means.

6. The image processing system according to claim 4, further comprising:

reproduced image generation means for generating a reproduced image which is an image reproducing the same illumination condition as an input image using the individual illumination basis group calculated by the second illumination basis calculation means; and

parameter update means for repeatedly updating the 3-dimensional shape parameter and texture parameter to be given to the generalized 3-dimensional object model such that the reproduced image generated by the reproduced image generation means is close to the input image until a predetermined convergence condition is satisfied.

7. The image processing system according to claim 6, wherein

the second illumination basis calculation means calculates the individual illumination basis corresponding to the individual 3-dimensional shape data that the generalized illumination basis model generation means generates using the generalized 3-dimensional object model by using the same parameter as the 3-dimensional shape parameter to be given to the generalized 3-dimensional object model.

8. The image processing system according to claim 7, wherein

the generalized illumination basis model generation means includes:

parameter setting means for setting a random 3-dimensional shape parameter to be given to the generalized 3-dimensional object model;

third illumination basis calculation means for calculating the illumination basis groups corresponding to the respective parameters based on the sample 3-dimensional shape group which is generated by giving the parameters set by the parameter setting means to the generalized 3-dimensional object model; and

basis calculation means for calculating the basis vector of the generalized illumination basis model using the illumination basis group calculated by the third illumination basis calculation means and the basis vector and eigenvalue of the generalized 3-dimensional object model.

9. The image processing system according to claim 8, wherein

the third illumination basis calculation means calculates the individual illumination basis using spherical harmonics based on the individual 3-dimensional shape data.

10. The image processing system according to claim 8, wherein

the third illumination basis calculation means includes:

illumination condition setting means for setting a plurality of different illumination conditions;

luminance calculation means for calculating the luminance value of each portion of the surface of the object under various illumination conditions set by the illumination condition setting means; and

calculation means for calculating the individual illumination basis group based on the luminance value calculated by the luminance calculation means by main component analysis.

14. An image processing method comprising:

an individual data generation step of generating individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

a first illumination basis calculation step of calculating an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions based on the generated individual 3-dimensional shape data and individual texture data;

a reproduced image generation step of generating a reproduced image which is an image reproducing the same illumination condition as an input image using the calculated individual illumination basis group; and

a parameter update step of repeatedly updating the 3-dimensional shape parameter and texture parameter to be given to the generalized 3-dimensional face model such that the generated reproduced image is close to the input image until a predetermined convergence condition is satisfied.

15. The image processing method according to claim 14, wherein

the first illumination basis calculation step calculates the individual illumination basis using spherical harmonics based on the individual 3-dimensional shape data generated in the individual data generation step.

16. The image processing method according to claim 14, wherein

the first illumination basis calculation step sets a plurality of different illumination conditions, calculates the luminance value of each portion of the surface of the object under the set various illumination conditions, and calculates the individual illumination basis group based on the calculated luminance value.

24. A 3-dimensional shape estimation method that uses the image processing method as claimed in claim 14 and estimates the 3-dimensional shape of an object, comprising

a shape output step of outputting, when a predetermined convergence condition is satisfied, a 3-dimensional shape generated by the generalized 3-dimensional object model based on the 3-dimensional shape parameter and texture parameter obtained at their convergence time.

25. An object position/posture estimation method that uses the image processing method as claimed in claim 14 and estimates the position and posture of an object, comprising

a position/posture output step of outputting, when a predetermined convergence condition is satisfied, a position/posture parameter representing the position and posture of an object obtained at its convergence time.

26. An image generation method that uses the image processing method as claimed in claim 14 and generates a reproduced image which is an image reproducing the same illumination condition as an input image, comprising

an image output step of outputting, when a predetermined convergence condition is satisfied, a reproduced image obtained at its convergence time.

17. An image processing method comprising:

an individual data generation step of generating individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

a generalized illumination basis model generation step of generating a generalized illumination basis model which corresponds to the generalized 3-dimensional object model and is used for generating illumination basis representing a basic variation in a reflection state caused due to a difference in the direction or intensity of illumination projected onto an individual object; and

a second illumination basis calculation step of calculating an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions using the generated generalized illumination basis model.

18. The image processing method according to claim 17, further comprising an output step of outputting the individual 3-dimensional shape data generated in the individual data generation step and individual illumination basis generated in the second illumination basis calculation step.

19. The image processing method according to claim 17, further comprising:

a reproduced image generation step of generating a reproduced image which is an image reproducing the same illumination condition as an input image using the individual illumination basis group calculated in the second illumination basis calculation step; and

a parameter update step of repeatedly updating the 3-dimensional shape parameter and texture parameter to be given to the generalized 3-dimensional object model such that the generated reproduced image is close to the input image until a predetermined convergence condition is satisfied.

20. The image processing method according to claim 19, wherein

the second illumination basis calculation step calculates the individual illumination basis corresponding to the individual 3-dimensional shape data that the generalized illumination basis model generation step generates using the generalized 3-dimensional object model by using the same parameter as the 3-dimensional shape parameter to be given to the generalized 3-dimensional object model.

21. The image processing method according to claim 20, further comprising:

a parameter setting step of setting a random 3-dimensional shape parameter to be given to the generalized 3-dimensional object model;

a third illumination basis calculation step of calculating the illumination basis groups corresponding to the respective parameters based on the sample 3-dimensional shape group which is generated by giving the set parameters to the generalized 3-dimensional object model; and

a basis calculation step of calculating the basis vector of the generalized illumination basis model using the calculated illumination basis group and the basis vector and eigenvalue of the generalized 3-dimensional object model.

22. The image processing method according to claim 21, wherein

the third illumination basis calculation step calculates the individual illumination basis using spherical harmonics based on the individual 3-dimensional shape data.

23. The image processing method according to claim 21, wherein

the third illumination basis calculation step sets a plurality of different illumination conditions, calculates the luminance value of each portion of the surface of the object under the set various illumination conditions, and calculates the individual illumination basis group based on the calculated luminance value by main component analysis.

27. An image processing program allowing a computer to execute:

individual data processing that generates individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

first illumination basis calculation processing that calculates an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions based on the generated individual 3-dimensional shape data and individual texture data;

reproduced image generation processing that generates a reproduced image which is an image reproducing the same illumination condition as an input image using the calculated individual illumination basis group; and

parameter update processing that repeatedly updates the 3-dimensional shape parameter and texture parameter to be given to the generalized 3-dimensional object model such that the generated reproduced image is close to the input image until a predetermined convergence condition is satisfied.

28. An image processing program allowing a computer to execute:

individual data generation processing that generates individual 3-dimensional shape data representing the 3-dimensional shape of an individual object and individual texture data representing the texture of the individual object based on a 3-dimensional shape parameter and texture parameter using a generalized 3-dimensional object model which is calculated based on 3-dimensional shape data and texture data of a plurality of objects similar to one another;

generalized illumination basis model generation processing that generates a generalized illumination basis model which corresponds to the generalized 3-dimensional object model and is used for generating illumination basis representing a basic variation in a reflection state caused due to a difference in the direction or intensity of illumination projected onto an individual object; and

second illumination basis calculation processing that calculates an individual illumination basis representing a variation in the luminance value of each portion of the 3-dimensional surface of the individual object under various illumination conditions using the generated generalized illumination basis model.