Method, apparatus and program for processing a three-dimensional image

• US 7,239,312 B2
• Filed: 10/28/2005
• Issued: 07/03/2007
• Est. Priority Date: 12/03/2001
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Urisaka, Shinya, Ochiai, Yoshinobu
• Primary Examiner(s): Nguyen; Kimbinh T.
• Application Number:
  US11/260,455
• Publication Number:
  US 20060033733A1
• Time in Patent Office: 613 Days
• Field of Search: 345/419, 345/420, 345/426, 345/427, 345/629, 345/443
• US Class Current: 345/419
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

An image processing apparatus and an image processing method reconstruct a three-dimensional image of an object which expresses a texture, glossiness, and three-dimensionality with realism. The image processing apparatus produces, from a physical object, data representing a shape and a surface feature of the physical object, holds the data as a three-dimensional model, and presents a three-dimensional image under conditions of an illumination environment and line of sight designated by a user when the three-dimensional image is reconstructed. To present the three-dimensional image at a high speed with realism, the apparatus and method use new and particularly advantageous features in a generation process and in a data format used in the three-dimensional model.

Claims

1. A three-dimensional image processing apparatus for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generator which generates, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizer which synthesizes the three-dimensional image from the three-dimensional model, and displays the three-dimensional image,

wherein the three-dimensional model generator comprises;

a three-dimensional shape acquisition unit which acquires three-dimensional data relating to a shape of the object,

a surface attribute acquisition unit which acquires surface attribute data relating to a surface attribute of the object, and

a three-dimensional data integrator which generates the three-dimensional model by integrating the three-dimensional data acquired by the three-dimensional shape acquisition unit and the surface attribute data acquired by the surface attribute acquisition unit, wherein

the three-dimensional image processing apparatus changes an image capturing environment to take an image data of an object according to an incoming signal.

2. A three-dimensional processing apparatus according to claim 1, wherein the surface attribute acquisition unit estimates the surface attribute of the object based on image data obtained from a captured image of the object.

3. A three-dimensional processing apparatus according to claim 2, wherein the surface attribute acquisition unit separates and extracts a diffuse reflection component and a specular reflection component from the image data to estimate the surface attribute.

4. A three-dimensional processing apparatus according to claim 2, wherein the surface attribute acquisition unit selects and uses data appropriate for estimating the surface attribute from the image data in the estimation of the surface attribute.

5. A three-dimensional processing apparatus according to claim 2, wherein the surface attribute acquisition unit comprises an image capturing environment controller which changes an image capturing environment to generate the image data in response to an operation by a user.

6. A three-dimensional processing apparatus according to claim 5, wherein the image capturing environment controller in the surface attribute acquisition unit controls at least one of a position, a color, and a shape of an illumination light source in the image capturing environment and a number of times of image capturing of the object.

7. A three-dimensional processing apparatus according to claim 1, wherein the three-dimensional data integrator generates the three-dimensional model by associating each point in a set of points forming the three-dimensional data with an independent piece of surface attribute data.

8. A three-dimensional processing apparatus according to claim 1, wherein the three-dimensional data integrator generates the three-dimensional model by associating a group of points belonging to a segment having a similar surface attribute, out of a set of points forming the three-dimensional data, with a common piece of surface attribute data.

9. A three-dimensional image processing apparatus for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generator which generates, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizer which synthesizes the three-dimensional image from the three-dimensional model, and displays the three-dimensional image,

wherein the three-dimensional image synthesizer comprises;

an operation device through which a user inputs an observation environment of the three-dimensional image, and

a three-dimensional image generator which generates the three-dimensional image of the object in the observation environment input through the operation device based on the three-dimensional model, wherein

the three-dimensional image processing apparatus changes an image capturing environment to take an image data of an object according to an incoming signal.

10. A three-dimensional image processing method for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generating step, of generating, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizing step, of synthesizing the three-dimensional image from the three-dimensional model, and displaying the three-dimensional image,

wherein the three-dimensional model generating step comprises;

a three-dimensional shape acquisition substep, acquiring three-dimensional data relating to a shape of the object,

a surface attribute acquisition substep, of acquiring surface attribute data relating to a surface attribute of the object, and

a three-dimensional data integrating substep, generating the three-dimensional model by integrating the three-dimensional data acquired in the three-dimensional shape acquisition substep and the surface attribute data acquired in the surface attribute acquisition substep, wherein

the three-dimensional image processing method changes an image capturing environment to take an image data of an object according to an incoming signal.

11. A three-dimensional processing method according to claim 10, wherein the surface attribute acquisition substep includes estimating the surface attribute of the object based on image data obtained from a captured image of the object.

12. A three-dimensional processing method according to claim 11, wherein the surface attribute acquisition substep includes separating and extracting a diffuse reflection component and a specular reflection component from the image data to estimate the surface attribute.

13. A three-dimensional processing method according to claim 11, wherein the surface attribute acquisition substep includes selecting and using data appropriate for estimating the surface attribute from the image data in the estimation of the surface attribute.

14. A three-dimensional processing method according to claim 11, wherein the surface attribute acquisition substep comprises an image capturing environment controlling process which changes an image capturing environment to generate the image data in response to an operation by a user.

15. A three-dimensional processing method according to claim 14, wherein the image capturing environment controlling process in the surface attribute acquisition substep controls at least one of a position, a color, and a shape of an illumination light source in the image capturing environment and a number of times of image capturing of the object.

16. A three-dimensional processing method according to claim 10, wherein the three-dimensional data integrating substep includes generating the three-dimensional model by associating each point in a set of points forming the three-dimensional data with an independent piece of surface attribute data.

17. A three-dimensional processing method according to claim 10, wherein the three-dimensional data integrating substep includes generating the three-dimensional model by associating a group of points belonging to a segment having a similar surface attribute, out of a set of points forming the three-dimensional data, with a common piece of surface attribute data.

18. A three-dimensional image processing method for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generating step, of generating, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizing step, of synthesizing the three-dimensional image from the three-dimensional model, and displaying the three-dimensional image,

wherein the three-dimensional image synthesizing step comprises;

an input substep, in which a user inputs an observation environment of the three-dimensional image, and

a three-dimensional image generating substep, of generating the three-dimensional image of the object in the observation environment input in the input substep based on the three-dimensional model, wherein

the three-dimensional image processing method changes an image capturing environment to take an image data of an object according to an incoming signal.

19. A computer-readable medium encoded with a computer program for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generating step, of generating, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizing step, of synthesizing the three-dimensional image from the three-dimensional model, and displays the three-dimensional image,

wherein the three-dimensional model generating step comprises;

a three-dimensional shape acquisition substep, of acquiring three-dimensional data relating to a shape of the object,

a surface attribute acquisition substep, of acquiring surface attribute data relating to a surface attribute of the object, and

a three-dimensional data integrating substep, of generating the three-dimensional model by integrating the three-dimensional data acquired in the three-dimensional shape acquisition substep and the surface attribute data acquired in the surface attribute acquisition substep, wherein

the three-dimensional image processing computer program changes an image capturing environment to take an image data of an object according to an incoming signal.

20. A computer-readable medium encoded with a computer program according to claim 19, wherein the surface attribute acquisition substep includes estimating the surface attribute of the object based on image data obtained from a captured image of the object.

21. A computer-readable medium encoded with a computer program according to claim 20, wherein the surface attribute acquisition substep includes separating and extracting a diffuse reflection component and a specular reflection component from the image data to estimate the surface attribute.

22. A computer-readable medium encoded with a computer program according to claim 20, wherein the surface attribute acquisition substep includes selecting and using data appropriate for estimating the surface attribute from the image data in the estimation of the surface attribute.

23. A computer-readable medium encoded with a computer program according to claim 20, wherein the surface attribute acquisition substep comprises an image capturing environment controlling process which changes an image capturing environment to generate the image data in response to an operation by a user.

24. A computer-readable medium encoded with a computer program according to claim 23, wherein the image capturing environment controlling process in the surface attribute acquisition substep controls at least one of a position, a color, and a shape of an illumination light source in the image capturing environment and a number of times of image capturing of the object.

25. A computer-readable medium encoded with a computer program according to claim 19, wherein the three-dimensional data integrating substep includes generating the three-dimensional model by associating each point in a set of points forming the three-dimensional data with an independent piece of surface attribute data.

26. A computer-readable medium encoded with a computer program according to claim 19, wherein the three-dimensional data integrating substep includes generating the three-dimensional model by associating a group of points belonging to a segment having a similar surface attribute, out of a set of points forming the three-dimensional data, with a common piece of surface attribute data.

27. A computer-readable medium encoded with a computer program for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generating step, of generating, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizing step, of synthesizing the three-dimensional image from the three-dimensional model, and displaying the three-dimensional image,

wherein the three-dimensional image synthesizing step comprises;

an input substep, in which a user inputs an observation environment of the three-dimensional image, and

a three-dimensional image generating substep, of generating the three-dimensional image of the object in the observation environment input in the input substep based on the three-dimensional model, wherein

the three-dimensional image processing computer program changes an image capturing environment to take an image data of an object according to an incoming signal.

28. A method for generating a three-dimensional model for use in reconstructing a three-dimensional image, the method comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of determining surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data in the second step is modified for every segment which is differentiated based on the three-dimensional data acquired in the first step.

29. A method according to claim 28, wherein, based on the three-dimensional data acquired in the first step, a differentiation is made between a point having a complex change in the surrounding geometry thereof and a point having a monotonic change or no change in the surrounding geometry thereof in connection with each point in a set of points forming the three-dimensional data at each segment of the object, and the manner of determining the surface attribute data in the second step is modified for every segment.

30. A method according to claim 29, wherein, based on the image data and the three-dimensional data, the second step includes determining the surface attribute data of a point which is differentiated as being a point having a monotonic change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data.

31. A method according to claim 29, wherein, in connection with a point which is differentiated as being a point having a complex change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data, the second step includes regarding the surface attribute data of a point, which is adjacent to the point having the complex change and is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, as surface attribute data of the point having the complex change.

32. A method for generating a three-dimensional model for use in reconstructing a three-dimensional image, the method comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of determining surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data in the second step is modified for every segment which is differentiated based on the image data.

33. A method according to claim 32, further comprising the steps of extracting a segment of the object on which the image data is substantially uniform, regarding the surface attribute data determined for at least one point in a set of points forming the three-dimensional data within the extracted segment as surface attribute data for all points in that segment, and determining surface attribute data for points in a non-extracted segment on a point-by-point basis.

34. A computer-readable medium encoded with a computer program for generating a three-dimensional model for use in reconstructing a three-dimensional image, the computer program comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of determining surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data in the second step is modified for every segment which is differentiated based on the three-dimensional data acquired in the first step.

35. A computer-readable medium encoded with a computer program according to claim 34, wherein, based on the three-dimensional data acquired in the first step, a differentiation is made between a point having a complex change in the surrounding geometry thereof and a point having a monotonic change or no change in the surrounding geometry thereof in connection with each point in a set of points forming the three-dimensional data at each segment of the object, and the manner of determining the surface attribute data in the second step is modified for every segment.

36. A computer-readable medium encoded with a computer program according to claim 35, wherein, based on the image data and the three-dimensional data, the second step includes determining the surface attribute data of a point which is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data.

37. A computer-readable medium encoded with a computer program according to claim 35, wherein in connection with a point which is differentiated as being a point having a complex change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data, the second step includes regarding the surface attribute data of a point, which is adjacent to the point having the complex change and is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, as surface attribute data of the point having the complex change.

38. A computer-readable medium encoded with a computer program for generating a three-dimensional model for use in reconstructing a three-dimensional image, the computer program comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of determining surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data in the second step is modified for every segment which is differentiated based on the image data.

39. A computer-readable medium encoded with a computer program according to claim 38, further comprising the steps of extracting a segment of the object on which the image data is substantially uniform, regarding the surface attribute data determined for at least one point in a set of points forming the three-dimensional data within the extracted segment as surface attribute data for all points in that segment, and determining surface attribute data for points in a non-extracted segment on a point-by-point basis.

40. A generator for generating a three-dimensional model for use in reconstructing a three-dimensional image, the generator comprising:

a three-dimensional shape acquisition unit for acquiring three-dimensional data relating to a shape of an object;

a surface attribute acquisition unit for acquiring surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a three-dimensional data integrator for generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attributed data by the surface attribute acquisition unit is modified for every segment which is differentiated based on the three-dimensional data acquired by said three-dimensional shape acquisition unit.

41. A generator according to claim 40, wherein based on the three-dimensional data acquired by the three-dimensional shape acquisition unit, a differentiation is made between a point having a complex change in the surrounding geometry thereof and a point having a monotonic change or no change in the surrounding geometry thereof in connection with each point in a set of points forming the three-dimensional data at each segment of the object, and the manner of determining the surface attribute data by the surface attribute acquisition unit is modified for every segment.

42. A generator according to claim 41, wherein the surface attribute acquisition unit determines, based on the image data and the three-dimensional data, the surface attribute data of a point which is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data.

43. A generator according to claim 41, wherein in connection with a point which is differentiated as being a point having a complex change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data, the surface attribute acquisition unit regards the surface attribute data of a point, which is adjacent to the point having the complex change and is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, as surface attribute data of the point having the complex change.

46. A three-dimensional image processing system comprising a generator for generating a three-dimensional model according to claim 40, a shape measurement device for measuring a shape of an object, and an image capturing device for capturing an image of the object.

44. A generator for generating a three-dimensional model for use in reconstructing a three-dimensional image, the generator comprising:

a three-dimensional shape acquisition unit for acquiring three-dimensional data relating to a shape of an object;

a surface attribute acquisition unit for acquiring surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a three-dimensional data integrator for generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data by the surface attribute acquisition unit is modified for every segment which is differentiated based on the image data.

45. A generator according to claim 44, wherein a segment of the object on which the image data is substantially uniform is extracted, the surface attribute data determined for at least one point in a set of points forming the three-dimensional data within the extracted segment is regarded as surface attribute data for all points in that segment, and surface attribute data for points in a non-extracted segment is determined point by point.

47. A three-dimensional image processing system comprising a generator for generating a three-dimensional model according to claim 44, a shape measurement device for measuring a shape of an object, and an image capturing device for capturing an image of the object.