Method, apparatus and program for processing a three-dimensional image

• US 20030107568A1
• Filed: 11/26/2002
• Published: 06/12/2003
• Est. Priority Date: 12/03/2001
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Urisaka, Shinya, Ochiai, Yoshinobu
• Granted Patent: US 7,034,820 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

An image processing apparatus and an image processing method reconstruct a three-dimensional image of an object which expresses a texture, glossiness, and three-dimensionality with realism. The image processing apparatus produces, from a physical object, data representing a shape and a surface feature of the physical object, holds the data as a three-dimensional model, and presents a three-dimensional image under conditions of an illumination environment and line of sight designated by a user when the three-dimensional image is reconstructed. To present the three-dimensional image at a high speed with realism, the apparatus and method use new and particularly advantageous features in a generation process and in a data format used in the three-dimensional model.

Claims

1. A three-dimensional image processing apparatus for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generator which generates, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizer which synthesizes the three-dimensional image from the three-dimensional model, and displays the three-dimensional image,

wherein the three-dimensional model generator comprises;

a three-dimensional shape acquisition unit which acquires three-dimensional data relating to a shape of the object,

a surface attribute acquisition unit which acquires surface attribute data relating to a surface attribute of the object, and

a three-dimensional data integrator which generates the three-dimensional model by integrating the three-dimensional data acquired by the three-dimensional shape acquisition unit and the surface attribute data acquired by the surface attribute acquisition unit.

2. A three-dimensional processing apparatus according to claim 1, wherein the surface attribute acquisition unit estimates the surface attribute of the object based on image data obtained from a captured image of the object.

3. A three-dimensional processing apparatus according to claim 2, wherein the surface attribute acquisition unit separates and extracts a diffuse reflection component and a specular reflection component from the image data to estimate the surface attribute.

4. A three-dimensional processing apparatus according to claim 2, wherein the surface attribute acquisition unit selects and uses data appropriate for estimating the surface attribute from the image data in the estimation of the surface attribute.

5. A three-dimensional processing apparatus according to claim 2, wherein the surface attribute acquisition unit comprises an image capturing environment controller which changes an image capturing environment to generate the image data in response to an operation by a user.

6. A three-dimensional processing apparatus according to claim 5, wherein the image capturing environment controller in the surface attribute acquisition unit controls at least one of a position, a color, and a shape of an illumination light source in the image capturing environment and a number of times of image capturing of the object.

7. A three-dimensional processing apparatus according to claim 1, wherein the three-dimensional data integrator generates the three-dimensional model by associating each point in a set of points forming the three-dimensional data with an independent piece of surface attribute data.

8. A three-dimensional processing apparatus according to claim 1, wherein the three-dimensional data integrator generates the three-dimensional model by associating a group of points belonging to a segment having a similar surface attribute, out of a set of points forming the three-dimensional data, with a common piece of surface attribute data.

9. A three-dimensional image processing apparatus for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generator which generates, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizer which synthesizes the three-dimensional image from the three-dimensional model, and displays the three-dimensional image,

wherein the three-dimensional image synthesizer comprises;

an operation device through which a user inputs an observation environment of the three-dimensional image, and

a three-dimensional image generator which generates the three-dimensional image of the object in the observation environment input through the operation device based on the three-dimensional model.

10. A three-dimensional image processing method for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generating step, of generating, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizing step, of synthesizing the three-dimensional image from the three-dimensional model, and displaying the three-dimensional image,

wherein the three-dimensional model generating step comprises;

a three-dimensional shape acquisition substep, acquiring three-dimensional data relating to a shape of the object,

a surface attribute acquisition substep, of acquiring surface attribute data relating to a surface attribute of the object, and

a three-dimensional data integrating substep, generating the three-dimensional model by integrating the three-dimensional data acquired in the three-dimensional shape acquisition substep and the surface attribute data acquired in the surface attribute acquisition substep.

11. A three-dimensional processing method according to claim 10, wherein the surface attribute acquisition substep includes estimating the surface attribute of the object based on image data obtained from a captured image of the object.

12. A three-dimensional processing method according to claim 11, wherein the surface attribute acquisition substep includes separating and extracting a diffuse reflection component and a specular reflection component from the image data to estimate the surface attribute.

13. A three-dimensional processing method according to claim 11, wherein the surface attribute acquisition substep includes selecting and using data appropriate for estimating the surface attribute from the image data in the estimation of the surface attribute.

14. A three-dimensional processing method according to claim 11, wherein the surface attribute acquisition substep comprises an image capturing environment controlling process which changes an image capturing environment to generate the image data in response to an operation by a user.

15. A three-dimensional processing method according to claim 14, wherein the image capturing environment controlling process in the surface attribute acquisition substep controls at least one of a position, a color, and a shape of an illumination light source in the image capturing environment and a number of times of image capturing of the object.

16. A three-dimensional processing method according to claim 10, wherein the three-dimensional data integrating substep includes generating the three-dimensional model by associating each point in a set of points forming the three-dimensional data with an independent piece of surface attribute data.

17. A three-dimensional processing method according to claim 10, wherein the three-dimensional data integrating substep includes generating the three-dimensional model by associating a group of points belonging to a segment having a similar surface attribute, out of a set of points forming the three-dimensional data, with a common piece of surface attribute data.

18. A three-dimensional image processing method for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generating step, of generating, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizing step, of synthesizing the three-dimensional image from the three-dimensional model, and displaying the three-dimensional image,

wherein the three-dimensional image synthesizing step comprises;

an input substep, in which a user inputs an observation environment of the three-dimensional image, and

a three-dimensional image generating substep, of generating the three-dimensional image of the object in the observation environment input in the input substep based on the three-dimensional model.

19. A three-dimensional image processing computer program for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generating step, of generating, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizing step, of synthesizing the three-dimensional image from the three-dimensional model, and displays the three-dimensional image,

wherein the three-dimensional model generating step comprises;

a three-dimensional shape acquisition substep, of acquiring three-dimensional data relating to a shape of the object,

a surface attribute acquisition substep, of acquiring surface attribute data relating to a surface attribute of the object, and

a three-dimensional data integrating substep, of generating the three-dimensional model by integrating the three-dimensional data acquired in the three-dimensional shape acquisition substep and the surface attribute data acquired in the surface attribute acquisition substep.

20. A three-dimensional processing computer program according to claim 19, wherein the surface attribute acquisition substep includes estimating the surface attribute of the object based on image data obtained from a captured image of the object.

21. A three-dimensional processing computer program according to claim 20, wherein the surface attribute acquisition substep includes separating and extracts a diffuse reflection component and a specular reflection component from the image data to estimate the surface attribute.

22. A three-dimensional processing computer program according to claim 20, wherein the surface attribute acquisition substep includes selecting and using data appropriate for estimating the surface attribute from the image data in the estimation of the surface attribute.

23. A three-dimensional processing computer program according to claim 20, wherein the surface attribute acquisition substep comprises an image capturing environment controlling process which changes an image capturing environment to generate the image data in response to an operation by a user.

24. A three-dimensional processing computer program according to claim 23, wherein the image capturing environment controlling process in the surface attribute acquisition substep controls at least one of a position, a color, and a shape of an illumination light source in the image capturing environment and a number of times of image capturing of the object.

25. A three-dimensional processing computer program according to claim 19, wherein the three-dimensional data integrating substep includes generating the three-dimensional model by associating each point in a set of points forming the three-dimensional data with an independent piece of surface attribute data.

26. A three-dimensional processing computer program according to claim 19, wherein the three-dimensional data integrating substep includes generating the three-dimensional model by associating a group of points belonging to a segment having a similar surface attribute, out of a set of points forming the three-dimensional data, with a common piece of surface attribute data.

27. A three-dimensional image processing computer program for reconstructing a three-dimensional image of an object, comprising:

a three-dimensional model generating step, of generating, from the object, a three-dimensional model which is data for reconstructing the three-dimensional image of the object; and

a three-dimensional image synthesizing step, of synthesizing the three-dimensional image from the three-dimensional model, and displaying the three-dimensional image,

wherein the three-dimensional image synthesizing step comprises;

an input substep, in which a user inputs an observation environment of the three-dimensional image, and

a three-dimensional image generating substep, of generating the three-dimensional image of the object in the observation environment input in the input substep based on the three-dimensional model.

28. A method for generating a three-dimensional model for use in reconstructing a three-dimensional image, the method comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of determining surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data in the second step is modified for every segment which is differentiated based on the three-dimensional data acquired in the first step.

29. A method according to claim 28, wherein, based on the three-dimensional data acquired in the first step, a differentiation is made between a point having a complex change in the surrounding geometry thereof and a point having a monotonic change or no change in the surrounding geometry thereof in connection with each point in a set of points forming the three-dimensional data at each segment of the object, and the manner of determining the surface attribute data in the second step is modified for every segment.

30. A method according to claim 29, wherein, based on the image data and the three-dimensional data, the second step includes determining the surface attribute data of a point which is differentiated as being a point having a monotonic change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data.

31. A method according to claim 29, wherein, in connection with a point which is differentiated as being a point having a complex change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data, the second step includes regarding the surface attribute data of a point, which is adjacent to the point having the complex change and is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, as surface attribute data of the point having the complex change.

32. A method for generating a three-dimensional model for use in reconstructing a three-dimensional image, the method comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of determining surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data in the second step is modified for every segment which is differentiated based on the image data.

33. A method according to claim 32, further comprising the steps of extracting a segment of the object on which the image data is substantially uniform, regarding the surface attribute data determined for at least one point in a set of points forming the three-dimensional data within the extracted segment as surface attribute data for all points in that segment, and determining surface attribute data for points in a non-extracted segment on a point-by-point basis.

34. A computer program for generating a three-dimensional model for use in reconstructing a three-dimensional image, the computer program comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of determining surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data in the second step is modified for every segment which is differentiated based on the three-dimensional data acquired in the first step.

35. A computer program according to claim 34, wherein, based on the three-dimensional data acquired in the first step, a differentiation is made between a point having a complex change in the surrounding geometry thereof and a point having a monotonic change or no change in the surrounding geometry thereof in connection with each point in a set of points forming the three-dimensional data at each segment of the object, and the manner of determining the surface attribute data in the second step is modified for every segment.

36. A computer program according to claim 35, wherein, based on the image data and the three-dimensional data, the second step includes determining the surface attribute data of a point which is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data.

37. A computer program according to claim 35, wherein in connection with a point which is differentiated as being a point having a complex change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data, the second step includes regarding the surface attribute data of a point, which is adjacent to the point having the complex change and is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, as surface attribute data of the point having the complex change.

38. A computer program for generating a three-dimensional model for use in reconstructing a three-dimensional image, the computer program comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of determining surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data in the second step is modified for every segment which is differentiated based on the image data.

39. A computer program according to claim 38, further comprising the steps of extracting a segment of the object on which the image data is substantially uniform, regarding the surface attribute data determined for at least one point in a set of points forming the three-dimensional data within the extracted segment as surface attribute data for all points in that segment, and determining surface attribute data for points in a non-extracted segment on a point-by-point basis.

40. A generator for generating a three-dimensional model for use in reconstructing a three-dimensional image, the generator comprising:

a three-dimensional shape acquisition unit for acquiring three-dimensional data relating to a shape of an object;

a surface attribute acquisition unit for acquiring surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a three-dimensional data integrator for generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data by the surface attribute acquisition unit is modified for every segment which is differentiated based on the three-dimensional data acquired by said three-dimensional shape acquisition unit.

41. A generator according to claim 40, wherein based on the three-dimensional data acquired by the three-dimensional shape acquisition unit, a differentiation is made between a point having a complex change in the surrounding geometry thereof and a point having a monotonic change or no change in the surrounding geometry thereof in connection with each point in a set of points forming the three-dimensional data at each segment of the object, and the manner of determining the surface attribute data by the surface attribute acquisition unit is modified for every segment.

42. A generator according to claim 41, wherein the surface attribute acquisition unit determines, based on the image data and the three-dimensional data, the surface attribute data of a point which is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data.

43. A generator according to claim 41, wherein in connection with a point which is differentiated as being a point having a complex change in the surrounding geometry thereof, out of the set of points forming the three-dimensional data, the surface attribute acquisition unit regards the surface attribute data of a point, which is adjacent to the point having the complex change and is differentiated as being a point having a monotonic change or no change in the surrounding geometry thereof, as surface attribute data of the point having the complex change.

46. A three-dimensional image processing system comprising a generator for generating a three-dimensional model according to one of claims 40 through 45, a shape measurement device for measuring a shape of an object, and an image capturing device for capturing an image of the object.

44. A generator for generating a three-dimensional model for use in reconstructing a three-dimensional image, the generator comprising:

a three-dimensional shape acquisition unit for acquiring three-dimensional data relating to a shape of an object;

a surface attribute acquisition unit for acquiring surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a three-dimensional data integrator for generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein the manner of determining the surface attribute data by the surface attribute acquisition unit is modified for every segment which is differentiated based on the image data.

45. A generator according to claim 44, wherein a segment of the object on which the image data is substantially uniform is extracted, the surface attribute data determined for at least one point in a set of points forming the three-dimensional data within the extracted segment is regarded as surface attribute data for all points in that segment, and surface attribute data for points in a non-extracted segment is determined point by point.

47. A method for generating a three-dimensional model for use in reconstructing a three-dimensional image, the method comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of acquiring surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein a user re-inputs at least one parameter used in the first step and the second step subsequent to one of the first step and the second step, and one of the first step and the second step is executed using at least one re-input parameter.

48. A method according to claim 47, wherein the re-inputting of the parameter and the re-execution of one of the first step and the second step are carried out a plurality of times.

49. A method according to claim 47, wherein the parameter re-input by the user and the data obtained from the first step and the second step are stored, and are read in response to a request from the user, and are used in a subsequent step.

50. A method according to claim 47, wherein the parameter re-input by the user and the data obtained from the first step and the second step are stored, and are read in response to a request from the user, are presented to the user.

51. A method according to claim 47, wherein the parameter re-input by the user is related to at least one of the following factors of a setup method:

an illumination light when data is acquired, a category designation of surface attribute relating to the object about which the data is acquired, a criterion according to which a method of segmenting the three-dimensional data is applied to segments to which different determination methods of the surface attribute are applied, and a criterion according to which the determination methods are applied to the segments of segmented three-dimensional data.

52. A computer program for generating a three-dimensional model for use in reconstructing a three-dimensional image, the computer program comprising:

a first step, of acquiring three-dimensional data relating to a shape of an object;

a second step, of acquiring surface attribute data from image data relating to an appearance of the object, and the three-dimensional data; and

a third step, of generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data,

wherein a user re-inputs at least one parameter used in the first step and the second step subsequent to one of the first step and the second step, and one of the first step and the second step is executed using at least one re-input parameter.

53. A computer program according to claim 52, wherein the re-inputting of the parameter and the re-execution of one of the first step and the second step are carried out a plurality of times.

54. A computer program according to claim 52, wherein the parameter re-input by the user and the data obtained from the first step and the second step are stored, and are read in response to a request from the user, and are used in a subsequent step.

55. A computer program according to claim 52, wherein the parameter re-input by the user and the data obtained from the first step and the second step are stored, and are read in response to a request from the user, are presented to the user.

56. A computer program according to claim 52, wherein the parameter re-input by the user is related to at least one of the following factors of a setup method:

an illumination light when data is acquired, a category designation of surface attribute relating to the object about which the data is acquired, a criterion according to which a method of segmenting the three-dimensional data is applied to segments to which different determination methods of the surface attribute are applied, and a criterion according to which the determination methods are applied to the segments of segmented three-dimensional data.

57. A generator for generating a three-dimensional model for use in reconstructing a three-dimensional image, the generator comprising:

a three-dimensional shape acquisition unit for acquiring three-dimensional data relating to a shape of an object;

a surface attribute acquisition unit for acquiring surface attribute data from image data relating to an appearance of the object, and the three-dimensional data;

a three-dimensional integrator for generating the three-dimensional model by integrating the three-dimensional data and the surface attribute data; and

an operation device for inputting parameters which are used by the three-dimensional shape acquisition unit and the surface attribute acquisition unit to result in the respective data,

wherein the operation device re-inputs at least one parameter in response to a user after one of the three-dimensional shape acquisition unit and the surface attribute acquisition unit acquires the respective data, and the operation device acquires again data through one of the three-dimensional shape acquisition unit and the surface attribute acquisition unit using at least one re-input parameter.

58. A generator according to claim 57, wherein the operation device performs the re-inputting of the parameter, and the data acquisition through one of the three-dimensional shape acquisition unit and the surface attribute acquisition unit a plurality of times.

59. A generator according to claim 57, wherein the operation device stores the parameter re-input by the user and the data obtained by the three-dimensional shape acquisition unit and the surface attribute acquisition unit, and uses the parameter and the data in a subsequent step in response to a request from the user.

60. A generator according to claim 57, the operation device stores the parameter re-input by the user and the data obtained by the three-dimensional shape acquisition unit and the surface attribute acquisition unit, and presents the parameter and the data to the user in response to a request from the user.

61. A generator according to claim 57, wherein, in response to a request from the user, the operation device inputs the parameter relating to at least one of the following factors of a setup method:

an illumination light when data is acquired, a category designation of surface attribute relating to the object about which the data is acquired, a criterion according to which a method of segmenting the three-dimensional data is applied to segments to which different determination methods of the surface attribute are applied, and a criterion according to which the determination methods are applied to the segments of segmented three-dimensional data.

62. A three-dimensional image processing apparatus comprising a three-dimensional image generator according to one of claims 57 through 61, and an image output device which displays or prints a three-dimensional image based on the three-dimensional model output from the three-dimensional model generator.