Virtual visual point image generating method and 3-d image display method and device

• US 20070122027A1
• Filed: 06/18/2004
• Published: 05/31/2007
• Est. Priority Date: 06/20/2003
• Status: Active Grant

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Suyama, Shiro, Kunita, Yutaka, Hashimoto, Akihiko
• Granted Patent: US 7,538,774 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G02B 30/50 the image being built up from image elements distributed over a 3D volume, e.g. voxels
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors

Abstract

A virtual viewpoint image generation method for generating a virtual viewpoint image that is an image of an object viewed from a virtual viewpoint based on plural images of the object taken by plural cameras is provided. In the virtual viewpoint image generation method, projection planes having a multi-layered structure are set, each corresponding point, on the images of the object, corresponding to a projection point of a projection plane is obtained, color information of each projection point is determined based on color information of corresponding points, for each of the projection points overlapping when viewed from a reference viewpoint in a space, a degree of probability that the object exists at a distance corresponding to a position of the projection point is determined based on a degree of correlation of the corresponding points, mixing processing is performed on color information of reference points overlapping when viewed from the virtual viewpoint according to the degree of probability of existence of the object so as to determine color information of each pixel of the virtual viewpoint image.

Claims

1. A virtual viewpoint image generation method comprising:

a step of obtaining plural images of an object taken by plural cameras;

a step of determining a virtual viewpoint that is a position from which the object is viewed; and

a step of generating a virtual viewpoint image that is an image of the object viewed from the virtual viewpoint based on the obtained images of the object,

the step of generating the virtual viewpoint image comprising;

a step 1 of setting projection planes having a multi-layered structure;

a step 2 of obtaining each corresponding point, on the images of the object, corresponding to a projection point of a projection plane;

a step 3 of determining color information or brightness information of each projection point based on color information or brightness information of corresponding points;

a step of calculating, for each of the projection points overlapping when viewed from a reference viewpoint in a space, a degree of probability that the object exists at a distance corresponding to a position of the projection point based on a degree of correlation of the corresponding points or neighborhoods of the corresponding points;

a step 5 of performing mixing processing on color information or brightness information of reference points overlapping when viewed from the virtual viewpoint according to the degree of probability of existence of the object so as to determine color information or brightness information of each pixel of the virtual viewpoint image; and

a step 6 of repeating steps from the step 1 to the steps 5 for every point corresponding to pixels of the virtual viewpoint image.

2. The virtual viewpoint image generation method as claimed in claim 1, the step 3 comprising:

mixing the color information or the brightness information of the corresponding points or selecting the color information or the brightness information of one corresponding point from the color information or the brightness information of the corresponding points.

3. The virtual viewpoint image generation method as claimed in claim 1 or 2, the step 4 or the step 5 comprising:

a step of setting, in each reference point on the projection plane, transparency having plural gradations from transparent to opaque by converting the degree of probability that the object exists; and

the step 5 comprising;

performing the mixing processing according to the transparency instead of the degree of probability that the object exists.

4. The virtual viewpoint image generation method as claimed in claim 3, the mixing processing of the step 5 comprising:

processing projection points successively from a projection point far from the virtual viewpoint to a projection point near the virtual viewpoint,

wherein color information or brightness information obtained by the mixing processing up to a projection point is obtained by calculating interior division between color information or brightness information at the projection point and color information or brightness information obtained by the mixing processing up to a previous projection point in a ratio according to the transparency.

5. The virtual viewpoint image generation method as claimed in any one of claims 1-4, wherein:

projection planes specific to each camera taking each image of the object are set in the step 1;

the color information or the brightness information of the step 3 are determined only by using color information or brightness information of the corresponding points of the images of the object taken by the plural cameras;

the degree of probability that the object exists in step 4 is calculated using, as the reference viewpoint, a viewpoint of the camera specific to the projection plane to which the projection point belongs; and

correction is performed based on position relationship between the virtual viewpoint and each reference viewpoint in the mixing processing of the color information or the brightness information.

6. A virtual viewpoint image generation apparatus comprising:

an object image obtaining means for obtaining plural images of an object taken by plural cameras;

a virtual viewpoint determination means for determining a virtual viewpoint that is a position from which the object is viewed; and

an image generation means for generating a virtual viewpoint image that is an image of the object viewed from the virtual viewpoint based on the obtained images of the object,

the image generation means comprising;

a projection plane determination means for determining projection planes having a multi-layered structure;

a reference viewpoint determination means for determining a position of the reference viewpoint;

a texture array keeping means for keeping an array of texture images to be mapped to the projection planes;

a corresponding point matching processing means for associating parts, in the images of the object, on which the same region of the object appears with each other;

a color information determination means for determining color information or brightness information in the array of the texture images by performing mixing processing on the images of the object;

an existence probability information determination means for calculating a degree of probability that the object exists at a distance corresponding to a position of the projection point in the array of the texture images based on the processing result of the corresponding point matching processing means;

a rendering means for rendering the projection planes viewed from the virtual viewpoint based on the color information or the brightness information determined by the color information determination means and the existence probability determined by the existence probability determination means.

7. The virtual viewpoint image generation apparatus as claimed in claim 6, the existence probability information determination means comprising:

a means for setting, in each reference point on the projection plane, transparency having plural gradations from transparent to opaque by converting the degree of possibility that the object exists;

wherein the rendering means performs rendering using the transparency instead of the degree of possibility that the object exists.

8. The virtual viewpoint image generation apparatus as claimed in claim 7, the rendering means comprising;

a means for processing projection points successively from a projection point far from the virtual viewpoint to a projection point near the virtual viewpoint,

wherein color information or brightness information obtained by the mixing processing up to a projection point is obtained by calculating interior division between color information or brightness information at the projection point and color information or brightness information obtained by the mixing processing up to a previous projection point in a ratio according to the transparency.

9. The virtual viewpoint image generation apparatus as claimed in any one of claims 6-8, wherein:

the projection plane determination means determines projection planes specific to each camera taking each image of the object;

the color information determination means determines the color information or the brightness information only by using color information or brightness information of the corresponding points of the images of the object taken by the plural cameras;

the existence probability information determination means calculates the degree of probability that the object exists by using, as the reference viewpoint, a viewpoint of the camera specific to the projection plane to which the projection point belongs; and

the rendering means includes a means for performing correction based on position relationship between the virtual viewpoint and each reference viewpoint.

10. A virtual viewpoint image generation program causing a computer to perform:

a step of obtaining plural images of an object taken by plural cameras;

a step of determining a virtual viewpoint that is a position from which the object is viewed; and

a step of generating a virtual viewpoint image that is an image of the object viewed from the virtual viewpoint based on the obtained images of the object,

the step of generating the virtual viewpoint image comprising;

a step 1 of setting projection planes having a multi-layered structure;

a step 2 of obtaining each corresponding point, on the images of the object, corresponding to a projection point of a projection plane;

a step 3 of determining color information or brightness information of each projection point based on color information or brightness information of corresponding points;

a step 4 of calculating, for each of the projection points overlapping when viewed from a reference viewpoint in a space, a degree of probability that the object exists at a distance corresponding to a position of the projection point based on a degree of correlation of the corresponding points or neighborhoods of the corresponding points;

a step 5 of performing mixing processing on color information or brightness information of reference points overlapping when viewed from the virtual viewpoint according to the degree of probability of existence of the object so as to determine color information or brightness information of each pixel of the virtual viewpoint image; and

a step 6 of repeating steps from the step 1 to the steps 5 for every point corresponding to pixels of the virtual viewpoint image.

11. A computer readable recording medium storing the virtual viewpoint image generation program as claimed in claim 10.

12. An image generation method comprising:

a step of obtaining images of an object taken from different viewpoints;

a step of obtaining a three-dimensional shape of the object based on the images; and

a step of generating an image of the object viewed from a viewpoint of an observer based on the obtained three-dimensional shape of the object,

the step of obtaining the three-dimensional shape of the object comprising;

a step of setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a step of determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a step of determining color information or brightness information of projection points that are points on the projection planes based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a step of calculating correlation degrees among corresponding points corresponding to the projection points;

a step of calculating, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that the object exists at the projection point based on the correlation degree of each projection point;

the step of calculating the correlation degree comprising;

a step of preparing plural groups of camera sets each being a group of some viewpoints selected from the plural viewpoints; and

a step of obtaining the correlation degree from the corresponding points on images included in each camera set,

the step of determining the existence probability comprising;

a step of calculating the existence probability based on the correlation degree of each projection point obtained for each camera set; and

a step of determining the existence probability of each projection point by performing integrating processing for the existence probability determined for each camera set.

13. The image display method as claimed in claim 12, the step of calculating the existence probability based on the correlation degree of each projection point obtained for each camera set comprising:

a step of calculating an evaluation reference value based on the correlation degree of each projection point calculated for each camera set;

a step of calculating a distribution function of the existence probability by performing statistical processing on the evaluation reference value of each projection point calculated for each camera set; and

a step of determining the existence probability of each projection point based on the distribution function of the existence probability.

14. The image generation method as claimed in claim 12 or 13, the step of generating the image of the object viewed from the viewpoint of the observer comprising:

mixing color information or brightness information of the projection points overlapping when viewed from the viewpoint of the observer in a ratio of the existence probability to determine color information or brightness information of each point on the image to be generated so as to generate a two-dimensional image.

15. The image generation method as claimed in claim 12 or 13, the step of generating the image of the object viewed from the viewpoint of the observer comprising:

a step of setting plural image generation planes at positions having different depths when viewed from the viewpoint of the observer; and

a step of converting the color information or brightness information and the existence probability of each projection point into color information or brightness information and a brightness distribution coefficient on each image generation plane based on position relationship among projection points overlapping when viewed from the viewpoint of the observer and points on each image generation plane.

16. An image generation apparatus comprising:

an object image obtaining means for obtaining images of an object taken from different viewpoints;

a three-dimensional shape obtaining means for obtaining a three-dimensional shape of the object based on the images; and

an object image generation means for generating an image of the object viewed from a viewpoint of an observer based on the obtained three-dimensional shape of the object,

the three-dimensional shape obtaining means comprising;

a means for setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a means for determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a means for determining color information or brightness information of projection points that are points on the projection planes based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a means for calculating correlation degrees among corresponding points corresponding to the projection points;

a means for determining, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that the object exists at the projection point based on the correlation degree of each projection point;

the means for calculating the correlation degree comprising;

a means for preparing plural groups of camera sets each being a group of some viewpoints selected from the plural viewpoints; and

a means for obtaining the correlation degree from the corresponding points on images included in each camera set,

the means for determining the existence probability comprising;

a means for calculating the existence probability based on the correlation degree of each projection point obtained for each camera set; and

a means for determining the existence probability of each projection point by performing integrating processing for the existence probability determined for each camera set.

17. The image generation apparatus as claimed in claim 16, the means for calculating the existence probability based on the correlation degree of each projection point obtained for each camera set comprising:

a means for calculating an evaluation reference value based on the correlation value of each projection point calculated for each camera set;

a means for calculating a distribution function of the existence probability by performing statistical processing on the evaluation reference value of each projection point calculated for each camera set; and

a means for determining the existence probability of each projection point based on the distribution function of the existence probability.

18. The image generation apparatus as claimed in claim 16 or 17, wherein the means for generating the image of the object viewed from the viewpoint of the observer is a means for:

mixing color information or brightness information of the projection points overlapping when viewed from the viewpoint of the observer in a ratio of the existence probability to determine color information or brightness information of each point on the image to be generated so as to determine a two-dimensional image.

19. The image generation apparatus as claimed in claim 16 or 17, the means for generating the image of the object viewed from the viewpoint of the observer comprising:

a means for setting plural image generation planes at positions having different depths when viewed from the viewpoint of the observer; and

a means for converting the color information or brightness information and the existence probability of each projection point into color information or brightness information and a brightness distribution coefficient on each image generation plane based on position relationship among projection points overlapping viewed from the viewpoint of the observer and points on the image generation planes.

20. An image generation program causing a computer to perform:

a step of obtaining images of an object taken from different viewpoints;

a step of obtaining a three-dimensional shape of the object based on the images; and

a step of generating an image of the object viewed from a viewpoint of an observer based on the obtained three-dimensional shape of the object,

the step of obtaining the three-dimensional shape of the object comprising;

a step of setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a step of determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a step of determining color information or brightness information of projection points that are points on the projection planes based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a step of calculating correlation degrees among corresponding points corresponding to the projection points;

a step of calculating, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that the object exists at the projection point based on the correlation degree of each projection point;

the step of calculating the correlation degree comprising;

a step of preparing plural groups of camera sets each being a group of some viewpoints selected from the plural viewpoints; and

a step of obtaining the correlation degree from the corresponding points on images included in each camera set,

the step of determining the existence probability comprising;

a step of calculating the existence probability based on the correlation degree of each projection point obtained for each camera set; and

a step of determining the existence probability of each projection point by performing integrating processing for the existence probability determined for each camera set.

21. A computer readable recording medium storing the image generation program as claimed in claim 20.

22. An image generation method comprising:

a step of obtaining images of an object taken by changing focusing distance;

a step of obtaining a virtual viewpoint that is a viewpoint from which the object appearing in the images is viewed;

a step of obtaining a three-dimensional shape of the object based on the images; and

a step of generating an image of the object viewed from the virtual viewpoint based on the obtained three-dimensional shape of the object,

the step of obtaining the three-dimensional shape of the object comprising;

a step of setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a step of determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a step of determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a step of determining focusing degrees of the projection points based on focusing degrees of the corresponding points corresponding to the projection points;

a step of calculating, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that the object exists at a distance corresponding to a position of the projection point based on the focusing degree of each projection point;

the step of generating the image of the object viewed from the virtual viewpoint comprising;

mixing color information or brightness information of the projection points overlapping when viewed from the virtual viewpoint in a ratio corresponding to the existence probability to determine color information or brightness information of each point of the image to be generated.

23. The image generation method as claimed in claim 22, the step of obtaining the three-dimensional shape of the object or the step of generating the image of the object viewed from the virtual viewpoint comprising:

a step of setting transparency having plural gradations from transparent to opaque on each projection point based on the existence probability of the projection points overlapping when viewed from the reference viewpoint or the virtual viewpoint;

the step of generating the image of the object viewed from the virtual viewpoint comprising;

mixing the color information or the brightness information of the projection points overlapping when viewed from the virtual viewpoint in a ratio according to the transparency set based on the existence probability so as to determine the color information or brightness information of each point of the image to be generated.

24. The image generation method as claimed in claim 23,

the step of generating the image of the object viewed from the virtual viewpoint comprising:

mixing the color information or the brightness information for projection points successively from a projection point far from the virtual viewpoint to a projection point near the virtual viewpoint,

wherein color information or brightness information obtained by the mixing processing up to a projection point is obtained by calculating interior division between color information or brightness information at the projection point and color information or brightness information obtained by the mixing processing up to a previous projection point in a ratio according to the transparency.

25. An image generation apparatus comprising;

an object image obtaining means for obtaining images of an object taken by changing focusing distance;

a virtual viewpoint setting means for setting a virtual viewpoint that is a viewpoint from which the object appearing in the images is viewed;

a three-dimensional shape obtaining means for obtaining a three-dimensional shape of the object based on the images; and

a rendering means for generating an image of the object viewed from the virtual viewpoint based on the obtained three-dimensional shape of the object,

the three-dimensional shape obtaining means comprising;

a means for setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a means for determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a means for determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a means for determining focusing degrees of the projection points based on focusing degrees of the corresponding points corresponding to the projection points;

a means for determining, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that the object exists at a distance corresponding to a position of the projection point based on the focusing degree of each projection point;

the rendering means comprising;

a means for mixing color information or brightness information of the projection points overlapping when viewed from the virtual viewpoint in a ratio corresponding to the existence probability to determine color information or brightness information of each point of the image to be generated.

26. The image generation apparatus as claimed in claim 25, the three-dimensional shape obtaining means or the rendering means comprising:

a means for setting transparency having plural gradations from transparent to opaque on each projection point based on the existence probability of the projection points overlapping when viewed from the reference viewpoint or the virtual viewpoint; and

the rendering means comprising;

a means for mixing the color information or the brightness information of the projection points overlapping when viewed from the virtual viewpoint in a ratio according to the transparency set based on the existence probability so as to determine the color information or brightness information of each point of the image to be generated.

27. The image generation apparatus as claimed in claim 26,

the rendering means comprising:

a means for mixing the color information or the brightness information for projection points successively from a projection point far from the virtual viewpoint to a projection point near the virtual viewpoint,

wherein color information or brightness information obtained by the mixing processing up to a projection point is obtained by calculating interior division between color information or brightness information at the projection point and color information or brightness information obtained by the mixing processing up to a previous projection point in a ratio according to the transparency.

28. An image generation program for causing a computer to perform:

a step of obtaining images of an object taken by changing focusing lengths;

a step of obtaining a virtual viewpoint that is a viewpoint from which the object appearing in the images is viewed;

a step of obtaining a three-dimensional shape of the object based on the images; and

a step of generating an image of the object viewed from the virtual viewpoint based on the obtained three-dimensional shape of the object,

the step of obtaining the three-dimensional shape of the object comprising;

a step of setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a step of determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a step of determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a step of determining focusing degrees of the projection points based on focusing degrees of the corresponding points corresponding to the projection points;

a step of calculating, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that the object exists at a distance corresponding to a position of the projection point based on the focusing degree of each projection point;

the step of generating the image of the object viewed from the virtual viewpoint comprising;

mixing color information or brightness information of the projection points overlapping when viewed from the virtual viewpoint in a ratio corresponding to the existence probability to determine color information or brightness information of each point of the image to be generated.

29. A computer readable recording medium storing the image generation program as claimed in claim 28.

30. An image generation method comprising;

a step of obtaining images of an object taken under different conditions;

a step of obtaining a three-dimensional shape of the object based on the images; and

a step of generating an image of the object viewed from a viewpoint of an observer based on the obtained three-dimensional shape of the object,

the step of obtaining the three-dimensional shape of the object comprising;

a step of setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a step of determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a step of determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a step of determining, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that a surface of the object exists at the projection point;

the step of determining the existence probability comprising;

a step of calculating an evaluation reference value of each projection point from image information of the corresponding points;

a step of performing statistical processing on the evaluation reference value of each projection point; and

a step of calculating the existence probability of each projection point based on the evaluation reference value on which the statistical processing has been performed.

31. The image generation method as claimed in claim 30, the step of obtaining the plural images comprising:

obtaining images by taking the object from different viewpoints;

the step of determining the existence probability comprising;

a step of obtaining correlation degrees among corresponding points corresponding to the projection points;

a step of calculating the evaluation reference value based on the correlation degree of each projection point;

a step of performing the statistical processing on the evaluation reference value; and

a step of calculating the existence probability of each projection point based on the evaluation reference value on which the statistical processing has been performed.

32. The image generation method as claimed in claim 30, the step of obtaining the plural images comprising:

obtaining the images of the object taken from a viewpoint by changing focusing distance;

the step of determining the existence probability comprising;

a step of calculating focusing degrees of the projection points from focusing degrees of the corresponding points corresponding to the projection points;

a step of calculating the evaluation reference value based on the focusing degree of each projection point;

a step of performing the statistical processing on the evaluation reference value; and

a step of calculating the existence probability of each projection point based on the evaluation reference value on which the statistical processing has been performed.

33. The image generation method as claimed in claim 30, the step of obtaining the plural images comprising:

obtaining images of the object taken from plural viewpoints, and images of the object taken from equal to one or more viewpoints among the plural viewpoints by changing focusing distance;

the step of determining the existence probability comprising;

a step of obtaining correlation degrees between the projection points and corresponding points on the plural images of different viewpoints;

a step of obtaining a first evaluation reference value based on the correlation degrees of each projection point; and

a step of performing the statistical processing on the first evaluation reference value;

a step of calculating focusing degrees of the projection points based on focusing degrees of the corresponding points on the images of different focusing distances;

a step of calculating a second evaluation reference value based on the focusing degrees of each projection point; and

a step of calculating the existence probability of each projection point based on the first evaluation reference value and the second evaluation reference value on which the statistical processing has been performed.

34. The image generation method as claimed in any one of claims 30-33, the step of generating the image of the object viewed from the viewpoint of the observer comprising:

mixing color information or brightness information of the projection points overlapping when viewed from the viewpoint of the observer in a ratio of the existence probability to determine color information or brightness information of each point on the image to be generated so as to determine a two-dimensional image.

35. The image generation method as claimed in any one of claims 30-33, the step of generating the image of the object viewed from the viewpoint of the observer comprising:

a step of setting plural image generation planes at positions having different depths when viewed from the viewpoint of the observer; and

a step of converting the color information or brightness information and the existence probability of each projection point into color information or brightness information and a brightness distribution coefficient on each image generation plane based on position relationship among projection points overlapping viewed from the viewpoint of the observer and points on the image generation planes.

36. An image generation apparatus comprising:

an object image obtaining means for obtaining images of an object taken under different conditions;

an object shape obtaining means for obtaining a three-dimensional shape of the object based on the images; and

an object image generation means for generating an image of the object viewed from a viewpoint of an observer based on the obtained three-dimensional shape of the object,

the object shape obtaining means comprising;

a means for setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a means for determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a means for determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a means for determining, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that a surface of the object exists at the projection point;

the means for determining the existence probability comprising;

a means for calculating an evaluation reference value of each projection point from image information of the corresponding points;

a means for performing statistical processing on the evaluation reference value of each projection point; and

a means for calculating the existence probability of each projection point based on the evaluation reference value on which the statistical processing has been performed.

37. The image generation apparatus as claimed in claim 36, wherein the object image obtaining means obtains images of the object taken from different viewpoints;

the means for determining the existence probability comprising;

a means for obtaining correlation degrees among corresponding points corresponding to the projection points;

a means for calculating the evaluation reference value based on the correlation degree of each projection point;

a means for performing the statistical processing on the evaluation reference value; and

a means for calculating the existence probability of each projection point based on the evaluation reference value on which the statistical processing has been performed.

38. The image generation apparatus as claimed in claim 36, wherein the object image obtaining means obtains the images of the object taken from a viewpoint by changing focusing distance;

the means for determining the existence probability comprising;

a means for calculating focusing degrees of the projection points from focusing degrees of the corresponding points corresponding to the projection points;

a means for calculating the evaluation reference value based on the focusing degree of each projection point;

a means for performing the statistical processing on the evaluation reference value; and

a means for calculating the existence probability of each projection point based on the evaluation reference value on which the statistical processing has been performed.

39. The image generation apparatus as claimed in claim 36, wherein the object image obtaining means obtains images of the object taken from plural viewpoints, and images of the object taken from equal to one or more viewpoints among the plural viewpoints by changing focusing distance;

the means for determining the existence probability comprising;

a means for obtaining correlation degrees between the projection points and corresponding points on the plural images having different projection points;

a means for obtaining a first evaluation reference value based on the correlation degrees of each projection point; and

a means for performing the statistical processing on the first evaluation reference value;

a means for calculating focusing degrees of the projection points based on focusing degrees of the corresponding points on the images of different focusing distances taken from a viewpoint;

a means for calculating a second evaluation reference value based on the focusing degrees of each projection point; and

a means for calculating the existence probability of each projection point based on the first evaluation reference value and the second evaluation reference value on which the statistical processing has been performed.

40. The image generation apparatus as claimed in any one of claims 36-39, the means for generating the image of the object viewed from the viewpoint of the observer comprising:

a means for mixing color information or brightness information of the projection points overlapping when viewed from the viewpoint of the observer in a ratio of the existence probability to determine color information or brightness information of each point on the image to be generated so as to determine a two-dimensional image.

41. The image generation apparatus as claimed in any one of claims 36-39, the means for generating the image of the object viewed from the viewpoint of the observer comprising:

a means for setting plural image generation planes at positions having different depths when viewed from the viewpoint of the observer; and

a means for converting the color information or brightness information and the existence probability of each projection point into color information or brightness information and a brightness distribution coefficient on each image generation plane based on position relationship among projection points overlapping viewed from the viewpoint of the observer and points on the image generation planes.

42. An image generation program causing a computer to perform:

a step of obtaining images of an object taken under different conditions;

a step of obtaining a three-dimensional shape of the object based on the images; and

a step of generating an image of the object viewed from a viewpoint of an observer based on the obtained three-dimensional shape of the object,

the step of obtaining the three-dimensional shape of the object comprising;

a step of setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a step of determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a step of determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a step of determining, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that a surface of the object exists at the projection point;

the step of determining the existence probability comprising;

a step of calculating an evaluation reference value of each projection point from image information of the corresponding points;

a step of performing statistical processing on the evaluation reference value of each projection point; and

a step of calculating the existence probability of each projection point based on the evaluation reference value on which the statistical processing has been performed.

43. A computer readable recording medium storing the image generation program as claimed in claim 42.

44. A three-dimensional image display method comprising:

a step of obtaining images of an object taken under different conditions;

a step of obtaining a three-dimensional shape of the object based on the images;

a step of setting a viewpoint position from which an observer observes plural image display planes existing at different depth positions when viewed from the observer;

a step of generating two-dimensional images to be displayed on each image display plane based on the obtained three-dimensional shape of the object; and

a step of presenting a three-dimensional image of the object by displaying the generated two-dimensional images on each display plane,

the step of obtaining the three-dimensional shape of the object comprising;

a step of setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a step of determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a step of determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a step of determining, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that a surface of the object exists at the projection point;

the step of generating the two-dimensional images comprising;

converting the color information or the brightness information and the existence probability of the projection point into color information or brightness information and existence probability of each display point that is a point on the image display plane corresponding to the projection plane on which the projection point exists so as to generate the two dimensional images;

the step of presenting the three-dimensional image of the object comprising;

displaying the color information or brightness information on each display point with brightness according to the existence probability.

45. The three-dimensional image display method as claimed in claim 44, the step of obtaining the plural images comprising:

obtaining images of the object taken from different viewpoints,

the step of determining the existence probability comprising;

a step of calculating correlation degrees among corresponding points corresponding to the projection points;

a step of determining, for projection points overlapping when viewed from the reference viewpoint, the existence probability of each projection point based on the correlation degree of each projection point.

46. The three-dimensional image display method as claimed in claim 44, the step of obtaining the plural images comprising:

obtaining images of the object taken from a viewpoint by changing focusing distance,

the step of determining the existence probability comprising;

a step of calculating focusing degrees of the projection points from focusing degrees of corresponding points corresponding to the projection points;

a step of determining, for projection points overlapping when viewed from the reference viewpoint, the existence probability of each projection point based on the focusing degree of each projection point.

47. The three-dimensional image display method as claimed in claim 44, the step of obtaining the plural images comprising:

obtaining images of the object taken from plural viewpoints, and images of the object taken from equal to one or more viewpoints among the plural viewpoints by changing focusing distance;

the step of determining the existence probability comprising;

a step of obtaining correlation degrees between the projection points and corresponding points on the plural images of different viewpoints;

a step of calculating focusing degrees of the projection points based on focusing degrees of the corresponding points on the images, having different focusing distance, of each viewpoint; and

a step of determining, for projection points overlapping when viewed from the reference viewpoints, the existence probability of each projection point based on the correlation degree and the focusing degree of each projection point.

48. A three-dimensional image display apparatus comprising:

an object image obtaining means for obtaining images of an object taken under different conditions;

a three-dimensional shape obtaining means for obtaining a three-dimensional shape of the object based on the images;

an observer viewpoint setting means for setting a viewpoint position from which an observer observes plural image display planes existing at different depth positions when viewed from the observer;

a two-dimensional image generation means for generating two-dimensional images to be displayed on each image display plane based on the obtained three-dimensional shape of the object; and

wherein the three-dimensional image display apparatus presents a three-dimensional image of the object by displaying the generated two-dimensional images on each display plane,

the three-dimensional shape obtaining means comprising;

a means for setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a means for determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a means for determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a means for determining, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that a surface of the object exists at the projection point;

the two-dimensional image generation means comprising;

a means for converting the color information or the brightness information and the existence probability of the projection point into color information or brightness information and existence probability of each display point that is a point on the image display plane corresponding to the projection plane on which the projection point exists to generate the two dimensional images;

wherein the three-dimensional image display apparatus displays the color information or brightness information on each display point with brightness according to the existence probability.

49. The three-dimensional image display apparatus as claimed in claim 48, wherein the object image obtaining means is a means for obtaining images of the object taken from different viewpoints,

the means for determining the existence probability comprising:

a means for calculating correlation degrees among corresponding points corresponding to the projection points;

a means for determining, for projection points overlapping when viewed from the reference viewpoint, the existence probability of each projection point based on the correlation degree of each projection point.

50. The three-dimensional image display apparatus as claimed in claim 48, wherein the object image obtaining means is a means for obtaining images of the object taken from a viewpoint by changing focusing distance,

the means for determining the existence probability comprising:

a means for calculating focusing degrees of the projection points from focusing degrees of corresponding points corresponding to the projection points; and

a means for determining, for projection points overlapping when viewed from the reference viewpoint, the existence probability of each projection point based on the focusing degree of each projection point.

51. The three-dimensional image display apparatus as claimed in claim 48, wherein the object image obtaining means is a means for obtaining images of the object taken from plural viewpoints, and images of the object taken from equal to one or more viewpoints among the plural viewpoints by changing focusing distance;

the means for determining the existence probability comprising;

a means for obtaining correlation degrees between the projection points and corresponding points on the plural images of different viewpoints;

a means for calculating focusing degrees of the projection points based on focusing degrees of the corresponding points on the images, having different focusing length, of each viewpoint; and

a means for determining, for projection points overlapping when viewed from the reference viewpoints, the existence probability of each projection point based on the correlation degree and the focusing degree of each projection point.

52. A three-dimensional image display program causing a computer to perform:

a step of obtaining images of an object taken under different conditions;

a step of obtaining a three-dimensional shape of the object based on the images;

a step of setting a viewpoint position from which an observer observes plural image display planes existing at different depth positions when viewed from the observer;

a step of generating two-dimensional images to be displayed on each image display plane based on the obtained three-dimensional shape of the object; and

a step of presenting a three-dimensional image of the object by displaying the generated two-dimensional images on each display plane,

the step of obtaining the three-dimensional shape of the object comprising;

a step of setting projection planes having a multi-layered structure in a virtual three-dimensional space;

a step of determining a reference viewpoint for obtaining the three-dimensional shape of the object;

a step of determining color information or brightness information of projection points, that are points on the projection planes, based on color information or brightness information of corresponding points, on the obtained images, corresponding to the projection points;

a step of determining, for each of the projection points overlapping when viewed from the reference viewpoint, an existence probability that is a probability that a surface of the object exists at the projection point;

the step of generating the two-dimensional images comprising;

a step of converting the color information or the brightness information and the existence probability of the projection point into color information or brightness information and existence probability of each display point that is a point on the image display plane corresponding to the projection plane on which the projection point exists to generate the two dimensional images;

the step of presenting the three-dimensional image of the object comprising;

a step of displaying the color information or brightness information on each display point with brightness according to the existence probability.

53. A computer readable recording medium storing the three-dimensional image generation program as claimed in claim 52.