Method and apparatus for reducing distortion in images

• US 20040012544A1
• Filed: 08/01/2003
• Published: 01/22/2004
• Est. Priority Date: 07/21/2000
• Status: Active Grant

• Current Assignee: Michael Grossberg, Rahul Swaminathan, Shree Nayar
• Original Assignee: Michael Grossberg, Rahul Swaminathan, Shree Nayar
• Inventors: Grossberg, Michael, Nayar, Shree, Swaminathan, Rahul
• Granted Patent: US 7,245,761 B2
• Time in Patent Office: Days
• US Class Current: 345/32
• CPC Class Codes: G02B 13/06 Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
• CPC Class Codes: G02B 27/0025 for optical correction, e.g. distorsion, aberration
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An algorithm is provided for reducing distortion in an image captured by a camera or other imaging system having more than one optical viewpoint. The algorithm can use a captured image, statistical information regarding the depths of scene objects, and information regarding the geometrical characteristics of the imaging system to generate an approximation of a perspective projection representing the scene. The virtual viewpoint of the approximate perspective projection is selected so as to reduce distortion. The resulting image has less distortion than that of an image generated under an assumption that scene objects are infinitely distant. In addition, the parameters of any distortion reducing function can be optimized by minimizing an objective function representing the amount of image distortion.

Claims

1. A method for imaging, comprising:

receiving an image generated by an image-sensing system having a plurality of viewpoints, the image being generated based upon radiation rays received by the image-sensing system, the radiation rays coming from a scene;

using first information for selecting a virtual viewpoint for a projection representing the scene, the first information comprising information regarding a statistical distribution associated with at least one depth value of the scene, the virtual viewpoint being selected for reducing distortion of the projection; and

using the image, the virtual viewpoint, and second information to generate the projection, the second information comprising information regarding at least one geometrical characteristic of the image-sensing system.

2. A method according to claim 1, further comprising using the first information to generate the projection.

3. A method according to claim 1, further comprising using the second information to select the virtual viewpoint.

4. A method according to claim 3, wherein the step of using the image, the virtual viewpoint, and the second information comprises:

using the second information to determine a portion of a ray surface of the image-sensing system, the portion of the ray surface having first and second ray surface points, the first ray surface point being associated with a first pixel of the image, and being further associated with a first ray passing through a first scene region represented by the first pixel, the first ray further passing through the first ray surface point, the second ray surface point being associated with a second pixel of the image, and being further associated with a second ray passing through a second scene region represented by the second pixel, the second ray further passing through the second ray surface point;

using the virtual viewpoint, the first ray, the first pixel, and the first information to determine a first estimated location within the projection, the first estimated location corresponding to a location of the first scene region;

using the virtual viewpoint, the second ray, the second pixel, and the first information to determine a second estimated location within the projection, the second estimated location corresonding to a location of the second scene region, wherein the virtual viewpoint is selected for minimizing respective uncertainties of the first and second estimated locations, each one of the respective uncertainties representing one of;

(a) a respective maximum size of first and second image regions within the projection, the first and second image regions corresponding respectively to the first and second scene regions, the first image region comprising a set of possible locations of the first estimated location, the second image region comprising a set of possible locations of the second estimated location, (b) a respective mean size of the first and second image regions, and (c) a total size of the first and second image regions; and

using the first and second estimated locations and the image to generate the projection.

5. A method according to claim 1, wherein the at least one geometrical characteristic of the imaging system comprises one of:

a spherical reflector characteristic;

a hyperbolic reflector characteristic; and

a parabolic reflector characteristic.

6. A method according to claim 1, wherein the projection is a perspective projection.

7. A method according to claim 1, wherein the projection represents a first portion of the scene and the virtual viewpoint is also for a further projection representing a second portion of the scene, the virtual viewpoint being further selected for reducing distortion of the further projection, and the method further comprising using the image, the virtual viewpoint, and the second information to generate the further projection.

8. A method according to claim 1, wherein the image-sensing system comprises at least two cameras.

9. A method for imaging, comprising:

receiving an image generated by an image-sensing system having a 5 plurality of viewpoints, the image being generated based upon radiation rays received by the image-sensing system, the radiation rays coming from a scene;

using first information to determine a portion of a caustic of the image-sensing system, the first information comprising information regarding at least one geometrical characteristic of the image-sensing system;

determining an average location of the portion of the caustic;

selecting the average location as a first virtual viewpoint for a projection representing the scene; and

using the image, the first virtual viewpoint, and the first information for generating the projection.

10. A method according to claim 9, further comprising using second information to perform an adjustment of the first virtual viewpoint, for generating a second virtual viewpoint for a further projection representing the scene, the second information comprising information regarding a distribution associated with at least one depth value of the scene, the projection having a first amount of distortion, the further projection having a second amount of distortion, the first amount of distortion being greater than the second amount of distortion.

11. A method according to claim 9, wherein the step of using the image, the first virtual viewpoint, and the first information comprises:

using the image and the first information to determine first and second rays, the first ray being associated with a first pixel of the image, and passing through a first scene region represented by the first pixel, the second ray being associated with a second pixel of the image, and passing through a second scene region represented by the second pixel;

using the virtual viewpoint, the first ray, the first pixel, and second information to determine a first estimated location within the projection, the first estimated location corresponding to the first pixel, and the second information comprising information regarding a statistical distribution associated with at least one depth value of the scene;

using the virtual viewpoint, the second ray, the second pixel, and the second information to determine a second estimated location within the projection, the second estimated location corresponding to the second pixel; and

using the first and second estimated locations and the image to generate the projection.

12. A method according to claim 9, wherein the at least one geometrical characteristic of the imaging system comprises one of:

a spherical reflector characteristic;

a hyperbolic reflector characteristic; and

a parabolic reflector characteristic.

13. A method according to claim 9, wherein the projection is a perspective projection.

19. An apparatus according to claim 12, wherein the projection is a perspective projection.

26. An apparatus according to claim 19, wherein the projection is a perspective projection.

14. An imaging apparatus, comprising:

an image-sensing system for generating an image based upon radiation rays received by the image-sensing system, the radiation rays coming from a scene, the image-sensing system having a plurality of viewpoints;

a virtual viewpoint processor for using first information to select a virtual viewpoint for a projection representing the scene, the first information comprising information regarding a statistical distribution associated with at least one depth value of the scene, and the virtual viewpoint being selected for reducing distortion of the projection; and

an image processor for using the image, the virtual viewpoint, and second information to generate the projection, the second information comprising information regarding at least one geometrical characteristic of the image-sensing system.

15. An apparatus according to claim 14, wherein the image processor comprises a processor for using the first information to generate the projection.

16. An apparatus according to claim 14, wherein the virtual viewpoint processor comprises a processor for using the second information to select the virtual viewpoint.

17. An apparatus according to claim 16, wherein the image processor comprises:

a first processor for using the second information to determine a portion of a ray surface of the image-sensing system, the portion of the ray surface having first and second ray surface points, the first ray surface point being associated with a first pixel of the image, and being further associated with a first ray passing through a first scene region represented by the first pixel, the first ray further passing through the first ray surface point, the second ray surface point being associated with a second pixel of the image, and being further associated with a second ray passing through a second scene region represented by the second pixel, the second ray further passing through the second ray surface point;

a second processor for using the virtual viewpoint, the first ray, the first pixel, and the first information to determine a first estimated location within the projection, the first estimated location corresponding to a location of the first scene region;

a third processor for using the virtual viewpoint, the second ray, the second pixel, and the first information to determine a second estimated location within the projection, the second estimated location corresponding to a location of the second scene region, wherein the virtual viewpoint processor comprises a fourth processor for selecting the virtual viewpoint for minimizing respective uncertainties of the first and second estimated locations, each one of the respective uncertainties representing one of;

(a) a respective maximum size of first and second image regions within the projection, the first and second image regions corresponding respectively to the first and second scene regions, the first image region comprising a set of possible locations of the first estimated location, the second image region comprising a set of possible locations of the second estimated location, (b) a respective mean size of the first and second image regions, and (c) a total size of the first and second image regions; and

a fifth processor for using the first and second estimated locations and the image to generate the projection.

18. An apparatus according to claim 14, wherein the at least one geometrical characteristic of the imaging system comprises one of:

a spherical reflector characteristic;

a hyperbolic reflector; and

a parabolic reflector characteristic.

20. An apparatus according to claim 14, wherein the projection represents a first portion of the scene, and the virtual viewpoint is also for a further projection representing a second portion of the scene, the virtual viewpoint processor comprising a first processor for selecting the virtual viewpoint for reducing distortion of the further projection, and the image processor comprising a second processor for using the image, the virtual viewpoint, and the second information to generate the further projection.

21. An apparatus according to claim 14, wherein the image-sensing system comprises at least two cameras.

22. An imaging apparatus, comprising:

an image-sensing system for generating an image based upon radiation rays received by the image-sensing system, the radiation rays coming from a scene, and the image-sensing system having a plurality of viewpoints;

a caustic determining processor for using first information to determine a portion of a caustic of the image-sensing system, the first information comprising information regarding at least one geometrical characteristic of the image-sensing system;

a caustic averaging processor for determining an average location of the portion of the caustic;

a virtual viewpoint processor for selecting the average location as a first virtual viewpoint for a projection representing the scene; and

an image processor for using the image, the first virtual viewpoint, and the first information for generating the projection.

23. An apparatus according to claim 22, further comprising a virtual viewpoint adjustment processor for using second information to perform an adjustment of the first virtual viewpoint, for generating a second virtual viewpoint for a further projection representing the scene, the second information comprising information regarding a distribution associated with at least one depth value of the scene, the projection having a first amount of distortion, the further projection having a second amount of distortion, the first amount of distortion being greater than the second amount of distortion.

24. An apparatus according to claim 22, wherein the image processor comprises:

a first processor for using the image and the first information to determine first and second rays, the first ray being associated with a first pixel of the image, and passing through a first scene region represented by the first pixel, the second ray being associated with a second pixel of the image, and passing through a second scene region represented by the second pixel;

a second processor for using the virtual viewpoint, the first ray, the first pixel, and second information to determine a first estimated location within the projection, the first estimated location corresponding to the first pixel, and the second information comprising information regarding a statistical distribution associated with at least one depth value of the scene;

a third processor for using the virtual viewpoint, the second ray, the second pixel, and the second information to determine a second estimated location within the projection, the second estimated location corresponding to the second pixel; and

a fourth processor for using the first and second estimated locations and the image to generate the projection.

25. An apparatus according to claim 22, wherein the at least one geometrical characteristic of the imaging system comprises one of:

a spherical reflector characteristic;

a hyperbolic reflector characteristic; and

a parabolic reflector characteristic.

27. A computer-readable medium having a set of instructions operable to direct a processor to perform the steps of:

receiving an image generated by an image-sensing system having a plurality of viewpoints, the image being generated based upon radiation rays received by the image-sensing system, the radiation rays coming from a scene;

using first information for selecting a virtual viewpoint for a projection representing the scene, the first information comprising information regarding a statistical distribution associated with at least one depth value of the scene, the virtual viewpoint being selected for reducing distortion of the projection; and

using the image, the virtual viewpoint, and second information to generate the projection, the second information comprising information regarding at least one geometrical characteristic of the image-sensing system.

28. A computer-readable medium according to claim 27, wherein the set of instructions further directs the processor to use the first information to generate the projection.

29. A computer-readable medium according to claim 27, wherein the set of instructions further directs the processor to use the second information to select the virtual viewpoint.

30. A computer-readable medium according to claim 29, wherein the step of using the image, the virtual viewpoint, and the second information comprises:

using the second information to determine a portion of a ray surface of the image-sensing system, the portion of the ray surface having first and second ray surface points, the first ray surface point being associated with a first pixel of the image, and being further associated with a first ray passing through a first scene region represented by the first pixel, the first ray further passing through the first ray surface point, the second ray surface point being associated with a second pixel of the image, and being further associated with a second ray passing through a second scene region represented by the second pixel, the second ray further passing through the second ray surface point;

using the virtual viewpoint, the first ray, the first pixel, and the first information to determine a first estimated location within the projection, the first estimated location corresponding to a location of the first scene region;

using the virtual viewpoint, the second ray, the second pixel, and the first information to determine a second estimated location within the projection, the second estimated location corresponding to a location of the second scene region, wherein the virtual viewpoint is selected for minimizing respective uncertainties of the first and second estimated locations, each one of the respective uncertainties representing one of;

(a) a respective maximum size of first and second image regions within the projection, the first and second image regions corresponding respectively to the first and second scene regions, the first image region comprising a set of possible locations of the first estimated location, the second image region comprising a set of possible locations of the second estimated location, (b) a respective mean size of the first and second image regions, and (c) a total size of the first and second image regions; and

using the first and second estimated locations and the image to generate the projection.

31. A computer-readable medium according to claim 27, wherein the at least one geometrical characteristic of the imaging system comprises one of:

a spherical reflector characteristic;

a hyperbolic reflector characteristic; and

a parabolic reflector characteristic.

32. A computer-readable medium according to claim 27, wherein the projection is a perspective projection.

33. A computer-readable medium according to claim 27, wherein the projection represents a first portion of the scene, and the virtual viewpoint is also for a further projection representing a second portion of the scene, the virtual viewpoint being further selected for reducing distortion of the further projection, and the set of instructions further directing the processor to use the image, the virtual viewpoint, and the second information to generate the further projection.

34. A computer-readable medium according to claim 27, wherein the image-sensing system comprises at least two cameras.

35. A computer-readable medium having a set of instructions operable to direct a processor to perform the steps of:

receiving an image generated by an image-sensing system having a plurality of viewpoints, the image being generated based upon radiation rays received by the image-sensing system, the radiation rays coming from a scene;

using first information to determine a portion of a caustic of the image-sensing system, the first information comprising information regarding at least one geometrical characteristic of the image-sensing system;

determining an average location of the portion of the caustic;

selecting the average location as a first virtual viewpoint for a projection representing the scene; and

using the image, the first virtual viewpoint, and the first information for generating the projection.

36. A computer-readable medium according to claim 35, wherein the set of instructions further directs the processor to use second information to perform an adjustment of the first virtual viewpoint, for generating a second virtual viewpoint for a further projection representing the scene, the second information comprising information regarding a distribution associated with at least one depth value of the scene, the projection having a first amount of distortion, the further projection having a second amount of distortion, and the first amount of distortion being greater than the second amount of distortion.

37. A computer-readable medium according to claim 35, wherein the step of using the image, the first virtual viewpoint, and the first information comprises:

using the image and the first information to determine first and second rays, the first ray being associated with a first pixel of the image, and passing through a first scene region represented by the first pixel, the second ray being associated with a second pixel of the image, and passing through a second scene region represented by the second pixel;

using the virtual viewpoint, the first ray, the first pixel, and second information to determine a first estimated location within the projection, the first estimated location corresponding to the first pixel, and the second information comprising information regarding a statistical distribution associated with at least one depth value of the scene;

using the virtual viewpoint, the second ray, the second pixel, and the second information to determine a second estimated location within the projection, the second estimated location corresponding to the second pixel; and

using the first and second estimated locations and the image to generate the projection.

38. A computer-readable medium according to claim 35, wherein the at least one geometrical characteristic of the imaging system comprises one of:

a spherical reflector characteristic;

a hyberbolic reflector characteristic; and

a parabolic reflector characteristic.

39. A computer-readable medium according to claim 35, wherein the projection is a perspective projection.