Digital Refocusing for Wide-Angle Images Using Axial-Cone Cameras

• US 20110316968A1
• Filed: 06/29/2010
• Published: 12/29/2011
• Est. Priority Date: 06/29/2010
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Taguchi, Yuichi, Agrawal, Amit K., Veeraraghavan, Ashok N., Ramalingam, Srikumar, Raskar, Ramesh
• Granted Patent: US 8,493,432 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A single camera acquires an input image of a scene as observed in an array of spheres, wherein pixels in the input image corresponding to each sphere form a sphere image. A set of virtual cameras are defined for each sphere on a line joining a center of the sphere and a center of projection of the camera, wherein each virtual camera has a different virtual viewpoint and an associated cone of rays, appearing as a circle of pixels on its virtual image plane. A projective texture mapping of each sphere image is applied to all of the virtual cameras on the virtual image plane to produce a virtual camera image comprising circle of pixels. Each virtual camera image for each sphere is then projected to a refocusing geometry using a refocus viewpoint to produce a wide-angle lightfield view, which are averaged to produce a refocused wide-angle image.

Claims

1. A method for obtaining a refocused wide-angle image of a scene, comprising:

acquiring, by a single stationary camera, an input image of a scene as observed in an array of spheres, wherein pixels in the input image corresponding to each sphere comprise a sphere image;

defining, for each sphere, a set of virtual cameras on a line joining a center of the sphere and a center of projection of the camera, wherein each virtual camera has a virtual image plane;

projecting, for each virtual camera of each sphere, the sphere image to the virtual image plane;

selecting, for each virtual camera of each sphere, a set of pixels corresponding to a cone of rays associated with the virtual camera to produce a virtual camera image;

projecting, for each sphere, all of the virtual camera images to a refocusing geometry assuming a refocus viewpoint to produce a view, wherein the view represents a wide-angle lightfield view;

averaging the views for all the spheres to produce the refocused wide-angle image; and

displaying the refocused wide-angle image on an output device.

2. The method of claim 1, wherein the projecting of the sphere images and the virtual camera images is performed by using projective texture mapping on a graphic processing unit (GPU).

3. The method of claim 1, wherein the views are generated for the refocusing geometry placed at multiple depths to estimate a wide field-of-view (FOV) depth map by evaluating a variance across the wide-angle views at each pixel.

4. The method of claim 3, wherein the projecting of all of the virtual camera images is performed for the refocusing geometry placed at multiple depths to generate a focal stack.

5. The method of claim 4, wherein the wide FOV depth map is estimated by evaluating a sharpness measure of the refocused images in the focal stack at each pixel.

6. The method of claim 3, wherein the wide FOV depth map is estimated by evaluating a consistency measure across all the views at each pixel.

7. The method of claim 3, wherein the wide FOV depth map is used as the refocusing geometry to obtain an all-in-focus output image.

8. The method of claim 3, wherein the refocusing geometry is modified according to the wide FOV depth map to bring multiple depths into focus simultaneously.

9. The method of claim 1, further comprising:

specifying the refocusing geometry and the refocus viewpoint by the user.

10. The method of claim 9, wherein the specifying is interactive.

11. The method of claim 1, wherein each sphere image is modeled as multiple axial-cone cameras, wherein each axial-cone camera has a different virtual viewpoint and an associated cone of rays.

12. The method of claim 1, wherein the camera is a video camera to acquire multiple input images of the array of spheres.

13. The method of claim 12, wherein each input image is processed independently.

14. The method of claim 1, wherein the array is regular.

15. The method of claim 1, wherein the array is irregular.

16. The method of claim 1, wherein the array is random.

17. The method of claim 1, wherein the array is two-dimensional.

18. The method of claim 1, wherein the array is three-dimensional space.

19. The method of claim 1, wherein a number of spheres is greater than one.

20. The method of claim 1, wherein the spheres are reflective.

21. The method of claim 1, wherein the spheres are refractive.

22. The method of claim 21, wherein a refractive index of the spheres is equal to that of glass or an acrylic.

23. An apparatus for obtaining a refocused wide-angle image of a scene, comprising:

an array of spheres;

a single camera configured to acquire an input image of a scene as observed in the array of spheres, wherein pixels in the input image corresponding to each sphere comprise a sphere image;

means for defining, for each sphere, a set of virtual cameras on a line joining a center of the sphere and a center of projection of the camera, wherein each virtual camera has a virtual image plane;

means for projecting, for each virtual camera of each sphere, the sphere image to the virtual image plane;

means for selecting, for each virtual camera of each sphere, a set of pixels corresponding to a cone of rays associated with the virtual camera to produce a virtual camera image;

means for projecting, for each sphere, all of the virtual camera images to a refocusing geometry assuming a refocus viewpoint to produce a view, wherein the view represents a wide-angle lightfield view;

means for averaging the views for all the spheres to produce the refocused wide-angle image; and

an output device configured to display the refocused wide-angle image.

24. The apparatus of claim 23, wherein the spheres are reflective.

25. The apparatus of claim 23, wherein the spheres are refractive.

26. The apparatus of claim 23, wherein the means for projecting is a graphic processing unit.