INTERACTIVE REFOCUSING OF ELECTRONIC IMAGES

• US 20140013273A1
• Filed: 09/10/2013
• Published: 01/09/2014
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Ng, Yi-Ren
• Granted Patent: US 9,530,195 B2
• Time in Patent Office: Days
• US Class Current: 715/802
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/20108 Interactive selection of 2D slice in a 3D data set
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H01L 27/14627 Microlenses

Abstract

A method is performed to refocus a digital photographic image comprising a plurality of pixels. In the method, a set of images is computed corresponding to the digital photographic image and focused at different depths. Refocus depths for at least a subset of the pixels are identified and stored in a look-up table. At least a portion of the digital photographic image is refocused at a desired refocus depth determined from the look-up table.

Claims

1. A method of refocusing a digital photographic image comprising a plurality of pixels, the method comprising:

computing a set of images corresponding to a scene in the digital photographic image, wherein each image comprises an image of the scene at a distinct focal depth;

identifying refocus depths for at least a subset of the pixels;

storing the refocus depths in a look-up table; and

refocusing at least a portion of the digital photographic image at a desired refocus depth determined from the look-up table.

2. The method of claim 1, wherein the different depths span a range of depths including closest and furthest distances to refocus.

3. The method of claim 2, wherein the range of depths corresponds to a depth of focus of a camera recording a light field corresponding to the set of images.

4. The method of claim 3, wherein the range of depths is extended beyond an effective depth of focus of the camera by a factor representing a tolerable mis-focus.

5. The method of claim 2, wherein the furthest distance is optical infinity.

6. The method of claim 2, wherein the closest distance corresponds to a closest object in a scene corresponding to the set of images, and wherein the furthest distance corresponds to a furthest object in the scene.

7. The method of claim 1, wherein the identifying operation comprises determining refocus depths for each pixel in the plurality of pixels.

8. The method of claim 1, wherein the identifying operation comprises determining refocus depths for respective regions in the digital photographic image.

9. The method of claim 1, wherein determining the refocus depth for a pixel includes searching other pixels surrounding the pixel until a pixel with a known refocus depth is found.

10. The method of claim 1, further comprising downsampling the look-up table.

11. The method of claim 10, wherein the downsampling operation includes a voting procedure wherein pixel votes are weighted.

12. The method of claim 1, wherein the desired refocus depth corresponds to a first refocus depth associated with a pixel at a user-specified point in the displayed image.

13. The method of claim 12, wherein a region of the displayed image that includes the user-specified point is refocused.

14. The method of claim 13, wherein the region is user-defined.

15. The method of claim 13, wherein the region corresponds to an object in the displayed image.

16. The method of claim 13, wherein a tag is associated with the region.

17. The method of claim 1, wherein the refocusing operation includes generating an animation comprising a series of images in the set having depths between a depth of the displayed image and the desired refocus depth.

18. The method of claim 1, wherein the refocusing operation includes blending at least a portion of two images focused at depths near to the desired refocus depth.

19. The method of claim 1, further comprising storing the set of images and the look-up table in a single file.

20. The method of claim 19, wherein the single file is one of a zip archive and a swf (Flash) file.

21. The method of claim 1, wherein computing a set of images corresponding to a scene comprises:

receiving a light field image comprising light field data;

determining a first depth corresponding to a nearest object in the scene;

determining a second depth corresponding to a farthest object in the scene;

digitally refocusing the light field image to generate a plurality of refocused images, each having a focus depth lying within the range from the first depth to the second depth;

storing the generated plurality of refocused images in a storage device;

for each of a plurality of two-dimensional positions in the light field image;

determining which of the plurality of refocused images is in focus at the two-dimensional position; and

storing, in the look-up table, an index entry associating the two-dimensional position with the identified refocused image;

and wherein refocusing at least a portion of the digital photographic image at a desired refocus depth comprises;

displaying one of the refocused images, the displayed image having a first focus depth;

receiving user input to select a two-dimensional position within the displayed image; and

in response to the received input;

retrieving, from the look-up table, an index entry identifying a stored refocused image corresponding to the selected two-dimensional position, the stored refocused image having a second focus depth different from the first focus depth;

retrieving the identified refocused image from the storage device; and

displaying the retrieved refocused image having the second focus depth.

22. The method of claim 21, wherein digitally refocusing the light field image to generate a plurality of refocused images comprises:

scoring the set of images using a metric based on image content; and

removing at least one image from the set, based on the metric.

23. A system for refocusing a digital photographic image comprising a plurality of pixels, the system comprising:

a processor, configured to perform the operations of;

computing a set of images corresponding to a scene in the digital photographic image, wherein each image comprises an image of the scene at a distinct focal depth; and

identifying refocus depths for at least a subset of the pixels;

a storage device, communicatively coupled to the processor, configured to store the refocus depths in a look-up table; and

a refocusing module, communicatively coupled to the storage device, configured to refocus at least a portion of the digital photographic image at a desired refocus depth determined from the look-up table.

24. The system of claim 23, wherein the different depths span a range of depths including closest and furthest distances to refocus.

25. The system of claim 24, wherein the closest distance corresponds to a closest object in a scene corresponding to the set of images, and wherein the furthest distance corresponds to a furthest object in the scene.

26. The system of claim 23, wherein the identifying operation comprises determining refocus depths for each pixel in the plurality of pixels.

27. The system of claim 23, wherein the identifying operation comprises determining refocus depths for respective regions in the digital photographic image.

28. The system of claim 23, wherein the storage device stores a downsampled look-up table.

29. The system of claim 23, wherein the desired refocus depth corresponds to a first refocus depth associated with a pixel at a user-specified point in the displayed image.

30. The system of claim 23, further comprising:

a sensor, communicatively coupled to the processor, configured to receive a light field image comprising light field data;

a display, communicatively coupled to the processor; and

an input device, communicatively coupled to the processor;

and wherein;

the processor is configured to compute the set of images corresponding to a scene by;

determining a first depth corresponding to a nearest object in the scene;

determining a second depth corresponding to a farthest object in the scene; and

digitally refocusing the light field image to generate a plurality of refocused images, each having a focus depth lying within the range from the first depth to the second depth; and

the storage device is configured to store the generated plurality of refocused images in a storage device;

and wherein;

for each of a plurality of two-dimensional positions in the light field image, the processor is further configured to determine which of the plurality of refocused images is in focus at the two-dimensional position; and

the storage device is configured to store, in the look-up table, an index entry associating the two-dimensional position with the identified refocused image;

the display is configured to display one of the refocused images, the displayed image having a first focus depth; and

the input device is configured to receive user input to select a two-dimensional position within the displayed image;

and wherein the refocusing module is configured to refocus at least a portion of the digital photographic image at a desired refocus depth by, in response to the received input;

retrieving, from the look-up table, an index entry identifying a stored refocused image corresponding to the selected two-dimensional position, the stored refocused image having a second focus depth different from the first focus depth;

retrieving the identified refocused image from the storage device; and

causing the display device to display the retrieved refocused image having the second focus depth.

31. The system of claim 30, wherein the processor is configured to digitally refocus the light field image to generate a plurality of refocused images by:

scoring the set of images using a metric based on image content; and

removing at least one image from the set, based on the metric.