Interactive refocusing of electronic images

• US 9,530,195 B2
• Filed: 09/10/2013
• Issued: 12/27/2016
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Ng, Yi-Ren
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): BLOOM, NATHAN J
• Application Number:
  US14/022,651
• Publication Number:
  US 20140013273A1
• Time in Patent Office: 1,204 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/20108 Interactive selection of 2D slice in a 3D data set
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H01L 27/14627 Microlenses

Abstract

A method is performed to refocus a digital photographic image comprising a plurality of pixels. In the method, a set of images is computed corresponding to the digital photographic image and focused at different depths. Refocus depths for at least a subset of the pixels are identified and stored in a look-up table. At least a portion of the digital photographic image is refocused at a desired refocus depth determined from the look-up table.

Claims

1. A method of refocusing an image, the method comprising:

in a processor, obtaining a set of refocused images generated from a single captured light field image comprising a plurality of pixels and representing a scene, wherein each refocused image comprises an image of the scene at a distinct focal depth;

in the processor, determining refocus depths for at least a portion of the captured image;

storing the refocus depths in a look-up table; and

in the same or a different processor, refocusing at least a portion of the captured image at a desired refocus depth determined from the look-up table;

wherein determining refocus depths for at least a portion of the captured image comprises, for each of at least a subset of the pixels, determining the refocus depth for the pixel by searching other pixels surrounding the pixel until a pixel with a known refocus depth is found.

2. The method of claim 1, wherein the refocus depths span a range of depths including closest and furthest distances to refocus.

3. The method of claim 2, wherein the single captured light field image comprises an image captured by a light field capture device, and wherein the range of depths corresponds to a depth of focus of the light field capture device.

4. The method of claim 3, wherein the range of depths is extended beyond an effective depth of focus of the light field capture device by a factor representing a tolerable mis-focus.

5. The method of claim 2, wherein the furthest distance is optical infinity.

6. The method of claim 2, wherein the closest distance corresponds to a closest object in the scene, and wherein the furthest distance corresponds to a furthest object in the scene.

7. The method of claim 1, wherein refocus depths for at least a subset of pixels in the plurality of pixels.

8. The method of claim 1, wherein determining refocus depths comprises determining refocus depths for respective regions in the captured image.

9. The method of claim 1, further comprising downsampling the look-up table.

10. The method of claim 9, wherein the downsampling operation includes a voting procedure wherein pixel votes are weighted.

11. The method of claim 1, further comprising displaying the captured image on a display device, and wherein the desired refocus depth corresponds to a first refocus depth associated with a pixel at a user-specified location in the displayed image.

12. The method of claim 11, wherein refocusing at least a portion of the captured image comprises refocusing a region of the displayed image that includes the user-specified location.

13. The method of claim 12, wherein the region is user-defined.

14. The method of claim 12, wherein the region corresponds to an object in the displayed image.

15. The method of claim 12, wherein a tag is associated with the region.

16. The method of claim 1, further comprising displaying the captured image on a display device, and wherein refocusing at least a portion of the captured image comprises generating an animation comprising a series of refocused images in the set having depths between a depth of the displayed image and the desired refocus depth.

17. The method of claim 1, wherein refocusing at least a portion of the captured image comprises blending at least a portion of two refocused images in the set focused at depths near the desired refocus depth.

18. The method of claim 1, further comprising storing the set of refocused images and the look-up table in a single file.

19. The method of claim 18, wherein the single file comprises one of a zip archive and a swf (Flash) file.

20. The method of claim 1, wherein obtaining a set of refocused images comprises:

receiving a captured light field image comprising light field data representing a scene;

determining a first depth corresponding to a nearest object in the scene;

determining a second depth corresponding to a farthest object in the scene;

digitally refocusing the light field image to generate a plurality of refocused images, each having a focus depth lying within the range from the first depth to the second depth;

storing the generated plurality of refocused images in a storage device; and

for each of a plurality of two-dimensional positions in the light field image;

determining which of the plurality of refocused images is in focus at the two-dimensional position; and

storing, in the look-up table, an index entry associating the two-dimensional position with the identified refocused image;

and wherein refocusing at least a portion of the captured image at a desired refocus depth comprises;

displaying one of the refocused images, the displayed image having a first focus depth;

receiving user input to select a two-dimensional position within the displayed image; and

in response to the received input;

retrieving, from the look-up table, an index entry identifying a stored refocused image corresponding to the selected two-dimensional position, the stored refocused image having a second focus depth different from the first focus depth;

retrieving the identified refocused image from the storage device; and

displaying the retrieved refocused image having the second focus depth.

21. The method of claim 20, wherein digitally refocusing the light field image to generate a plurality of refocused images comprises:

scoring the set of images using a metric based on image content; and

removing at least one image from the set, based on the metric.

22. A system for refocusing an image, the system comprising:

a processor, configured to perform the operations of;

obtaining a set of refocused images generated from a single captured light field image comprising a plurality of pixels and representing a scene, wherein each refocused image comprises an image of the scene at a distinct focal depth; and

determining refocus depths for at least a portion of the captured image; and

a storage device, communicatively coupled to the processor, configured to store the refocus depths in a look-up table;

wherein the processor is further configured to refocus at least a portion of the captured image at a desired refocus depth determined from the look-up table;

wherein determining refocus depths for at least a portion of the captured image comprises, for each of at least a subset of the pixels, determining the refocus depth for the pixel by searching other pixels surrounding the pixel until a pixel with a known refocus depth is found.

23. The system of claim 22, wherein the refocus depths span a range of depths including closest and furthest distances to refocus.

24. The system of claim 23, wherein the closest distance corresponds to a closest object in the scene, and wherein the furthest distance corresponds to a furthest object in the scene.

25. The system of claim 22, wherein the captured image comprises a plurality of pixels, and wherein determining refocus depths comprises determining refocus depths for at least a subset of pixels in the plurality of pixels.

26. The system of claim 22, wherein determining refocus depths comprises determining refocus depths for respective regions in the captured image.

27. The system of claim 22, wherein the storage device stores a downsampled look-up table.

28. The system of claim 22, further comprising:

a display device, communicatively coupled to the processor, configured to display the captured image; and

an input device, communicatively coupled to the processor, configured to receive user input;

and wherein the desired refocus depth corresponds to a first refocus depth associated with a pixel at a location in the displayed image, wherein the location is specified by a user via the input device.

29. The system of claim 22, further comprising:

a sensor, communicatively coupled to the processor, configured to receive a light field image comprising light field data representing a scene;

a display device, communicatively coupled to the processor, configured to display the captured image; and

an input device, communicatively coupled to the processor, configured to receive user input;

and wherein;

the processor is configured to obtain the set of refocused images by;

receiving a captured light field image comprising light field data representing a scene;

determining a first depth corresponding to a nearest object in the scene;

determining a second depth corresponding to a farthest object in the scene; and

digitally refocusing the light field image to generate a plurality of refocused images, each having a focus depth lying within the range from the first depth to the second depth; and

the storage device is configured to store the generated plurality of refocused images;

and wherein;

for each of a plurality of two-dimensional positions in the light field image;

the processor is further configured to determine which of the plurality of refocused images is in focus at the two-dimensional position; and

the storage device is configured to store, in the look-up table, an index entry associating the two-dimensional position with the identified refocused image;

the display is configured to display one of the refocused images, the displayed image having a first focus depth; and

the input device is configured to receive user input to select a two-dimensional position within the displayed image;

and wherein the processor is configured to refocus at least a portion of the captured image at a desired refocus depth by, in response to the received input;

retrieving, from the look-up table, an index entry identifying a stored refocused image corresponding to the selected two-dimensional position, the stored refocused image having a second focus depth different from the first focus depth;

retrieving the identified refocused image from the storage device; and

causing the display device to display the retrieved refocused image having the second focus depth.

30. The system of claim 29, wherein the processor is configured to digitally refocus the light field image to generate a plurality of refocused images by:

scoring the set of images using a metric based on image content; and

removing at least one image from the set, based on the metric.