Interactive Refocusing of Electronic Images

• US 20080131019A1
• Filed: 11/30/2007
• Published: 06/05/2008
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Ng, Yi-Ren
• Granted Patent: US 8,559,705 B2
• Time in Patent Office: Days
• US Class Current: 382/255
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/20108 Interactive selection of 2D slice in a 3D data set
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H01L 27/14627 Microlenses

Abstract

A method is performed to refocus a digital photographic image comprising a plurality of pixels. In the method, a set of images is computed corresponding to the digital photographic image and focused at different depths. Refocus depths for at least a subset of the pixels are identified and stored in a look-up table. At least a portion of the digital photographic image is refocused at a desired refocus depth determined from the look-up table.

Claims

1. A method of refocusing a digital photographic image comprising a plurality of pixels, the method comprising:

computing a set of images corresponding to a scene in the digital photographic image, wherein each image comprises an image of the scene at a distinct focal depth;

identifying refocus depths for at least a subset of the pixels;

storing the refocus depths in a look-up table; and

refocusing at least a portion of the digital photographic image at a desired refocus depth determined from the look-up table.

2. The method of claim 1, wherein the different depths span a range of depths including closest and furthest distances to refocus.

3. The method of claim 2, wherein the range of depths corresponds to a depth of focus of a camera recording a light field corresponding to the set of images.

4. The method of claim 3, wherein the range of depths is extended beyond an effective depth of focus of the camera by a factor representing a tolerable mis-focus.

5. The method of claim 2, wherein the furthest distance is optical infinity.

6. The method of claim 2, wherein the closest distance corresponds to a closest object in a scene corresponding to the set of images, and wherein the furthest distance corresponds to a furthest object in the scene.

7. The method of claim 1, wherein the identifying operation comprises determining a refocus depth for each pixel in the plurality of pixels.

8. The method of claim 1, wherein the identifying operation comprises determining refocus depths for respective regions in the digital photographic image.

9. The method of claim 1, wherein determining the refocus depth for a pixel includes searching other pixels surrounding the pixel until a pixel with a known refocus depth is found.

10. The method of claim 1, further comprising downsampling the look-up table.

11. The method of claim 10, wherein the downsampling operation includes a voting procedure wherein pixel votes are weighted.

12. The method of claim 1, wherein the desired refocus depth corresponds to a first refocus depth associated with a pixel at a user-specified point in the displayed image.

13. The method of claim 12, wherein a region of the displayed image that includes the user-specified point is refocused.

14. The method of claim 13, wherein the region is user-defined.

15. The method of claim 13, wherein the region corresponds to an object in the displayed image.

16. The method of claim 13, wherein a tag is associated with the region.

17. The method of claim 1, wherein the refocusing operation includes generating an animation comprising a series of images in the set having depths between a depth of the displayed image and the desired refocus depth.

18. The method of claim 1, wherein the refocusing operation includes blending at least a portion of two images focused at depths near to the desired refocus depth.

19. The method of claim 1, further comprising storing the set of images and the look-up table in a single file.

21. A method of refocusing a digital photographic image of a scene, the method comprising:

compiling a set of digital photographic images, wherein each image comprises an image of the scene at a distinct focal depth;

identifying refocus depths for regions within the scene;

storing the refocus depths in a look-up table; and

refocusing at least a portion of an image from the set at a desired refocus depth determined from the look-up table.

20. The method of claim 21, wherein the single file is one of a zip archive or a swf (Flash) file.

22. A method of refocusing a digital photographic image, the method comprising:

displaying the image at a first focal depth;

displaying a slider icon, the slider icon having a first position corresponding to the first focal depth;

interactively receiving input to move the slider icon to a second position corresponding to a second focal depth; and

in response to the received input, refocusing the image at the second focal depth.

23. The method of claim 22, wherein the refocusing comprises blending stored images having focal depths near to the second focal depth.

24. A method of refocusing a digital photographic image, the method comprising:

displaying the image at a first focal depth;

displaying an icon to select a position in the image;

interactively receiving input to modify the image by refocusing at least a portion of the image to a second focal depth corresponding to the selected position; and

in response to the received input, displaying the modified image.

25. The method of claim 24, wherein the icon comprises a pointer.

28. The method of claim 24, wherein the icon comprises a brush tool to specify a region surrounding the selected position to refocus.

29. The method of claim 24, wherein the refocusing operation comprises blending stored images having focal depths near to the desired refocus depth.

26. The method of claim 26, wherein the selected position corresponds to an object in the image, the object having a focal depth corresponding to the second focal depth.

27. The method of claim 26, wherein the selected position corresponds to a face in the image, and wherein refocusing at least a portion of the image includes altering the focus of pixels determined to be part of the face and not altering the focus of pixels determined to be part of the background.