Interactive refocusing of electronic images

• US 8,559,705 B2
• Filed: 11/30/2007
• Issued: 10/15/2013
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Ng, Yi-Ren
• Primary Examiner(s): Chu, Randolph I
• Assistant Examiner(s): BLOOM, NATHAN J
• Application Number:
  US11/948,901
• Publication Number:
  US 20080131019A1
• Time in Patent Office: 2,146 Days
• Field of Search: 382/100, 382/103, 382153-154, 382/162, 382/173, 382/181, 382/190, 382254-255, 382/276, 382/282, 382/293, 382295-296, 382305-307, 348/61, 348/63, 348/121, 348/135, 348/143, 348/159, 348/207.99, 345418-419, 345/427, 345/629, 345633-634, 345/646, 345649-650, 345655-659
• US Class Current: 382/162
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/20108 Interactive selection of 2D slice in a 3D data set
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H01L 27/14627 Microlenses

Abstract

A method is performed to refocus a digital photographic image comprising a plurality of pixels. In the method, a set of images is computed corresponding to the digital photographic image and focused at different depths. Refocus depths for at least a subset of the pixels are identified and stored in a look-up table. At least a portion of the digital photographic image is refocused at a desired refocus depth determined from the look-up table.

Claims

1. A method of refocusing a light field image in response to user input, the method comprising:

receiving a light field image comprising light field data;

digitally refocusing the light field image to generate a plurality of refocused images, each having a focus depth;

storing the generated plurality of refocused images in a storage device;

for each of a plurality of two-dimensional positions in the light field image;

determining which of the plurality of refocused images is in focus at the two-dimensional position; and

storing, in a look-up table, an index entry associating the two-dimensional position with the identified refocused image;

displaying one of the refocused images, the displayed image having a first focus depth;

receiving user input to select a two-dimensional position within the displayed image; and

in response to the received input;

retrieving, from the look-up table, an index entry identifying a stored refocused image corresponding to the selected two-dimensional position, the stored refocused image having a second focus depth different from the first focus depth;

retrieving the identified refocused image from the storage device; and

displaying the retrieved refocused image having the second focus depth.

2. The method of claim 1, wherein receiving user input to select a two-dimensional position within the displayed image comprises:

displaying an icon to select a position within the displayed image;

receiving user input, via a pointing device, to move the displayed icon.

3. The method of claim 2, wherein the icon comprises a pointer.

4. The method of claim 2, wherein the icon comprises a brush tool to specify a region surrounding the selected position to refocus.

5. The method of claim 1, wherein identifying a stored refocused image in the plurality having the second focus depth comprises:

identifying at least two of the refocused images having different focus depths; and

blending the at least two identified images.

6. The method of claim 1, wherein receiving user input to select a two-dimensional position within the displayed image comprises:

detecting contact with a touch-sensitive screen, the contact specifying a position within the displayed image.

7. The method of claim 1, wherein receiving user input to select a two-dimensional position within the displayed image comprises:

detecting a user interaction gesture specifying a position within the displayed image.

8. The method of claim 1, further comprising:

displaying a smooth transition from the display of the image having the first focus depth to the display of at least one refocused image having an intermediate focus depth between the first focus depth and the second focus depth; and

displaying a smooth transition from the display of the at least one refocused image having an intermediate focus depth to the display of the refocused image having the second focus depth.

9. The method of claim 1, wherein:

displaying the image having the first focus depth comprises;

retrieving a first stored image having the first focus depth; and

displaying the retrieved first image; and

displaying the refocused image having the second focus depth comprises;

retrieving a second stored image having the second focus depth; and

displaying the retrieved second image.

10. The method of claim 1, wherein:

displaying one of the refocused images comprises displaying a web page comprising the refocused image;

receiving user input to select a two-dimensional position within the displayed image comprises receiving a signal indicating a user interaction with the web page; and

displaying the refocused image having the second focus depth comprises displaying a web page comprising the refocused image having the second focus.

11. The method of claim 1, further comprising:

receiving an indication of at least one tag, each tag being associated with a position in the image, each tag being further associated with a description;

wherein;

receiving user input to select a two-dimensional position within the displayed image comprises receiving user input to select a description; and

displaying the refocused image having the second focus depth comprises refocusing the image to a focus depth corresponding to a position in the image associated with a tag associated with the selected description.

12. The method of claim 8, wherein displaying a smooth transition from the display of the image having the first focus depth to the display of at least one refocused image having an intermediate focus depth comprises:

generating at least one intermediate image having a focus depth between the first focus depth and the second focus depth by determining an interpolated focus depth between the first focus depth and the second focus depth; and

displaying the generated an image at the interpolated focus depth.

13. The method of claim 1, wherein digitally refocusing light field data to generate a plurality of refocused images comprises generating a plurality of projections of the light field image at different focus depths.

14. The method of claim 1, wherein determining which of the plurality of refocused images is in focus at the two-dimensional position comprises detecting edges at the two-dimensional position for each of the refocused images.

15. The method of claim 1, wherein determining which of the plurality of refocused images is in focus at the two-dimensional position comprises:

for each of the refocused images, computing a gradient magnitude at the two-dimensional position, each refocused image having a focus depth; and

comparing the computed gradient magnitudes at the two-dimensional position.

16. The method of claim 1, wherein:

storing an index entry associating the two-dimensional position with identified refocused image comprises storing, in a look-up table, an index entry associating the two-dimensional position with the focus depth of the identified refocused image; and

identifying a stored refocused image in the plurality associated with the selected two-dimensional position comprises;

retrieving, from the look-up table, an index entry identifying a second focus depth corresponding to the selected two-dimensional position; and

identifying a stored refocused image in the plurality having the second focus depth.