Depth map generation using bokeh detection

• US 9,087,405 B2
• Filed: 12/16/2013
• Issued: 07/21/2015
• Est. Priority Date: 12/16/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Seitz, Steven Maxwell, Esteban, Carlos Hernandez, Suwajanakorn, Supasorn
• Primary Examiner(s): Hoang, Peter
• Application Number:
  US14/107,163
• Publication Number:
  US 20150170400A1
• Time in Patent Office: 582 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

In one aspect, one or more computing devices receive a set of image frames. Each image frame includes pixels. The computing devices align image frames in order to identify flows of the pixels in the set of image frames. Regions of bokeh effect are identified in each image frame by measuring the sizes of areas of expansion across image frames using a set of assumptions and the identified flows. The computing devices adjust the alignment of the set of image frames based at least in part on the identified regions of bokeh effect. For each image frame, the computing devices generates an index map of focus values for each of the pixels that image frame using the improved alignment. A depth map is generated by the computing devices based at least in part on the index maps.

Claims

1. A method for generating depth maps, the method comprising:

receiving, by one or more computing devices, a set of image frames, each image frame of the set of image frames including pixels;

aligning, by the one or more computing devices, the set of image frames to provide flows of the pixels in the set of image frames;

identifying, by the one or more computing devices, any region of bokeh effect in each image frame of the set of image frames by measuring sizes of areas of expansion across image frames of the set of image frames using an assumption that contour expansion due to a bokeh effect has consistent movements through frames of the set of frames and the flows;

adjusting, by the one or more computing devices, the alignment of the set of image frames based at least in part on the identified regions of bokeh effect;

generating, by the one or more computing devices, for each particular image frame of the set of image frames an index map of focus values for the pixels of that particular image frame using the improved alignment; and

generating, by the one or more computing devices, a depth map based at least in part on the index maps.

2. The method of claim 1, further comprising capturing the set of image frames.

3. The method of claim 1, wherein aligning the set of image frames further includes using a flow concatenation process that determines optical flow between consecutive images.

4. The method of claim 3, wherein the flow concatenation process further produces new flows that warp each image frame of the set of image frames to a reference frame, and wherein the new flows are used to identify the regions of bokeh effect.

5. The method of claim 1, wherein identifying the regions of bokeh effect includes using a voting scheme that provides for each pixel in a first image frame of the set of image frames to vote for a corresponding pixel in each of the other image frames of the set of image frames.

6. The method of claim 1, wherein identifying any region of bokeh effect in each image frame of the set of image frames by measuring sizes of areas of expansion further includes using second assumption that a given region of bokeh effect has a high color intensity relative to other regions without bokeh effect.

7. The method of claim 1, further comprising generating a composite image of focused pixels from the set of image frames based at least in part on the index map, wherein the composite image is used to generate the depth map.

8. A system for generating depth maps, the system comprising one or more computing devices configured to:

receive a set of image frames, each image frame of the set of image frames including pixels;

align the set of image frames to provide flows of the pixels in the set of image frames;

identify any region of bokeh effect in each image frame of the set of image frames by measuring sizes of areas of expansion across image frames of the set of image frames using an assumption that contour expansion due to a bokeh effect has consistent movements through frames of the set of frames and the flows;

adjust the alignment of the set of image frames based at least in part on the identified regions of bokeh effect;

generate, for each particular image frame of the set of image frames an index map of focus values for the pixels of that particular image frame using the improved alignment; and

generate a depth map based at least in part on the index maps.

9. The system of claim 8, wherein the one or more computing devices are further configured to capture the set of image frames.

10. The system of claim 8, wherein the one or more computing devices are further configured to align using a flow concatenation process that determines optical flow between consecutive images.

11. The system of claim 10, wherein the one or more computing devices are further configured to produces new flows that warp each image frame of the set of image frames to a reference frame and the one or more computing devices are further configured to use the new flows to identify the bokeh regions.

12. The system of claim 8, wherein the one or more computing devices are further configured to identify the regions of bokeh effect by using a voting scheme that provides for each pixel in a first image frame of the set of image frames to vote for a corresponding pixel in each of the other image frames of the set of image frames.

13. The system of claim 8, wherein the one or more computing devices are further configured to:

generate a composite image of focused pixels from the set of image frames based at least in part on the index map, and

to use the composite image to generate the depth map.

14. A non-transitory, tangible computer-readable recording medium storing instructions, the instructions, when executed by one or more processors, cause the one or more processors to perform a method for generating depth maps, the method comprising:

receiving a set of image frames, each image frame of the set of image frames including pixels;

aligning the set of image frames to provide flows of the pixels in the set of image frames;

identifying any region of bokeh effect in each image frame of the set of image frames by measuring sizes of areas of expansion across image frames of the set of image frames using a an assumption that contour expansion due to a bokeh effect has consistent movements through frames of the set of frames and the flows;

adjusting the alignment of the set of image frames based at least in part on the identified regions of bokeh effect;

generating for each particular image frame of the set of image frames an index map of focus values for the pixels of that particular image frame using the improved alignment; and

generating a depth map based at least in part on the index maps.

15. The medium of claim 14, wherein the method further includes capturing the set of image frames.

16. The medium of claim 14, wherein the method further includes aligning the set of image frames by using a flow concatenation process that determines optical flows between consecutive images.

17. The medium of claim 16, wherein the flow concatenation process further produces new flows that warp each of the image frames of the set of image frames to a reference frame and the method further includes using the new flows to identify the regions of bokeh effect.

18. The medium of claim 14, wherein the method further comprises identifying the regions of bokeh effect by using a voting scheme that provides for each pixel in a first image frame of the set of image frames to vote for a corresponding pixel in each of the other image frames of the set of image frames.

19. The medium of claim 14, wherein the method further comprises:

generating a composite image of focused pixels from the set of image frames based at least in part on the index map,

wherein the composite image is used to generate the depth map.