Depth map generation using bokeh detection

• US 9,256,948 B1
• Filed: 06/16/2015
• Issued: 02/09/2016
• Est. Priority Date: 12/16/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Seitz, Steven Maxwell, Esteban, Carlos Hernandez, Suwajanakorn, Supasorn
• Primary Examiner(s): Hoang, Peter
• Application Number:
  US14/740,511
• Time in Patent Office: 238 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

In one aspect, one or more computing devices receive a set of image frames. Each image frame includes pixels. The computing devices align image frames in order to identify flows of the pixels in the set of image frames. Regions of bokeh effect are identified in each image frame by measuring the sizes of areas of expansion across image frames using a set of assumptions and the identified flows. The computing devices adjust the alignment of the set of image frames based at least in part on the identified regions of bokeh effect. For each image frame, the computing devices generates an index map of focus values for each of the pixels that image frame using the improved alignment. A depth map is generated by the computing devices based at least in part on the index maps.

Claims

1. A method for generating depth maps, the method comprising:

receiving, by one or more computing devices having one or more processors, a set of image frames, each image frame of the set of image frames including pixels;

aligning, by the one or more computing devices, the set of image frames to provide flows of the pixels in the set of image frames;

for each pixel of each image frame of the set of image frames, determining, by the one or more computing devices, a vote value based at least in part on a voting scheme where each pixel of an image frames of the set of image frames votes for a corresponding pixel in another image frame of the set of image frames according to the flows;

identifying, by the one or more computing devices, any region of bokeh effect in each image frame of the set of image frames based at least in part the determined vote values;

adjusting, by the one or more computing devices, the alignment of the set of image frames based at least in part on the identified regions of bokeh effect; and

generating, by the one or more computing devices, a depth map based at least in part on the adjusted alignment.

2. The method of claim 1, wherein determining the vote value based at least in part on the voting scheme includes using each pixel in a given image frame of the set of image frames vote for at least one corresponding pixel in every other image frame of the set of image frames.

3. The method of claim 1, wherein identifying regions of bokeh effect includes identifying areas of a given image frame of the set of image frames having higher vote values as compared to other areas of the given image frame.

4. The method of claim 1, further comprising:

for each identified region of bokeh effect in each given image frame of the set of image frames, propagating back a highest voting value of the identified region of bokeh effect to corresponding pixels for that identified region of bokeh effect in the given image frame in all other image frames of the set of image frames, and

wherein adjusting the alignment is further based on the propagated highest voting values.

5. The method of claim 4, further comprising:

using the propagated highest voting values to generate a confidence map, and

wherein adjusting the alignment is further based on the confidence map.

6. The method of claim 1, wherein determining the vote value based at least in part on the voting scheme includes:

pairing each image frame of the set of image frames with each of the other image frames of the set of image frames; and

conducting the voting scheme between each of the paired image frames.

7. The method of claim 1, wherein identifying any region of bokeh effect includes comparing a voting values for a given pixel of a given image frames of the set of image frames to a threshold to determine whether the given pixel is within a region of bokeh effect.

8. A system for generating depth maps, the system comprising one or more computing devices having one or more processors configured to:

receive a set of image frames, each image frame of the set of image frames including pixels;

align by the one or more computing devices, the set of image frames to provide flows of the pixels in the set of image frames;

for each pixel of each image frame of the set of image frames, determine a vote value based at least in part on a voting scheme where each pixel of an image frames of the set of image frames votes for a corresponding pixel in another image frame of the set of image frames according to the flows;

identify any region of bokeh effect in each image frame of the set of image frames based at least in part the determined vote values;

adjust the alignment of the set of image frames based at least in part on the identified regions of bokeh effect; and

generate a depth map based at least in part on the adjusted alignment.

9. The system of claim 8, wherein the one or more processors are further configured to determine the vote value based at least in part on the voting scheme by using each pixel in a given image frame of the set of image frames vote for at least one corresponding pixel in every other image frame of the set of image frames.

10. The system of claim 8, wherein the one or more processors are further configured to identify regions of bokeh effect by identifying areas of a given image frame of the set of image frames having higher vote values as compared to other areas of the given image frame.

11. The system of claim 8, wherein the one or more processors are further configured to for each identified region of bokeh effect in each given image frame of the set of image frames, propagate back a highest voting value of the identified region of bokeh effect to corresponding pixels for that identified region of bokeh effect in the given image in all other image frames of the set of image frames, and

adjust the alignment further based on the propagated highest voting values.

12. The system of claim 11, wherein the one or more processors are further configured to:

use the propagated highest voting values to generate a confidence map, and

adjusting the alignment further based on the confidence map.

13. The system of claim 8, wherein the one or more processors are further configured to determine the vote value based at least in part on the voting scheme by:

pairing each image frame of the set of image frames with each of the other images of the set of image frames; and

conducting the voting scheme between each of the paired images.

14. The system of claim 8, wherein the one or more processors are further configured to identify any region of bokeh effect by comparing a voting values for a given pixel of a given image frames of the set of image frames to a threshold to determine whether the given pixel is within a region of bokeh effect.

15. A non-transitory, tangible computer-readable recording medium storing instructions, the instructions, when executed by one or more processors, cause the one or more processors to perform a method for generating depth maps, the method comprising:

receiving a set of image frames, each image frame of the set of image frames including pixels;

aligning the set of image frames to provide flows of the pixels in the set of image frames;

for each pixel of each image frame of the set of image frames, determining a vote value based at least in part on a voting scheme where each pixel of an image frames of the set of image frames votes for a corresponding pixel in another image frame of the set of image frames according to the flows;

identifying any region of bokeh effect in each image frame of the set of image frames based at least in part on the determined vote values;

adjusting the alignment of the set of image frames based at least in part on the identified regions of bokeh effect; and

generating a depth map based at least in part on the adjusted alignment.

16. The medium of claim 15, wherein determining the vote value based at least in part on the voting scheme includes using each pixel in a given image frame of the set of image frames vote for at least one corresponding pixel in every other image frame of the set of image frames.

17. The medium of claim 15, wherein identifying regions of bokeh effect includes identifying areas of a given image frame of the set of image frames having higher vote values as compared to other areas of the given image frame.

18. The medium of claim 15, wherein the method further comprises:

for each identified region of bokeh effect in each given image frame of the set of image frames, propagating back a highest voting value of the identified region of bokeh effect to corresponding pixels for that identified region of bokeh effect in the given image frame in all other image frames of the set of image frames, and

wherein adjusting the alignment is further based on the propagated highest voting values.

19. The medium of claim 15, wherein determining the vote value based at least in part on the voting scheme includes:

pairing each image frame of the set of image frames with each of the other images of the set of image frames; and

conducting the voting scheme between each of the paired images.

20. The medium of claim 15, wherein identifying any region of bokeh effect includes comparing a voting values for a given pixel of a given image frames of the set of image frames to a threshold to determine whether the given pixel is within a region of bokeh effect.