Image capture for later refocusing or focus-manipulation

• US 9,501,834 B2
• Filed: 01/09/2012
• Issued: 11/22/2016
• Est. Priority Date: 08/18/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm Technologies Incorporated (Qualcomm, Inc.)
• Inventors: Tzur, Meir
• Primary Examiner(s): HSU, AMY R
• Application Number:
  US13/346,602
• Publication Number:
  US 20130044254A1
• Time in Patent Office: 1,779 Days
• Field of Search: 348/345
• US Class Current: 1/1
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

A system, method, and computer program product for capturing images for later refocusing. Embodiments estimate a distance map for a scene, determine a number of principal depths, capture a set of images, with each image focused at one of the principal depths, and process captured images to produce an output image. The scene is divided into regions, and the depth map represents region depths corresponding to a particular focus step. Entries having a specific focus step value are placed into a histogram, and depths having the most entries are selected as the principal depths. Embodiments may also identify scene areas having important objects and include different important object depths in the principal depths. Captured images may be selected according to user input, aligned, and then combined using blending functions that favor only scene regions that are focused in particular captured images.

Claims

1. A processor-implemented method for capturing images for later refocusing, comprising:

estimating a depth map for a scene;

determining a number of principal depths via selecting a plurality of depths having the largest areas in the depth map as the principal depths;

capturing a set of images, each image focused at one of the principal depths; and

processing captured images to produce an output image.

2. The method of claim 1, wherein the estimating comprises depth from defocus.

3. The method of claim 1, wherein the estimating comprises acquiring images at different focus settings and finding the focus setting yielding a maximum spatial derivative for at least a portion of the scene.

4. The method of claim 1, wherein the estimating comprises employing an external rangefinder.

5. The method of claim 1, wherein the determining further comprises:

dividing the scene into a plurality of rectangles, wherein the depth map represents depths of the rectangles, the depths corresponding to one of a predetermined number of focus steps; and

selecting depths having the most depth map entries as the principal depths.

6. The method of claim 1, wherein the determining further comprises:

dividing the scene into a plurality of rectangles, wherein the depth map represents depths of the rectangles, the depths corresponding to one of a predetermined number of focus steps;

counting depth map entries having a specific focus step value;

sorting counted depth map entries into a histogram; and

selecting depths having the most depth map entries as the principal depths.

7. The method of claim 6, wherein the number of principal depths is a fraction of the number of focus steps, the fraction calculated according to a minimum number of focus steps from the histogram required to cover a predetermined percentage of the rectangles.

8. The method of claim 1, wherein the determining further comprises:

identifying scene areas having important objects; and

including different important object depths in the principal depths.

9. The method of claim 8, wherein the important objects include at least one of faces, pets, cars, people, and recognized faces of specific persons.

10. The method of claim 8, wherein the important object depths are placed first in the principal depths.

11. The method of claim 10, further comprising, if the number of important objects is less than a predetermined number, adding depths corresponding to highest histogram counts but not corresponding to important objects to the principal depths.

12. The method of claim 8, further comprising, if the number of important objects is at least a predetermined number, the number of important objects determines the number of principal depths.

13. The method of claim 8, wherein the number of principal depths is limited by at least one of available memory and the number of images that can be captured substantially simultaneously.

14. The method of claim 1, wherein the processing comprises selecting one image from the set according to at least one of user input and automated selection criteria.

15. The method of claim 14, wherein the selection criteria include at least one of which image has a recognized face in focus, and which image has the highest depth map histogram count.

16. The method of claim 1, wherein the processing comprises:

selecting a plurality of images from the set according to at least one of user input and automated selection criteria;

aligning the selected images; and

combining the selected images using blending functions.

17. The method of claim 16, wherein each blending function equals one in areas of a corresponding selected image that are focused, zero in areas of other selected images that are focused, and values between zero and one in other areas.

18. The method of claim 16, wherein the plurality includes all the captured images.

19. The method of claim 1, further comprising generating an output video clip based on at least part of the processed images.

20. A computer program product comprising a machine-readable medium tangibly embodying non-transitory program instructions thereon that, when executed by the machine, cause the machine to capture images for later refocusing by:

estimating a depth map for a scene;

determining a number of principal depths via selecting a plurality of depths having the largest areas in the depth map as the principal depths;

capturing a set of images, each image focused at one of the principal depths; and

processing captured images to produce an output image.

21. A system for capturing images for later refocusing, comprising:

means for estimating a depth map for a scene;

means for determining a number of principal depths, wherein the means for determining comprises means for selecting a plurality of depths having the largest areas in the depth map as the principal depths;

means for capturing a set of images, each image focused at one of the principal depths; and

means for processing captured images to produce an output image.

22. A system for capturing images for later refocusing, comprising:

a processor; and

a memory that stores instructions, the processor instruction execution causing the processor to;

estimate a depth map for a scene;

determine a number of principal depths via selecting a plurality of depths having the largest areas in the depth map as the principal depths;

capture a set of images, each image focused at one of the principal depths; and

process captured images to produce an output image.

23. An integrated circuit for capturing images for later refocusing, comprising:

a first circuit element for estimating a depth map for a scene;

a second circuit element for determining a number of principal depths via selecting a plurality of depths having the largest areas in the depth map as the principal depths;

a third circuit element for capturing a set of images, each image focused at one of the principal depths; and

a fourth circuit element for processing captured images to produce an output image.