Phase detection autofocus using subaperture images

• US 10,334,151 B2
• Filed: 05/23/2016
• Issued: 06/25/2019
• Est. Priority Date: 04/22/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Romanenko, Yuriy, Chung, Edmond, Deguara, Joachim
• Primary Examiner(s): Henn, Timothy J
• Application Number:
  US15/162,426
• Publication Number:
  US 20160269620A1
• Time in Patent Office: 1,128 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/672 based on the phase difference signals
• CPC Class Codes: H04N 23/673 based on contrast or high frequency components of image signals, e.g. hill climbing method
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A light-field camera may have enhanced autofocus functionality that does not require a dedicated optical pathway or dedicated sensors, and yet brings the camera rapidly into focus for image capture. According to one method, light may be received through the aperture of a light-field camera, with the main lens of the light-field camera at a first focus setting. The light may be directed to a sensor of the light-field camera via an optical pathway. The sensor may capture the light to generate preliminary light-field data. In a processor, the preliminary light-field data may be used to generate an offset, which may be used to refocus the main lens to a second focus setting. Additional light may be directed along the same optical pathway and captured with the main lens at the second focus setting to capture a focused light-field image.

Claims

1. A method for focusing a light-field camera, the method comprising:

through an aperture of the light-field camera, receiving first light with a main lens of the light-field camera at a first focus setting;

directing the first light to a sensor of the light-field camera via an optical pathway;

in the sensor, capturing the first light to generate preliminary light-field data;

in a processor, using the preliminary light-field data to generate an offset;

using the offset to refocus the main lens to a second focus setting; and

wherein using the preliminary light-field data to generate the offset comprises;

using the preliminary light-field data to generate a plurality of subaperture images;

computing relative displacements between the subaperture images; and

based on the relative displacements, generating the offset using an averaging of a set of focus offsets.

2. The method of claim 1, wherein computing relative displacements between the subaperture images comprises:

cropping each of the subaperture images to provide a cropped subaperture image such that all cropped subaperture images depict the same image portion;

identifying at least one feature of each of the cropped subaperture images; and

comparing positions of the features within the cropped subaperture images to determine the relative displacements.

3. The method of claim 2, wherein calculating a plurality of relative positions of the features for adjacent pairs of the cropped subaperture images yields the set of focus offsets.

4. The method of claim 1, further comprising:

refocusing the main lens through a sweep from the second focus setting to a third focus setting;

at the sensor, at each of a plurality of intermediate focus settings between the second focus setting and the third focus setting, capturing intermediate light-field data;

at the processor, for each intermediate light-field data, computing a contrast level; and

comparing the contrast levels to identify an optimal intermediate focus setting of the intermediate focus settings at which contrast is the highest.

5. The method of claim 4, further comprising:

refocusing the main lens to the optimal intermediate focus setting;

through the aperture, receiving second light with the main lens at the optimal intermediate focus setting;

directing the second light to the sensor via the optical pathway; and

in the sensor, capturing the second light to generate a light-field image.

6. The method of claim 1, further comprising:

at the processor, computing a confidence factor; and

using the confidence factor to determine whether the light-field camera is properly focused.

7. The method of claim 1, further comprising:

through the aperture, receiving second light with the main lens at the second focus setting;

directing the second light to the sensor via the optical pathway; and

in the sensor, capturing the second light to generate a light-field image.

8. The method of claim 1, further comprising:

determining whether the light-field camera is properly focused with the main lens at the second focus setting; and

until the light-field camera is properly focused, iterating the steps of;

through the aperture, receiving additional light with the main lens;

directing the additional light to the sensor via the optical pathway;

in the sensor, capturing the additional light to generate additional preliminary light-field data;

in the processor, using the additional preliminary light-field data to generate an additional offset;

using the additional offset to refocus the main lens; and

determining whether the light-field camera is properly focused with the main lens.

9. A light-field camera comprising:

an aperture configured to receive first light with a main lens of the light-field camera at a first focus setting such that the first light is directed to a sensor of the light-field camera via an optical pathway;

a sensor configured to capture the first light to generate preliminary light-field data; and

a processor configured to;

use the preliminary light-field data to generate an offset;

use the offset to cause the main lens to be refocused to a second focus setting; and

wherein the processor is configured to use the preliminary light-field data to generate the offset by;

using the preliminary light-field data to generate a plurality of subaperture images;

computing relative displacements between the subaperture images; and

based on the relative displacements, generating the offset using an averaging of a set of focus offsets.

10. The light-field camera of claim 9, wherein computing relative displacements between the subaperture images comprises:

cropping each of the subaperture images to provide a cropped subaperture image such that all cropped subaperture images depict the same image portion;

identifying at least one feature of each of the cropped subaperture images; and

comparing positions of the features within the cropped subaperture images to determine the relative displacements.

11. The light-field camera of claim 9, wherein:

the processor is further configured to cause the main lens to be refocused through a sweep from the second focus setting to a third focus setting;

the sensor is further configured, at each of a plurality of intermediate focus settings between the second focus setting and the third focus setting, to capture intermediate light-field data;

the processor is further configured, for each intermediate light-field data, to compute a contrast level; and

the processor is further configured to compare the contrast levels to identify an optimal intermediate focus setting of the intermediate focus settings at which contrast is the highest.

12. The light-field camera of claim 11, wherein:

the processor is further configured to cause the main lens to be refocused to the optimal intermediate focus setting;

the aperture is further configured to receive second light with the main lens at the optimal intermediate focus setting such that the second light is directed to the sensor via the optical pathway; and

the sensor is further configured to capture the second light to generate a light-field image.

13. The light-field camera of claim 9, wherein the processor is further configured to:

computing a confidence factor; and

use the confidence factor to determine whether the light-field camera is properly focused.

14. The light-field camera of claim 9, wherein:

the aperture is further configured to receive second light with the main lens at the second focus setting such that the second light is directed to the sensor via the optical pathway; and

the sensor is further configured to capture the second light to generate a light-field image.

15. The light-field camera of claim 9, wherein the processor is further configured to:

determine whether the light-field camera is properly focused with the main lens at the second focus setting; and

until the light-field camera is properly focused, cause the light-field camera to iterate the steps of;

through the aperture, receiving additional light with the main lens such that the additional light is directed to the sensor via the optical pathway;

in the sensor, capturing the additional light to generate additional preliminary light-field data;

using the additional preliminary light-field data to generate an additional offset;

using the additional offset to refocus the main lens; and

determining whether the light-field camera is properly focused with the main lens.

16. The light-field image camera of claim 9, wherein computing relative displacements between the subaperture images comprises:

cropping each of the subaperture images to provide a cropped subaperture image such that all cropped subaperture images depict the same image portion;

identifying at least one feature of each of the cropped subaperture images;

comparing positions of the features within the cropped subaperture images to determine the relative displacements; and

calculating a plurality of relative position of the features for adjacent pairs of cropped subaperture images to yield the set of focus offsets.