Determining an image capture payload burst structure based on a metering image capture sweep

• US 9,118,841 B2
• Filed: 12/24/2014
• Issued: 08/25/2015
• Est. Priority Date: 12/13/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Levoy, Marc Stewart, Geiss, Ryan, Hasinoff, Samuel William
• Primary Examiner(s): AGGARWAL, YOGESH K
• Application Number:
  US14/582,470
• Publication Number:
  US 20150109478A1
• Time in Patent Office: 244 Days
• Field of Search: 3482211-2291, 348/362
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

A first plurality of images of a scene may be captured. Each image of the first plurality of images may be captured with a different total exposure time (TET). Based at least on the first plurality of images, a TET sequence may be determined for capturing images of the scene. A second plurality of images of the scene may be captured. Images in the second plurality of images may be captured using the TET sequence. Based at least on the second plurality of images, an output image of the scene may be constructed.

Claims

1. A method comprising:

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured with a different total exposure time (TET), wherein histograms are stored in a training database and are based on at least two captured images of respective scenes, and wherein the stored histograms are associated with respective target pixel values;

determining a TET sequence for capturing images of the scene, wherein determining the TET sequence comprises determining a scene histogram based on at least one of the images in the first plurality of images of the scene, comparing the scene histogram to at least one stored histogram, determining one or more target pixel values for the scene based on the respective target pixel values, and selecting one or more TET values to use in the TET sequence based on the one or more determined target pixel values;

capturing, by the image sensor, a second plurality of images of the scene, wherein images in the second plurality of images are captured using the TET sequence; and

based at least on the second plurality of images, constructing an output image of the scene.

2. The method of claim 1, wherein the scene histogram is based on downsampling and combining the images in the first plurality of images of the scene.

3. The method of claim 1, wherein the stored histograms are also associated with respective dynamic range parameters that indicate whether the respective scenes exhibit low dynamic range (LDR) or high dynamic range (HDR), the method further comprising:

determining a dynamic range parameter for the scene from the respective dynamic range parameters, wherein the TET sequence is also based on the dynamic range parameter for the scene.

4. The method of claim 3, wherein the dynamic range parameter for the scene indicates that the scene is LDR, and wherein determining the TET sequence comprises selecting a single TET value to use in the TET sequence.

5. The method of claim 4, wherein constructing the output image of the scene comprises aligning and combining one or more of the images in the second plurality of images captured using the single TET value.

6. The method of claim 3, wherein the dynamic range parameter for the scene indicates that the scene is HDR, and wherein determining the TET sequence comprises selecting a short TET value and a long TET value to use in the TET sequence.

7. The method of claim 6, wherein constructing the output image of the scene comprises:

aligning and combining (i) one or more of the images in the second plurality of images captured with the short TET value, and (ii) one or more of the images in the second plurality of images captured with the long TET value.

8. The method of claim 6, wherein determining the TET sequence also comprises selecting a fallback TET value to use in the TET sequence.

9. The method of claim 8, wherein constructing the output image of the scene comprises:

attempting to align (i) one or more of the images in the second plurality of images captured with the short TET value, with (ii) one or more of the images in the second plurality of images captured with the long TET value;

determining that the attempted alignment has failed; and

in response to determining that the attempted alignment has failed, aligning and combining one or more of the images in the second plurality of images captured with the fallback TET value to form the output image.

10. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured with a different total exposure time (TET), wherein histograms are stored in a training database and are based on at least two captured images of respective scenes, and wherein the stored histograms are associated with respective target pixel values;

determining a TET sequence for capturing images of the scene, wherein determining the TET sequence comprises determining a scene histogram based on at least one of the images in the first plurality of images of the scene, comparing the scene histogram to at least one stored histogram, determining one or more target pixel values for the scene based on the respective target pixel values, and selecting one or more TET values to use in the TET sequence based on the one or more determined target pixel values;

capturing, by the image sensor, a second plurality of images of the scene, wherein images in the second plurality of images are captured using the TET sequence; and

based at least on the second plurality of images, constructing an output image of the scene.

11. The article of manufacture of claim 10, wherein the scene histogram is based on downsampling and combining the images in the first plurality of images of the scene.

12. The article of manufacture of claim 10, wherein the stored histograms are also associated with respective dynamic range parameters that indicate whether the respective scenes exhibit low dynamic range (LDR) or high dynamic range (HDR), the operations further comprising:

determining a dynamic range parameter for the scene from the respective dynamic range parameters, wherein the TET sequence is also based on the dynamic range parameter for the scene.

13. The article of manufacture of claim 12, wherein the dynamic range parameter for the scene indicates that the scene is LDR, and wherein determining the TET sequence comprises selecting a single TET value to use in the TET sequence.

14. The article of manufacture of claim 13, wherein constructing the output image of the scene comprises aligning and combining one or more of the images in the second plurality of images captured using the single TET value.

15. The article of manufacture of claim 12, wherein the dynamic range parameter for the scene indicates that the scene is HDR, and wherein determining the TET sequence comprises selecting a short TET value and a long TET value to use in the TET sequence.

16. A computing device comprising:

at least one processor;

an image sensor;

memory; and

program instructions, stored in the memory, that upon execution by the at least one processor cause the computing device to perform operations including;

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured with a different total exposure time (TET), wherein histograms are stored in a training database and are based on at least two captured images of respective scenes, and wherein the stored histograms are associated with respective target pixel values;

determining a TET sequence for capturing images of the scene, wherein determining the TET sequence comprises determining a scene histogram based on at least one of the images in the first plurality of images of the scene, comparing the scene histogram to at least one stored histogram, determining one or more target pixel values for the scene based on the respective target pixel values, and selecting one or more TET values to use in the TET sequence based on the one or more determined target pixel values;

capturing, by the image sensor, a second plurality of images of the scene, wherein images in the second plurality of images are captured using the TET sequence; and

based at least on the second plurality of images, constructing an output image of the scene.

17. The computing device of claim 16, wherein the stored histograms are also associated with respective dynamic range parameters that indicate whether the respective scenes exhibit low dynamic range (LDR) or high dynamic range (HDR), the operations further comprising:

determining a dynamic range parameter for the scene from the respective dynamic range parameters, wherein the TET sequence is also based on the dynamic range parameter for the scene.

18. The computing device of claim 17, wherein the dynamic range parameter for the scene indicates that the scene is LDR, and wherein determining the TET sequence comprises selecting a single TET value to use in the TET sequence.

19. The computing device of claim 18, wherein constructing the output image of the scene comprises aligning and combining one or more of the images in the second plurality of images captured using the single TET value.

20. The computing device of claim 17, wherein the dynamic range parameter for the scene indicates that the scene is HDR, and wherein determining the TET sequence comprises selecting a short TET value and a long TET value to use in the TET sequence.