Determining an Image Capture Payload Burst Structure Based on a Metering Image Capture Sweep

• US 20140168468A1
• Filed: 12/13/2012
• Published: 06/19/2014
• Est. Priority Date: 12/13/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Geiss, Ryan, Hasinoff, Samuel William, Levoy, Marc Stewart
• Granted Patent: US 8,866,927 B2
• Time in Patent Office: Days
• US Class Current: 348/229.1
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/009 Global, i.e. based on properties of the image as a whole applied in cameras using an electronic image sensor H04N23/80, H04N23/70
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

A first plurality of images of a scene may be captured. Each image of the first plurality of images may be captured with a different total exposure time (TET). Based at least on the first plurality of images, a TET sequence may be determined for capturing images of the scene. A second plurality of images of the scene may be captured. Images in the second plurality of images may be captured using the TET sequence. Based at least on the second plurality of images, an output image of the scene may be constructed.

Claims

1. A method comprising:

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured with a different total exposure time (TET);

based at least on the first plurality of images, determining a TET sequence for capturing images of the scene;

capturing, by the image sensor, a second plurality of images of the scene, wherein images in the second plurality of images are captured using the TET sequence; and

based at least on the second plurality of images, constructing an output image of the scene.

2. The method of claim 1, wherein determining the TET sequence comprises determining a scene histogram based on at least one of the images in the first plurality of images of the scene.

3. The method of claim 2, wherein the scene histogram is based on downsampling and combining the images in the first plurality of images of the scene.

4. The method of claim 2, wherein histograms are stored in a training database and are based on at least two images captured of respective scenes, wherein the histograms in the training database are associated with respective dynamic range parameters, wherein the respective dynamic range parameters indicate whether the respective scenes exhibit low dynamic range (LDR) or high dynamic range (HDR), and wherein determining the TET sequence comprises:

comparing the scene histogram to at least one histogram in the training database; and

based on an outcome of the comparison, determining a dynamic range parameter for the scene from the respective dynamic range parameters, wherein determining the TET sequence is further based on the dynamic range parameter for the scene.

5. The method of claim 4, wherein the dynamic range parameter for the scene indicates that the scene is LDR, and wherein determining the TET sequence comprises selecting a single TET value to use in the TET sequence.

6. The method of claim 4, wherein the dynamic range parameter for the scene indicates that the scene is HDR, and wherein determining the TET sequence comprises selecting a short TET value and a long TET value to use in the TET sequence.

7. The method of claim 6, wherein determining the TET sequence also comprises selecting a fallback TET value to use in the TET sequence.

8. The method of claim 4, wherein the histograms in the training database are also associated with respective target pixel values, wherein determining the TET sequence further comprises:

based on the respective target pixel values in the training database, determining one or more target pixel values for the scene; and

based on the one or more determined target pixel values, selecting one or more TET values to use in the TET sequence.

9. The method of claim 1, wherein determining the TET sequence comprises determining that the scene is a low dynamic range (LDR) scene and defining a common value for TETs in the TET sequence, and wherein constructing the output image of the scene comprises aligning and combining one or more of the images in the second plurality of images captured using the common value.

10. The method of claim 1, wherein determining the TET sequence comprises determining that the scene is a high dynamic range (HDR) scene and defining a short TET value and a long TET value, and wherein constructing the output image of the scene comprises aligning and combining (i) one or more of the images in the second plurality of images captured with the short TET value, and (ii) one or more of the images in the second plurality of images captured with the long TET value.

11. The method of claim 1, wherein determining the TET sequence comprises determining that the scene is a high dynamic range (HDR) scene and defining a short TET value, a long TET value, and a fallback TET value.

12. The method of claim 11, wherein constructing the output image of the scene comprises:

attempting to align (i) one or more of the images in the second plurality of images captured with the short TET value, with (ii) one or more of the images in the second plurality of images captured with the long TET value;

determining that alignment of (i) the one or more of the images in the second plurality of images captured with the short TET value, and (ii) the one or more of the images in the second plurality of images captured with the long TET value has failed; and

in response to determining that the alignment has failed, aligning and combining one or more of the images in the second plurality of images captured with the fallback TET value to form the output image.

13. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

capturing, by an image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured with a different total exposure time (TET);

based at least on the first plurality of images, determining a TET sequence for capturing images of the scene;

capturing, by the image sensor, a second plurality of images of the scene, wherein images in the second plurality of images are captured using the TET sequence; and

based at least on the second plurality of images, constructing an output image of the scene.

14. The article of manufacture of claim 13, wherein determining the TET sequence comprises determining a scene histogram based on at least one of the images in the first plurality of images of the scene.

15. The article of manufacture of claim 14, wherein the scene histogram is based on downsampling and combining the images in the first plurality of images of the scene.

16. The article of manufacture of claim 14, wherein histograms are stored in a training database and are based on at least two images captured of respective scenes, wherein the histograms in the training database are associated with respective dynamic range parameters, wherein the respective dynamic range parameters indicate whether the respective scenes exhibit low dynamic range (LDR) or high dynamic range (HDR), and wherein determining the TET sequence further comprises:

comparing the scene histogram to at least one of the histograms in the training database; and

based on an outcome of the comparison, determining a dynamic range parameter for the scene from the respective dynamic range parameters, wherein determining the TET sequence is further based on the dynamic range parameter for the scene.

17. A computing device comprising:

at least one processor;

an image sensor;

data storage; and

program instructions, stored in the data storage, that upon execution by the at least one processor cause the computing device to perform operations including;

capturing, at the image sensor, a first plurality of images of a scene, wherein each image of the first plurality of images is captured with a different total exposure time (TET);

based at least on the first plurality of images, determining a TET sequence for capturing images of the scene;

capturing, at the image sensor, a second plurality of images of the scene, wherein images in the second plurality of images are captured using the TET sequence; and

based at least on the second plurality of images, constructing an output image of the scene.

18. The computing device of claim 17, wherein determining the TET sequence comprises determining a scene histogram based on at least one of the images in the first plurality of images of the scene.

19. The computing device of claim 18, wherein the scene histogram is based on downsampling and combining the images in the first plurality of images of the scene.

20. The computing device of claim 18, wherein histograms are stored in a training database and are based on at least two images captured of respective scenes, wherein the histograms in the training database are associated with respective dynamic range parameters, wherein the respective dynamic range parameters indicate whether the respective scenes exhibit low dynamic range (LDR) or high dynamic range (HDR), and wherein determining the TET sequence further comprises:

comparing the scene histogram to at least one of the histograms in the training database; and

based on an outcome of the comparison, determining a dynamic range parameter for the scene from the respective dynamic range parameters, wherein determining the TET sequence is further based on the dynamic range parameter for the scene.