Interleaved capture for high dynamic range image acquisition and synthesis

• US 9,100,589 B1
• Filed: 04/16/2013
• Issued: 08/04/2015
• Est. Priority Date: 09/11/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Geiss, Ryan
• Primary Examiner(s): Dagnew, Mekonnen
• Application Number:
  US13/863,981
• Time in Patent Office: 840 Days
• Field of Search: 348/221.1, 348/229.1, 348362-368, 348/218.1
• US Class Current: 1/1
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A computing device may capture a first image using a first exposure length. It may be determined that x pixels in the captured first image are properly-exposed and that y pixels in the captured first image are improperly-exposed. Based on the y improperly-exposed pixels, a second exposure length may be determined. A second image may be captured using the second exposure length. The second image may include y′ pixels that correspond to the y improperly-exposed pixels in the first image. At least some of the x properly-exposed pixels from the first image may be combined with at least some of the y′ pixels from the second image to form a composite image.

Claims

1. A method comprising:

obtaining a first image that was captured using a first exposure length, wherein the first image includes a first set of two or more properly-exposed pixels and a second set of two or more improperly-exposed pixels;

determining a second exposure length based on pixel values of the second set of two or more improperly-exposed pixels in the first image;

obtaining a second image that was captured using the second exposure length, wherein the second image includes a third set of two or more pixels that correspond to the second set of two or more improperly-exposed pixels in the first image;

determining a third exposure length based on the first set of two or more properly-exposed pixels in the first image and a first target average pixel value;

obtaining a third image that was captured using the third exposure length, wherein the third image includes a fourth set of two or more properly-exposed pixels and a fifth set of two or more improperly-exposed pixels;

determining a fourth exposure length based on the fifth set of two or more improperly-exposed pixels in the third image and a second target average pixel value, wherein the second target average pixel value is different from the first target average pixel value;

obtaining a fourth image that was captured using the fourth exposure length, wherein the fourth image includes a sixth set of two or more pixels that correspond to the fifth set of two or more improperly-exposed pixels in the third image; and

forming, by a computing device, a fifth image by selecting pixels from the fourth set of properly-exposed pixels in the third image and pixels from the sixth set of two or more pixels in the fourth image that correspond to the fifth set of two or more improperly-exposed pixels in the third image.

2. The method of claim 1, wherein the captured first image and the captured second image each consist of a particular number of pixels.

3. The method of claim 1, wherein the second set of two or more improperly-exposed pixels in the first image and the third set of two or more pixels in the second image each consist of a particular number of pixels.

4. The method of claim 1, wherein each pixel is associated with a pixel value inclusively between 0 and 255, wherein pixel values of pixels in the first set of two or more properly-exposed pixels in the first image are within a particular range, and wherein endpoints of the particular range are inclusively between 96 and 240.

5. The method of claim 1, wherein the first exposure length is based on input from an ambient light sensor.

6. The method of claim 1, wherein pixel values of pixels in the first set of two or more properly-exposed pixels in the first image are between an under-exposure threshold value and an over-exposure threshold value.

7. The method of claim 6, wherein pixel values of pixels in the second set of two or more improperly-exposed pixels in the first image are above the over-exposure threshold value, and wherein the second exposure length is shorter than the first exposure length.

8. The method of claim 6, wherein pixel values of pixels in the second set of two or more improperly-exposed pixels in the first image are below the under-exposure threshold value, and wherein the second exposure length is longer than the first exposure length.

9. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

obtaining a first image that was captured using a first exposure length, wherein the first image includes a first set of two or more properly-exposed pixels and a second set of two or more improperly-exposed pixels;

determining a second exposure length based on pixel values of the second set of two or more improperly-exposed pixels of the first image;

obtaining a second image that was captured using the second exposure length, wherein the second image includes a third set of two or more pixels that correspond to the second set of two or more improperly-exposed pixels in the first image;

determining a third exposure length based on the first set of two or more properly-exposed pixels in the first image and a first target average pixel value;

obtaining a third image that was captured using the third exposure length, wherein the third image includes a fourth set of two or more properly-exposed pixels and a fifth set of two or more improperly-exposed pixels;

determining a fourth exposure length based on the fifth set of two or more improperly-exposed pixels in the third image and a second target average pixel value, wherein the second target average pixel value is different from the first target average pixel value;

obtaining a fourth image that was captured using the fourth exposure length, wherein the fourth image includes a sixth set of two or more pixels that correspond to the fifth set of two or more improperly-exposed pixels in the third image; and

forming a fifth image by selecting pixels from the fourth set of properly-exposed pixels in the third image and pixels from the sixth set of two or more pixels in the fourth image that correspond to the fifth set of two or more improperly-exposed pixels in the third image.

10. The article of manufacture of claim 9, wherein the captured first image and the captured second image each consist of a particular number of pixels.

11. The article of manufacture of claim 9, wherein the second set of two or more improperly-exposed pixels of the first image and the third set of two or more pixels in the second image each consist of a particular number of pixels.

12. The article of manufacture of claim 9, wherein each pixel is associated with a pixel value inclusively between 0 and 255, wherein pixel values of pixels in the first set of two or more properly-exposed pixels in the first image are within a particular range, and wherein endpoints of the particular range are inclusively between 96 and 240.

13. The article of manufacture of claim 9, wherein the first exposure length is based on input from an ambient light sensor.

14. The article of manufacture of claim 9, wherein pixel values of pixels in the first set of two or more properly-exposed pixels in the first image are between an under-exposure threshold value and an over-exposure threshold value.

15. The article of manufacture of claim 14, wherein pixel values of pixels in the second set of two or more improperly-exposed pixels in the first image are above the over-exposure threshold value, and wherein the second exposure length is shorter than the first exposure length.

16. The article of manufacture of claim 14, wherein pixel values of pixels in the second set of two or more improperly-exposed pixels in the first image are below the under-exposure threshold value, and wherein the second exposure length is longer than the first exposure length.

17. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

capturing a sequence of m image pairs, wherein m is at least, wherein the first image of each ith image pair is captured using a respective exposure length of e(i), wherein i ranges from 1 to m inclusively and m is an integer, wherein x(i) pixels in the first image of the ith image pair are properly-exposed and y(i) pixels in the first image of the ith image pair are improperly-exposed, wherein the second image in the ith pair is captured using a respective exposure length of f(i) that is based on pixel values of the y(i) improperly-exposed pixels in the first image of the ith image pair, and wherein the second image in the ith image pair includes y′

(i) pixels that correspond to the y(i) improperly-exposed pixels, wherein e(k) is based a target average pixel value and the properly-exposed pixels in the first image of the (k-1)th image pair when k is greater than 1; and

forming a new image by selecting pixels from the x(m) properly-exposed pixels and pixels from the y′

(m) pixels.

18. The article of manufacture of claim 17, wherein determining that the x(i) pixels are properly-exposed comprises determining that pixel values of the x(i) pixels are between an under-exposure threshold value and an over-exposure threshold value.

19. The article of manufacture of claim 18, wherein determining that the y(i) pixels are improperly-exposed comprises determining that the pixel values of the y(i) pixels are above the over-exposure threshold value, and wherein f(i) is shorter than e(i).

20. The article of manufacture of claim 18, wherein determining that the y(i) pixels are improperly-exposed comprises determining that the pixel values of the y(i) pixels are below the under-exposure threshold value, and wherein f(i) is longer than e(i).