Determining exposure times using split paxels

• US 9,172,888 B2
• Filed: 09/17/2014
• Issued: 10/27/2015
• Est. Priority Date: 12/18/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Geiss, Ryan
• Primary Examiner(s): AGGARWAL, YOGESH K
• Application Number:
  US14/488,891
• Publication Number:
  US 20150002733A1
• Time in Patent Office: 405 Days
• Field of Search: 3482211-2291, 348/362
• US Class Current: 1/1
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof

Abstract

A plurality of images of a scene may be captured. Each image of the plurality of images may be captured using a different total exposure time (TET). The images in the plurality of images may be downsampled. A pixel value histogram based on pixel values of the downsampled images may be constructed. The pixel value histogram may be compared to one or more reference pixel value histograms. A payload TET may be determined based on comparing the pixel value histogram to the one or more reference pixel value histograms.

Claims

1. A method comprising:

capturing, by an image sensor, a plurality of images of a scene, wherein each image of the plurality of images is captured using a different total exposure time (TET);

downsampling the images in the plurality of images;

constructing a pixel value histogram based on pixel values of the downsampled images;

comparing the pixel value histogram to one or more reference pixel value histograms, wherein comparing the pixel value histogram to the one or more reference pixel value histograms comprises determining, for the pixel value histogram and each of the one or more reference pixel value histograms, respective similarity metrics, and determining respective weights based on the similarity metrics;

determining a payload TET based on comparing the pixel value histogram to the one or more reference pixel value histograms; and

capturing at least one additional image of the scene using the payload TET.

2. The method of claim 1, wherein downsampling the images in the plurality of images comprises forming a plurality of high-average downsampled images and a second plurality of low-average downsampled images, and wherein constructing the pixel value histogram based on pixel values of the downsampled images comprises constructing the pixel value histogram based on pixel values of the plurality of high-average downsampled images and the plurality of low-average downsampled images.

3. The method of claim 2, wherein forming each image of the plurality of high-average downsampled images comprises:

dividing each image in the plurality of high-average downsampled images into respective non-overlapping matrices of paxels, wherein each paxel in the non-overlapping matrices of paxels represents at least a 2×

2 pixel tile of the respective image in the plurality of images;

calculating average pixel values of each paxel, wherein the average pixel values are respective average values of all pixels within the respective paxel; and

calculating high-average pixel values of each paxel, wherein the high-average pixel values are respective average values of all pixels within the respective paxel with values greater than or equal to the average pixel value of the respective paxel.

4. The method of claim 3, wherein forming each image of the plurality of low-average downsampled images comprises:

calculating low-average pixel values of each paxel, wherein the low-average pixel values are respective average values of all pixels within the respective paxel with values less than the average pixel value of the respective paxel.

5. The method of claim 4, wherein downsampling the images in the plurality of images further comprises:

calculating respective high pixel fractions for each paxel, wherein the high pixel fractions are respective ratios of (i) pixels within the respective paxel with values greater than or equal to the average pixel value of the respective paxel, to (ii) total pixels within the respective paxel; and

calculating respective low pixel fractions for each paxel, wherein the low pixel fractions are respective ratios of (i) pixels within the respective paxel with values less than the average pixel value of the respective paxel, to (ii) total pixels within the respective paxel.

6. The method of claim 5, wherein constructing the pixel value histogram based on the pixel values of the downsampled images comprises:

combining images from the plurality of high-average downsampled images into a combined high-average downsampled image; and

combining images from the plurality of low-average downsampled images into a combined low-average downsampled image.

7. The method of claim 6, wherein combining images from first plurality of high-average downsampled images into the combined high-average downsampled image comprises for each pixel location in the combined high-average downsampled image, selecting a high-average pixel value from the same pixel location in one of the high-average downsampled images, and wherein combining images from the second plurality of low-average downsampled images into the combined low-average downsampled image comprises for each pixel location in the combined low-average downsampled image, selecting a low-average pixel value from the same pixel location in one of the low-average downsampled images.

8. The method of claim 7, wherein constructing the pixel value histogram based on the pixel values of the downsampled images further comprises:

adding, to the pixel value histogram, a first set of quantities representing each high-average pixel value in the combined high-average downsampled image, wherein the quantities in the first set of quantities are based on the respective associated high pixel fractions; and

adding, to the pixel value histogram, a second set of quantities representing each low-average pixel value in the combined low-average downsampled image, wherein the quantities in the second set of quantities are based on the respective associated low pixel fractions.

9. The method of claim 1, wherein the one or more reference pixel value histograms are associated with respective target average pixel values, and wherein the payload TET is based on a sum of the respective weights applied to the respective target average pixel values.

10. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

capturing, by an image sensor, a plurality of images of a scene, wherein each image of the plurality of images is captured using a different total exposure time (TET);

downsampling the images in the plurality of images;

constructing a pixel value histogram based on pixel values of the downsampled images;

comparing the pixel value histogram to one or more reference pixel value histograms, wherein comparing the pixel value histogram to the one or more reference pixel value histograms comprises determining, for the pixel value histogram and each of the one or more reference pixel value histograms, respective similarity metrics, and determining respective weights based on the similarity metrics;

determining a payload TET based on comparing the pixel value histogram to the one or more reference pixel value histograms; and

capturing at least one additional image of the scene using the payload TET.

11. The article of manufacture of claim 10, wherein downsampling the images in the plurality of images comprises forming a plurality of high-average downsampled images and a second plurality of low-average downsampled images, and wherein constructing the pixel value histogram based on pixel values of the downsampled images comprises constructing the pixel value histogram based on pixel values of the plurality of high-average downsampled images and the plurality of low-average downsampled images.

12. The article of manufacture of claim 11, wherein forming each image of the plurality of high-average downsampled images comprises:

dividing each image in the plurality of high-average downsampled images into respective non-overlapping matrices of paxels, wherein each paxel in the non-overlapping matrices of paxels represents at least a 2×

2 pixel tile of the respective image in the plurality of images;

calculating average pixel values of each paxel, wherein the average pixel values are respective average values of all pixels within the respective paxel; and

calculating high-average pixel values of each paxel, wherein the high-average pixel values are respective average values of all pixels within the respective paxel with values greater than or equal to the average pixel value of the respective paxel.

13. The article of manufacture of claim 12, wherein forming each image of the plurality of low-average downsampled images comprises:

calculating low-average pixel values of each paxel, wherein the low-average pixel values are respective average values of all pixels within the respective paxel with values less than the average pixel value of the respective paxel.

14. The article of manufacture of claim 13, wherein downsampling the images in the plurality of images further comprises:

calculating respective high pixel fractions for each paxel, wherein the high pixel fractions are respective ratios of (i) pixels within the respective paxel with values greater than or equal to the average pixel value of the respective paxel, to (ii) total pixels within the respective paxel; and

calculating respective low pixel fractions for each paxel, wherein the low pixel fractions are respective ratios of (i) pixels within the respective paxel with values less than the average pixel value of the respective paxel, to (ii) total pixels within the respective paxel.

15. The article of manufacture of claim 14, wherein constructing the pixel value histogram based on the pixel values of the downsampled images comprises:

combining images from the plurality of high-average downsampled images into a combined high-average downsampled image; and

combining images from the plurality of low-average downsampled images into a combined low-average downsampled image.

16. The article of manufacture of claim 15, wherein combining images from first plurality of high-average downsampled images into the combined high-average downsampled image comprises for each pixel location in the combined high-average downsampled image, selecting a high-average pixel value from the same pixel location in one of the high-average downsampled images, and wherein combining images from the second plurality of low-average downsampled images into the combined low-average downsampled image comprises for each pixel location in the combined low-average downsampled image, selecting a low-average pixel value from the same pixel location in one of the low-average downsampled images.

17. The article of manufacture of claim 16, wherein constructing the pixel value histogram based on the pixel values of the downsampled images further comprises:

adding, to the pixel value histogram, a first set of quantities representing each high-average pixel value in the combined high-average downsampled image, wherein the quantities in the first set of quantities are based on the respective associated high pixel fractions; and

adding, to the pixel value histogram, a second set of quantities representing each low-average pixel value in the combined low-average downsampled image, wherein the quantities in the second set of quantities are based on the respective associated low pixel fractions.

18. A computing device comprising:

at least one processor;

an image sensor;

data storage; and

program instructions, stored in the data storage, that upon execution by the at least one processor cause the computing device to perform operations including;

capturing, by the image sensor, a plurality of images of a scene, wherein each image of the plurality of images is captured using a different total exposure time (TET);

downsampling the images in the plurality of images;

constructing a pixel value histogram based on pixel values of the downsampled images;

comparing the pixel value histogram to one or more reference pixel value histograms, wherein comparing the pixel value histogram to the one or more reference pixel value histograms comprises determining, for the pixel value histogram and each of the one or more reference pixel value histograms, respective similarity metrics, and determining respective weights based on the similarity metrics;

determining a payload TET based on comparing the pixel value histogram to the one or more reference pixel value histograms; and

capturing at least one additional image of the scene using the payload TET.

19. The computing device of claim 18, wherein downsampling the images in the plurality of images comprises forming a plurality of high-average downsampled images and a second plurality of low-average downsampled images, and wherein constructing the pixel value histogram based on pixel values of the downsampled images comprises constructing the pixel value histogram based on pixel values of the plurality of high-average downsampled images and the plurality of low-average downsampled images.

20. The computing device of claim 19, wherein forming each image of the plurality of high-average downsampled images comprises:

dividing each image in the plurality of high-average downsampled images into respective non-overlapping matrices of paxels, wherein each paxel in the non-overlapping matrices of paxels represents at least a 2×

2 pixel tile of the respective image in the plurality of images;

calculating average pixel values of each paxel, wherein the average pixel values are respective average values of all pixels within the respective paxel; and

calculating high-average pixel values of each paxel, wherein the high-average pixel values are respective average values of all pixels within the respective paxel with values greater than or equal to the average pixel value of the respective paxel.