Noise models for image processing

• US 9,686,537 B2
• Filed: 02/05/2013
• Issued: 06/20/2017
• Est. Priority Date: 02/05/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhou, Changyin, Geiss, Ryan, Hasinoff, Samuel William
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US13/759,749
• Publication Number:
  US 20140218540A1
• Time in Patent Office: 1,596 Days
• Field of Search: 3482221, 3482231, 3482291, 3482301, 348239, 348241, 348242, 348255, 348362, 382162, 382167, 382267, 382270, 382274, 382300
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A plurality of images of a scene may be obtained. These images may have been captured by an image sensor, and may include a first image and a second image. A particular gain may have been applied to the first image. An effective color temperature and a brightness of a first pixel in the first image may be determined, and a mapping between pixel characteristics and noise deviation of the image sensor may be selected. The pixel characteristics may include pixel brightness. The selected mapping may be used to map at least the brightness of the first pixel to a particular noise deviation. The brightness of the first pixel and the particular noise deviation may be compared to a brightness of a second pixel of the second image. The comparison may be used to determine whether to merge the first pixel and the second pixel.

Claims

1. A method comprising:

obtaining, by a computing device, a plurality of images of a scene, wherein the plurality of images were captured by an image sensor, and wherein the plurality of images includes a first image and a second image, wherein a particular gain has been applied to the first image;

determining an effective color temperature and a brightness of a first pixel in the first image;

based on the effective color temperature and the particular gain, selecting a mapping between pixel characteristics and noise deviation of the image sensor, wherein the pixel characteristics include pixel brightness, and wherein the noise deviation characterizes error introduced by the image sensor when capturing images with the effective color temperature and the particular gain;

using the selected mapping to map at least the brightness of the first pixel to a particular noise deviation;

comparing the brightness of the first pixel and the particular noise deviation to a brightness of a second pixel of the second image; and

based on the comparison, determining whether to merge the first pixel and the second pixel.

2. The method of claim 1, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.

3. The method of claim 2, wherein merging the first pixel and the second pixel comprises:

adding the brightness of the first pixel and the brightness of the second pixel to provide a pixel value sum; and

dividing the pixel value sum by two.

4. The method of claim 2, wherein the defined range is configured such that the brightness of the second pixel is within the defined range when the first pixel and the second pixel are subject to a sub-pixel alignment error.

5. The method of claim 1, wherein a first tile of pixels contains the first pixel and a second tile of pixels contains the second pixel, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation;

determining that the root-mean-square difference of pixel brightness values of the first tile and pixel brightness values of the second tile exceeds a root-mean-square difference threshold; and

based on the root-mean-square difference exceeding the root-mean-square difference threshold, merging no pixels from the second tile with the pixels from the first tile.

6. The method of claim 1, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is outside of a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being outside of the defined range, not merging the first pixel and the second pixel.

7. The method of claim 1, wherein a noise model for the image sensor associates effective color temperatures and gains to mappings between pixel characteristics and noise deviation of the image sensor, and wherein selecting the mapping between pixel characteristics and noise deviation of the image sensor based on the effective color temperature and the particular gain comprises selecting the mapping from the noise model for the image sensor.

8. The method of claim 7, wherein the noise model comprises a plurality of discrete points, each point defined by a respective color temperature in combination with a respective gain and associated with a respective mapping, and wherein selecting the mapping between pixel characteristics and noise deviation of the image sensor based on the effective color temperature and the particular gain comprises:

determining that the effective color temperature in combination with the particular gain does not correspond to any of the discrete points;

based on the effective color temperature in combination with the particular gain not corresponding to any of the discrete points, selecting a discrete point defined by a target color temperature and a target gain, wherein the target gain is higher than the particular gain and the target color temperature is a closest color temperature to the effective color temperature; and

selecting the mapping associated with the discrete point.

9. The method of claim 1, wherein mapping at least the brightness of the first pixel to the particular noise deviation comprises looking up at least the brightness of the first pixel in a pre-determined table to derive the particular noise deviation.

10. The method of claim 9, wherein mapping at least the brightness of the first pixel to the particular noise deviation further comprises looking up a color value of the first pixel in the pre-determined table to derive the particular noise deviation.

11. The method of claim 1, wherein at least two images of the plurality of images were captured using different total exposure times (TETs).

12. The method of claim 1, wherein the plurality of images were captured by the image sensor of a camera device, wherein the mapping between pixel brightness and noise deviation of the image sensor comprises a mapping between pixel brightness and noise deviation of the image sensor and other components of the camera device.

13. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

obtaining a plurality of images of a scene captured by an image sensor, wherein the plurality of images includes a first image and a second image, and wherein a particular gain has been applied to the first image;

determining an effective color temperature and a brightness of a first pixel in the first image;

based on the effective color temperature and the particular gain, selecting a mapping between pixel characteristics and noise deviation of the image sensor, wherein the pixel characteristics include pixel brightness, and wherein the noise deviation characterizes error introduced by the image sensor when capturing images with the effective color temperature and the particular gain;

using the selected mapping to map at least the brightness of the first pixel to a particular noise deviation;

comparing the brightness of the first pixel and the particular noise deviation to a brightness of a second pixel of the second image; and

based on the comparison, determining whether to merge the first pixel and the second pixel.

14. The article of manufacture of claim 13, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.

15. The article of manufacture of claim 14, wherein merging the first pixel and the second pixel comprises:

adding the brightness of the first pixel and the brightness of the second pixel to provide a pixel value sum; and

dividing the pixel value sum by two.

16. The article of manufacture of claim 13, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is outside of a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being outside of the defined range, not merging the first pixel and the second pixel.

17. The article of manufacture of claim 13, wherein a noise model for the image sensor associates effective color temperatures and gains to mappings between pixel characteristics and noise deviation of the image sensor, and wherein selecting the mapping between pixel characteristics and noise deviation of the image sensor based on the effective color temperature and the particular gain comprises selecting the mapping from the noise model for the image sensor.

18. A computing device comprising:

at least one processor;

data storage; and

program instructions, stored in the data storage, that upon execution by the at least one processor cause the computing device to perform operations including;

obtaining a plurality of images of a scene captured by an image sensor, wherein the plurality of images includes a first image and a second image, and wherein a particular gain has been applied to the first image;

determining an effective color temperature and a brightness of a first pixel in the first image;

based on the effective color temperature and the particular gain, selecting a mapping between pixel characteristics and noise deviation of the image sensor, wherein the pixel characteristics include pixel brightness, and wherein the noise deviation characterizes error introduced by the image sensor when capturing images with the effective color temperature and the particular gain;

using the selected mapping to map at least the brightness of the first pixel to a particular noise deviation;

comparing the brightness of the first pixel and the particular noise deviation to a brightness of a second pixel of the second image; and

based on the comparison, determining whether to merge the first pixel and the second pixel.

19. The computing device of claim 18, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.

20. The computing device of claim 18, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is outside of a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being outside of the defined range, not merging the first pixel and the second pixel.