Noise models for image processing

• US 9,749,551 B2
• Filed: 04/19/2017
• Issued: 08/29/2017
• Est. Priority Date: 02/05/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Geiss, Ryan, Zhou, Changyin, Hasinoff, Samuel William
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US15/491,429
• Publication Number:
  US 20170223282A1
• Time in Patent Office: 132 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A plurality of images of a scene may be obtained. These images may have been captured by an image sensor, and may include a first image and a second image. A particular gain may have been applied to the first image. An effective color temperature and a brightness of a first pixel in the first image may be determined, and a mapping between pixel characteristics and noise deviation of the image sensor may be selected. The pixel characteristics may include pixel brightness. The selected mapping may be used to map at least the brightness of the first pixel to a particular noise deviation. The brightness of the first pixel and the particular noise deviation may be compared to a brightness of a second pixel of the second image. The comparison may be used to determine whether to merge the first pixel and the second pixel.

Claims

1. A method comprising:

determining a color temperature of a first pixel and a brightness of the first pixel, wherein the first pixel appears in a first image that was captured by an image sensor;

selecting a mapping between pixel brightness and noise deviation of the image sensor, wherein the noise deviation characterizes error in pixels that have the color temperature of the first pixel, wherein the error is introduced when the image sensor is capturing an image;

using the selected mapping to map at least the brightness of the first pixel to a particular noise deviation; and

based on the brightness of the first pixel, the particular noise deviation, and a brightness of a second pixel, determining whether to merge the first pixel and the second pixel, wherein the second pixel appears in a second image that was captured by the image sensor.

2. The method of claim 1, wherein a particular gain has been applied to the first image, and wherein the noise deviation characterizes the error in pixels that have the color temperature of the first pixel and have had the particular gain applied.

3. The method of claim 2, wherein the mapping is selected based on the color temperature of the first pixel and the particular gain.

4. The method of claim 1, wherein determining whether to merge the first pixel and the second pixel comprises comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel.

5. The method of claim 4, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel of the second image comprises:

determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.

6. The method of claim 5, wherein merging the first pixel and the second pixel comprises:

adding the brightness of the first pixel and the brightness of the second pixel to provide a pixel value sum; and

dividing the pixel value sum by two to form a merged pixel.

7. The method of claim 5, wherein the defined range is configured such that the brightness of the second pixel is within the defined range when the first pixel and the second pixel are subject to a sub-pixel alignment error.

8. The method of claim 4, wherein a first tile of pixels contains the first pixel and a second tile of pixels contains the second pixel, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation;

determining that the root-mean-square difference of pixel brightness values of the first tile and pixel brightness values of the second tile exceeds a root-mean-square difference threshold; and

based on the root-mean-square difference exceeding the root-mean-square difference threshold, merging no pixels from the second tile with the pixels from the first tile.

9. The method of claim 4, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is outside of a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being outside of the defined range, not merging the first pixel and the second pixel.

10. The method of claim 1, wherein a noise model for the image sensor associates color temperatures to mappings between pixel characteristics and the noise deviation of the image sensor, and wherein selecting the mapping between the pixel characteristics and the noise deviation of the image sensor comprises selecting the mapping from the noise model for the image sensor.

11. The method of claim 10, wherein the noise model comprises a plurality of discrete points, each point defined by a respective color temperature and associated with a respective mapping, and wherein selecting the mapping between the pixel characteristics and the noise deviation of the image sensor comprises:

determining that the color temperature of the first pixel does not correspond to any of the discrete points;

selecting a discrete point defined by a target color temperature, wherein target color temperature is a closest color temperature to the color temperature of the first pixel; and

selecting the mapping associated with the discrete point.

12. The method of claim 1, wherein mapping at least the brightness of the first pixel to the particular noise deviation comprises looking up at least the brightness of the first pixel in a pre-determined table to derive the particular noise deviation.

13. The method of claim 12, wherein mapping at least the brightness of the first pixel to the particular noise deviation further comprises looking up a color value of the first pixel in the pre-determined table to derive the particular noise deviation.

14. The method of claim 1, wherein the first image and the second image were captured using different total exposure times.

15. The method of claim 1, wherein the image sensor is disposed within a camera device, wherein the mapping between the pixel brightness and the noise deviation of the image sensor comprises a mapping between the pixel brightness and a combination noise deviation of the image sensor and other components of the camera device.

16. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

determining a color temperature of a first pixel and a brightness of the first pixel, wherein the first pixel appears in a first image that was captured by an image sensor;

selecting a mapping between pixel brightness and noise deviation of the image sensor, wherein the noise deviation characterizes error in pixels of the color temperature of the first pixel, wherein the error is introduced when the image sensor is capturing an image;

using the selected mapping to map at least the brightness of the first pixel to a particular noise deviation; and

based on the brightness of the first pixel, the particular noise deviation, and a brightness of a second pixel, determining whether to merge the first pixel and the second pixel, wherein the second pixel appears in a second image that was captured by the image sensor.

17. The article of manufacture of claim 16, wherein a particular gain has been applied to the first image, and wherein the noise deviation characterizes the error in pixels that have the color temperature of the first pixel and have had the particular gain applied.

18. The article of manufacture of claim 17, wherein comparing the brightness of the first pixel and the particular noise deviation to the brightness of the second pixel comprises:

determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the particular noise deviation; and

based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.

19. The article of manufacture of claim 17, wherein merging the first pixel and the second pixel comprises:

adding the brightness of the first pixel and the brightness of the second pixel to provide a pixel value sum; and

dividing the pixel value sum by two to form a merged pixel.

20. A computing device comprising:

one or more processors;

memory; and

program instructions, stored in the memory, that upon execution by the one or more processors cause the computing device to perform operations including;

determining a color temperature of a first pixel and a brightness of the first pixel, wherein the first pixel appears in a first image that was captured by an image sensor;

selecting a mapping between pixel brightness and noise deviation of the image sensor, wherein the noise deviation characterizes error in pixels of the color temperature of the first pixel, wherein the error is introduced when the image sensor is capturing an image;

using the selected mapping to map at least the brightness of the first pixel to a particular noise deviation; and

based on the brightness of the first pixel, the particular noise deviation, and a brightness of a second pixel, determining whether to merge the first pixel and the second pixel, wherein the second pixel appears in a second image that was captured by the image sensor.