System, method, and computer program product implementing an image processing pipeline for high-dynamic range images

• US 9,071,765 B2
• Filed: 12/28/2012
• Issued: 06/30/2015
• Est. Priority Date: 12/28/2012
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Motta, Ricardo Jansson
• Primary Examiner(s): Tran, Nhan T
• Assistant Examiner(s): NGUYEN, CHAN T H
• Application Number:
  US13/730,639
• Publication Number:
  US 20140184894A1
• Time in Patent Office: 914 Days
• Field of Search: 348/229.1, 348/222.1, 348/218.1, 348/239
• US Class Current: 1/1
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 25/583 with different integration times

Abstract

A system, method, and computer program product for generating high-dynamic range image data is disclosed. The method includes the steps of receiving image sensor data from an interleaved image sensor. The interleaved the image sensor includes a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time that is shorter than the first exposure time. The method further includes the steps of identifying a first subset of pixels in the second portion having an intensity value above a first threshold value, identifying a second subset of pixels in the first portion having an intensity value below a second threshold value, and generating high-dynamic range (HDR) data based on the first subset and the second subset.

Claims

1. A method comprising:

capturing image sensor data from an interleaved image sensor that includes a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time that is less than the first exposure time by;

resetting the pixels in the first portion at a first reset time,

resetting the pixels in the second portion at a second reset time, and

sampling the pixels in the first portion and the second portion after a sampling time has elapsed since the first reset time;

receiving the image sensor data from the interleaved image sensor;

identifying a first subset of pixels in the second portion having an intensity value above a first threshold value;

identifying a second subset of pixels in the first portion having an intensity value below a second threshold value; and

generating high-dynamic range (HDR) data based on the first subset and the second subset,

wherein the difference between the sampling time and the first reset time is equal to the first exposure time and the difference between the sampling time and the second reset time is equal to the second exposure time.

2. The method of claim 1, wherein the interleaved image sensor includes a Bayer pattern color filter array arranged in a plurality of quad rows, and wherein the first portion comprises odd quad rows of the interleaved image sensor and the second portion comprises even quad rows of the interleaved image sensor.

3. The method of claim 1, wherein identifying the first subset of pixels in the second portion comprises generating a first mask that identifies the pixels in the second portion that have an intensity value greater than the first threshold value, and wherein identifying the second subset of pixels in the first portion comprises generating a second mask that identifies the pixels in the first portion that have an intensity value less than the second threshold value.

4. The method of claim 1, wherein generating high-dynamic range data comprises generating an intensity value for each pixel in the HDR data by:

determining whether a corresponding pixel associated with an index for the pixel is included in the first portion or the second portion; and

if the corresponding pixel is included in the first portion, then;

determining whether a neighboring pixel of the corresponding pixel is included in the first subset, and

if the neighboring pixel is included in the first subset, then generating the intensity value for the pixel by filtering one or more sampled values in the second portion, or

if the neighboring pixel is not included in the first subset, then generating the intensity value for the pixel by selecting the intensity value for the corresponding pixel;

or

if the corresponding pixel is included in the second portion, then;

determining whether the neighboring pixel is included in the second subset, and

if the neighboring pixel is included in the second subset, then generating the intensity value for the pixel by filtering one or more sampled values in the first portion to generate a filtered value and scaling the filtered value, or

if the neighboring pixel is not included in the second subset, then generating the intensity value for the pixel by scaling the intensity value for the corresponding pixel by an exposure ratio.

5. The method of claim 4, wherein determining whether the neighboring pixel is included in the first subset comprises sampling a first mask, and wherein determining whether the neighboring pixel is included in the second subset comprises sampling a second mask.

6. The method of claim 5, wherein the first mask is generated by performing a comparison operation between the intensity level of the neighboring pixel and the first threshold value, and wherein the second mask is generated by performing a comparison operation between the intensity level of the neighboring pixel and the second threshold value.

7. The method of claim 4, wherein filtering comprises performing a linear interpolation between two sampled values.

8. The method of claim 4, wherein filtering comprises taking a weighted sum of a plurality of sampled values.

9. The method of claim 4, wherein filtering is combined with demosaic processing that is based on the weighted sum of a plurality of sampled values.

10. The method of claim 1, further comprising modifying the HDR data by companding the HDR data.

11. The method of claim 10, further comprising transmitting the modified HDR data to an image signal processor that is configured to perform at least one of the functions of noise reduction, demosaicing, color conversion, and gamma correction.

12. The method of claim 1, further comprising:

identifying a third subset of pixels in the second portion having an intensity value above a third threshold value and below the first threshold value;

identifying a fourth subset of pixels in the first portion having an intensity value below a fourth threshold value and above a second threshold value;

determining that a neighboring pixel is included in the third subset, then blending a first intensity value for the pixel generated by filtering one or more values in the second portion with a second intensity value for the pixel generated by selecting the intensity value for a corresponding pixel; and

determining that a neighboring pixel is included in the fourth subset, then blending a third intensity value for the pixel by filtering one or more values in the first portion with a fourth intensity value for the pixel generated by scaling the intensity value for the corresponding pixel by an exposure ratio.

13. The method of claim 12, wherein blending comprises performing a linear interpolation based on the intensity value of the neighboring pixel.

14. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform steps comprising:

capturing image sensor data from an interleaved image sensor that includes a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time that is less than the first exposure time by;

resetting the pixels in the first portion at a first reset time,

resetting the pixels in the second portion at a second reset time, and

sampling the pixels in the first portion and the second portion after a sampling time has elapsed since the first reset time;

receiving the image sensor data from the interleaved image sensor;

identifying a first subset of pixels in the second portion having an intensity value above a first threshold value;

identifying a second subset of pixels in the first portion having an intensity value below a second threshold value; and

generating high-dynamic range (HDR) data based on the first subset and the second subset,

wherein the difference between the sampling time and the first reset time is equal to the first exposure time and the difference between the sampling time and the second reset time is equal to the second exposure time.

15. The non-transitory computer-readable storage medium of claim 14, wherein generating high-dynamic range data comprises generating an intensity value for each pixel in an HDR image by:

determining whether a corresponding pixel associated with an index for the pixel is included in the first portion or the second portion; and

if the corresponding pixel is included in the first portion, then;

determining whether a neighboring pixel of the corresponding pixel is included in the first subset, and

if the neighboring pixel is included in the first subset, then generating the intensity value for the pixel by filtering one or more values in the second portion, or

if the neighboring pixel is not included in the first subset, then generating the intensity value for the pixel by scaling the intensity value for the corresponding pixel by an exposure ratio;

or

if the corresponding pixel is included in the second portion, then;

determining whether the neighboring pixel is included in the second subset, and

if the neighboring pixel is included in the second subset, then generating the intensity value for the pixel by filtering one or more values in the first portion, or

if the neighboring pixel is not included in the second subset, then generating the intensity value for the pixel by selecting the intensity value for the corresponding pixel.

16. The non-transitory computer-readable storage medium of claim 15, wherein filtering comprises performing a linear interpolation between two sampled values.

17. A system, comprising:

an interleaved image sensor that includes a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time that is less than the first exposure time, the interleaved image sensor configured to capture image sensor data by;

resetting the pixels in the first portion at a first reset time,

resetting the pixels in the second portion at a second reset time, and

sampling the pixels in the first portion and the second portion after a sampling time has elapsed since the first reset time; and

an image processing pipeline coupled to the interleaved image sensor and configured to;

receive the image sensor data from the interleaved image sensor,

identify a first subset of pixels in the second portion having an intensity value above a first threshold value,

identify a second subset of pixels in the first portion having an intensity value below a second threshold value, and

generate high-dynamic range (HDR) data based on the first subset and the second subset,

wherein the difference between the sampling time and the first reset time is equal to the first exposure time and the difference between the sampling time and the second reset time is equal to the second exposure time.

18. The system of claim 17, wherein generating high-dynamic range data comprises generating an intensity value for each pixel in an HDR image by:

determining whether a corresponding pixel associated with an index for the pixel is included in the first portion or the second portion; and

if the corresponding pixel is included in the first portion, then;

determining whether a neighboring pixel of the corresponding pixel is included in the first subset, and

if the neighboring pixel is included in the first subset, then generating the intensity value for the pixel by filtering one or more values in the second portion, or

if the neighboring pixel is not included in the first subset, then generating the intensity value for the pixel by scaling the intensity value for the corresponding pixel by an exposure ratio;

or

if the corresponding pixel is included in the second portion, then;

determining whether the neighboring pixel is included in the second subset, and

if the neighboring pixel is included in the second subset, then generating the intensity value for the pixel by filtering one or more values in the first portion, or

if the neighboring pixel is not included in the second subset, then generating the intensity value for the pixel by selecting the intensity value for the corresponding pixel.

19. The system of claim 17, wherein the image processing pipeline is implemented as a shader program configured to be executed by a graphics processing unit (GPU).