Method and apparatus for obtaining high dynamic range images

• US 7,084,905 B1
• Filed: 02/23/2000
• Issued: 08/01/2006
• Est. Priority Date: 02/23/2000
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Sony Corporation (Sony Group Corp.)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Sony Corporation (Sony Group Corp.)
• Inventors: Nayar, Shree K., Mitsunaga, Tomoo
• Primary Examiner(s): Garber, Wendy R.
• Assistant Examiner(s): Yoder, III, Chriss S.
• Application Number:
  US09/511,469
• Time in Patent Office: 2,351 Days
• Field of Search: 348187-188, 348/221.1, 348/222.1, 348/229.1, 348/230.1, 348/234, 348/231.3, 348/241, 348/231.6, 348/246, 348/247, 348/362
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 25/585 with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes
• CPC Class Codes: H04N 3/155 Control of the image-sensor operation, e.g. image processing within the image-sensor

Abstract

Disclosed are method and apparatus for obtaining relatively high dynamic range images using a relatively low dynamic range image sensor without significant loss of resolution. The image sensor has an array of light-sensing elements with different sensitivity levels in accordance with a predetermined spatially varying sensitivity pattern for the array of light-sensing elements. An image of a scene is captured with the image sensor and stored as brightness values at respective pixel positions in a linear or two-dimensional uniform grid. The brightness values of the captured image at the pixel positions are then used to estimate the brightness values at off-grid positions of a uniform off-grid array located at respective interstices of the pixel position grid. The estimated off-grid brightness values are either used directly as the pixel brightness values of a relatively high dynamic output image or interpolated to derive resampled on-grid brightness values at the pixel positions of the pixel position grid to provide a relatively high dynamic range output image. Alternatively, the brightness values of the captured image are interpolated by an on-grid interpolation filter to derive pixel brightness values of a relatively high dynamic range output image, each pixel brightness value of the output image being derived from a corresponding plurality of the captured image brightness values. In each instance, either the captured image brightness values or the pixel brightness values of the output image may be compensated for non-linearities of the radiometric response function of the light-sensing elements of the image sensor.

Claims

1. A system for obtaining a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor adapted to be exposed to incident light from the scene for capturing an image thereof comprising:

a) an image sensor having a multiplicity of light-sensing elements in an array, each one of the light-sensing elements having a particular one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements and having a respective response function, each light-sensing element in response to incident light from the scene producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light-sensing elements;

b) a captured image memory for storing the captured image brightness values produced by the light-sensing elements at pixel positions of the pixel position array;

c) a first pixel position generator for providing the pixel positions of the pixel position array;

d) a sensitivity pattern memory for storing data indicative of the sensitivity level corresponding to each one of the first pixel positions of the pixel position array;

e) a response function compensator receiving pixel positions from the first pixel position generator, data indicative of the sensitivity levels corresponding to the received pixel positions from the sensitivity pattern memory and captured image brightness values at the received pixel positions from the captured image memory, and being responsive to a pixel position received from the first pixel position generator and data indicative of the sensitivity level corresponding to the received pixel position for retrieving from the captured image memory the captured image brightness value at the received pixel position and for compensating the retrieved captured image brightness value at the received pixel position by the inverse of the response function of a light-sensing element having the sensitivity level corresponding to the received pixel position to provide a respective compensated brightness value at the pixel position received from the first pixel position generator;

f) a compensated on-grid brightness value memory for storing respective compensated captured image values at the pixel positions received from the first pixel position generator;

g) a second pixel position generator for providing the pixel positions of the pixel position array;

h) an on-grid brightness value generator receiving pixel positions from the second pixel position generator and being responsive to each pixel position received from the second pixel position generator for deriving from the compensated captured image brightness values in the compensated on-grid brightness value memory a respective output image brightness value at each received pixel position; and

i) an output image memory for storing the respective output image brightness value at each pixel position received from the second pixel position generator.

2. The system of claim 1, wherein the response function compensator comprises a lookup table memory storing separate lookup table data representing the inverse of respective response functions of light-sensing elements having different ones of the plurality of sensitivity levels, and a mapper receiving captured image brightness values from the captured image memory and being responsive to a captured image brightness value at a pixel position received from the first pixel position generator for deriving a respective index for the lookup table memory, and retrieving therefrom a compensated on-grid brightness value corresponding to the respective index from lookup table data representing the inverse of the response function of a light-sensing element having the sensitivity level corresponding to the received pixel position, the retrieved compensated on-grid brightness value being provided to the compensated on-grid brightness value memory.

3. The system of claim 1, wherein the array of light-sensing elements of the image sensor is a two-dimensional array, and the pixel position array is a two-dimensional array having pixel positions located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns.

4. The system of claim 1, wherein the array of light-sensing elements of the image sensor is a linear array for capturing a line image and the pixel position array is a linear array having respective regularly spaced pixel positions of a linear pixel row, and wherein the pixel positions provided by the pixel position generator have coordinates x, where x has values from 2.5 to xSize-2.5 and xSize is the dimension of the captured line image.

5. The system of claim 4, wherein the on-grid brightness value generator is responsive to a pixel position x received from the second pixel position generator for deriving from the compensated captured image brightness values in the compensated on-grid brightness memory a respective output image brightness value at the received pixel position, the on-grid brightness value generator comprising:

i) a memory for storing a 5×

1 Gaussian interpolation filter kernel G;

ii) a comparator for comparing each one of five compensated on-grid brightness values I(k) at pixel positions k having values from x−

2 to x+2 with a predetermined low threshold value and a predetermined high threshold value;

iii) a multiplier for calculating a quantity I(k)G(k-x) for each value of k at which I(k) is greater than the predetermined low threshold value and less than the predetermined high threshold value, where G(k-x) is the value of the Gaussian interpolation kernel G at position (k-x);

iv) a first adder for adding the quantities I(k)G(k-x) calculated by the multiplier for each value of k where I(k) is greater than the predetermined low threshold value and less than the predetermined high threshold value to derive a first sum p;

v) a second adder for adding the values G(k-x) of the Gaussian interpolation kernel in the memory for each value of k where I(k) is greater than the predetermined low threshold value and less than the predetermined high threshold value to derive a second sum q; and

vi) a divider for deriving a respective output image brightness value I_o(x) at the position x received from the second pixel position generator by dividing the sum p by the sum q.

6. The system of claim 5, wherein the predetermined low threshold value is the noise level of a light-sensing element having the sensitivity level corresponding to the pixel position of the compensated on-grid brightness value being compared therewith compensated by the inverse of the response function of the light-sensing element, and the predetermined high threshold value is the saturation or near saturation brightness value of a light-sensing element having the sensitivity level corresponding to the pixel position of the compensated on-grid brightness value being compared therewith.

7. The system of claim 5, wherein the 5×

1 Gaussian interpolator filter kernel has the form;