Method and apparatus for obtaining high dynamic range images

• US 7,999,858 B2
• Filed: 10/05/2004
• Issued: 08/16/2011
• Est. Priority Date: 02/23/2000
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Sony Corporation (Sony Group Corp.)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Sony Corporation (Sony Group Corp.)
• Inventors: Nayar, Shree K., Mitsunaga, Tomoo
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): Yoder, III; Chriss S
• Application Number:
  US10/959,679
• Publication Number:
  US 20050099504A1
• Time in Patent Office: 2,506 Days
• Field of Search: 348/188, 348/241, 348/246, 348/247, 348/187, 348/362, 348/229.1, 348/230.1, 348/222.1, 348/221.1, 348/231.99, 348/231.6, 348/234, 358/521, 358/523, 358/524, 382/300
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 25/585 with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes
• CPC Class Codes: H04N 3/155 Control of the image-sensor operation, e.g. image processing within the image-sensor

Abstract

The application provides techniques for obtaining a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor exposed to incident light from the scene for capturing an image. The image sensor has a multiplicity of light-sensing elements in an array and each light sensing element has a particular one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements and has a response function. Each light sensing element is responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array. Each one of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light sensing elements.

Claims

1. A system for obtaining a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor adapted to be exposed to incident light from the scene for capturing an image thereof comprising:

an image sensor having a multiplicity of light-sensing elements in an array, each light-sensing element having a particular one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements and a respective response function, each light-sensing element being responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light-sensing elements;

a captured image memory for storing the captured image brightness values produced by the light-sensing elements at corresponding ones of the multiplicity of pixel positions of the pixel position array;

an off-grid estimator for deriving from the captured image brightness values in the captured image memory respective estimated off-grid brightness values at a multiplicity of off-grid positions located at respective interstices of the pixel position array so as to form a regular off-grid position array, the off-grid estimator comprising;

an off-grid position generator for providing the off-grid positions of the off-grid position array;

a sensitivity pattern memory for storing data indicative of the sensitivity level corresponding to each one of the pixel positions of the pixel position array;

a plurality of sensitivity level off-grid brightness value estimators, each corresponding to a respective one of the sensitivity levels, for receiving off-grid positions from the off-grid position generator and sensitivity pattern data from the sensitivity pattern memory, each one of the plurality of sensitivity level off-grid brightness value estimators being responsive to an off-grid position received from the off-grid position generator and the sensitivity pattern data from the sensitivity pattern memory for deriving from the captured image brightness values in the captured image memory a respective sensitivity level off-grid brightness value for the corresponding sensitivity level at the received off-grid position;

an accumulator for combining the respective sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimator for the corresponding sensitivity levels at each off-grid position received from the off-grid position generator to derive a respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator; and

a response function compensator for compensating the respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator by the inverse of a combined response function of the light-sensing elements to derive a respective estimated off-grid brightness value at each off-grid position received from the off-grid position generator, the combined response function being the sum of weighted response functions of light-sensing elements having different ones of the plurality of sensitivity levels, each one of the weighted response functions being a response function of a light-sensing element having a respective one of the sensitivity levels multiplied by a redetermined weighting factor for the sensitivity levels; and

an output image generator for deriving pixel brightness values of an output image from the estimated off-grid brightness values.

2. The system of claim 1, wherein the array of light-sensing elements of the image sensor is a linear array for capturing a line image and the pixel position array is a linear array having the multiplicity of pixel positions located at respective regularly spaced positions in a pixel row so as to form a linear captured image brightness value array, the captured image memory for storing the captured image brightness values produced by the light-sensing elements.

3. The system of claim 2, wherein the off-grid positions of the off-grid position array are located between respective adjacent pairs of pixel positions of the linear pixel position array.

4. The system of claim 3, wherein the predetermined sensitivity pattern has a predetermined first sensitivity level and a predetermined second sensitivity level, and adjacent pixel positions of the linear pixel position array correspond to different ones of the predetermined first sensitivity level and the predetermined second sensitivity level.

5. The system of claim 1, wherein the array of light-sensing elements of the image sensor is a linear array and the pixel position array is a two-dimensional array having pixel positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the linear array of light-sensing elements of the image sensor is adapted to be sequentially exposed to successive regularly spaced linear regions of the scene in multiple exposures, each one of the exposures producing a respective linear captured image brightness value array, the respective linear captured image brightness value arrays produced by the multiple exposures being concatenated to form a two-dimensional captured image brightness value array having respective captured image brightness values at the pixel positions of the pixel position array, the concatenated linear captured image brightness value arrays being stored in the captured image memory as respective captured image brightness values at the pixel positions of the two-dimensional pixel position array.

6. The system of claim 5, wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate columns and pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

7. The system of claim 6, wherein the predetermined sensitivity pattern has a first predetermined sensitivity level and a second predetermined sensitivity level, and wherein adjacent pixel positions in each one of the pixel rows correspond to different ones of the first and second predetermined sensitivity levels, and adjacent pixel positions in each one of the pixel columns correspond to the same one of the first and second predetermined sensitivity levels.

8. The system of claim 1, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, the captured image memory for storing the captured image brightness values produced by the light-sensing elements at corresponding ones of the multiplicity of pixel positions of the pixel position array.

9. The system of claim 8 wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each intermediate row being parallel to the pixel rows and extending between a respective adjacent pair of pixel rows and each intermediate column being parallel to the pixel columns and extending between a respective adjacent pair of pixel columns.

10. The system of claim 9, wherein the predetermined sensitivity pattern of the array of light-sensing elements has four different predetermined sensitivity levels and corresponds to a repetitive pattern of groups of four nearest neighbor pixel positions, each one of the four pixel positions in each group corresponding to a different one of the four predetermined sensitivity levels in a common predetermined positional order.

11. The system of claim 8, wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate rows and the plurality of pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of pixel columns.

12. The system of claim 11, wherein the predetermined sensitivity pattern of the array of light-sensing elements has first and second predetermined sensitivity levels, and wherein adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of the pixel columns correspond to different ones of the first and second predetermined sensitivity levels.

13. The system of claim 1, wherein the output image generator comprises an output image memory for storing the respective estimated off-grid brightness values derived by the off-grid estimator as the pixel brightness values of the output image.

14. The system of claim 1, wherein the response function compensator comprises a lookup table memory storing lookup table data representing the inverse of the combined response function of the light-sensing elements, and a mapper for deriving from the respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator a respective index for the lookup table memory and providing a data value in the lookup table memory corresponding to the respective index as the respective estimated off-grid brightness value at the off-grid position received from the off-grid position generator.

15. The system of claim 1, wherein the output image generator comprises:

an off-grid brightness value memory for storing the respective estimated off-grid brightness value derived by the response function compensator at each off-grid position received from the off-grid position generator;

an on-grid resampler for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective resampled on-grid brightness values at the pixel positions of the pixel position array; and

an output image memory for storing the respective resampled on-grid brightness values derived by the on-grid resampler as the pixel brightness values of the output image,

and wherein the on-grid resampler comprises;

a pixel position generator for providing pixel positions of the pixel position array; and

an interpolator for receiving pixel positions provided by the pixel position generator and being responsive to a pixel position received from the pixel position generator for deriving from the estimated off-grid brightness values in the off-grid brightness value memory a respective resampled on-grid brightness value at the received pixel position.

16. The system of claim 1, wherein the accumulator of the off-grid estimator comprises an adder for adding the respective sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators for the corresponding sensitivity levels at each off-grid position received from the off-grid position generator to provide the combined sensitivity level off-grid brightness values at each off-grid position received from the off-grid position generator.

17. The system of claim 1, wherein the accumulator of the off-grid estimator comprises a weighting factor memory for storing a plurality of predetermined weighting factors each corresponding to a different one of the plurality of sensitivity levels of the light-sensing elements, a plurality of multipliers each for multiplying the sensitivity level off-grid brightness value corresponding to a respective one of the sensitivity levels at a received off-grid position by the weighting factor corresponding to the respective one of the sensitivity levels to provide a weighted sensitivity level off-grid brightness value for the corresponding sensitivity level at the off-grid position received from the off-grid position generator, and an adder for summing the weighted sensitivity level off-grid brightness values at the received off-grid position provided by the plurality of multipliers to derive the combined sensitivity level off-grid brightness value at the off-grid positions received from the off-grid position generator.

18. The system of claim 17, wherein the array of light-sensing elements of the image sensor is a linear array and the pixel position array is a linear array having a multiplicity of pixel positions at respective regularly spaced pixel positions in a pixel row, and wherein the off-grid positions of the off-grid position array are each located between a respective adjacent pair of pixel positions of the pixel position array and the sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators for an off-grid position received from the off-grid position generator are respective captured image brightness values at a predefined number of pixel positions that are nearest neighbors to the received off-grid position, each one of the predefined number of nearest neighbor pixel positions corresponding to a different one of the plurality of sensitivity levels of the light-sensing elements.

19. The system of claim 18, wherein each one of the off-grid positions of the off-grid position array is located midway between a respective adjacent pair of pixel positions of the linear pixel position array and the predetermined sensitivity pattern of the light-sensing elements has a first and a second predetermined sensitivity level, and wherein the combined sensitivity level off-grid brightness value at an off-grid position x′

received from the off-grid position generator is expressed as

B(x′

)=W₁I_c(x′

−

0.5)+W₂I_c(x′

+0.5),

where B(x′

) is the combined sensitivity level off-grid brightness value at the received off-grid position x′

, W₁ is the weighting factor for the sensitivity level corresponding to pixel position (x′

−

0.5), W₂ is the weighting factor for the sensitivity level corresponding to pixel position (x′

+0.5), I_c(x′

−

0.5) is the captured image brightness value at pixel position (x′

−

0.5) and I_c(x′

+0.5) is the captured image brightness value at pixel position (x′

+0.5), and the combined response function of the light-sensing elements is expressed as

S(E)=W₁P₁(E)+W₂P₂(E),

where P₁(E) is the radiometric response function of a light-sensing element having the sensitivity level corresponding to pixel position (x′

−

0.5) and P₂(E) is the radiometric response function of a light-sensing element having the sensitivity level corresponding to pixel position (x′

+0.5).

20. The system of claim 17, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the pixel columns, the multiplicity of off-grid positions are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

21. The system of claim 20, wherein the predetermined sensitivity pattern of the array of light-sensing elements corresponds to a pixel position array having repetitive disposed groups of four nearest neighbor pixel positions, where each one of the four nearest neighbor pixel positions in each group corresponds to a respective one of four predetermined sensitivity levels in a common predetermined positional order.

22. The system of claim 17, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the pixel columns, and wherein the multiplicity of off-grid positions are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

23. The system of claim 17, wherein the predetermined sensitivity pattern of the array of light-sensing elements has a first and a second predetermined sensitivity level, and adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of the pixel columns correspond to different ones of the predetermined first and second sensitivity levels.

24. The system of claim 17, wherein the array of light-sensing elements of the image sensor is a linear array and the multiplicity of pixel positions are located at respective regularly spaced pixel positions in a pixel row so as to form a linear captured image brightness value array, and wherein the linear array of light-sensing elements of the image sensor is adapted to be sequentially exposed to successive regularly spaced linear regions of the scene in multiple exposures, each one of the exposures producing a respective linear captured image brightness value array, the respective linear captured image brightness value arrays produced by the multiple exposures being concatenated to form a two-dimensional captured image brightness value array having respective captured image brightness values at pixel positions of a pixel position array located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the multiplicity of off-grid positions are located at respective intersections of the pixel rows and a plurality of regularly spaced intermediate columns, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

25. The system of claim 24, wherein the predetermined sensitivity pattern of the array of light-sensing elements has a first and a second predetermined sensitivity level, and wherein adjacent pixel positions in each pixel row correspond to different ones of the first and the second predetermined sensitivity levels and adjacent pixel positions in each pixel column correspond to the same one of the first and the second predetermined sensitivity levels.

26. The system of claim 1, wherein each one of the sensitivity level off-grid brightness value estimators is responsive to an off-grid position received from the off-grid position generator and the data indicative of the sensitivity level corresponding to each one of the pixel positions from the sensitivity pattern memory for determining a pixel position corresponding to the same sensitivity level as the sensitivity level off-grid brightness value estimator and located nearest the received off-grid position, and for estimating a sensitivity level off-grid brightness value at the received off-grid position to be equal to the captured image brightness value at the nearest pixel position corresponding to the same sensitivity level as the sensitivity level off-grid brightness value estimator.

27. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device having respective integrated lenses formed over the photosensitive surfaces of predetermined ones of the photodiodes, each integrated lens concentrating the incident light onto the respective light-sensing surface, whereby the sensitivity level of each one of the photodiodes is determined by whether or not an integrated lens is formed over the light-sensing surface thereof.

28. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, each one of the photodiodes having an integrated light filter formed above the light-sensing surface thereof, the transparency of each integrated light filter being fixed during fabrication of the charge coupled device, whereby the sensitivity level of each one of the photodiodes in the array is determined by the transparency of the integrated light filter above the light-sensing surface thereof.

29. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device being formed with an opaque layer overlying the array of photodiodes with a respective aperture formed in the opaque layer above the light-sensing surface of each one of the photodiodes, the respective size of the aperture above the light-sensing surface of each one of the photodiodes being fixed during fabrication of the charge coupled device, whereby the sensitivity level of each one of the photodiodes in the array is determined by the size of the aperture above the light-sensing surface thereof.

30. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having a multiplicity of photodiodes in an array of spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device having an interline structure in which the columns of photodiodes are each disposed adjacent a corresponding vertical shift register, each photodiode having an associated transfer gate electrode for controlling the transfer of stored photogenerated charge from the photodiode to the corresponding vertical shift register, the stored photogenerated charge in each one of the photodiodes being operatively periodically removed by the application of a periodic reset pulse to the device, the photogenerated charge stored in each one of the photodiodes being operatively transferred to a respective stage of the corresponding vertical shift register by the application of a transfer trigger pulse to the associated transfer gate electrode and being accumulated therein for an exposure interval, the transfer trigger pulse being applied immediately preceding a reset pulse, the respective transfer trigger pulses being operatively applied to the transfer gate electrodes associated with the photodiodes in the array at predetermined frequencies, whereby the sensitivity level of each one of the photodiodes in the array is determined by the frequency of transfer trigger pulses applied to the transfer gate electrode associated therewith.