Apparatus and method for high dynamic range imaging using spatially varying exposures

• US 6,864,916 B1
• Filed: 06/04/1999
• Issued: 03/08/2005
• Est. Priority Date: 06/04/1999
• Status: Expired due to Term

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Sony Corporation (Sony Group Corp.)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Sony Corporation (Sony Group Corp.)
• Inventors: Nayar, Shree K., Mitsunaga, Tomoo
• Primary Examiner(s): MOE, AUNG SOE
• Application Number:
  US09/326,422
• Time in Patent Office: 2,104 Days
• Field of Search: 348/224.1, 348/342, 348/256, 348/295, 348/229.1, 348/222.1, 348/221.1, 348/219.1, 348/218.1, 348/362, 358/518, 358/504, 358/520, 358/525, 358/523, 358/534, 382/274, 382/254, 382/26, 382/260
• US Class Current: 348/224.1
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Apparatus and methods are provided for obtaining high dynamic range images using a low dynamic range image sensor. The image of a scene is captured with an image sensor using a spatially varying exposure function. The spatially varying exposure function can be implemented in a number of ways, such as by using as an optical mask with a fixed spatial attenuation pattern or by using an array of light sensing elements having spatially varying photosensitivities. The captured image is then normalized with respect to the spatially varying exposure function. The normalized image data can be interpolated to account for pixels that are either saturated or blackened to enhance the dynamic range of the image sensor.

Claims

1. A system for high dynamic range imaging comprising:

an image sensor comprising an array of light-sensing elements, the image sensor sensing an image of a scene and providing corresponding pixel values representing light intensities impinging on respective ones of the light-sensing elements, the image sensor providing a saturated pixel value when the intensity of light impinging on a corresponding one of the light-sensing elements is greater than a first threshold level and providing a blackened pixel value when the intensity of light impinging on a corresponding one of the light-sensing elements is below a second threshold level, the image sensor having a low dynamic range relative to the range of light intensities of the image of the scene;

a mask interposed between the scene and the array of light-sensing elements of the image sensor, the mask having a multiplicity of light-transmitting cells that attenuate the intensity of scene light transmitted though the mask on to the list-sensing elements, wherein the light-transmitting cells are configured so that the intensity of scene light transmitted on to the light sensing elements is attenuated by brightness level amounts that vary spatially across the array of light-sensing elements, and whereby the scene is exposed to the light sensing elements in a spatially varying manner, each of the light-sensing elements having a corresponding exposure value indicative of the transparency of the cell through which light impinging on the light-sensing element passes;

a first memory storing exposure values corresponding to the light-sensing elements; and

an image processor coupled to the image sensor for receiving the pixel values provided thereby and coupled to the first memory for receiving the exposure values corresponding to the pixel values, the image processor comprising a normalizer for mapping the pixel values by a function of the exposure values to derive corresponding normalized pixel values.

2. The system according to claim 1, wherein the normalizer divides each of the pixel values with the exposure value corresponding to the light-sensing element receiving light intensity represented by the pixel value to derive corresponding normalized pixel values.

3. The system according to claim 1, wherein the exposure values corresponding to the light-sensing elements are fixed and the image processor further comprises an interpolator for interpolating the normalized pixel values to derive interpolated pixel values at respective positions of a second array overlapping the array of the light-sensing elements.

4. The system according to claim 3, wherein the interpolator of the image processor interpolates said normalized pixel values, excluding said normalized pixel values corresponding to saturated pixel values or blacked pixel values.

5. The system according to claim 3, wherein the interpolator of the image processor interpolates the normalized pixel values multiplied by respective weighting values.

6. The system according to claim 5, wherein the weighting values are respective signal-to-noise ratios of the corresponding pixel values.

7. The system according to claim 3, wherein the interpolator of the image processor applies an interpolation filter to the normalized pixel values.

8. The system according to claim 3, wherein the light-sensing elements of the array are disposed at respective intersections of a first grid defined by spaced intersecting perpendicular lines extending in a vertical direction and a horizontal direction, and the interpolator derives interpolated pixel values at respective intersections of a second grid defined by spaced intersecting perpendicular lines extending in the vertical direction and in the horizontal direction, the first and second grids overlapping one another.

9. The system according to claim 8, wherein the interpolator of the image processor separately interpolates said normalized pixel values in each different region of the first grid, each region overlapping adjacent regions and encompassing a predetermined number of intersections of the first grid.

10. The system according to claim 9, wherein the mask comprises repetitively disposed identical groups of cells having different transparencies.

11. The system according to claim 10, wherein each of the repetitively disposed identical groups of cells of the mask consists of four neighboring cells having different transparencies, each cell controlling the light exposure of a different one of the light-sensing elements, and the first and second grids have identical dimensions and are displaced from one another by one-half grid position in the horizontal direction and one-half grid position it the vertical direction.

12. The system according to claim 9, wherein the interpolator of the image processor derives a respective interpolated pixel value in each different region of the first grid by taking the average of all normalized pixel values in the region which do not correspond to said saturated pixel value or said blackened pixel value.

13. The system according to claim 3 further comprising an output image memory coupled to the image processor for receiving and storing the interpolated pixel values.

14. The system according to claim 1 further comprising a second memory storing a response function of the image sensor, wherein the image processor is coupled to the second memory for receiving the response function and further comprises a calibrator for converting the pixel values provided by the image sensor to linear response pixel values in accordance with the response fiction before the pixel values are normalized by the normalizer.

15. The system according to claim 1, wherein the array of light sensing elements is a solid state device image sensing device.

16. The system according to claim 15, wherein the solid state device is a charge-coupled device light sensing array.

17. The system according to claim 16, wherein the solid state dice is CMOS light-sensing array.

18. The system according to claim 15, wherein the mask is integrated with the light-sensing elements on the same substrate.

19. The system according to claim 1, wherein the mask is made of a nonlinear optical material.

20. A system for high dynamic range imaging comprising:

an image sensor comprising an array of light-sensing elements, wherein the light-sensing elements are configured to have respective photosensitivities that vary spatially across the array so that when the image sensor is exposed to list from a scene an exposure time then the array of light sensing elements impose a spatial variation the scene exposure, the image sensor sensing an image of a scene and providing corresponding pixel values representing light intensities impinging on respective ones of the light-sensing elements, each light-sensing element having a respective first threshold level so as to cause the image sensor to provide a saturated pixel value when the intensity of list impinging on the light-sensing element is greater than the respective first threshold level, each light-sensing element having a respective second threshold value so as to cause the image sensor to provide a blackened pixel value when the intensity of light impinging on the light-sensing element is below the respective second threshold level, the image sensor having a low dynamic range relative to the range of light intensities of the image of the scene, the light-sensing elements having respective photosensitivity values indicative of their respective photosensitivities;

a first memory storing photosensitivity values corresponding to each of the light-sensing elements; and

an image processor coupled to the array of light-sensing elements for receiving the pixel values provided by the image sensor and coupled to the first memory for receiving the photosensitivity values corresponding to the light-sensing elements, the image processor comprising a normalizer for mapping the pixel values by a function of the photosensitivity values to derive corresponding normalized pixel values, and an interpolator for interpolating the normalized pixel values to derive interpolated pixel values at respective positions of a second array overlapping the array of light-sensing elements.

21. The system according to claim 20, where the normalizer of the image processor maps the pixel values by a function of the photosensitivity values by dividing each of the pixel values by the photosensitivity value corresponding to the light-sensing element receiving light intensity represented by the pixel value.

22. The system according to claim 20, wherein the light-sensing elements of the image sensor are disposed at respective intersections of a first grid defined by spaced intersecting perpendicular lines extending in a vertical direction and a horizontal direction, and the positions of the interpolated pixel values are respective intersections of a second grid defined by spaced intersecting lines extending in the vertical direction and in the horizontal direction.

23. The system according to claim 22, wherein the interpolator of the image processor separately interpolates normalized pixel values in each different region of the first grid, each region overlapping adjacent regions and encompassing a predetermined number of intersections of the first grid.

24. The system according to claim 23, wherein the interpolator of the image processor derives a respective interpolated pixel value in each different region of the first grid by taking the average of all normalized pixel values in the region which do not correspond to said saturated pixel value or said blackened pixel value.

25. The system according to claim 22, wherein the array of light-sensing elements of the image sensor comprise repetitively disposed identical groups of light-sensing elements having different photosensitivities.

26. The system according to claim 25, where each of the repetitively disposed identical groups of light-sensing elements consists of four neighboring light-sensing elements having different photosensitivities, and the first and second grids having identical dimensions and are displaced from one another by one-half grid position in the horizontal direction and one-half grid position in the vertical direction.

27. The system according to claim 20, wherein the image sensor has a respective response function to light impinging on each of the light-sensing elements and the system further comprises a second memory storing the respective response functions of the image sensor, and wherein the image processor is coupled to the second memory for receiving the response functions and further comprises a calibrator for converting the pixel values provided by the image sensor to linear response pixel values in accordance with the respective response functions before the pixel values are normalized by the normalizer.

28. The system according to claim 20, further comprising an output image memory coupled to the image processor for receiving and storing the interpolated pixel values.

29. A method for high dynamic range imaging comprising the steps of:

exposing an image sensor to light from a scene to sense an image frame, wherein the image sensor comprises an array of light sensing elements that is configured so that a sensing capability of the image sensor varies spatially across the array during the sensing of the image frame, whereby the sensed image formed is an image formed with a spatially varying exposure of the scene, the image sensor sensing the spatially varying exposure image and providing corresponding pixel values representing light intensities impinging on respective ones of the light-sensing elements, the image sensor providing a saturated pixel value when the intensity of light impinging on a corresponding one of the light-sensing elements is greater than a first threshold level and providing a blackened pixel value when the intensity of the light impinging on a corresponding one of the light-sensing elements is below a second threshold level, the image sensor having a low dynamic range relative to the range of light intensities of the image of the scene; and

normalizing the pixel values provided by the image sensor with respect to the spatially varying exposure of the scene by the light-sensing elements to derive corresponding normalized pixel values.

30. The method according to claim 29, wherein the step of exposing the image sensor using a spatially varying exposure includes the step of using a mask having a multiplicity of light transmitting cells each controlling the exposure of a respective one or more of the light-sensing elements of the image sensor to light from the scene, each of the light-sensing elements having a corresponding exposure value indicative of the transparency of the mask cell through which light impinging on the light-sensing element pass and the step of normalizing the pixel values provided by the image sensor includes the step of mapping the pixel values by a function of the exposure values to derive corresponding normalized pixel values.

31. The method of claim 30 fiber comprising the steps of projecting an image of the scene onto the mask, and projecting the image of the ene as transmitted through the mask onto the array of light-sensing elements.

32. The method according to claim 30, wherein the step of mapping the pixel values by a function of the exposure values comprises dividing each of the pixel values with the exposure value corresponding to the light-sensing element receiving light intensity resented by the pixel value.

33. The method according to claim 29, wherein the exposure values corresponding to the light-sensing elements are fixed, and further comprising the step of interpolating the normalized pixel values to derive interpolated pixel values at respective positions of a second array overlapping the array of the light-sensing elements.

34. The method according to claim 33, wherein the step of interpolating said normalized pixel values includes the steps of separately interpolating said normalized pixel values in each different region of the array of the light-sensing elements, each region overlapping adjacent regions and encompassing a predetermined number of the light-sensing element positions.

35. The method of claim 34, wherein the steps of separately interpolating normalized pixel values in each different region of the array of light-sensing elements each include the step of taking the average of all normalized peel values in the region, except for ones that correspond to said saturated pixel value or said blackened pixel value, to derive a respective one of the interpolated pixel values.

36. The method according to claim 33 further comprising the step of storing the interpolated pixel values in an output image memory.

37. The method according to claim 29, wherein the step of interpolating the normalized pixel values include the step of applying an interpolation filter to the normalized pixel values.

38. The method, of claim 29 further comprising the step of calibrating the pixel values provided by the image sensor according to a response function of the image sensor to derive linear response pixel values before the step of normalizing the pixel values.