Method and apparatus for obtaining high dynamic range images

• US 8,610,789 B1
• Filed: 05/10/2011
• Issued: 12/17/2013
• Est. Priority Date: 02/23/2000
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Sony Corporation (Sony Group Corp.)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University), Sony Corporation (Sony Group Corp.)
• Inventors: Nayar, Shree K., Mitsunaga, Tomoo
• Primary Examiner(s): Moe, Aung S
• Assistant Examiner(s): YODER III, CHRISS S
• Application Number:
  US13/104,606
• Time in Patent Office: 952 Days
• Field of Search: 348/187, 348/188, 348/221.1, 348/222.1, 348/229.1, 348/230.1, 348/231.6, 348/231.99, 348/234, 348/241, 348/246, 348/247, 348/362, 358/521, 358/523, 358/524, 382/300
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 25/585 with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes
• CPC Class Codes: H04N 3/155 Control of the image-sensor operation, e.g. image processing within the image-sensor

Abstract

The application provides techniques for obtaining a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor exposed to incident light from the scene for capturing an image. The image sensor has a multiplicity of light-sensing elements in an array and each light sensing element has a particular one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements and has a response function. Each light sensing element is responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array. Each one of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light sensing elements.

Claims

1. A system for obtaining a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor adapted to be exposed to incident light from the scene for capturing an image thereof comprising:

an image sensor having a multiplicity of light-sensing elements in an array, each light-sensing element having a particular one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements and a respective response function, the predetermined sensitivity pattern of the array of light-sensing elements having at least two different predetermined sensitivity levels and corresponding to a repetitive pattern of groups of pixel positions, each one of the pixel positions in each group corresponding to a different one of the at least two predetermined sensitivity levels in a common predetermined positional order, each light-sensing element being responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light-sensing elements;

a captured image memory for storing the captured image brightness values produced by the light-sensing elements at corresponding ones of the multiplicity of pixel positions of the pixel position array;

an off-grid estimator for deriving from the captured image brightness values in the captured image memory respective estimated off-grid brightness values at a multiplicity of off-grid positions located at respective interstices of the pixel position array so as to form a regular off-grid position array; and

an output image generator for deriving pixel brightness values of an output image from the estimated off-grid brightness values, comprising;

an off-grid brightness value memory for storing the estimated off-grid brightness values derived by the off-grid estimator;

an on-grid resampler for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective resampled on-grid brightness values at the pixel positions of the pixel position array; and

an output image memory for storing the respective resampled on-grid brightness values derived by the on-grid resampler as the pixel brightness values of the output image.

2. The system of claim 1, wherein the array of light-sensing elements of the image sensor is a linear array for capturing a line image and the pixel position array is a linear array having the multiplicity of pixel positions located at respective regularly spaced positions in a pixel row so as to form a linear captured image brightness value array, the captured image memory for storing the captured image brightness values produced by the light-sensing elements.

3. The system of claim 2, wherein the off-grid positions of the off-grid position array are located between respective adjacent pairs of pixel positions of the linear pixel position array.

4. The system of claim 3, wherein the predetermined sensitivity pattern has a predetermined first sensitivity level and a predetermined second sensitivity level, and adjacent pixel positions of the linear pixel position array correspond to different ones of the predetermined first sensitivity level and the predetermined second sensitivity level.

5. The system of claim 1, wherein the array of light-sensing elements of the image sensor is a linear array and the pixel position array is a two-dimensional array having pixel positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the linear array of light-sensing elements of the image sensor is adapted to be sequentially exposed to successive regularly spaced linear regions of the scene in multiple exposures, each one of the exposures producing a respective linear captured image brightness value array, the respective linear captured image brightness value arrays produced by the multiple exposures being concatenated to form a two-dimensional captured image brightness value array having respective captured image brightness values at the pixel positions of the pixel position array, the concatenated linear captured image brightness value arrays being stored in the captured image memory as respective captured image brightness values at the pixel positions of the two-dimensional pixel position array.

6. The system of claim 5, wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate columns and pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

7. The system of claim 6, wherein the predetermined sensitivity pattern has a first predetermined sensitivity level and a second predetermined sensitivity level, and wherein adjacent pixel positions in each one of the pixel rows correspond to different ones of the first and second predetermined sensitivity levels, and adjacent pixel positions in each one of the pixel columns correspond to the same one of the first and second predetermined sensitivity levels.

8. The system of claim 1, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, the captured image memory for storing the captured image brightness values produced by the light-sensing elements at corresponding ones at the multiplicity of pixel positions of the pixel position array.

9. The system of claim 8 wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each intermediate row being parallel to the pixel rows and extending between a respective adjacent pair of pixel rows and each intermediate column being parallel to the pixel columns and extending between a respective adjacent pair of pixel columns.

10. The system of claim 8, wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate rows and the plurality of pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of pixel columns.

11. The system of claim 10, wherein the predetermined sensitivity pattern of the array of light-sensing elements has first and second predetermined sensitivity levels, and wherein adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of the pixel columns correspond to different ones of the first and second predetermined sensitivity levels.

12. The system of claim 1, wherein the output image generator comprises an output image memory for storing the respective estimated off-grid brightness values derived by the off-grid estimator as the pixel brightness values of the output image.

13. The system of claim 1, wherein the off-grid estimator comprises:

an off-grid position generator for providing the off-grid positions of the off-grid position array;

a sensitivity pattern memory for storing data indicative of the sensitivity level corresponding to each one of the pixel positions of the pixel position array;

a plurality of sensitivity level off-grid brightness value estimators each corresponding to a respective one of the plurality of sensitivity levels of the light-sensing elements, and receiving off-grid positions from the off-grid position generator and sensitivity pattern data from the sensitivity pattern memory, each one of the sensitivity level off-grid brightness value estimators being responsive to an off-grid position received from the off-grid position generator and the sensitivity pattern data from the sensitivity pattern memory for deriving from the captured image brightness values in the captured image memory a respective sensitivity level off-grid brightness value for the corresponding sensitivity level at the received off-grid position; and

an accumulator for combining the respective sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators for the corresponding sensitivity levels at each off-grid position received from the off-grid position generator to derive a respective estimated off-grid brightness value at each off-grid position received from the off-grid position generator.

14. The system of claim 13, wherein the accumulator comprises an adder for adding the sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimator for corresponding sensitivity levels at each received off-grid position to derive the respective estimated off-grid brightness value at each off-grid positions received from the off-grid position generator.

15. The system of claim 13, wherein the accumulator comprises a weighting factor memory for storing a plurality of predetermined weighting factors each corresponding to a different one of the plurality of sensitivity levels of the light-sensing elements, a plurality of multipliers each for multiplying the sensitivity level off-grid brightness value corresponding to a respective one of the sensitivity levels at a received off-grid position by the weighting factor corresponding to the respective one of the sensitivity levels to provide a respective weighted sensitivity level off-grid brightness value for the corresponding sensitivity level at the received off-grid position, and an adder for summing the respective weighted sensitivity level off-grid brightness values provided by the plurality of multipliers to derive a respective estimated off-grid brightness value at the received off-grid position received from the off-grid position generator.

16. The system of claim 1, wherein the on-grid resampler comprises:

a pixel position generator for providing the pixel positions of the pixel position array;

an interpolator for receiving pixel positions provided by the pixel position generator and being responsive to a pixel position received from the pixel position generator for deriving from the estimated off-grid brightness values in the off-grid brightness value memory a respective interpolated on-grid brightness value at the received pixel position; and

a response function compensator for compensating the respective interpolated on-grid brightness value derived by the interpolator at each pixel position received from the pixel position generator by the inverse of a combined response function of the light-sensing elements to derive a respective resampled on-grid brightness value at each pixel position received from the pixel position generator, the combined response function of the light-sensing elements being the sum of weighted response functions of light-sensing elements having different ones of the sensitivity levels, each one of the weighted response functions being a response function of a light-sensing element having a respective one of the sensitivity levels multiplied by a predetermined weighting factor for the respective one of the sensitivity levels.

17. The system of claim 16, wherein the response function compensator comprises a lookup table memory storing data representing the inverse of the combined response function of the light-sensing elements and a mapper for deriving from the respective interpolated on-grid brightness value derived by the interpolator at a received pixel position a respective index for the lookup table memory and providing a data value in the lookup table memory corresponding to the respective index to the output image memory, the data value being the pixel brightness value of the output image at the received pixel position.

18. The system of claim 16, wherein the interpolator is responsive to a pixel position received from the pixel position generator for computing a product of the estimated off-grid brightness values at a predetermined number of off-grid positions that are nearest neighbors to the received pixel position and an interpolation kernel having the same dimensions as the predetermined number of off-grid positions.

19. The system of claim 16, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel columns being orthogonal to the pixel rows, and the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each intermediate column being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, and wherein the interpolator derives the interpolated on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position generator using the relation

20. The system of claim 16, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel columns and a plurality of regularly spaced pixel rows, the pixel rows being orthogonal to the pixel columns, and the off-grid positions provided by the off-grid generator are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, and wherein the interpolator derives the interpolated on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position generator using the relation

21. The system of claim 1, wherein the off-grid estimator comprises:

an off-grid position generator for providing the off-grid positions of the off-grid position array;

a sensitivity pattern memory for storing data indicative of the sensitivity level corresponding to each one of the pixel positions of the pixel position array;

a plurality of sensitivity level off-grid brightness value estimators, each corresponding to a respective one of the sensitivity levels, for receiving off-grid positions from the off-grid position generator and sensitivity pattern data from the sensitivity pattern memory, each one of the plurality of sensitivity level off-grid brightness value estimators being responsive to an off-grid position received from the off-grid position generator and the sensitivity pattern data from the sensitivity pattern memory for deriving from the captured image brightness values in the captured image memory a respective sensitivity level off-grid brightness value for the corresponding sensitivity level at the received off-grid position;

an accumulator for combining the respective sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimator for the corresponding sensitivity levels at each off-grid position received from the off-grid position generator to derive a respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator; and

a response function compensator for compensating the respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator by the inverse of a combined response function of the light-sensing elements to derive a respective estimated off-grid brightness value at each off-grid position received from the off-grid position generator, the combined response function being the sum of weighted response functions of light-sensing elements having different ones of the plurality of sensitivity levels, each one of the weighted response functions being a response function of a light-sensing element having a respective one of the sensitivity levels multiplied by a predetermined weighting factor for the respective one of the sensitivity levels.

22. The system of claim 21, wherein the response function compensator comprises a lookup table memory storing lookup table data representing the inverse of the combined response function of the light-sensing elements, and a mapper for deriving from the respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator a respective index for the lookup table memory and providing a data value in the lookup table memory corresponding to the respective index as the respective estimated off-grid brightness value at the off-grid position received from the off-grid position generator.

23. The system of claim 21, wherein the output image generator comprises:

an off-grid brightness value memory for storing the respective estimated off-grid brightness value derived by the response function compensator at each off-grid position received from the off-grid position generator;

an on-grid resampler for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective resampled on-grid brightness values at the pixel positions of the pixel position array; and

an output image memory for storing the respective resampled on-grid brightness values derived by the on-grid resampler as the pixel brightness values of the output image,

and wherein the on-grid resampler comprises;

a pixel position generator for providing pixel positions of the pixel position array; and

an interpolator for receiving pixel positions provided by the pixel position generator and being responsive to a pixel position received from the pixel position generator for deriving from the estimated off-grid brightness values in the off-grid brightness value memory a respective resampled on-grid brightness value at the received pixel position.

24. The system of claim 23, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and the off-grid positions provided by the of grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, and wherein the interpolator derives the resampled on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position generator by the relation

25. The system of claim 24, wherein the 4×

4 cubic interpolation kernel φ

has the form;

26. The system of claim 23, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, and wherein the interpolator derives the resampled on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position generator using the relation

27. The system of claim 23, wherein the pixel positions provided by the pixel position generator are located at respective regularly spaced pixel positions in a pixel row defining a linear pixel position array and the off-grid positions provided by the off-grid position generator are each located midway between a respective adjacent pair of pixel positions of the pixel position row, and wherein the interpolator derives the resampled on-grid brightness values at each one of the pixel positions x received from the pixel position generator using the relation

28. The system of claim 27, wherein the 4×

1 cubic interpolation kernel x has the form;

29. The system of claim 21, wherein the accumulator of the off-grid estimator comprises an adder for adding the respective sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators for the corresponding sensitivity levels at each off-grid position received from the off-grid position generator to provide the combined sensitivity level off-grid brightness values at each off-grid position received from the off-grid position generator.

30. The system of claim 21, wherein the accumulator of the off-grid estimator comprises a weighting factor memory for storing a plurality of predetermined weighting factors each corresponding to a different one of the plurality of sensitivity levels of the light-sensing elements, a plurality of multipliers each for multiplying the sensitivity level off-grid brightness value corresponding to a respective one of the sensitivity levels at a received off-grid position by the weighting factor corresponding to the respective one of the sensitivity levels to provide a weighted sensitivity level off-grid brightness value for the corresponding sensitivity level at the off-grid position received from the off-grid position generator, and an adder for summing the weighted sensitivity level off-grid brightness values at the received off-grid position provided by the plurality of multipliers to derive the combined sensitivity level off-grid brightness value at the off-grid positions received from the off-grid position generator.

31. The system of claim 30, wherein the array of light-sensing elements of the image sensor is a linear array and the pixel position array is a linear array having a multiplicity of pixel positions at respective regularly spaced pixel positions in a pixel row, and wherein the off-grid positions of the off-grid position array are each located between a respective adjacent pair of pixel positions of the pixel position array and the sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators for an off-grid position received from the off-grid position generator are respective captured image brightness values at a predefined number of pixel positions that are nearest neighbors to the received off-grid position, each one of the predefined number of nearest neighbor pixel positions corresponding to a different one of the plurality of sensitivity levels of the light-sensing elements.

32. The system of claim 31, wherein each one of the off-grid positions of the off-grid position array is located midway between a respective adjacent pair of pixel positions of the linear pixel position array and the predetermined sensitivity pattern of the light-sensing elements has a first and a second predetermined sensitivity level, and wherein the combined sensitivity level off-grid brightness value at an off-grid position x′

received from the off-grid position generator is expressed as

B(x′

)=W₁I_c(x′

−

0.5)+W₂I_c(x′

+0.5),

where B(x′

) is the combined sensitivity level off-grid brightness value at the received off-grid position x′

, W₁ is the weighting factor for the sensitivity level corresponding to pixel position (x′

−

0.5), W₂ is the weighting factor for the sensitivity level corresponding to pixel position (x′

+0.5), I_c(x′

−

0.5) is the captured image brightness value at pixel position (x′

−

0.5) and I_c(x′

+0.5) is the captured image brightness value at pixel position (x′

+0.5), and the combined response function of the light-sensing elements is expressed as

S(E)=W₁P₁(E)+W₂P₂(E),

where P₁(E) is the radiometric response function of a light-sensing element having the sensitivity level corresponding to pixel position (x′

−

0.5) and P2(E) is the radiometric response function of a light-sensing element having the sensitivity level corresponding to pixel position (x′

+0.5).

33. The system of claim 30, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the pixel columns, the multiplicity of off-grid positions are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

34. The system of claim 33, wherein the predetermined sensitivity pattern of the array of light-sensing elements corresponds to a pixel position array having repetitive disposed groups of four nearest neighbor pixel positions, where each one of the four nearest neighbor pixel positions in each group corresponds to a respective one of four predetermined sensitivity levels in a common predetermined positional order.

35. The system of claim 30, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the pixel columns, and wherein the multiplicity of off-grid positions are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

36. The system of claim 30, wherein the predetermined sensitivity pattern of the array of light-sensing elements has a first and a second predetermined sensitivity level, and adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of the pixel columns correspond to different ones of the predetermined first and second sensitivity levels.

37. The system of claim 30, wherein the array of light-sensing elements of the image sensor is a linear array and the multiplicity of pixel positions are located at respective regularly spaced pixel positions in a pixel row so as to form a linear captured image brightness value array, and wherein the linear array of light-sensing elements of the image sensor is adapted to be sequentially exposed to successive regularly spaced linear regions of the scene in multiple exposures, each one of the exposures producing a respective linear captured image brightness value array, the respective linear captured image brightness value arrays produced by the multiple exposures being concatenated to form a two-dimensional captured image brightness value array having respective captured image brightness values at pixel positions of a pixel position array located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the multiplicity of off-grid positions are located at respective intersections of the pixel rows and a plurality of regularly spaced intermediate columns, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

38. The system of claim 37, wherein the predetermined sensitivity pattern of the array of light-sensing elements has a first and a second predetermined sensitivity level, and wherein adjacent pixel positions in each pixel row correspond to different ones of the first and the second predetermined sensitivity levels and adjacent pixel positions in each pixel column correspond to the same one of the first and the second predetermined sensitivity levels.

39. The system of claim 21, wherein each one of the sensitivity level off-grid brightness value estimators is responsive to an off-grid position received from the off-grid position generator and the data indicative of the sensitivity level corresponding to each one of the pixel positions from the sensitivity pattern memory for determining a pixel position corresponding to the same sensitivity level as the sensitivity level off-grid brightness value estimator and located nearest the received off-grid position, and for estimating a sensitivity level off-grid brightness value at the received off-grid position to be equal to the captured image brightness value at the nearest pixel position corresponding to the same sensitivity level as the sensitivity level off-grid brightness value estimator.

40. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device having respective integrated lenses formed over the photosensitive surfaces of predetermined ones of the photodiodes, each integrated lens concentrating the incident light onto the respective light-sensing surface, whereby the sensitivity level of each one of the photodiodes is determined by whether or not an integrated lens is formed over the light-sensing surface thereof.

41. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, each one of the photodiodes having an integrated light filter formed above the light-sensing surface thereof, the transparency of each integrated light filter being fixed during fabrication of the charge coupled device, whereby the sensitivity level of each one of the photodiodes in the array is determined by the transparency of the integrated light filter above the light-sensing surface thereof.

42. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device being formed with an opaque layer overlying the array of photodiodes with a respective aperture formed in the opaque layer above the light-sensing surface of each one of the photodiodes, the respective size of the aperture above the light-sensing surface of each one of the photodiodes being fixed during fabrication of the charge coupled device, whereby the sensitivity level of each one of the photodiodes in the array is determined by the size of the aperture above the light-sensing surface thereof.

43. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having a multiplicity of photodiodes in an array of spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device having an interline structure in which the columns of photodiodes are each disposed adjacent a corresponding vertical shift register, each photodiode having an associated transfer gate electrode for controlling the transfer of stored photogenerated charge from the photodiode to the corresponding vertical shift register, the stored photogenerated charge in each one of the photodiodes being operatively periodically removed by the application of a periodic reset pulse to the device, the photogenerated charge stored in each one of the photodiodes being operatively transferred to a respective stage of the corresponding vertical shift register by the application of a transfer trigger pulse to the associated transfer gate electrode and being accumulated therein for an exposure interval, the transfer trigger pulse being applied immediately preceding a reset pulse, the respective transfer trigger pulses being operatively applied to the transfer gate electrodes associated with the photodiodes in the array at predetermined frequencies, whereby the sensitivity level of each one of the photodiodes in the array is determined by the frequency of transfer trigger pulses applied to the transfer gate electrode associated therewith.

44. The system of claim 1, wherein the off-grid estimator comprises:

a pixel position generator for providing the pixel positions of the pixel position array;

a sensitivity pattern memory for storing the sensitivity level corresponding to each one of the pixel positions;

a response function compensator receiving pixel positions of the pixel position array from the pixel position generator, data indicative of the sensitivity levels corresponding to the received pixel positions from the sensitivity pattern memory and captured image brightness values at the received pixel positions from the captured image memory, and being responsive to a pixel position received from the pixel position generator and data indicative of the sensitivity level corresponding to the received pixel position for retrieving from the captured image memory the captured image brightness value at the received pixel position and for compensating the retrieved captured imaged brightness value at the received pixel positions by the inverse of a response function of a light-sensing element having the sensitivity level corresponding to the received pixel position to provide a respective compensated brightness value at the pixel position received from the pixel position generator;

a compensated on-grid brightness value memory for storing respective compensated captured image brightness values derived by the response function compensator;

an off-grid position generator for providing the off-grid positions of the off-grid position array;

a pseudoinverse estimator receiving off-grid positions from the off-grid position generator and being responsive to an off-grid position received from the off-grid position generator for deriving from the compensated captured image brightness values in the compensated on-grid brightness value memory a respective estimated off-grid brightness value at the off-grid positions received from the off-grid position generator.

45. The system of claim 44, wherein the output image generator comprises

an off-grid brightness value memory for storing the respective estimated off-grid brightness values derived by the pseudoinverse estimator;

an on-grid resampler for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective on-grid resampled brightness values at the pixel positions of the pixel position array; and

an output image memory for storing the respective resampled on-grid brightness values derived by the on-grid resampler as pixel brightness values of the output image, and wherein the on-grid resampler comprises;

a second pixel position generator for providing the pixel positions of the pixel position array; and

an interpolator receiving pixel positions from the pixel position generator and being responsive to a pixel position received from the second pixel position generator for deriving from the estimated off-grid brightness values in the off-grid brightness value memory the resampled on-grid brightness value at the pixel position received from the second pixel position generator.

46. The system of claim 45, wherein the pixel positions of the pixel position array provided by the second pixel position generator are located at respective regularly spaced pixel positions of a linear pixel position array, and the off-grid positions of the off-grid position array provided by the off-grid position generator are each located between a respective adjacent pair of pixel positions of the linear pixel position array.

47. The system of claim 46, wherein the off-grid positions provided by the off-grid position generator are each located midway between a respective adjacent pair of pixel positions of the linear pixel position array, the pixel positions provided by the second pixel position generator having coordinate x, where x has values from 3.5 to xSize−

3.5 and xSize is the dimension of the captured line image.

48. The system of claim 47, wherein the interpolator is responsive to a pixel position received from the second pixel position generator for deriving a respective resampled on-grid position brightness value in accordance with the relation

49. The system of claim 48, wherein the 4×

1 cubic interpolation kernel Ψ

has the form;

50. The system of claim 45, wherein the pixel positions of the pixel position array provided by the second pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions of the off-grid position array provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intelluediate rows and a plurality of regularly spaced intermediate columns, each intermediate row being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows and each intermediate column being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.

51. The system of claim 50, wherein the interpolator derives a respective resampled on-grid brightness value at a pixel position received from the second pixel position generator by computing the product of respective estimated off-grid brightness values at a predetermined number of off-grid positions that are nearest neighbors to the received pixel position and an interpolation kernel having the same dimensions as the predetermined number of off-grid positions.

52. The system of claim 51, wherein the plurality of intermediate rows each extend medially between a respective adjacent pair of pixel rows and the plurality of regularly spaced intermediate columns each extend medially between a respective adjacent pair of pixel columns, and wherein the pixel positions provided by the second pixel position generator have coordinates (x,y) where x has values from 2.5 to xSize−

2.5 and, y has values from 2.5 to ySize 2.5, xSize is the dimension of the captured image in the direction of the pixel rows and ySize is the dimension of the captured image in the direction of the pixel columns.

53. The system of claim 52, wherein the interpolator is responsive to a pixel position (x,y) received from the second pixel position generator for deriving a respective resampled on-grid brightness value in accordance with the relation

54. The method of claim 53, wherein the 4×

4 cubic interpolation kernel φ

has the form;

55. The system of claim 51, wherein the interpolation kernel is a 2-dimensional Gaussian kernel.

56. The system of claim 55, wherein the 2-dimensional Gaussian kernel has the form:

57. The system of claim 51, wherein the interpolation kernel is a bi-cubic B-spline interpolation filter kernel.

58. The system of claim 57, wherein the bi-cubic B-spline interpolation filter kernel has the form:

59. The system of claim 51, wherein the interpolation kernel is a bi-linear interpolation filter kernel.

60. The system of claim 59, wherein the bi-linear interpolation kernel has the form:

61. The system of claim 45, wherein the pixel positions of the pixel position array provided by the second pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions of the off-grid position array provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate pixel columns being parallel to the pixel columns and extending between a respective adjacent pair of pixel columns.

62. The system of claim 61, wherein the interpolator includes means for shifting the off-grid position array to coincide with the pixel position array, whereby the resampled on-grid brightness value at each pixel position of the pixel position array is equal to the estimated off-grid brightness value at a coincident off-grid position of the shifted off-grid position array.

63. The system of claim 45, wherein the pixel positions provided by the second pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each intermediate column being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, the pixel positions provided by the second pixel position generator having coordinates (x,y) where x has values from 2.5 to xSize−

2.5, y has values from 2.5 to ySize−

2.5, xSize is the dimension of the captured image in the direction of the pixel rows and ySize is the dimension of the captured image in the direction of the pixel columns.

64. The system of claim 63, wherein the interpolator is responsive to a pixel position received from the second pixel position generator for deriving a respective resampled on-grid brightness value in accordance with the relation

65. The system of claim 44, wherein the response function compensator comprises a lookup table memory storing separate lookup table data representing the inverse of respective response functions of light-sensing elements having different ones of the plurality of sensitivity levels, and a mapper receiving from the captured image memory the captured image brightness value at the pixel position received from the first pixel position generator for deriving therefrom a respective index for the lookup table data representing the inverse of the response function of a light-sensing element having the sensitivity level corresponding to the received pixel position, and providing a compensated on-grid brightness value corresponding to the index to the compensated on-grid brightness value memory.

66. The system of claim 44, wherein the pseudoinverse estimator comprises means responsive to an off-grid positions received from the off-grid position generator for retrieving from the compensated on-grid brightness value memory respective compensated on-grid brightness values at a predetermined number of pixel positions that are nearest neighbors to the received off-grid position, a comparator for comparing each one of the respective compensated on-grid brightness values at the nearest neighbor pixel positions to a predetermined low threshold value and a predetermined high threshold value, an adder means for computing a sum of compensated on-grid brightness values at the nearest neighbor pixel positions that are greater than the predetermined low threshold value and less than the predetermined high threshold value, and a divider for deriving the estimated off-grid brightness value at the received off-grid position by dividing the sum computed by the adder by the number of compensated on-grid brightness values included in the sum.

67. The system of claim 44, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each intermediate column being parallel to the pixel columns and extending medially between a respective adjacent pair of pixel columns, and wherein the off-grid positions provided by the off-grid position generator have coordinates (x′

,y′

) where x′

has values from 1 to xSize−

1 and y′

has values from 1 to ySize−

1, xSize being the dimension of the captured image in the direction of the pixel rows and ySize being the dimension of the captured image in the direction of the pixel columns.

68. The system of claim 67, wherein the predetermined sensitivity pattern of the array of light-sensing elements corresponds to a pixel position array having repetitively disposed groups of four nearest neighbor pixel positions, where each of the four nearest neighbor pixel positions in each group corresponds to a respective one of four predetermined sensitivity levels in a common predetermined positional order.

69. The system of claim 67, wherein the pseudoinverse estimator for estimating the respective brightness value at an off-grid position (x′

,y′

) received from the off-grid position generator comprises;

a memory for storing a 4×

4 cubic interpolator kernel φ

;

a comparator for comparing the compensated on-grid brightness value at each one of sixteen pixel positions (u,v) that are nearest neighbors to the received off-grid position (x′

,y′

) with a predetermined low threshold value and a predetermined high threshold value;

a first computing means for calculating the indices i and j of the 4×

4 cubic interpolation kernel φ

using the relations i=x′

−

u−

1.5 and j=y′

−

v−

1.5 for each one of the sixteen nearest neighbor pixel positions (u,v) having a compensated on-grid brightness value I(u,v) that is greater than the predetermined low threshold value and less than the predetermined high threshold value, where u has values from x′

−

1.5 to x′

+1.5 and v has values from y′

−

1.5 to y′

+1.5;

a second computing means for calculating the quantity

70. The system of claim 44, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, and the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each intermediate column being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, the off-grid positions provided by the off-grid position generator having coordinates (x′

,y′

) where x′

has values from 1 to xSize−

1 and y′

has values from 0.5 to ySize−

0.5 for off-grid positions located at respective intersections of the pixel rows and the intermediate columns, and where x′

has values from 0.5 to ySize−

0.5 and y′

has values from 1 to ySize−

1 for off-grid positions located at respective intersections of the pixel columns and the intermediate rows, xSize being the dimension of the captured image in the direction of the pixel rows and ySize being the dimension of the captured image in the direction of the pixel columns.

71. The system of claim 70, wherein the predetermined sensitivity pattern of the array of light-sensing elements has first and second predetermined sensitivity levels, and adjacent pixel positions in each pixel row and adjacent pixel positions in each pixel column correspond to different ones of the first and second predetermined sensitivity levels.

72. The system of claim 70, wherein the pseudoinverse estimator for estimating the respective off-grid brightness value at an off-grid position (x′

,y′

) received from the off-grid position generator comprises;

a memory for storing a 4×

4 cubic interpolation kernel φ

;

a first computing means responsive to an off-grid position (x′

,y′

) received from the off-grid position generator for computing coordinates (u,v) using the relations u=x′

+0.5 i−

0.5j and v=y′

−

1.5+0.5i+0.5j, where i and j are indices of the 4×

4 cubic interpolation kernel dφ

, and each one of the indices i and j has values from 0 to 3;

means responsive to each pair of coordinates (u,v) computed by the first computing means for determining whether the coordinates (u,v) are those of a pixel position;

a comparator responsive to the coordinates (uv) being coordinates of a pixel position for comparing the compensated on-grid brightness value I(u,v) at the pixel position (u,v) with a predetermined low threshold value and a predetermined high threshold value;

a second computing means responsive to the coordinates (u,v) being the coordinates of a pixel position and the compensated on-grid brightness value I(uv) at the pixel position (u,v) being greater than a predetermined low threshold value and less than a predetermined high threshold value for calculating the quantity

73. The system of claim 72, wherein the 4×

4 cubic interpolation kernel φ

has the form;

74. The system of claim 44, wherein the pixel positions provided by the pixel position generator are located at respective regularly spaced pixel positions in a linear pixel position array and the off-grid positions provided by the off-grid position generator are each located midway between a respective adjacent pair of pixel positions of the linear pixel position array, and wherein the predetermined sensitivity pattern of the light-sensing elements has a first and a second predetermined sensitivity level, and adjacent pixel positions of the linear pixel position array correspond to different ones of the first and second predetermined sensitivity levels.

75. The system of claim 74, wherein the pixel positions provided by the pixel position generator has values from 0.5 to xSize−

0.5, where xSize is the dimension of the captured line image, and the off-grid positions provided by the off-grid position generator have values from 2 to xSize−

2.

76. The system of claim 75, wherein the pseudoinverse estimator for estimating the respective brightness values at an off-grid position x′

received from the off-grid position generator comprises;

a memory for storing a 4□

1 Gaussian interpolation kernel G;

a comparator for comparing the compensated on-grid brightness value I(k) at each one of four pixel positions k that are nearest neighbors to the received off-grid position x′

with a predetermined low threshold value and a predetermined high threshold value, where k has values from x′

−

1.5 to x′

+1.5;

a first computing means for calculating the quantity I(k)G(kW−

x′

) for each one of the compensated on-grid brightness values I(k) at the four nearest neighbor pixel positions k that is greater than the predetermined low threshold value and less than the predetermined high threshold value, where G(k−

x′

) is the value of the Gaussian interpolation kernel G at position (k′

x′

);

a first adder for adding the quantities I(k)G(k−

x′

) for all values of I(k) greater than the predetermined low threshold value and less than the predetermined high threshold value to derive a first sum p;

a second adder for adding the Gaussian interpolation kernel values G(k−

x′

) for all values of k where I(k) is greater than the predetermined low threshold value and less than the predetermined high threshold value to derive a second sum q; and

a divider for deriving the estimated off-grid brightness value B(x′

) at the off-grid position x′

received from the off-grid position generator.

77. The system of claim 76, wherein the predetermined low threshold value is the noise level of a light-sensing element having a sensitivity level corresponding to the pixel position of the compensated on-grid brightness value being compared therewith compensated by the inverse of the response function of the light-sensing element, and the predetermined high threshold value is the saturation or near saturation brightness value of a light-sensing element having the sensitivity level corresponding to the pixel position of the compensated on-grid brightness value being compared therewith.