Photocurrent estimation from multiple captures for simultaneous SNR and dynamic range improvement in CMOS image sensors

• US 7,009,636 B2
• Filed: 11/13/2001
• Issued: 03/07/2006
• Est. Priority Date: 11/13/2001
• Status: Expired due to Term

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Gamal, Abbas El, Liu, Xinqiao
• Primary Examiner(s): Ometz, David
• Assistant Examiner(s): Daniels, Anthony J.
• Application Number:
  US09/992,479
• Publication Number:
  US 20030098919A1
• Time in Patent Office: 1,575 Days
• Field of Search: 348/241, 348/246, 348/229.1
• US Class Current: 348/208.1
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

The dynamic range and the noise immunity of a digital imaging system are increased by basing an estimate of the illumination on a sensor on a series of measurements of the accumulated illumination at intervals within an exposure period. The measuring may occur destructively, or alternatively the photocurrent of the sensor may continue to accumulate over the exposure period. The estimate may use statistical signal processing of the measurements, based on various noise models and various optimization criteria. The estimate may be computed recursively over the multiplicity of measurements, using a set of recursive values that may include but is not limited to the estimated illumination, a current weighting coefficient, a variance of the current measurement and a variance over the series of measurements.

Claims

1. An estimation method for recursively estimating an optimal illumination on a sensor capable of capturing non-destructively a plurality of image samples during an exposure period, said method comprising:

measuring an illumination indication from said sensor, said measuring occurs two or more times at intervals during said exposure period, producing a multiplicity of measurements;

determining an estimated illumination on said sensor from all or essentially all of said multiplicity of measurements non-destructively captured before motion/saturation, said determining step occurring recursively over said multiplicity of measurements and including statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model; and

maintaining a plurality of parameters during said measuring step, said plurality of parameters comprising;

said estimated illumination;

a weighting coefficient of a particular one of said multiplicity of measurements;

a variance between said particular one of said multiplicity of measurements and said multiplicity of measurements; and

an overall variance of said multiplicity of measurements.

2. The estimation method of claim 1, wherein said sensor is a photodiode and said illumination indication is a charge accumulated from photocurrent produced by said photodiode.

3. The estimation method of claim 1, wherein said measuring occurs non-destructively and said charge accumulates over said exposure period.

4. The estimation method of claim 1, wherein said determining includes statistical signal processing of said multiplicity of measurements, said signal processing being based on maximizing a likelihood of accuracy of said estimated illumination.

5. The estimation method of claim 1, wherein said determining includes statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing an error of said estimated illumination.

6. The estimation method of claim 1, wherein said determining includes statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing a linear mean square error of said estimated illumination.

7. The estimation method of claim 1, wherein said sensor is configured in a sensor array, a pixel sensor in a digital camera, a pixel sensor in a video camera, a pixel sensor in a stereo digital camera or a pixel sensor in a stereo video camera.

8. An estimation method for recursively estimating an optimal illumination on a sensor capable of capturing non-destructively a plurality of image samples during an exposure period, said method comprising:

measuring an illumination indication from said sensor, said measuring occurs two or more times at intervals during said exposure period, producing a multiplicity of measurements;

determining an estimated illumination on said sensor from all or essentially all of said multiplicity of measurements non-destructively captured before motion/saturation, said determining step occurring recursively over said multiplicity of measurements and including statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model; and

maintaining a plurality of parameters during said measuring step, said plurality of parameters comprising;

said estimated illumination;

a weighting coefficient applied to a difference between a present one of said multiplicity of measurements and said estimated illumination corresponding to a previous one of said multiplicity of measurements;

a mean square error of said estimated illumination; and

a covariance of said estimated illumination with said present one of said multiplicity of measurements.

9. The estimation method of claim 8, wherein said sensor is a photodiode and said illumination indication is a charge accumulated from photocurrent produced by said photodiode.

10. The estimation method of claim 8, wherein said measuring step occurs non-destructively and said charge accumulates over said exposure period.

11. The estimation method of claim 8, wherein said determining includes statistical signal processing of said multiplicity of measurements, said signal processing being based on maximizing a likelihood of accuracy of said estimated illumination.

12. The estimation method of claim 8, wherein said determining includes statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing an error of said estimated illumination.

13. The estimation method of claim 8, wherein said determining includes statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing a linear mean square error of said estimated illumination.

14. The estimation method of claim 8, wherein said sensor is configured in a sensor array, a pixel sensor in a digital camera, a pixel sensor in a video camera, a pixel sensor in a stereo digital camera or a pixel sensor in a stereo video camera.

15. An apparatus configured to estimate illumination on a sensor during an exposure period, said apparatus comprising:

a sampling means configured to measure, at a multiplicity of time intervals during said exposure period, an illumination indication from said sensor, and configured to produce a multiplicity of measurements; and

an estimation means configured to determine an estimated illumination on said sensor from said multiplicity of measurements, said estimation means being configured to compute recursively over said multiplicity of measurements and to maintain recursively a plurality of parameters over said multiplicity of measurements, said plurality of parameters comprising;

said estimated illumination;

a weighting coefficient of a particular one of said multiplicity of measurements;

a variance between said particular one of said multiplicity of measurements and said multiplicity of measurements; and

an overall variance of said multiplicity of measurements.

16. The apparatus of claim 15, wherein said sensor is implemented in a sensor array, a pixel sensor in a single chip imaging device, a pixel sensor in a digital camera, a pixel sensor in a video camera, a pixel sensor in a stereo digital camera or a pixel sensor in a stereo video camera.

17. The apparatus of claim 15, wherein said sensor is a photodiode and said illumination indication is a charge accumulated from photocurrent produced by said photodiode.

18. The apparatus of claim 15, wherein said sampling means operates non-destructively and said charge accumulates over said exposure period.

19. The apparatus of claim 15, wherein said estimation means can perform statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model.

20. The apparatus of claim 15, wherein said estimation means can perform statistical signal processing of said multiplicity of measurements, said signal processing being based on maximizing a likelihood of accuracy of said estimated illumination.

21. The apparatus of claim 15, wherein said estimation means can perform statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing an error of said estimated illumination.

22. The apparatus of claim 15, wherein said estimation means can perform statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing a linear mean square error of said estimated illumination.

23. An apparatus configured to estimate illumination on a sensor during an exposure period, said apparatus comprising:

a sampling means configured to measure, at a multiplicity of time intervals during said exposure period, an illumination indication from said sensor, and configured to produce a multiplicity of measurements; and

an estimation means configured to determine an estimated illumination on said sensor from said multiplicity of measurements, wherein said estimation means is configured to perform statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model, and wherein said estimation means is configured to compute recursively over said multiplicity of measurements and to maintain recursively a plurality of parameters over said multiplicity of measurements, said plurality of parameters comprising;

said estimated illumination;

a weighting coefficient applied to a difference between a present one of said multiplicity of measurements and said estimated illumination corresponding to a previous one of said multiplicity of measurements;

a mean square error of said estimated illumination; and

a covariance of said estimated illumination with said present one of said multiplicity of measurements.

24. The apparatus of claim 23, wherein said sensor is implemented in a sensor array, a pixel sensor in a single chip imaging device, a pixel sensor in a digital camera, a pixel sensor in a video camera, a pixel sensor in a stereo digital camera or a pixel sensor in a stereo video camera.

25. The apparatus of claim 23, wherein said sensor is a photodiode and said illumination indication is a charge accumulated from photocurrent produced by said photodiode.

26. The apparatus of claim 23, wherein said sampling means operates non-destructively and said charge accumulates over said exposure period.

27. The apparatus of claim 23, wherein said estimation means can perform statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model.

28. The apparatus of claim 23, wherein said estimation means can perform statistical signal processing of said multiplicity of measurements, said signal processing being based on maximizing a likelihood of accuracy of said estimated illumination.

29. The apparatus of claim 23, wherein said estimation means can perform statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing an error of said estimated illumination.

30. The apparatus of claim 23, wherein said estimation means can perform statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing a linear mean square error of said estimated illumination.