Photocurrent estimation from multiple captures for simultaneous SNR and dynamic range improvement in CMOS image sensors

• US 20030098919A1
• Filed: 11/13/2001
• Published: 05/29/2003
• Est. Priority Date: 11/13/2001
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Liu, Xinqiao, Gamal, Abbas El
• Granted Patent: US 7,009,636 B2
• Time in Patent Office: Days
• US Class Current: 348/297
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

The dynamic range and the noise immunity of a digital imaging system are increased by basing an estimate of the illumination on a sensor on a series of measurements of the accumulated illumination at intervals within an exposure period. The measuring may occur destructively, or alternatively the photocurrent of the sensor may continue to accumulate over the exposure period. The estimate may use statistical signal processing of the measurements, based on various noise models and various optimization criteria. The estimate may be computed recursively over the multiplicity of measurements, using a set of recursive values that may include but is not limited to the estimated illumination, a current weighting coefficient, a variance of the current measurement and a variance over the series of measurements.

Claims

9. An estimation method for non-recursively estimating an optimal illumination on a sensor capable of capturing non-destructively a plurality of image samples during an exposure period, said method comprising the steps of:

measuring an illumination indication from said sensor;

storing said illumination indication, wherein said measuring and storing steps occur a multiplicity of times during said exposure period thereby collecting a multiplicity of measurements; and

performing a non-recursive optimal illumination estimation on said sensor from said collected multiplicity of measurements.

1. An estimation method for estimating illumination on a sensor capable of capturing non-destructively a plurality of image samples during an exposure period, said method comprising the steps of:

measuring an illumination indication from said sensor, said measuring occurs a multiplicity of times at intervals during said exposure period thereby producing a multiplicity of measurements; and

determining, based on weighted averaging, an estimated illumination on said sensor from said multiplicity of measurements.

2. The estimation method of claim 1, wherein said sensor is a photodiode and said illumination indication is a charge accumulated from photocurrent produced by said photodiode.

3. The estimation method of claim 2, wherein said measuring step occurring non-destructively and said charge accumulating over said exposure period.

4. The estimation method of claim 1, wherein said determining step including statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model.

5. The estimation method of claim 1, wherein said determining step including statistical signal processing of said multiplicity of measurements, said signal processing being based on maximizing a likelihood of accuracy of said estimated illumination.

6. The estimation method of claim 1, wherein said determining step including statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing an error of said estimated illumination.

7. The estimation method of claim 1, wherein said determining step including statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing a linear mean square error of said estimated illumination.

8. The estimation method of claim 1, wherein said sensor is configured in a sensor array, a pixel sensor in a digital camera, a pixel sensor in a video camera, a pixel sensor in a stereo digital camera or a pixel sensor in a stereo video camera.

10. The estimation method of claim 9, wherein said sensor is a photodiode and said illumination indication is a charge accumulated from photocurrent produced by said photodiode.

11. The estimation method of claim 10, wherein said measuring step occurring non-destructively and said charge accumulating over said exposure period.

12. The estimation method of claim 9, wherein said determining step including statistical signal processing of said multiplicity of measurements, said signal processing being based on maximizing a likelihood of accuracy of said estimated illumination.

13. The estimation method of claim 9, wherein said determining step further comprising statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing an error of said estimated illumination.

14. The estimation method of claim 9, wherein said determining step further comprising statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing a linear mean square error of said estimated illumination.

15. The estimation method of claim 9, wherein said sensor is configured in a sensor array, a pixel sensor in a digital camera, a pixel sensor in a video camera, a pixel sensor in a stereo digital camera or a pixel sensor in a stereo video camera.

16. An estimation method for recursively estimating an optimal illumination on a sensor capable of capturing non-destructively a plurality of image samples during an exposure period, said method comprising the steps of:

measuring an illumination indication from said sensor, said measuring occurs a multiplicity of times at intervals during said exposure period thereby producing a multiplicity of measurements; and

determining an estimated illumination on said sensor from said multiplicity of measurements, said determining step occurring recursively over said multiplicity of measurements and including statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model.

17. The estimation method of claim 16 further comprising a step of maintaining a plurality of parameters during said measuring step, said plurality of parameters comprising:

said estimated illumination;

means for weighting a particular one of said multiplicity of measurements;

means for indicating variance between said particular one of said multiplicity of measurements and said multiplicity of measurements; and

means for indicating overall variance of said multiplicity of measurements.

18. The estimation method of claim 16 further comprising a step of maintaining a plurality of parameters during said measuring step, said plurality of parameters comprising:

said estimated illumination;

a weighting coefficient applied to a difference between a present one of said multiplicity of measurements and said estimated illumination corresponding to a previous one of said multiplicity of measurements;

a mean square error of said estimated illumination; and

a covariance of said estimated illumination with said present one of said multiplicity of measurements.

19. The estimation method of claim 16, wherein said sensor is a photodiode and said illumination indication is a charge accumulated from photocurrent produced by said photodiode.

20. The estimation method of claim 16, wherein said measuring step occurring non-destructively and said charge accumulating over said exposure period.

21. The estimation method of claim 16, wherein said determining step including statistical signal processing of said multiplicity of measurements, said signal processing being based on maximizing a likelihood of accuracy of said estimated illumination.

22. The estimation method of claim 16, wherein said determining step further comprising statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing an error of said estimated illumination.

23. The estimation method of claim 16, wherein said determining step further comprising statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing a linear mean square error of said estimated illumination.

24. The estimation method of claim 16, wherein said sensor is configured in a sensor array, a pixel sensor in a digital camera, a pixel sensor in a video camera, a pixel sensor in a stereo digital camera or a pixel sensor in a stereo video camera.

25. An apparatus configured to estimate illumination on a sensor during an exposure period, said apparatus comprising:

a sampling means configured to measure, at a multiplicity of time intervals during said exposure period, an illumination indication from a sensor, and configured to thereby produce a multiplicity of measurements; and

an estimation means configured to determine, based on weighted averaging, an estimated illumination on said sensor from said multiplicity of measurements.

26. The apparatus of claim 25, wherein said sensor is implemented in a sensor array, a pixel sensor in a single chip imaging device, a pixel sensor in a digital camera, a pixel sensor in a video camera, a pixel sensor in a stereo digital camera or a pixel sensor in a stereo video camera.

27. The apparatus of claim 25, wherein said sensor is a photodiode and said illumination indication is a charge accumulated from photocurrent produced by said photodiode.

28. The apparatus of claim 27, wherein said sampling means operates non-destructively and said charge accumulates over said exposure period.

29. The apparatus of claim 25, wherein said estimation means being configured to perform statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model.

30. The apparatus of claim 25, wherein said estimation means being configured to perform statistical signal processing of said multiplicity of measurements, said signal processing being based on maximizing a likelihood of accuracy of said estimated illumination.

31. The apparatus of claim 25, wherein said estimation means being configured to perform statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing an error of said estimated illumination.

32. The apparatus of claim 25, wherein said estimation means being configured to perform statistical signal processing of said multiplicity of measurements, said signal processing being based on minimizing a linear mean square error of said estimated illumination.

33. An apparatus configured to estimate illumination on a sensor during an exposure period, said apparatus comprising:

a sampling means configured to measure, at a multiplicity of time intervals during said exposure period, an illumination indication from a sensor, and configured to thereby produce a multiplicity of measurements; and

an estimation means configured to determine an estimated illumination on said sensor from said multiplicity of measurements, said estimation means being configured to compute recursively over said multiplicity of measurements and to maintain recursively a plurality of parameters over said multiplicity of measurements, said plurality of parameters comprising;

said estimated illumination;

means for weighting a particular one of said multiplicity of measurements;

means for indicating variance between said particular one of said multiplicity of measurements and said multiplicity of measurements; and

means for indicating overall variance of said multiplicity of measurements.

34. An apparatus configured to estimate illumination on a sensor during an exposure period, said apparatus comprising:

a sampling means configured to measure, at a multiplicity of time intervals during said exposure period, an illumination indication from a sensor, and configured to thereby produce a multiplicity of measurements; and

an estimation means configured to determine an estimated illumination on said sensor from said multiplicity of measurements, wherein said estimation means being configured to perform statistical signal processing of said multiplicity of measurements, said signal processing being based on a noise model selected from a fixed pattern noise model, a reset noise model, a shot noise model and a read noise model, and wherein said estimation means being configured to compute recursively over said multiplicity of measurements and to maintain recursively a plurality of parameters over said multiplicity of measurements, said plurality of parameters comprising;

said estimated illumination;

a weighting coefficient applied to a difference between a present one of said multiplicity of measurements and said estimated illumination corresponding to a previous one of said multiplicity of measurements;

a mean square error of said estimated illumination; and

a covariance of said estimated illumination with said present one of said multiplicity of measurements.

35. An apparatus configured to estimate illumination on a sensor during an exposure period for simultaneously reducing noise and improving dynamic range at low illumination end, where said sensor is configured in a complementary metal oxide semiconductor (CMOS) image sensor system capable of capturing multiple image samples during said exposure period, said apparatus comprising:

means for measuring, at a multiplicity of intervals during said exposure period, actual photocurrent from said sensor, said means for measuring thereby producing a multiplicity of photocurrent measurements; and

means for estimating optimal photocurrent on said sensor from said multiplicity of measurements.