Motion/saturation detection system and method for synthesizing high dynamic range motion blur free images from multiple captures

• US 7,061,524 B2
• Filed: 11/13/2001
• Issued: 06/13/2006
• Est. Priority Date: 11/13/2001
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: El Gamal, Abbas, Liu, Xinqiao
• Primary Examiner(s): Vu, Ngoc-Yen
• Assistant Examiner(s): Daniels, Anthony J.
• Application Number:
  US09/992,480
• Publication Number:
  US 20030095189A1
• Time in Patent Office: 1,673 Days
• Field of Search: 348/241, 348/296, 348/208.4, 348/208.12, 348/221.1, 250/208.1, 382/107, 382/274, 382/318
• US Class Current: 348/208.4
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Motion/Saturation detection system and method for synthesizing high dynamic range motion blur free images from multiple captures, the system and method utilizing photocurrent estimation to reduce read noise and enhance dynamic range at the low illumination end, saturation detection to enhance dynamic range at the high illumination end, and motion blur detection to ensure the photocurrent estimation is not corrupted by motion. Motion blur detection also makes it possible to extend exposure time and to capture more images, which can be used to further enhance dynamic range at the low illumination end. The present invention operates completely locally, making it well suited for single chip digital camera implementations.

Claims

1. A method for preventing motion/saturation from corrupting image capturing during a global exposure time of a sensor, comprising:

performing for each pixel of said sensor;

a) determining a difference between an illumination measurement obtained during current image capturing and an illumination estimation generated during previous image capturing;

b) comparing said difference with a threshold value;

c) determining, based on step b), whether motion/saturation has occurred;

d1) accepting said illumination measurement and updating said illumination estimation if motion/saturation has not occurred;

d2) updating said illumination estimation with said illumination measurement if motion/saturation has occurred and if terminating said performing act is deferred; and

d3) terminating said performing act and outputting said illumination estimation as final illumination estimation if motion/saturation has occurred.

2. The method of claim 1, wherein said threshold value is generated based on a predetermined parameter and a prediction variable, said predetermined parameter being configured to achieve desired tradeoff between signal-to-noise ratio and motion blur.

3. The method of claim 1, wherein said image capturing occurs a multiplicity of times during said global exposure time thereby producing a multiplicity of measurements and wherein an optimal illumination estimation for said sensor is generated based on said multiplicity of measurements.

4. The method of claim 3, wherein said optimal illumination estimation is generated based on maximum likelihood.

5. The method of claim 3, wherein said optimal illumination estimation is generated based on linear minimum mean square error.

6. The method of claim 1, wherein said image capturing occurs a multiplicity of times during said global exposure time thereby producing a multiplicity of measurements and wherein an optimal illumination estimation for said sensor is generated based on selectively accepted multiplicity of measurements.

7. The method of claim 6, wherein said optimal illumination estimation is generated based on maximum likelihood.

8. The method of claim 6, wherein said optimal illumination estimation is generated based on linear minimum mean square error.

9. The method of claim 1, wherein each pixel'"'"'s effective exposure time is adaptive to its own lighting condition thereby enabling performing for each pixel independently of other pixels'"'"' lighting conditions.

10. The method of claim 1, wherein each pixel is capable of terminating its own exposure time thereby enabling extending said global exposure time.

11. The method of claim 10, wherein said global exposure time is limited by motion and saturation only thereby enabling said sensor to achieve higher signal-to-noise ratio and dynamic range.

12. The method of claim 1, wherein said steps a)–

c) are performed based on parameters calculated recursively, said parameters including a weighting coefficient, an overall variance, and a covariance.

13. The method of claim 1, wherein said steps a)–

c) are performed based on parameters calculated non-recursively.

14. The method of claim 1, wherein said illumination estimation is generated based on maximum likelihood.

15. The method of claim 1, wherein said illumination estimation is generated based on linear minimum mean square error.

16. The method of claim 1, further comprising

utilizing a soft decision rule for preventing error accumulation due to slow motion.

17. The method of claim 16, wherein said threshold value is characterized by a first range of values and a second range of values that include said first range of values, wherein said first range of values is characterized by a first constant parameter m₁ and said second range of values is characterized by a second constant parameter m₂ where 0<

m₁<

m₂, and wherein m₁, m₂, and length of global exposure time are chosen so to achieve a desirable balance between highest possible signal-to-noise ratio and least possible motion blur.

18. The method of claim 17, wherein step c) further comprises:

c1) indicating no motion/saturation has occurred and updating said illumination estimation with said illumination measurement, if said difference falls within said first range of values; and

c2) indicating motion/saturation has occurred and,

1) if said difference falls outside said second range of values, terminating said performing act and using said illumination estimation for generating an optimal illumination estimation for said sensor; and

2) if said difference falls between said first range of values and said second range of values, deferring terminating said performing act and updating said illumination estimation with said illumination measurement.

19. A method for synthesizing from multiple captures high dynamic range motion blur free images, said method comprising the steps of:

a) capturing a first image sample;

b) generating for each pixel a current illumination estimation based on said first captured image sample;

c) capturing a next image sample;

d) determining for each pixel whether motion/saturation has occurred and whether to include said next image sample;

wherein said step d) further comprises;

d1) if motion/saturation has occurred, using said current illumination as final illumination estimation; and

d2) if no motion/saturation has occurred or a decision is deferred, including said next image sample and updating said current illumination;

e) repeat steps c) and d) until no more image samples are to be captured.

20. A system having a sensor capable of capturing a multiplicity of image samples during a global exposure time, comprising:

motion/saturation detecting means for determining for each pixel whether motion/saturation has occurred between a previous capturing and a current capturing;

processing means for determining for each pixel whether to accept an image sample captured during said current capturing;

estimating means for generating an optimal illumination estimation for said sensor based on selectively accepted multiplicity of image samples captured during said global exposure time thereby Preventing motion/saturation from corrupting image capturing;

means for determining for each pixel a difference between an illumination measurement obtained during said current capturing and an illumination estimation generated during said previous capturing;

means for comparing for each pixel said difference with a threshold value;

means for updating for each pixel said illumination estimation with an accepted or deferred illumination measurement; and

means for outputing for each pixel a final illumination estimation.

21. The system of claim 20, wherein said threshold value is generated based on a predetermined parameter and a prediction variable, said predetermined parameter is configured to achieve desired tradeoff between signal-to-noise ratio and motion blur.

22. The system of claim 20, wherein said threshold value is characterized by a first range of values and a second range of values that include said first range of values, wherein said first range of values is characterized by a first constant parameter mi and said second range of values is characterized by a second constant parameter m₂ where 0>

m₁>

m₂, and wherein m₁, m₂, and length of global exposure time are chosen so to achieve a desirable balance between highest possible signal-to-noise ratio and least possible motion blur.

23. The system of claim 22, further comprising:

a soft decision means for preventing error accumulation due to slow motion, said soft decision means

indicating no motion/saturation has occurred and updating said illumination estimation with said illumination measurement, if said difference falls within said first range of values;

indicating motion/saturation has occurred and outputting said illumination estimation as said final illumination estimation, if said difference falls outside said second range of values; and

indicating motion/saturation has occurred and said illumination measurement is deferred, and updating said illumination estimation with said deferred illumination measurement, if said difference falls between said first range of values and said second range of values.

24. The system of claim 20, wherein each pixel'"'"'s effective exposure time is adaptive to its own lighting condition thereby enabling performing for each pixel independently of other pixels'"'"' lighting conditions.

25. The system of claim 20, wherein each pixel is capable of terminating its own exposure time thereby enabling extending said global exposure time.

26. The system of claim 25, wherein said global exposure time is limited by motion and saturation only thereby enabling said sensor to achieve higher signal-to-noise ratio and dynamic range.

27. The system of claim 20, wherein said motion/saturation detecting means utilizes parameters calculated recursively.

28. The system of claim 20, wherein said motion/saturation detecting means utilizes parameters calculated non-recursively.

29. The system of claim 20, wherein said estimating means is characterized as recursive.

30. The system of claim 20, wherein said estimating means is characterized as non-recursive.

31. The system of claim 20, wherein said threshold value is generated based on a predetermination parameter and a prediction variable, said predetermined parameter being configured to achieve desired tradeoff between signal-to-noise ratio and motion blur.

32. The system of claim 20, wherein said estimating means is configured based on linear minimum mean square error.

33. The system of claim 20, wherein said motion/saturation detecting means and said estimating means are implemented based on a self-reset pixel architecture.

34. The system of claim 33, wherein said self-reset pixel architecture utilizes self-reset digital pixel sensors.

35. The system of claim 20, wherein said system is implemented on a single chip.

36. The system of claim 20, wherein said sensor is a digital pixel sensor.

37. The system of claim 20, wherein said sensor is a photodiode and said illumination measurement represents a charge accumulated from photocurrent produced by said photodiode.