Low-light video frame enhancement

• US 8,199,222 B2
• Filed: 06/16/2009
• Issued: 06/12/2012
• Est. Priority Date: 03/05/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Drimbarean, Alexandru, Zamfir, Adrian, Albu, Felix, Poenaru, Vlad, Florea, Corneliu, Bigioi, Petronel, Steinberg, Eran, Corcoran, Peter
• Primary Examiner(s): WHIPKEY, JASON T
• Application Number:
  US12/485,316
• Publication Number:
  US 20090303343A1
• Time in Patent Office: 1,092 Days
• Field of Search: 348/218.1, 348/222.1, 348/239, 348/252, 348/262
• US Class Current: 348/239
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/84 for processing colour signals

Abstract

A method of combining image data from multiple frames to enhance one or more parameters of video image quality includes acquiring a first image at a first exposure duration, as well as acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that the first and second images include approximately a same first scene. In this way, the second image is relatively sharp and under-exposed, while the first image is relatively well-exposed and less sharp than the second image. Brightness and/or color information are extracted from the first image and applied to the second image to generate an enhanced version of the second image.

Claims

1. A method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising using a processor and at least one digital image acquisition system including a lens and an image sensor in:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is relatively well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

displaying, storing, transmitting, or streaming the enhanced version of the second image;

acquiring a third image at or near said first exposure duration when a predetermined number of multiple frames has been recorded since the acquiring of the first image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately a same second scene different from or same as the first scene;

extracting brightness or color, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

7. The method of any of claims 1-3 or 6, further comprising utilizing separate image acquisition subsystems (IAS'"'"'s) for acquiring the first and second images.

8. The method of claim 7, wherein the separate IAS'"'"'s comprise first and second IAS'"'"'s, wherein the first IAS is configured to acquire images at a faster frame rate than the second IAS, such that the first IAS is specially configured to capture relatively sharp, under-exposed images and the second IAS is specially configured to capture well-exposed and less sharp images.

9. The method of claim 8, wherein the second IAS has a variable frame rate configured for capturing optimally-exposed images under different lighting conditions.

10. The method of claim 7, further comprising acquiring both the first and second images with a same lens as optical input, and splitting the optical input between first and second image sensors.

11. The method of claim 7, further comprising acquiring the first and second images, respectively, with a first lens/sensor combination and a second lens/sensor combination.

12. The method of claim 11, further comprising applying parallax correction.

13. The method of any of claims 1-3 or 6, wherein the first and second images comprise video images of a video stream.

2. A method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising using a processor and at least one digital image acquisition system including a lens and an image sensor in:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is relatively well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

displaying, storing, transmitting, or streaming the enhanced version of the second image;

acquiring a third image at or near said first exposure duration when a predetermined amount of camera movement has been detected since the acquiring of the first image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately a same second scene different from the first scene due at least to said camera movement;

extracting brightness or color, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

3. A method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising using a processor and at least one digital image acquisition system including a lens and an image sensor in:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is relatively well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

displaying, storing, transmitting, or streaming the enhanced version of the second image;

acquiring a third image at or near said first exposure duration including combining some of the data from the first image with some or all of the data from the second image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately said same first scene;

extracting brightness or color information, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

4. The method of claim 3, further comprising repeatedly iterating the method as the scene evolves with time or camera movement or both.

5. The method of claim 3, further comprising:

iteratively alternating acquisition of well-exposed and sharp, under-exposed images;

extracting brightness or color information, or both, from the well-exposed images; and

applying to the sharp, under-exposed images the brightness or color information, or both, extracted from the well-exposed images to generate enhanced versions of the sharp, under-exposed images enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced versions of the sharp, under-exposed images.

6. A method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising using a processor and at least one digital image acquisition system including a lens and an image sensor in:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is relatively well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

displaying, storing, transmitting, or streaming the enhanced version of the second image; and

wherein the method further comprising iteratively performing the following;

acquiring two sharp, under-exposed frames and a well-exposed frame;

extracting brightness or color information, or both, from the well-exposed frame;

applying to each pair of sharp, under exposed images respectively captured immediately before and immediately after the acquiring of each well-exposed frame the brightness or color information, or both, extracted from the well-exposed frame to generate enhanced versions of each of the pair of sharp, under-exposed images enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced versions of the pair of sharp, under-exposed images.

14. A method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising using a processor and at least one digital image acquisition system including a lens and an image sensor in:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is relatively well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

displaying, storing, transmitting, or streaming the enhanced version of the second image; and

wherein the method further comprising utilizing separate image acquisition subsystems (IAS'"'"'s) for acquiring the first and second images, and

wherein the separate IAS'"'"'s comprise first and second IAS'"'"'s, wherein the first IAS is configured to acquire images at a faster frame rate than the second IAS, such that the first IAS is specially configured to capture relatively sharp, under-exposed images and the second IAS is specially configured to capture well-exposed and less sharp images, and

wherein the first IAS has a fixed frame rate.

15. A digital image acquisition system, comprising a lens, an image sensor for capturing a stream of multiple digital images, a processor and a memory having processor-readable code embedded therein for programming the processor to perform a method of combining image data from multiple frames to enhance one or more parameters of digital image quality, wherein the method comprises:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

acquiring a third image at or near said first exposure duration when a predetermined number of multiple frames has been recorded since the acquiring of the first image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately a same second scene different from or same as the first scene;

extracting brightness or color, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

21. The system of any of claims 15-17 or 20, wherein the method further comprises separate image acquisition subsystems (IAS'"'"'s) for acquiring the first and second images.

22. The system of claim 21, wherein the separate IAS'"'"'s comprise first and second IAS'"'"'s, wherein the first IAS is configured to acquire images at a faster frame rate than the second IAS, such that the first IAS is specially configured to capture relatively sharp, under-exposed images and the second IAS is specially configured to capture well-exposed and less sharp images.

23. The system of claim 22, wherein the first IAS has a fixed frame rate.

24. The system of claim 22, wherein the second IAS has a variable frame rate configured for capturing optimally-exposed images under different lighting conditions.

25. The system of claim 21, wherein the method further comprises acquiring both the first and second images with a same lens as optical input, and splitting the optical input between first and second image sensors.

26. The system of claim 21, wherein the method further comprises acquiring the first and second images, respectively, with a first lens/sensor combination and a second lens/sensor combination.

27. The system of claim 26, wherein the method further comprises applying parallax correction.

28. The system of any of claims 15-17 or 20, wherein the first and second images comprise video images of a video stream.

16. A digital image acquisition system, comprising a lens, an image sensor for capturing a stream of multiple digital images, a processor and a memory having processor-readable code embedded therein for programming the processor to perform a method of combining image data from multiple frames to enhance one or more parameters of digital image quality, wherein the method comprises:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

acquiring a third image at or near said first exposure duration when a predetermined amount of camera movement has been detected since the acquiring of the first image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately a same second scene different from the first scene due at least to said camera movement;

extracting brightness or color, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

17. A digital image acquisition system, comprising a lens, an image sensor for capturing a stream of multiple digital images, a processor and a memory having processor-readable code embedded therein for programming the processor to perform a method of combining image data from multiple frames to enhance one or more parameters of digital image quality, wherein the method comprises:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

acquiring a third image at or near said first exposure duration including combining some of the data from the first image with some or all of the data from the second image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately said same first scene;

extracting brightness or color information, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

18. The system of claim 17, wherein the method further comprises repeatedly iterating the method as the scene evolves with time or camera movement or both.

19. The system of claim 17, wherein the method further comprises:

iteratively alternating acquisition of well-exposed and sharp, under-exposed images;

extracting brightness or color information, or both, from the well-exposed images; and

applying to the sharp, under-exposed images the brightness or color information, or both, extracted from the well-exposed images to generate enhanced versions of the sharp, under-exposed images enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced versions of the sharp, under-exposed images.

20. A digital image acquisition system, comprising a lens, an image sensor for capturing a stream of multiple digital images, a processor and a memory having processor-readable code embedded therein for programming the processor to perform a method of combining image data from multiple frames to enhance one or more parameters of digital image quality, wherein the method comprises:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both, and

wherein the method further comprises iteratively performing the following;

acquiring two sharp, under-exposed frames and a well-exposed frame;

extracting brightness or color information, or both, from the well-exposed frame;

applying to each pair of sharp, under exposed images respectively captured immediately before and immediately after the acquiring of each well-exposed frame the brightness or color information, or both, extracted from the well-exposed frame to generate enhanced versions of each of the pair of sharp, under-exposed images enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced versions of the pair of sharp, under-exposed images.

29. One or more non-transitory processor-readable media having processor-readable code embedded therein for programming one or more processors to perform a method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

acquiring a third image at or near said first exposure duration when a predetermined number of multiple frames has been recorded since the acquiring of the first image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately a same second scene different from or same as the first scene;

extracting brightness or color, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

35. The one or more processor-readable media of any of claims 29-31 or 34, wherein the method further comprises utilizing separate image acquisition subsystems (IAS'"'"'s) for acquiring the first and second images.

36. The one or more processor-readable media of claim 35, wherein the separate IAS'"'"'s comprise first and second IAS'"'"'s, wherein the first IAS is configured to acquire images at a faster frame rate than the second IAS, such that the first IAS is specially configured to capture relatively sharp, under-exposed images and the second IAS is specially configured to capture well-exposed and less sharp images.

37. The one or more processor-readable media of claim 36, wherein the first IAS has a fixed frame rate.

38. The one or more processor-readable media of claim 36, wherein the second IAS has a variable frame rate configured for capturing optimally-exposed images under different lighting conditions.

39. The one or more processor-readable media of claim 35, wherein the method further comprises acquiring both the first and second images with a same lens as optical input, and splitting the optical input between first and second image sensors.

40. The one or more processor-readable media of claim 35, wherein the method further comprises acquiring the first and second images, respectively, with a first lens/sensor combination and a second lens/sensor combination.

41. The one or more processor-readable media of claim 40, wherein the method further comprises applying parallax correction.

42. The one or more processor-readable media of any of claims 29-31 or 34, wherein the first and second images comprise video images of a video stream.

30. One or more non-transitory processor-readable media having processor-readable code embedded therein for programming one or more processors to perform a method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

acquiring a third image at or near said first exposure duration when a predetermined amount of camera movement has been detected since the acquiring of the first image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately a same second scene different from the first scene due at least to said camera movement;

extracting brightness or color, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

31. One or more non-transitory processor-readable media having processor-readable code embedded therein for programming one or more processors to perform a method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both;

acquiring a third image at or near said first exposure duration including combining some of the data from the first image with some or all of the data from the second image;

acquiring a fourth image at or near said second exposure duration and at a time just before, just after or overlapping in time with the third image, such that said third and fourth images comprise approximately said same first scene;

extracting brightness or color information, or both, from the third image;

applying to the fourth image the brightness or color information, or both, extracted from the third image to generate an enhanced version of the fourth image enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced version of the fourth image.

32. The one or more processor-readable media of claim 31, wherein the method further comprises repeatedly iterating the method as the scene evolves with time or camera movement or both.

33. The one or more processor-readable media of claim 31, wherein the method further comprises:

iteratively alternating acquisition of well-exposed and sharp, under-exposed images;

extracting brightness or color information, or both, from the well-exposed images; and

applying to the sharp, under-exposed images the brightness or color information, or both, extracted from the well-exposed images to generate enhanced versions of the sharp, under-exposed images enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced versions of the sharp, under-exposed images.

34. One or more non-transitory processor-readable media having processor-readable code embedded therein for programming one or more processors to perform a method of combining image data from multiple frames to enhance one or more parameters of digital image quality, comprising:

acquiring a first image at a first exposure duration;

acquiring a second image at a second exposure duration shorter than the first exposure duration and at a time just before, just after or overlapping in time with acquiring the first image, such that said first and second images comprise approximately a same first scene, and whereby the second image is relatively sharp and under-exposed and the first image is well-exposed and less sharp than the second image;

extracting brightness or color information, or both, from the first image;

applying to the second image the brightness or color information, or both, extracted from the first image to generate an enhanced version of the second image enhanced in terms of brightness or color quality, or both; and

wherein the method further comprises iteratively performing the following;

acquiring two sharp, under-exposed frames and a well-exposed frame;

extracting brightness or color information, or both, from the well-exposed frame;

applying to each pair of sharp, under exposed images respectively captured immediately before and immediately after the acquiring of each well-exposed frame the brightness or color information, or both, extracted from the well-exposed frame to generate enhanced versions of each of the pair of sharp, under-exposed images enhanced in terms of brightness or color quality, or both; and

displaying, storing, transmitting, or streaming the enhanced versions of the pair of sharp, under-exposed images.