Tone mapping for low-light video frame enhancement

• US 8,698,924 B2
• Filed: 11/08/2010
• Issued: 04/15/2014
• Est. Priority Date: 03/05/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Drimbarean, Alexandru, Zamfir, Adrian, Albu, Felix, Poenaru, Vlad, Florea, Corneliu, Bigioi, Petronel, Steinberg, Eran, Corcoran, Peter
• Primary Examiner(s): WHIPKEY, JASON T
• Application Number:
  US12/941,983
• Publication Number:
  US 20110205381A1
• Time in Patent Office: 1,254 Days
• Field of Search: 382/254, 348/222.1, 348/239, 348/241, 348/242, 348/252, 348/263, 348/254, 348/255, 348/256
• US Class Current: 348/253
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20008 Globally adaptive
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/86 for controlling the colour saturation of colour signals, e.g. automatic chroma control circuits
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
• CPC Class Codes: H04N 9/68 for controlling the amplitude of colour signals, e.g. automatic chroma control circuits H04N9/71, H04N9/73 take precedence; camera processing pipelines for controlling the colour saturation of colour signals H04N23/86

Abstract

A technique is provided for generating sharp, well-exposed, color images from low-light images. A series of under-exposed images is acquired. A mean image is computed and a sum image is generated each based on the series of under-exposed images. Chrominance variables of pixels of the mean image are mapped to chrominance variables of pixels of the sum image. Chrominance values of pixels within the series of under-exposed images are replaced with chrominance values of the sum image. A set of sharp, well-exposed, color images is generated based on the series of under-exposed images with replaced chrominance values.

Claims

1. A method of generating sharp, well-exposed, color images from low-light images, comprising:

acquiring a series of under-exposed images;

computing a mean image based on an average of multiple images of the series of under-exposed images;

generating a sum image based on a sum of multiple images of the series of under-exposed images;

mapping chrominance variables of pixels of the mean image to chrominance variables of pixels of the sum image;

replacing chrominance values of pixels within the series of under-exposed images with chrominance values of the sum image; and

generating a set of sharp, well-exposed, color images based on the series of under-exposed images with replaced chrominance values.

2. The method of claim 1, wherein the acquiring of said series of under-exposed images comprises capturing raw images and generating processed RGB images based on the raw images.

3. The method of claim 2, wherein the computing of said mean image comprises applying a tone-mapping transformation to the series of under-exposed images.

4. The method of claim 3, wherein the tone-mapping transformation comprises a logarithmic transformation.

5. The method of claim 2, wherein the acquiring of said series of under-exposed images further comprises applying a tone-mapping transformation to the series of under-exposed images.

6. The method of claim 5, wherein the tone-mapping transformation comprises a logarithmic transformation.

7. The method of claim 1, wherein the acquiring of said series of under-exposed images comprises applying a tone-mapping transformation to the series of under-exposed images.

8. The method of claim 7, wherein the tone-mapping transformation comprises a logarithmic transformation.

9. The method of claim 1, further comprising transforming the mean and sum images to YUV space prior to mapping and replacing said chrominance values.

10. The method of claim 9, further comprising transforming the under-exposed images with replaced chrominance values back to RGB space.

11. The method of claim 1, wherein the set of sharp, well-exposed, color images comprises multiple video images.

12. The method of claim 1, wherein the set of sharp, well-exposed, color images comprise one or more still images.

13. A digital video acquisition device, comprising:

a lens and an image sensor for capturing a stream of multiple digital video or still images;

a processor; and

a memory having processor-readable code embedded therein for programming the processor to perform a method of generating sharp, well-exposed, color images from under-exposed, low-light images, wherein the method comprises;

acquiring a series of under-exposed images;

computing a mean image based on an average of multiple images of the series of under-exposed images;

generating a sum image based on a sum of multiple images of the series of under-exposed images;

mapping chrominance variables of pixels of the mean image to chrominance variables of pixels of the sum image;

replacing chrominance values of pixels within the series of under-exposed images with chrominance values of the sum image; and

generating a set of sharp, well-exposed, color images based on the series of under-exposed images with replaced chrominance values.

14. The device of claim 13, wherein the acquiring of said series of under-exposed images comprises capturing raw images and generating processed RGB images based on the raw images.

15. The device of claim 14, wherein the computing of said mean image comprises applying a tone-mapping transformation to the series of under-exposed images.

16. The device of claim 15, wherein the tone-mapping transformation comprises a logarithmic transformation.

17. The device of claim 14, wherein the acquiring of said series of under-exposed images further comprises applying a tone-mapping transformation to the series of under-exposed images.

18. The device of claim 17, wherein the tone-mapping transformation comprises a logarithmic transformation.

19. The device of claim 13, wherein the acquiring of said series of under-exposed images comprises applying a tone-mapping transformation to the series of under-exposed images.

20. The device of claim 19, wherein the tone-mapping transformation comprises a logarithmic transformation.

21. The device of claim 13, wherein the method further comprises transforming the mean and sum images to YUV space prior to mapping and replacing said chrominance values.

22. The device of claim 21, wherein the method further comprises transforming the under-exposed images with replaced chrominance values back to RGB space.

23. The device of claim 13, wherein the set of sharp, well-exposed, color images comprises multiple video images.

24. The device of claim 13, wherein the set of sharp, well-exposed, color images comprise one or more still images.

25. One or more non-transitory processor-readable media having processor-readable code embedded therein for programming one or more processors to perform a method of generating sharp, well-exposed, color images from under-exposed, low-light images, wherein the method comprises:

acquiring a series of under-exposed images;

computing a mean image based on an average of multiple images of the series of under-exposed images;

generating a sum image based on a sum of multiple images of the series of under-exposed images;

mapping chrominance variables of pixels of the mean image to chrominance variables of pixels of the sum image;

replacing chrominance values of pixels within the series of under-exposed images with chrominance values of the sum image; and

generating a set of sharp, well-exposed, color images based on the series of under-exposed images with replaced chrominance values.

26. The one or more non-transitory processor-readable media of claim 25, wherein the acquiring of said series of under-exposed images comprises capturing raw images and generating processed RGB images based on the raw images.

27. The one or more non-transitory processor-readable media of claim 26, wherein the computing of said mean image comprises applying a tone-mapping transformation to the series of under-exposed images.

28. The one or more non-transitory processor-readable media of claim 27, wherein the tone-mapping transformation comprises a logarithmic transformation.

29. The one or more non-transitory processor-readable media of claim 26, wherein the acquiring of said series of under-exposed images further comprises applying a tone-mapping transformation to the series of under-exposed images.

30. The one or more non-transitory processor-readable media of claim 29, wherein the tone-mapping transformation comprises a logarithmic transformation.

31. The one or more non-transitory processor-readable media of claim 25, wherein the acquiring of said series of under-exposed images comprises applying a tone-mapping transformation to the series of under-exposed images.

32. The one or more non-transitory processor-readable media of claim 31, wherein the tone-mapping transformation comprises a logarithmic transformation.

33. The one or more non-transitory processor-readable media of claim 25, wherein the method further comprises transforming the mean and sum images to YUV space prior to mapping and replacing said chrominance values.

34. The one or more non-transitory processor-readable media of claim 33, wherein the method further comprises transforming the under-exposed images with replaced chrominance values back to RGB space.

35. The one or more non-transitory processor-readable media of claim 25, wherein the set of sharp, well-exposed, color images comprises multiple video images.

36. The one or more non-transitory processor-readable media of claim 25, wherein the set of sharp, well-exposed, color images comprise one or more still images.