Tone mapping for low-light video frame enhancement

• US 9,307,212 B2
• Filed: 07/27/2015
• Issued: 04/05/2016
• Est. Priority Date: 03/05/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Vranceanu, Ruxandra
• Primary Examiner(s): MOE, AUNG SOE
• Application Number:
  US14/810,297
• Publication Number:
  US 20150334283A1
• Time in Patent Office: 253 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20008 Globally adaptive
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 23/86 for controlling the colour saturation of colour signals, e.g. automatic chroma control circuits
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
• CPC Class Codes: H04N 9/68 for controlling the amplitude of colour signals, e.g. automatic chroma control circuits H04N9/71, H04N9/73 take precedence; camera processing pipelines for controlling the colour saturation of colour signals H04N23/86

Abstract

A technique is provided for generating sharp, well-exposed, color images front low-light images. A series of under-exposed images is acquired. A mean image is computed and a sum image is generated each based on the series of under-exposed images. Chrominance variables of pixels of the mean image are mapped to chrominance variables of pixels of the sum image. Chrominance values of pixels within the series of under-exposed images are replaced with chrominance values of the sum image. A set of sharp, well-exposed, color images is generated based on the series of under-exposed images with replaced chrominance values.

Claims

1. An image processing method, comprising:

acquiring a first well-exposed image frame of a scene;

acquiring additional image frames of the same scene;

wherein the additional image frames include a second image frame and a third image frame;

generating one or more combined images based on the well-exposed image frame of the scene and at least one of the additional image frames;

determining a frame-to-frame motion between the second image frame and the third image frame; and

responsive to said frame-to-frame motion exceeding a threshold, acquiring a second well-exposed image frame of the scene.

2. The image processing method of claim 1, further comprising:

after acquiring the second well-exposed image frame, acquiring second additional image frames of the scene;

generating one or more combined images based on the second well-exposed image frame and at least one of the second additional image frames; and

acquiring a third well-exposed image frame responsive to the number of second additional image frames exceeding a threshold.

3. The image processing method of claim 1,

wherein the frame-to-frame motion is determined from a registration of the additional image frames.

4. The image processing method of claim 1,

wherein the additional image frames are frames that are exposed using an exposure determined from a maximum frame-rate.

5. The image processing method of claim 1,

wherein the one or more combined images are generated based on the well-exposed image frame of the scene and the additional image frames of the same scene until one or more conditions are satisfied, wherein the one or more conditions include:

the frame-to-frame motion between two particular additional image frames, from the additional image frames, exceeds the threshold, or

a certain number of frames, from the additional image frames, have been acquired.

6. The image processing method of claim 1,

wherein generating one or more combined images includes generating a first combined image by:

extracting brightness and/or color information from the well-exposed image frame; and

using the brightness and/or color information from the well-exposed image frame to enhance the second image frame.

7. The image processing method of claim 1,

wherein generating one or more combined images includes generating a second combined image by:

extracting brightness and/or color information from the first combined image; and

using the brightness and/or color information from the first combined image to enhance the third image frame.

8. The image processing method of claim 1,

wherein an exposure time for capturing the additional image frames is determined using an implementation of an auto-exposure algorithm executed when a camera is set in a video mode.

9. The image processing method of claim 8,

wherein the camera is configured to acquire the first well-exposed image frame using an auto-exposure set for a time period that is larger than one imposed by a maximum frame rate.

10. An image acquisition and processing device, comprising:

an image acquisition unit configured to;

acquiring a first well-exposed image frame of a scene;

acquiring additional image frames of the same scene;

wherein the additional image frames include a second image frame and a third image frame;

an image combining unit configured to;

generating one or more combined images based on the well-exposed image frame of the scene and at least one of the additional image frames;

determining a frame-to-frame motion between the second image frame and the third image frame; and

responsive to said frame-to-frame motion exceeding a threshold, acquiring a second well-exposed image frame of the scene.

11. The image acquisition and processing device of claim 10, wherein the image combining unit is further configured to:

after acquiring the second well-exposed image frame, acquiring second additional image frames of the scene;

generating one or more combined images based on the second well-exposed image frame and at least one of the second additional image frames; and

acquiring a third well-exposed image frame responsive to the number of second additional image frames exceeding a threshold.

12. The image acquisition and processing device of claim 10,

wherein the frame-to-frame motion is determined from a registration of the additional image frames.

13. The image acquisition and processing device of claim 10,

wherein the additional image frames are frames that are exposed using an exposure determined from a maximum frame-rate.

14. The image acquisition and processing device of claim 10,

wherein the one or more combined images are generated based on the well-exposed image frame of the scene and the additional image frames of the same scene until one or more conditions are satisfied, wherein the one or more conditions include:

the frame-to-frame motion between two particular additional image frames, from the additional image frames, exceeds the threshold, or

a certain number of frames, from the additional image frames, has been recorded.

15. The image acquisition and processing device of claim 10,

wherein the one or more combined images are generated by extracting brightness and/or color information from the well-exposed image frame to enhance the at least one of the additional image frames.

16. The image acquisition and processing device of claim 10,

wherein an exposure time for capturing the additional image frames is determined using an implementation of an auto-exposure algorithm executed when a camera is set in a video mode.

17. The image acquisition and processing device of claim 16,

wherein the camera is configured to acquire the first well-exposed image frame using an auto-exposure set for a time period that is larger than one imposed by a maximum frame rate.

18. A non-transitory computer readable storage medium storing one or more instructions which, when executed by a processor, cause the processor to perform:

acquiring a first well-exposed image frame of a scene;

acquiring additional image frames of the same scene;

wherein the additional image frames include a second image frame and a third image frame;

generating one or more combined images based on the well-exposed image frame of the scene and at least one of the additional image frames;

determining a frame-to-frame motion between the second image frame and the third image frame; and

responsive to said frame-to-frame motion exceeding a threshold, acquiring a second well-exposed image frame of the scene.

19. The non-transitory computer readable storage medium of claim 18,

wherein the instructions include instructions for:

after acquiring the second well-exposed image frame, acquiring second additional image frames of the scene;

generating one or more combined images based on the second well-exposed image frame and at least one of the second additional image frames; and

acquiring a third well-exposed image frame responsive to the number of second additional image frames exceeding a threshold.

20. The non-transitory computer readable storage medium of claim 18,

wherein the frame-to-frame motion is determined from a registration of the additional image frames.

21. The non-transitory computer readable storage medium of claim 18,

wherein the additional image frames are frames that are exposed using an exposure determined from a maximum frame-rate.

22. The non-transitory computer readable storage medium of claim 18,

wherein the one or more combined images are generated based on the well-exposed image frame of the scene and the additional image frames of the same scene until one or more conditions are satisfied:

the frame-to-frame motion between two particular additional image frames, from the additional image frames, exceeds the threshold, or

a certain number of frames, from the additional image frames, has been recorded.

23. The non-transitory computer readable storage medium of claim 18,

wherein the one or more combined images are generated by extracting brightness and/or color information from the well-exposed image frame to enhance the at least one of the additional image frames.

24. The non-transitory computer readable storage medium of claim 18,

wherein an exposure time for capturing the additional image frames is determined using an implementation of an auto-exposure algorithm executed when a camera is set in a video mode.

25. The non-transitory computer readable storage medium of claim 24,

wherein the camera is configured to acquire the first well-exposed image frame using an auto-exposure set for a time period that is larger than one imposed by a maximum frame rate.