WDR IMAGING WITH LED FLICKER MITIGATION

• US 20180109715A1
• Filed: 10/18/2016
• Published: 04/19/2018
• Est. Priority Date: 10/18/2016
• Status: Active Grant

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: MANGLA, MAYANK, DABRAL, SHASHANK, LABAZIEWICZ, PETER
• Granted Patent: US 10,992,876 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/745 Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination

Abstract

A method of WDR imaging. Exposure times (ETs) are set for first and second frames for an image sensor to avoid second frame saturating by setting a second ET>1/a PWM frequency applied to an LED illuminating a scene to generate second longer ET pixel data (PD). The first frame has first PD and a first ET<the second ET. A high and low intensity threshold are calculated from a full well capacity. Raw image signals are obtained originating from the image sensor of the scene. Flicker is detected by comparing the first and second frame intensity values to the high and low threshold to determine whether the first PD is flickering data. A WDR merge is performed by selecting weightings from the first and second PD for each pixel including increasing weighting of the second PD for flicker. A final image is formed from the weighted WDR merge.

Claims

1. A method of wide dynamic range (WDR) imaging, comprising:

setting exposure times for a plurality of frames including a first frame and at least a second frame for an image sensor which includes a plurality of pixels to avoid saturating at least said second frame by setting a second exposure time greater than (>

) 1/a pulse-width modulation (PWM) frequency applied to a light emitting diode (LED) that illuminates a scene to generate second longer exposure time pixel data (second pixel data), wherein said first frame has a first exposure time<

said second exposure time generating first shorter exposure time pixel data (first pixel data);

computing a high intensity threshold from a full well capacity (FWC) of said image sensor and a low intensity threshold from said FWC divided by an intensity ratio;

obtaining raw image signals originating from said image sensor generated from said scene, wherein said second frame has second intensity values and said first frame has first intensity values for each of said plurality of pixels;

detecting whether flicker is present in each of said plurality of pixels based on comparing said first and said second intensity values to said high intensity threshold and to said low intensity threshold to determine whether said first pixel data is flickering data;

performing a WDR merge using a WDR merge algorithm by selecting weightings from said first and said second pixel data for each of said plurality of pixels including increasing a weighting of said second pixel data in a case of said flickering data, and

forming a final image from said weightings after said WDR merge.

2. The method of claim 1, wherein said increasing said weighting of said second pixel data comprises ignoring said first pixel data.

3. The method of claim 1, wherein said plurality of frames further comprises a third frame providing a third exposure time generating third intermediate exposure time pixel data, wherein said third frame has third intensity values.

4. The method of claim 1, wherein said image sensor comprises a complementary metal oxide semiconductor (CMOS) WDR sensor.

5. The method of claim 1, further comprising performing tone mapping after said WDR merge to provide an output toned mapped image.

6. The method of claim 1, wherein said computing, said obtaining, said detecting said flicker, said performing the WDR merge and said forming said final image are all real-time processed.

7. The method of claim 1, wherein said method is used in an Advanced Driver Assistance System (ADAS) system, said ADAS system comprising:

an image recognition system coupled to receive said raw image signals from said image sensor including a video recognition processor and a transceiver, and

a processor block including a processor core coupled to said image recognition system, said processor core coupled to utilize at least one memory having a plurality of algorithms including a flicker mitigation algorithm for implementing said method.

8. The method of claim 1, wherein said LED comprises a plurality of said LEDs in said scene emitting at different said PWM frequencies.

9. A signal processing chain for wide dynamic range (WDR) multi-frame processing, comprising:

a pre-processing including statistics generation block for receiving pixel data from a plurality of frames;

a WDR merge block having a flicker mitigation algorithm, said flicker mitigation algorithm for;

setting exposure times for said plurality of frames including a first frame and at least a second frame for an image sensor which includes a plurality of pixels to avoid saturating at least said second frame by setting a second exposure time greater than (>

) 1/a pulse-width modulation (PWM) frequency applied to a light emitting diode (LED) that illuminates a scene to generate second longer exposure time pixel data (second pixel data), wherein said first frame has a first exposure time<

said second exposure time generating first shorter exposure time pixel data (first pixel data);

computing a high intensity threshold from a full well capacity (FWC) of said image sensor and a low intensity threshold from said FWC divided by an intensity ratio;

obtaining raw image signals originating from said image sensor generated from said scene, wherein said second frame has second intensity values and said first frame has first intensity values for each of said plurality of pixels;

detecting whether flicker is present in each of said plurality of pixels based on comparing said first and said second intensity values to said high intensity threshold and to said low intensity threshold to determine whether said first pixel data is flickering data;

performing a WDR merge using a WDR merge algorithm by selecting weightings from said first and said second pixel data for each of said plurality of pixels including increasing a weighting of said second pixel data in a case of said flickering data, and

forming a final image from said weightings after said WDR merge.

10. The signal processing chain of claim 9, wherein said increasing said weighting of said second pixel data comprises ignoring said first pixel data.

11. The signal processing chain of claim 9, wherein said plurality of frames further comprises a third frame providing a third exposure time generating third intermediate exposure time pixel data, wherein said third frame has third intensity values.

12. The signal processing chain of claim 9, further comprising a tone mapping block for performing tone mapping after said WDR merge to provide an output toned mapped image.

13. The signal processing chain of claim 9, wherein said computing, said obtaining, said detecting said flicker, said performing the WDR merge and said forming said final image are all real-time processed.

14. An Advanced Driver Assistance System (ADAS) system, comprising:

an image sensor for generating image data from a scene;

an image recognition system coupled to receive said image data from said image sensor including a video recognition processor and a transceiver;

a processor block including a processor core coupled to said image recognition system, said processor core coupled to utilize at least one memory having a plurality of including a flicker mitigation algorithm for implementing a method of wide dynamic range (WDR) imaging, comprising;

setting exposure times for a plurality of frames including a first frame and at least a second frame for said image sensor which includes a plurality of pixels to avoid saturating at least said second frame by setting a second exposure time greater than (>

) 1/a pulse-width modulation (PWM) frequency applied to a light emitting diode (LED) that illuminates said scene to generate second longer exposure time pixel data (second pixel data), wherein said first frame has a first exposure time<

said second exposure time generating first shorter exposure time pixel data (first pixel data);

computing a high intensity threshold from a full well capacity (FWC) of said image sensor and a low intensity threshold from said FWC divided by an intensity ratio;

obtaining raw image signals originating from said image sensor generated from said scene, wherein said second frame has second intensity values and said first frame has first intensity values for each of said plurality of pixels;

detecting whether flicker is present in each of said plurality of pixels based on comparing said first and said second intensity values to said high intensity threshold and to said low intensity threshold to determine whether said first pixel data is flickering data;

performing a WDR merge using a WDR merge algorithm by selecting weightings from said first and said second pixel data for each of said plurality of pixels including increasing a weighting of said second pixel data in a case of said flickering data, and

forming a final image from said weightings after said WDR merge.

15. The ADAS system of claim 14, wherein said increasing said weighting of said second pixel data comprises ignoring said first pixel data.

16. The ADAS system of claim 14, wherein said plurality of frames further comprises a third frame providing a third exposure time generating third intermediate exposure time pixel data, wherein said third frame has third intensity values.

17. The ADAS system of claim 14, further comprising a tone mapping block for performing tone mapping after said WDR merge to provide an output toned mapped image.

18. The ADAS system of claim 14, wherein said computing, said obtaining, said detecting said flicker, said performing the WDR merge and said forming said final image are all real-time processed.

19. The ADAS system of claim 14, wherein said image sensor comprises a complementary metal oxide semiconductor (CMOS) WDR sensor.