Multi-frame range gating for lighting-invariant depth maps for in-motion applications and attenuating environments

• US 10,873,738 B2
• Filed: 08/09/2019
• Issued: 12/22/2020
• Est. Priority Date: 03/03/2016
• Status: Active Grant

• Current Assignee: 4D Intellectual Properties, LLC
• Original Assignee: 4D Intellectual Properties, LLC
• Inventors: Retterath, James E.
• Primary Examiner(s): Habib, Irfan
• Application Number:
  US16/537,305
• Publication Number:
  US 20200036958A1
• Time in Patent Office: 501 Days
• CPC Class Codes: G01S 17/18 wherein range gates are used
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4863 Detector arrays, e.g. charge-transfer gates
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/005 Aspects relating to the "3D+depth" image format
• CPC Class Codes: H04N 23/671 in combination with active ranging signals, e.g. using light or sound signals emitted toward objects

Abstract

Range gated imaging systems utilize short light pulses and synchronized detector cycles to image objects that are contained within a distance range. Single-image range gating allows depth estimation for static scenes, but this technique is not well-suited for in-motion applications or attenuating environments. In various embodiments, a multi-frame range gating system uses a series of range gate frames that are produced within a brief timing window, whereby each range-gate frame illuminates a different lighting-invariant depth range within the field of view. Multi-frame range gate cycles of as little as three frames are sufficient to produce precise lighting-invariant depth information for each pixel in a sensor for in-motion applications. Lighting-invariant depth information is produced with multi-frame range gate cycles of as little as four frames for in-motion applications for attenuating environments like fog, rain, dust, smoke, smog, and underwater.

Claims

1. An active image sensor system configured to generate a lighting-invariant depth map of a scene comprising:

at least one emitter configured to emit a set of light pulses toward the scene;

an array of detectors configured to receive light for a field of view that includes at least a portion of the scene;

control circuitry operably coupled to the at least one emitter and the array of detectors and configured to cause the at least one emitter to begin to emit the set of light pulses at a first time and to cause the array of detectors to begin to receive light for the field of view and store in one or more frame buffers at least three successive frames of data as a multi-frame representation of the field of view during an emitter/detector cycle that includes at least a second time, a third time and a fourth time after the first time, wherein the control circuitry is configured to vary an elapsed time during the emitter/detector cycle between the first time and the second time, between the first time and the third time and between the first time and the fourth time; and

a processing system operably coupled to the control circuitry and the one or more frame buffers, the processing system configured to generate the lighting-invariant depth map by analyzing the frame buffers for each of a plurality of the emitter/detector cycles to process the at least three successive frames of data for each emitter/detector cycle to determine intensity values and to generate an additional frame of data as a two-dimensional representation of distance values to objects in the field of view by performing the following for each detector in the detector array;

analyze the at least three successive frames of data for the detector and determine a minimum non-zero intensity value due to ambient light, a maximum intensity value, and a frame of the at least three successive frames at which the maximum intensity value occurs; and

determine a distance to an object based on the frame at which the maximum intensity value occurs.

2. The active image sensor system of claim 1, wherein each detector in the array of detectors is configured to produce a linearized response to a number of incident photons of light that includes ambient light.

3. The active image sensor system of claim 1, wherein the emitter/detector cycle is less than 50 microseconds.

4. The active image sensor system of claim 1, wherein the image sensor system is in motion relative to the scene.

5. The active image sensor system of claim 1, wherein the distance is determined based on a calculation of a time of flight.

6. The active image sensor system of claim 1, wherein the set of light pulses is a single light pulse.

7. The active image sensor system of claim 1, wherein the at least one emitter comprises one or more active light sources selected from the set consisting of an LED, a laser, a laser diode, and a vehicle headlamp.

8. The system of claim 1, wherein the light pulses comprise electromagnetic energy selected from the wavelength ranges of:

ultraviolet (UV)—

100-400 nm, visible—

400-700 nm, near infrared (NIR)—

700-1400 nm, infrared (IR)—

1400-8000 nm, long-wavelength IR (LWIR)—

8 um-15 um, or far IR (FIR)—

15 um-1000 um.

9. The active image sensor system of claim 1, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector and the control circuitry is configured to capture at least one color image frame corresponding to each of the plurality of multi-color filter elements.

10. The active image sensor of claim 1, wherein the intensity values for each of the detectors in the one or more frame buffers represent pixels.

11. An active image sensor system configured to generate a lighting-invariant depth map of a scene within an attenuating environment comprising:

at least one emitter configured to emit a set of light pulses toward the scene;

an array of detectors configured to receive light for a field of view that includes at least a portion of the scene, wherein the light includes a varying amount of light due to ambient light and to light pulse reflections from the attenuating environment;

control circuitry operably coupled to the at least one emitter and the array of detectors and configured to cause the at least one emitter to begin to emit the set of light pulses at a first time and to cause the array of detectors to begin to receive light for the field of view and store in one or more frame buffers at least five successive frames of data as a multi-frame representation of the field of view during an emitter/detector cycle that includes at a second time, a third time, a fourth time, a fifth time and a sixth time after the first time, wherein the control circuitry is configured to vary an elapsed time during the emitter/detector cycle between the first time and the second time, between the first time and the third time, between the first time and the fourth time, between the first time and the fifth time and between the first time and the sixth time to generate the at least five successive frames of data; and

a processing system operably coupled to the control circuitry and the one or more frame buffers to generate the lighting-invariant depth map by analyzing the frame buffers for each of a plurality of the emitter/detector cycles to process the at least five successive frames of data for each emitter/detector cycle to determine intensity values and to generate an additional frame of data as a two-dimensional representation of distance values to objects in the field of view by performing the following for each detector in the detector array;

analyze the at least five successive frames of data for the detector and determine a minimum intensity value due to ambient light, a maximum intensity value due to light reflected from the attenuating environment, and a frame of the at least five successive frames at which the minimum intensity value occurs; and

determine a distance to an object based on the frame at which the minimum intensity value occurs.

12. The active image sensor system of claim 11, wherein the attenuating environment is selected from the set that includes one or more of fog, rain, dust, smoke, smog, and underwater.

13. The active image sensor system of claim 11, wherein each detector in the array of detectors is configured to produce a linearized response to a number of incident photons of light that includes ambient light.

14. The active image sensor system of claim 11, wherein the emitter/detector cycle is less than 50 microseconds.

15. The active image sensor system of claim 11, wherein the image sensor system is in motion relative to the scene.

16. The active image sensor system of claim 11, wherein the distance is determined based on a calculation of a time of flight.

17. The active image sensor system of claim 11, wherein the set of light pulses is a single light pulse.

18. The active image sensor system of claim 11, wherein the at least one emitter comprises one or more active light sources selected from the set consisting of an LED, a laser, a laser diode, and a vehicle headlamp.

19. The system of claim 11, wherein the light pulses comprise electromagnetic energy selected from the wavelength ranges of:

ultraviolet (UV)—

100-400 nm, visible—

400-700 nm, near infrared (NIR)—

700-1400 nm, infrared (IR)—

1400-8000 nm, long-wavelength IR (LWIR)—

8 um-15 um, or far IR (FIR)—

15 um-1000 um.

20. The active image sensor system of claim 11, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector and the control circuitry is configured to capture at least one color image frame corresponding to each of the plurality of multi-color filter elements.

21. The active image sensor of claim 11, wherein the intensity values for each of the detectors in the one or more frame buffers represent pixels.

22. An active image sensor system configured to generate a depth map of a scene within an attenuating environment comprising:

at least one emitter configured to emit a set of light pulses toward the scene;

an array of detectors configured to receive light for a field of view that includes at least a portion of the scene, wherein the light includes a varying amount of light due to ambient light and to light pulse reflections from the attenuating environment;

control circuitry operably coupled to the at least one emitter and the array of detectors and configured to cause the at least one emitter to begin to emit the set of light pulses at a first time and to cause the array of detectors to begin to receive light for the field of view and store in one or more frame buffers at least four successive frames of data as a multi-frame representation of the field of view during an emitter/detector cycle that includes at a second time, a third time, a fourth time and a fifth time after the first time, wherein the control circuitry is configured to vary an elapsed time during the emitter/detector cycle between the first time second time, between the first time and the third time, between the first time and the fourth time, and between the first time and the fifth time to generate the at least four successive frames of data; and

a processing system operably coupled to the control circuitry and the one or more frame buffers to generate the depth map by analyzing the frame buffers for each of a plurality of the emitter/detector cycles to process the at least five successive frames of data for each emitter/detector cycle to determine intensity values and to generate an additional frame of data as a two-dimensional representation of distance values to objects in the field of view by performing the following for each detector in the detector array;

analyze the at least four successive of data for the detector and determine a minimum intensity value due to ambient light, a maximum intensity value due to light reflected from the attenuating environment, and a frame of the at least four successive frames, exclusive of a first frame of the at least four successive frames, at which the minimum intensity value occurs; and

determine a distance to an object based on the frame at which the minimum intensity value occurs.

23. The active image sensor system of claim 22, wherein the attenuating environment is selected from the set that includes one or more of fog, rain, dust, smoke, smog, and underwater.

24. The active image sensor system of claim 22, wherein each detector in the array of detectors is configured to produce a linearized response to a number of incident photons of light that includes ambient light.

25. The active image sensor system of claim 22, wherein the emitter/detector cycle is less than 50 microseconds.

26. The active image sensor system of claim 22, wherein the image sensor system is in motion relative to the scene.

27. The active image sensor system of claim 22, wherein the distance is determined based on a calculation of a time of flight.

28. The active image sensor system of claim 22, wherein the set of light pulses is a single light pulse.

29. The active image sensor system of claim 22, wherein the at least one emitter comprises one or more active light sources selected from the set consisting of an LED, a laser, a laser diode, and a vehicle headlamp.

30. The system of claim 22, wherein the light pulses comprise electromagnetic energy selected from the wavelength ranges of:

ultraviolet (UV)—

100-400 nm, visible—

400-700 nm, near infrared (NIR)—

700-1400 nm, infrared (IR)—

1400-8000 nm, long-wavelength IR (LWIR)—

8 um-15 um, or far IR (FIR)—

15 um-1000 um.

31. The active image sensor system of claim 22, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector and the control circuitry is configured to capture at least one color image frame corresponding to each of the plurality of multi-color filter elements.

32. The active image sensor of claim 22, wherein the intensity values for each of the detectors in the one or more frame buffers represent pixels.