Methods and apparatus for a lighting-invariant image sensor for automated object detection and vision systems

• US 10,382,742 B2
• Filed: 10/22/2018
• Issued: 08/13/2019
• Est. Priority Date: 03/03/2016
• Status: Active Grant

• Current Assignee: 4D Intellectual Properties, LLC
• Original Assignee: 4D Intellectual Properties, LLC
• Inventors: Retterath, James E.
• Primary Examiner(s): Patel, Jayanti K
• Assistant Examiner(s): Habib, Irfan
• Application Number:
  US16/167,196
• Publication Number:
  US 20190058867A1
• Time in Patent Office: 295 Days
• CPC Class Codes: G01S 17/18 wherein range gates are used
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4863 Detector arrays, e.g. charge-transfer gates
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/005 Aspects relating to the "3D+depth" image format
• CPC Class Codes: H04N 23/671 in combination with active ranging signals, e.g. using light or sound signals emitted toward objects

Abstract

An active camera system images successive frames of a scene utilizing an array of detectors configured to produce a response having a linear relationship to a number of incident photons. Multi-frame capture is utilized with differentiated frame illumination for successive frames. Frame processing allows the camera to produce, at the pixel level, different image intensities representing maximum signal intensity, ambient signal intensity, and object signal intensity. The linearized response of detectors in the lighting-invariant image sensor is used to establish non-attenuated signal strength that enables shadow removal and glare removal for one or more of the image signals. Embodiments of a lighting-invariant image sensor may be used in Autonomous and Semi-Autonomous Vehicle Control/Assist Systems.

Claims

1. An active camera system configured to generate a lighting-invariant image of a scene comprising:

at least one emitter configured to emit an active light pulse toward the scene;

an array of detectors configured to receive light for a field of view that includes at least a portion of the scene, each detector in the array of detectors configured to produce a linearized response to a number of incident photons of light;

control circuitry operably coupled to the at least one emitter and the array of detectors and configured to store at least two successive frames of data as frame pixels in one or more frame buffers, wherein each frame pixel is based on the linearized response from a corresponding one or more of the detectors in the array of detectors, wherein a duration of a capture cycle of the array of detectors is constant for the at least two successive frames but an intensity of the active light pulse emitted during each capture cycle is different for the at least two successive frames; and

a processing system operably coupled to the control circuitry and the one or more frame buffers to generate the lighting-invariant image of the scene, the processing system configured to;

analyze the at least two successive frames of data to determine a minimum intensity value due to ambient light in the scene and a maximum intensity value for frame pixels; and

generate the lighting-invariant image based on a difference between the minimum intensity values and the maximum intensity values of the frame pixels such that the image is based on a reflected portion of the frame data resulting from the active light pulse.

2. The system of claim 1, wherein the at least one emitter comprises one or more LEDs, lasers, or laser diodes.

3. The system of claim 1, wherein the at least one emitter comprises one or more vehicle headlamps.

4. The system of claim 1, wherein the control circuitry is configured such that the relative difference in timing has no overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the at least two successive frames.

5. The system of claim 1, wherein the control circuitry is configured such that the relative difference in timing has complete overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the at least two successive frames.

6. The system of claim 1, wherein the control circuitry is configured such that the relative difference in timing has:

no overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the at least two successive frames; and

complete overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least another of the at least two successive frames.

7. The system of claim 1, wherein each detector in the array of detectors has a linear response to a number of incident photons, and wherein the reflected portion of the pulse of emitted light for each image pixel is defined as the maximum intensity value minus the minimum intensity value.

8. The system of claim 1, wherein each detector in the array of detectors has a non-linear response to the number of incident photons, and wherein the reflected portion of the pulse of emitted light for each image pixel is defined as the maximum intensity value minus the minimum intensity value, and further modified by the non-linear response characteristics of the detector corresponding to that image pixel.

9. The system of claim 1, wherein the lighting-invariant image model further comprises a distance for at least some of the common points in the field of view of the scene for each image that is determined based on the reflected portion of the pulse of emitted light for the image pixels corresponding to the common point in the field of view of the scene.

10. The system of claim 1, wherein the emitted radiation is electromagnetic energy selected from the wavelength ranges of:

ultraviolet (UV)—

100-400 nm, visible—

400-700 nm, near infrared (NIR)—

700-1400 nm, infrared (IR)—

1400-8000 nm, long-wavelength IR (LWIR)—

8 um-15 um, or far IR (FIR)—

15 um-1000 um.

11. The system of claim 1, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector and the control circuitry is configured to capture a color image frame corresponding to each of the plurality of multi-color filter elements.

12. The system of claim 1, wherein the active camera system is attached to a vehicle and is configured to generate lighting-invariant images of the scene that is in motion relative to the vehicle.

13. The system of claim 1, wherein each detector in the array of detectors has an individual field of view that is a different subset of the field of view of the array of detectors.

14. A method of generating a lighting-invariant image from two or more frames of image data of a scene, comprising:

causing at least one emitter to generate and emit light within a defined frequency range throughout a field of view of the scene;

causing an array of detectors to receive light within the defined frequency range for a field of view of the array of detectors, the field of view of the scene defined by the field of view of the array of detectors;

controlling a start time and a duration of an image capture cycle of the array of detectors that correspond to a frame of image pixels that integrate image data of the scene, a start time of a pulse of emitted light from the emitter, and a relative difference in timing between the start time of the pulse of emitted light and the start time of the duration of the image capture cycle of the array of detectors;

producing at least two successive frames of image pixels in which at least one of the relative difference in timing between the start times and the duration of the image capture cycle of the array of detectors is different;

analyzing the frames of image pixels with a processing system to determine a minimum intensity value due to ambient light in the scene and a maximum intensity value of image pixels corresponding to a common point in the field of view of the scene; and

generating the image based on the minimum intensity values and the maximum intensity values of the image pixels for the at least two successive frames of image data such that the image represents a reflected portion of the pulse of emitted light captured by the image pixels in that image.

15. The method of claim 14, wherein controlling the relative difference in timing is performed such that there is no overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the at least two successive frames.

16. The method of claim 14, wherein controlling the relative difference in timing is performed such that there is complete overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the at least two successive frames.

17. The method of claim 14, wherein controlling the relative difference in timing is performed such that there is:

no overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the at least two successive frames; and

complete overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least another of the at least two successive frames.

18. The method of claim 14, wherein each detector in the array of detectors has a linear response to a number of incident photons, and wherein the reflected portion of the pulse of emitted light for each image pixel is defined as the maximum intensity value minus the minimum intensity value.

19. The method of claim 14, wherein each detector in the array of detectors has a non-linear response to the number of incident photons, and wherein a reflected portion of the pulse of emitted light for each image pixel is defined as the maximum intensity value minus the minimum intensity value, and further modified by the non-linear response characteristics of the detector corresponding to that image pixel.

20. The method of claim 14, wherein generating the lighting-invariant image model further comprising determining a distance for at least some of the common points in the field of view of the scene based on the reflected portion of the pulse of emitted light for the image pixels corresponding to the common point in the field of view of the scene.

21. The method of claim 14, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector, producing two or more frame of image pixels of the scene includes capturing a color image frame corresponding to each of the plurality of multi-color filter elements.

22. An active camera system attached to a vehicle and configured to generate lighting-invariant images of a scene that is in motion relative to the vehicle, the system comprising:

at least one emitter configured to generate and emit light within a defined frequency range throughout a field of view of the scene;

an array of detectors configured to receive light within the defined frequency range for a field of view of the array of detectors, the field of view of the scene defined by the field of view of the array of detectors;

control circuitry operably coupled to the at least one emitter and the array of detectors and configured to;

control a start time and a duration of an image capture cycle of the array of detectors that correspond to a frame of image pixels that integrate image data of the scene, a start time of a pulse of emitted light from the emitter, and a relative difference in timing between the start time of the pulse of emitted light and the start time of the duration of the image capture cycle of the array of detectors;

produce at least two successive frames of image pixels of the scene in which at least one of the relative difference in timing and the duration of the image capture cycle of the array of detectors is different;

a processing system operably coupled to the control circuitry to generate the images of the scene and configured to;

analyze the at least two successive frames of image pixels to determine a minimum intensity value due to ambient light in the scene and a maximum intensity value of image pixels corresponding to a common point in the field of view of the scene; and

generate each of the lighting-invariant images of the scene that is in-motion relative to the vehicle based on the minimum intensity values and the maximum intensity values of the image pixels for the at least two successive frames of image data.

23. The system of claim 22, wherein the control circuitry is configured to control the relative difference in timing to achieve one of the following relative differences:

there is no overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the two or more successive frames;

there is complete overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the two or more successive frames;

or

there is no overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least one of the two or more successive frames, and complete overlap between a duration of the pulse of emitted light and the duration of the image capture cycle of the array of detectors for at least another of the two or more successive frames.

24. The system of claim 22, wherein each detector in the array of detectors has one of a linear response to a number of incident photons and a non-linear response to the number of incident photons,

wherein for the linear response to the number of incident photons the reflected portion of the pulse of emitted light for each image pixel is defined as the maximum intensity value minus the minimum intensity value, and

wherein for the non-linear response to the number of incident photons a reflected portion of the pulse of emitted light for each image pixel is defined as the maximum intensity value minus the minimum intensity value, and further modified by the non-linear response characteristics of the detector corresponding to that image pixel.

25. The system of claim 22, wherein the processing system further is configured determine a distance for at least some of the common points in the field of view of the scene based on the reflected portion of the pulse of emitted light for the image pixels corresponding to the common point in the field of view of the scene that can be used to generate each of the lighting-invariant images of the scene.

26. The system of claim 22, wherein the array of detectors further comprises a plurality of multi-color filter elements for each detector, and the two or more frame of image pixels of the scene produced by the control circuitry includes capturing a color image frame corresponding to each of the plurality of multi-color filter elements.

27. An active camera system attached to a vehicle and configured to generate lighting-invariant images of a scene that is in motion relative to the vehicle, the system comprising:

at least one emitter configured to emit active light within a defined frequency range throughout a field of view of the scene;

an array of detectors configured to receive light within the defined frequency range for a field of view of the array of detectors, the field of view of the scene defined by the field of view of the array of detectors;

control circuitry operably coupled to the at least one emitter and the array of detectors and configured to;

control a start time and a duration of an image capture cycle of the array of detectors that correspond to a frame of image pixels that integrate image data of the scene;

control the at least one emitter to emit active light during an entire period corresponding to the duration of the image capture cycle;

produce at least two successive frames of image pixels of the scene in which there is a relative difference in the start time or the duration of the image capture cycle of the array of detectors;

a processing system operably coupled to the control circuitry to generate the images of the scene and configured to;

analyze the at least two successive frames of image pixels to determine a minimum intensity value due to ambient light in the scene and a maximum intensity value of image pixels corresponding to a common point in the field of view of the scene; and

generate each of the lighting-invariant images of the scene that is in-motion relative to the vehicle based on the minimum intensity values and the maximum intensity values of the image pixels for the at least two successive frames of image data.

28. The system of claim 27, wherein the at least one emitter is controlled to emit active light such that:

the at least one emitter only emits active light corresponding to the duration of the image capture cycle;

the at least one emitter only emits active light at least during some portion of the duration of the image capture cycle;

or

the at least one emitter constantly emits active light.

29. The system of claim 27, wherein each detector in the array of detectors has one of a linear response to a number of incident photons and a non-linear response to the number of incident photons,

wherein for the linear response to the number of incident photons the reflected portion of the pulse of emitted light for each image pixel is defined as the maximum intensity value minus the minimum intensity value, and

wherein for the non-linear response to the number of incident photons a reflected portion of the pulse of emitted light for each image pixel is defined as the maximum intensity value minus the minimum intensity value, and further modified by the non-linear response characteristics of the detector corresponding to that image pixel.

30. The system of claim 27, wherein the processing system is further configured to determine a distance for at least some of the common points in the field of view of the scene based on the reflected portion of the pulse of emitted light for the image pixels corresponding to the common point in the field of view of the scene that can be used to generate each of the lighting-invariant images of the scene.