Combining lidar and camera data

• US 10,677,897 B2
• Filed: 04/16/2018
• Issued: 06/09/2020
• Est. Priority Date: 04/14/2017
• Status: Active Grant

• Current Assignee: Luminar Technologies, Inc.
• Original Assignee: Luminar Technologies, Inc.
• Inventors: LaChapelle, Joseph G., Eichenholz, Jason M.
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Tarko, Asmamaw G
• Application Number:
  US15/954,427
• Publication Number:
  US 20180299534A1
• Time in Patent Office: 785 Days
• Field of Search: 348138
• CPC Class Codes: B60Y 2400/3015 Optical cameras
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

A system includes a lidar, a camera, and a controller communicatively coupled to the camera and the lidar. The lidar includes a laser configured to emit pulses of light, a scanner configured to direct the emitted pulses in accordance with a scan pattern, and a receiver configured to detect the emitted pulse of light scattered by one or more remote targets to collect a set of lidar pixels of a scan frame, in a sequence defined by the scan pattern. The camera has a field of regard that at least partially overlaps the field of regard of the lidar. The controller is configured to cause the camera to capture images while the receiver of the lidar module collects the complete set of lidar pixels of the scan frame, and align lidar pixels with corresponding pixels in the captured images.

Claims

1. A system operating in a vehicle, the system comprising:

a lidar module including;

a laser configured to emit pulses of light according to a pulse rate scheme and a power scheme,

a scanner configured to direct the emitted pulses of light in accordance with a scan pattern to illuminate a field of regard of the lidar module, and

a receiver configured to detect the emitted pulses of light scattered by one or more remote targets, wherein the receiver comprises;

an avalanche photodiode configured to produce an electrical-current pulse corresponding to a received pulse of light, and

a transimpedance amplifier configured to convert the electrical-current pulse into a voltage pulse that corresponds to the electrical-current pulse;

the system further comprising;

a camera with a field of view at least partially overlapping the field of regard of the lidar module; and

a controller communicatively coupled to the camera and the lidar module, the controller configured to;

determine, using the camera, an object within the field of view of the camera and the field of regard of the lidar module, and

cause the lidar module to modify at least one of (i) the scan pattern, (ii) the pulse rate scheme, and (iii) the power scheme of the lidar module based on the determined object.

2. The system of claim 1, wherein:

the controller is configured to determine, using the camera, the object by identifying that the object is a person, and

the controller is further configured to cause the lidar module to modify the power scheme of the lidar module by lowering an optical power of the emitted pulses of light over a region of interest based on the person.

3. The system of claim 1, wherein:

the lidar module is configured to capture a scan frame by following the scan pattern;

the camera is configured to capture at least a first image and a second image; and

the controller is further configured to;

align a first lidar pixel of the scan frame to a first camera pixel from the first image, and

align a second lidar pixel of the scan frame to a second camera pixel from the second image.

4. The system of claim 1, wherein the controller is further configured to:

identify, based on the determined object, a region of interest within the field of regard of the lidar module; and

modify the scan pattern by temporarily focusing the scan pattern on the identified region of interest.

5. The system of claim 1, wherein laser is configured to emit the pulses of light with a pulse duration less than 20 nanoseconds and a pulse repetition frequency less than 5 MHz.

6. The system of claim 1, wherein the scanner comprises one or more of (i) a galvanometer scanner, (ii) a polygonal scanner, and (iii) a microelectromechanical systems (MEMS) device.

7. The system of claim 1, wherein the scanner is configured to synchronously scan a field of view of the laser and a field of view of the receiver, wherein the laser field of view and the receiver field of view are scanned along the scan pattern at equal scanning speeds.

8. The system of claim 1, wherein determining the object within the field of view of the camera and the field of regard of the lidar module comprises identifying the object or determining a shape of the object.

9. The system of claim 1, wherein:

the receiver further comprises;

a comparator configured to produce an electrical-edge signal when the voltage pulse rises above or falls below a particular threshold voltage, and

a time-to-digital converter configured to receive the electrical-edge signal and determine an interval of time between emission of one of the emitted pulses of light and receipt of the electrical-edge signal; and

the controller is further configured to determine a distance from the system to the object based at least in part on the determined interval of time.

10. The system of claim 1, wherein:

the receiver further comprises;

a gain circuit configured to amplify the voltage pulse,

a comparator configured to produce an electrical-edge signal when the amplified voltage pulse rises above or falls below a particular threshold voltage, and

a time-to-digital converter configured to receive the electrical-edge signal and determine an interval of time between emission of one of the emitted pulses of light and receipt of the electrical-edge signal; and

the controller is further configured to determine a distance from the system to the object based at least in part on the determined interval of time.

11. A method for controlling a lidar module, the method comprising:

emitting, from a laser, pulses of light according to a pulse rate scheme and a power scheme;

scanning, using a scanner, the emitted pulses of light in accordance with a scan pattern to illuminate a field of regard of the lidar module;

detecting, using a receiver, the emitted pulses of light scattered by one or more remote targets in the field of regard of the lidar module, wherein the receiver comprises;

an avalanche photodiode configured to produce an electrical-current pulse corresponding to a received pulse of light, and

a transimpedance amplifier configured to convert the electrical-current pulse into a voltage pulse that corresponds to the electrical-current pulse;

obtaining, using a camera, a first image that at least partially overlaps the field of regard of the lidar module;

identifying, using the first image, an object within the field of regard of the lidar module;

identifying, based on the object, a region within the field of regard of the lidar module; and

modifying, over the region, a lidar parameter, wherein the lidar parameter is at least one of (i) the scan pattern, (ii) the pulse rate scheme, and (iii) the power scheme.

12. The method of claim 11, wherein:

identifying the object within the field of regard of the lidar module comprises identifying a person; and

modifying the lidar parameter comprises lowering an optical power of the emitted pulses of light over the region.

13. The method of claim 11, further comprising:

collecting a scan frame in a sequence defined by the scan pattern, wherein the scan frame includes a first lidar pixel and a second lidar pixel;

obtaining, using the camera, a second image that at least partially overlaps the field of regard of the lidar module;

aligning the first lidar pixel of the scan frame to a first camera pixel from the first image;

aligning the second lidar pixel of the scan frame to a second camera pixel from the second image.

14. The method of claim 13, further comprising:

before aligning the first lidar pixel, identifying that a capture time of the first image is most proximate in time to a capture time of the first lidar pixel;

before aligning the second lidar pixel, identifying that a capture time of the second image is most proximate in time to a capture time of the second lidar pixel.

15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to perform operations comprising:

instructing a laser of a lidar module to emit pulses of light according to a pulse rate scheme and a power scheme;

instructing a scanner of the lidar module to scan the emitted pulses of light in accordance with a scan pattern to illuminate a field of regard of the lidar module;

receiving lidar pixels corresponding to the emitted pulses of light scattered by one or more remote targets in the field of regard of the lidar module, wherein the emitted pulses of light scattered by the remote targets are detected using a receiver of the lidar module, the receiver comprising;

an avalanche photodiode configured to produce an electrical-current pulse corresponding to a received pulse of light, and

a transimpedance amplifier configured to convert the electrical-current pulse into a voltage pulse that corresponds to the electrical-current pulse;

collecting a scan frame from the lidar module;

obtaining an image from a camera;

identifying, using the image, an object in the field of regard of the lidar module;

identifying, based on the object, a region within the image; and

modifying a lidar parameter of the lidar module, wherein the lidar parameter is at least one of (i) a scan pattern, (ii) a pulse rate scheme, and (iii) a power scheme.

16. The storage media of claim 15, wherein:

identifying the object comprises identifying a person; and

modifying the lidar parameter comprises lowering an optical power of the emitted pulses of light over the region.