Combining Lidar and Camera Data

• US 20180299534A1
• Filed: 04/16/2018
• Published: 10/18/2018
• Est. Priority Date: 04/14/2017
• Status: Active Grant

• Current Assignee: Luminar Technologies, Inc.
• Original Assignee: Luminar Technologies, Inc.
• Inventors: LaChapelle, Joseph G., Eichenholz, Jason M.
• Granted Patent: US 10,677,897 B2
• Time in Patent Office: Days
• CPC Class Codes: B60Y 2400/3015 Optical cameras
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming

Abstract

A system includes a lidar, a camera, and a controller communicatively coupled to the camera and the lidar. The lidar includes a laser configured to emit pulses of light, a scanner configured to direct the emitted pulses in accordance with a scan pattern, and a receiver configured to detect the emitted pulse of light scattered by one or more remote targets to collect a set of lidar pixels of a scan frame, in a sequence defined by the scan pattern. The camera has a field of regard that at least partially overlaps the field of regard of the lidar. The controller is configured to cause the camera to capture images while the receiver of the lidar module collects the complete set of lidar pixels of the scan frame, and align lidar pixels with corresponding pixels in the captured images.

Claims

1. A system operating in a vehicle, the system comprising:

a lidar module including;

a laser configured to emit pulses of light,

a scanner configured to direct the emitted pulses of light in accordance with a scan pattern to illuminate a field of regard of the lidar module, and

a receiver configured to detect the emitted pulse of light scattered by one or more remote target to collect a set of lidar pixels of a scan frame, in a sequence defined by the scan pattern;

the system further comprising;

a camera with a field of view that at least partially overlaps the field of regard of the lidar module; and

a controller communicatively coupled to the camera and the lidar module, the controller configured to;

cause the camera to capture a plurality of images while the receiver of the lidar module collects the complete set of lidar pixels of the scan frame,

align a first lidar pixel in the complete set with one or more pixels of a first one of the plurality of captured images, and

align a second lidar pixel in the complete set with one or more pixels of a second one of the plurality of captured images.

2. A system operating in a vehicle, the system comprising:

a lidar module including;

a laser configured to emit pulses of light according to a pulse rate scheme and a power scheme,

a scanner configured to direct the emitted pulses of light in accordance with a scan pattern to illuminate a field of regard of the lidar module, and

a receiver configured to detect the emitted pulse of light scattered by one or more remote targets;

the system further comprising;

a camera with a field of view at least partially overlapping the field of regard of the lidar module; and

a controller communicatively coupled to the camera and the lidar module, the controller configured to;

determine, using the camera, an object within the field of the camera and the field of regard of the lidar module, and

cause the lidar module to modify at least one of (i) the scan pattern, (ii) the pulse rate scheme, and (iii) the power scheme of the lidar module based on the determined object.