System and method for calibrating video and lidar subsystems

• US 9,134,402 B2
• Filed: 03/15/2013
• Issued: 09/15/2015
• Est. Priority Date: 08/13/2012
• Status: Active Grant

• Current Assignee: Aeva, Inc.
• Original Assignee: Digital Signal Corporation (StereoVision Imaging, Inc.)
• Inventors: Sebastian, Richard L., Zheleznyak, Anatoley T.
• Primary Examiner(s): RATCLIFFE, LUKE D
• Application Number:
  US13/839,216
• Publication Number:
  US 20150160332A1
• Time in Patent Office: 914 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/34 using transmission of continuous, frequency-modulated waves while heterodyning the received signal, or a signal derived therefrom, with a locally-generated signal related to the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/4972 Alignment of sensor

Abstract

A system uses range and Doppler velocity measurements from a lidar system and images from a video system to estimate a six degree-of-freedom trajectory of a target. The system calibrates the lidar subsystem with the video subsystem in two stages. In a first stage, the system calibrates the lidar subsystem so that lidar measurements provide 3D lidar coordinates. In a second stage, the system relates the 3D lidar coordinates with a video image obtained from the video subsystem.

Claims

1. A system for calibrating a lidar subsystem and a video subsystem, the system comprising:

a lidar subsystem configured to direct a first beam toward a reference grid target and generate a lidar measurement for each of a plurality of points on the reference grid target for the first beam, wherein the reference grid target comprises a plurality of highly reflective tapes along each edge of the reference grid target and a plurality of video targets interspersed throughout a surface of the reference grid target;

a video subsystem configured to provide a plurality of two-dimensional images of the reference grid target; and

a processor configured to;

receive, from the lidar subsystem, the lidar measurement for each of the plurality of points on the reference grid target,

receive, from the video subsystem, the plurality of images of the reference grid target,

determine a lidar measurement for each of the plurality of highly reflective tapes along each edge of the reference grid target from among the received lidar measurement for each of the plurality of points on the reference grid target;

determine 3D coordinates of each of the plurality of video targets based at least on the lidar measurement for each of the plurality of highly reflective tapes along each edge of the reference grid target; and

relate the 3D coordinates of each of the plurality of video targets with pixel coordinates of each of the plurality of video targets, wherein the pixel coordinates are obtained from an image of the reference grid target.

2. The system of claim 1, wherein the lidar measurement for each of the plurality of points comprises 3D lidar coordinates corresponding to the lidar measurement of range and Doppler velocity at a particular azimuth, elevation, and a time at which the measurement was taken.

3. The system of claim 1, wherein the lidar subsystem is configured to generate a first lidar measurement for each of the plurality of points on the reference grid target at a first range, wherein the video subsystem is configured to provide a first image of the reference grid target at the first range, and wherein the processor is further configured to:

determine a first lidar measurement for each of the plurality of highly reflective tapes along each edge of the reference grid target at the first range;

determine, at the first range, first 3D coordinates of each of the plurality of video targets based at least on the first lidar measurement for each of the plurality of highly reflective tapes; and

relates the first 3D coordinates of each of the plurality of video targets with pixel coordinates of the plurality of video targets, wherein the pixel coordinates are obtained from the first image.

4. The system of claim 3, wherein the lidar subsystem is configured to generate a second lidar measurement for each of the plurality of points on the reference grid target at a second range, wherein the video subsystem is configured to provide a second image of the reference grid target at the second range, and wherein the processor is further configured to:

determine a second lidar measurement for each of the plurality of highly reflective tapes along each edge of the reference grid target at the second range;

determines, at the second range, second 3D coordinates of each of the plurality of video targets based at least on the second lidar measurement for each of the plurality of highly reflective tapes; and

relates the second 3D coordinates of each of the plurality of video targets with pixel coordinates of the plurality of video targets, wherein the pixel coordinates are obtained from the second image.