System and Method for Tracking Objects Using Lidar and Video Measurements

• US 20170248698A1
• Filed: 10/12/2016
• Published: 08/31/2017
• Est. Priority Date: 02/20/2009
• Status: Active Grant

• Current Assignee: Aeva, Inc.
• Original Assignee: Digital Signal Corporation (StereoVision Imaging, Inc.)
• Inventors: Sebastian, Richard L., Zheleznyak, Anatoley T.
• Granted Patent: US 10,429,507 B2
• Time in Patent Office: Days
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01S 17/34 using transmission of continuous, frequency-modulated waves while heterodyning the received signal, or a signal derived therefrom, with a locally-generated signal related to the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4815 using multiple transmitters
• CPC Class Codes: G01S 7/491 Details of non-pulse systems
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 13/211 using temporal multiplexing

Abstract

A system uses range and Doppler velocity measurements from a lidar system and images from a video system to estimate a six degree-of-freedom trajectory of a target. The system estimates this trajectory in two stages: a first stage in which the range and Doppler measurements from the lidar system along with various feature measurements obtained from the images from the video system are used to estimate first stage motion aspects of the target (i.e., the trajectory of the target); and a second stage in which the images from the video system and the first stage motion aspects of the target are used to estimate second stage motion aspects of the target. Once the second stage motion aspects of the target are estimated, a three-dimensional image of the target may be generated.

Claims

1. A system for tracking a target from lidar measurements and video images, the system comprising:

a lidar subsystem configured to direct at least four beams toward the target and to provide a range measurement and a Doppler velocity measurement for each of a plurality of points on the target based on information obtained from the at least four beams;

a video subsystem configured to capture a plurality of two-dimensional images of the target; and

a processor configured to;

receive, from the lidar subsystem, the range measurement and the Doppler velocity measurement for each of the plurality of points on the target,

receive, from the video subsystem, the plurality of images of the target,

estimate a translational velocity component v_z^trans and angular velocity components ω

_x and ω

_y for each of the plurality of points on the target from the range measurements and the Doppler velocity measurements from the lidar subsystem,

estimate translational velocity components v_x^trans and v_y^trans and the angular velocity component ω

_z for each of the plurality of points on the target from two-dimensional position and velocity measurements of at least one feature of the target obtained from the plurality of images of the target and from the estimated velocity component v_z^trans and the estimated angular velocity components ω

_x and ω

_y, and

generate a three-dimensional image of the target using the estimated translational velocity components v_x^trans, v_y^trans, and v_z^trans and the estimated angular velocity components ω

_x, ω

_y and ω

_z.

2. The system of claim 1, wherein the processor is further configured to update the estimated translational velocity components v_x^trans, v_y^trans, and v_z^trans and the estimated angular velocity components ω

_x, ω

_y, and co, using a range measurement and a Doppler velocity measurement for each of an additional plurality of points on the target.

3. The system of claim 1,

wherein at least two of the at least four beams are displaced from one another by a known distance or angle along a first axis.

4. The system of claim 3,

wherein two beams of the at least four beams are displaced from one another by a known distance or angle along a second axis.

5. The system of claim 4, wherein the first axis and the second axis are orthogonal to one another.

6. The system of claim 1, further comprising a fifth beam and wherein the processor is configured to direct and maintain the fifth beam on the target.

7. The system of claim 6, wherein the lidar subsystem is configured to provide a range measurement and a Doppler velocity measurement for the position on the target.

8. The system of claim 1, wherein the position on the target is a feature of the target.

9. A method for tracking a target from lidar measurements from a lidar subsystem and video images from a video subsystem, the method comprising:

receiving, from the lidar subsystem, a range measurement and a Doppler velocity measurement for each of a plurality of points on the target, wherein the plurality of points is collectively generated from information obtained from at least four lidar beams directed toward the target;

receiving, from the video subsystem, a plurality of two-dimensional images of the target;

estimating, by a processor, a translational velocity component v_z^trans and angular velocity components ω

_x and ω

_y for each of the plurality of points on the target from the range measurements and the Doppler velocity measurements from the lidar subsystem;

estimating translational velocity components v_x^trans and v_y^trans and the angular velocity component ω

_z for each of the plurality of points on the target from two-dimensional position and velocity measurements of at least one feature of the target obtained from the plurality of images of the target and from the estimated velocity component v_z^trans and the estimated angular velocity components ω

_x and ω

_y; and

generating a three-dimensional image of the target using the estimated translational velocity components v_x^trans, v_y^trans, and v_z^trans and the estimated angular velocity components ω

_y, ω

_y and ω

_z.

10. The method of claim 9, further comprising updating the estimated translational velocity components v_x^trans, v_y^trans, and v_z^trans and the estimated angular velocity components ω

_y, ω

_y and ω

_z using a range measurement and a Doppler velocity measurement for each of an additional plurality of points on the target.

11. The method of claim 9, wherein

wherein at least two of the at least four lidar beams are displaced from one another by a known distance or angle along a first axis.

12. The method of claim 11,

wherein two lidar beams of the at least four lidar beams are displaced from one another by a known distance or angle along a second axis.

13. The method of claim 12, wherein the first axis and the second axis are orthogonal to one another.

14. The method of claim 13, further comprising directing and maintaining a fifth lidar beam of the lidar subsystem on the target.

15. The method of claim 14, further comprising receiving a range measurement and a Doppler velocity measurement for the position on the target from the fifth lidar beam of the lidar subsystem.

16. The method claim 10, further comprising:

locating a position on the target using the three-dimensional image of the target; and

monitoring the position on the target.

17. The method of claim 16, wherein the position on the target is a feature of the target.