System and Method for Generating Three Dimensional Images Using Lidar and Video Measurements

• US 20100271615A1
• Filed: 02/22/2010
• Published: 10/28/2010
• Est. Priority Date: 02/20/2009
• Status: Active Grant

• Current Assignee: Aeva, Inc.
• Original Assignee: Digital Signal Corporation (StereoVision Imaging, Inc.)
• Inventors: Zheleznyak, Anatoley T., Sebastian, Richard Lee
• Granted Patent: US 8,717,545 B2
• Time in Patent Office: Days
• US Class Current: 356/4.10
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01S 17/34 using transmission of continuous, frequency-modulated waves while heterodyning the received signal, or a signal derived therefrom, with a locally-generated signal related to the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4815 using multiple transmitters
• CPC Class Codes: G01S 7/491 Details of non-pulse systems
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 13/211 using temporal multiplexing

Abstract

A system uses range and Doppler velocity measurements from a lidar system and images from a video system to estimate a six degree-of-freedom trajectory of a target. The system estimates this trajectory in two stages: a first stage in which the range and Doppler measurements from the lidar system along with various feature measurements obtained from the images from the video system are used to estimate first stage motion aspects of the target (i.e., the trajectory of the target); and a second stage in which the images from the video system and the first stage motion aspects of the target are used to estimate second stage motion aspects of the target. Once the second stage motion aspects of the target are estimated, a three-dimensional image of the target may be generated.

Claims

1. A system for generating a three-dimensional image of a target from lidar measurements and video images, the system comprising:

a lidar subsystem that directs at least two beams toward the target and that generates a range measurement and a Doppler velocity measurement for each of a plurality of points on the target for each of the at least two beams;

a video subsystem that provides a plurality of two-dimensional images of the target; and

a processor that;

receives, from the lidar subsystem, the range measurement and the Doppler velocity measurement for each of the plurality of points on the target,

receives, from the video subsystem, the plurality of images of the target,

estimates first stage motion aspects of the target from range measurements and Doppler velocity measurements from the lidar subsystem and from two-dimensional position and velocity measurements of at least one feature of the target obtained from the plurality of images of the target,

estimates second stage motion aspects of the target by subtracting a first one of the plurality of images from a second one of the plurality of images, wherein the first one of the plurality of images and the second one of the plurality of images are each referenced to a particular time using the first stage motion aspects of the target, and

generates the three-dimensional image of the target using the second stage motion aspects.

2. The system of claim 1, wherein the first stage motion aspects comprise a three-dimensional translational velocity and a three-dimensional angular velocity.

3. The system of claim 1, wherein the processor estimates first stage motion aspects of the target using a least squares estimator.

4. The system of claim 1, wherein the lidar subsystem directs at least four beams toward a target and generates a range measurement and a Doppler velocity measurement for each of a plurality of points on the target for each of the at least four beams.

5. The system of claim 1, wherein the at least two beams are displaced from one another by a known distance or angle.

6. The system of claim 4, wherein at least two of the at least four beams are displaced from one another by a known distance or angle along a first axis and wherein at least two of the at least four beams are displaced from one another by a known distance or angle along a second axis.

7. The system of claim 6, wherein the first axis and the second axis are orthogonal to one another.

8. The system of claim 1, wherein the processor estimates first stage motion aspects of the target from range measurements and Doppler velocity measurements from the lidar subsystem and from position and velocity measurements of at least one feature of the target obtained from the plurality of images of the target by:

estimating two angular velocity components and a translational velocity component of the target using range measurements and Doppler velocity measurements from the lidar subsystem, and

estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target and the position and velocity measurements of the at least one feature of the target.

9. The system of claim 8, wherein estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target and the position and velocity measurements of the at least one feature of the target further comprises estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target, the position and velocity measurements of the at least one feature of the target, and an estimate of a distance component to the at least one feature of the target.

10. The system of claim 9, wherein the estimate of the distance component to the at least one feature of the target comprises determining an average distance component based on the range measurements from the lidar subsystem.

11. The system of claim 8, wherein estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target and the position and velocity measurements of the at least one feature of the target further comprises iteratively estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target, the position and velocity measurements of the at least one feature of the target, and an estimate of a distance component to the at least one feature of the target.

12. The system of claim 11, wherein during a first iteration, the estimate of the distance component to the at least one feature of the target comprises determining an average distance component based on the range measurements from the lidar subsystem.

13. The system of claim 12, wherein during a second iteration, the estimate of the distance component to the at least one feature of the target comprises a distance to the at least one feature of the target determined from the first motion aspects of the target that were estimated during the first iteration.

14. A method for generating a three-dimensional image of a target from lidar measurements from a lidar subsystem and video images from a video subsystem, the system comprising:

receiving, from the lidar subsystem, a range measurement and a Doppler velocity measurement for each of a plurality of points on the target for each of at least two beams of the lidar subsystem;

receiving, from the video subsystem, a plurality of two-dimensional images of the target;

determining two-dimensional position and velocity measurements of at least one feature of the target using the plurality of images of the target;

estimating first stage motion aspects of the target from the range measurements and Doppler velocity measurements from the lidar subsystem and from the position and velocity measurements of the at least one feature of the target determined from the plurality of images of the target;

estimating second stage motion aspects of the target by subtracting a first one of the plurality of images from a second one of the plurality of images, wherein the first one of the plurality of images and the second one of the plurality of images are each referenced to a particular time using the first stage motion aspects of the target; and

generating a three-dimensional image of the target using the second stage motion aspects of the target.

15. The method of claim 14, wherein estimating first stage motion aspects of the target from the range measurements and Doppler velocity measurements from the lidar subsystem and from the position and velocity measurements of the at least one feature of the target determined from the plurality of images of the target comprises estimating, using at least one least squares estimator, first stage motion aspects of the target from the range measurements and Doppler velocity measurements from the lidar subsystem and from the position and velocity measurements of the at least one feature of the target determined from the plurality of images of the target.

16. The method of claim 14, wherein receiving, from the lidar subsystem, a range measurement and a Doppler velocity measurement for each of a plurality of points on the target for each of at least two beams of the lidar subsystem comprises receiving, from the lidar subsystem, a range measurement and a Doppler velocity measurement for each of a plurality of points on the target for each of at least four beams of the lidar subsystem.

17. The method of claim 14, wherein estimating first stage motion aspects of the target from the range measurements and Doppler velocity measurements from the lidar subsystem and from the position and velocity measurements of the at least one feature of the target determined from the plurality of images of the target comprises:

estimating two angular velocity components and a translational velocity component of the target using range measurements and Doppler velocity measurements from the lidar subsystem; and

estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target and the position and velocity measurements of the at least one feature of the target.

18. The method of claim 17, wherein estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target and the position and velocity measurements of the at least one feature of the target comprises:

iteratively estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target, the position and velocity measurements of the at least one feature of the target, and an estimate of a distance component to the at least one feature of the target.

19. The method of claim 18, wherein iteratively estimating the first stage motion aspects of the target using the estimated two angular velocity components and the translational velocity component of the target, the position and velocity measurements of the at least one feature of the target, and an estimate of a distance component to the at least one feature of the target comprises:

during a first iteration, determining an average distance component to the at least one feature of the target based on the range measurements from the lidar subsystem as the estimate of the distance component to the at least one feature of the target; and

during a second iteration, determining a distance component to the at least one feature of the target based on the first stage motion aspects of the target determined during the first iteration as the estimate of the distance component to the at least one feature of the target.

20. The method of claim 19, further comprising:

after the second iteration, estimating the first stage motion aspects of the target using the two angular velocity components and the translational velocity component of the target estimated during the second iteration and using the three-dimensional position and two-dimensional velocity for each of the points on the target determined from the first stage motion aspects of the target estimated during the second iteration.

21. The method of claim 14, wherein estimating second stage motion aspects of the target by subtracting a first one of the plurality of images from a second one of the plurality of images, wherein the first one of the plurality of images and the second one of the plurality of images are each referenced to a particular time using the first stage motion aspects of the target comprises mapping the plurality of images from the video subsystem onto the plurality of points on the target using the first stage motion aspects of the target.