System and method for vehicle position and velocity estimation based on camera and lidar data

• US 10,552,691 B2
• Filed: 04/25/2017
• Issued: 02/04/2020
• Est. Priority Date: 04/25/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: Tusimple
• Inventors: Li, Chenyang, Hou, Xiaodi, Liu, Siyuan
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US15/496,113
• Publication Number:
  US 20180308250A1
• Time in Patent Office: 1,015 Days
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

A vehicle position and velocity estimation based on camera and LIDAR data are disclosed. A particular embodiment includes: receiving input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device; determining a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device; tracking a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generating tracking data; determining a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data; determining a velocity of the proximate object using the 3D position and the tracking data; and outputting the 3D position and velocity of the proximate object relative to the autonomous vehicle.

Claims

1. A system comprising:

a data processor; and

a vehicle position and velocity estimation module, executable by the data processor, the vehicle position and velocity estimation module being configured to perform a proximate object position and velocity estimation operation for an autonomous vehicle, the proximate object position and velocity estimation operation being configured to;

receive input object data from a subsystem of the autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device being one or more light imaging, detection, and ranging (LIDAR) sensors;

determine a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device and semantic segmentation processing of the image data;

track a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generate tracking data;

correlate the proximate object identified from the image data with the proximate object identified and tracked from the distance data, the correlation being configured to match the 2D position of the proximate object detected in the image data with the 3D position of the same proximate object detected in the distance data;

determine a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data;

determine a velocity of the proximate object using the 3D position and the tracking data; and

output the 3D position and velocity of the proximate object relative to the autonomous vehicle.

2. The system of claim 1 wherein the image generating device is one or more cameras.

3. The system of claim 1 wherein the proximate object position and velocity estimation operation being further configured to retain position tracking data from a previous cycle.

4. The system of claim 1 wherein the image data corresponds to at least one frame from a video stream generated by one or more cameras.

5. The system of claim 1 wherein the distance data corresponds to a point cloud generated by the one or more LIDAR sensors.

6. The system of claim 1 wherein the 3D position and velocity of the proximate object is output to a vehicle control subsystem causing the autonomous vehicle to avoid the proximate object.

7. The system of claim 1 wherein the proximate object position and velocity estimation operation being configured to generate the 3D position and velocity of the proximate object even when intermittent, errant, or unstable image data or distance data is received.

8. A method comprising:

receiving input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device being one or more light imaging, detection, and ranging (LIDAR) sensors;

determining a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device and semantic segmentation processing of the image data;

tracking a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generating tracking data;

correlating the proximate object identified from the image data with the proximate object identified and tracked from the distance data, the correlating including matching the 2D position of the proximate object detected in the image data with the 3D position of the same proximate object detected in the distance data;

determining a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data;

determining a velocity of the proximate object using the 3D position and the tracking data; and

outputting the 3D position and velocity of the proximate object relative to the autonomous vehicle.

9. The method of claim 8 wherein the image generating device is one or more cameras.

10. The method of claim 8 including retaining position tracking data from a previous cycle.

11. The method of claim 8 wherein the image data corresponds to at least one frame from a video stream generated by one or more cameras.

12. The method of claim 8 wherein the distance data corresponds to a point cloud generated by the one or more LIDAR sensors.

13. The method of claim 8 wherein the 3D position and velocity of the proximate object is output to a vehicle control subsystem causing the autonomous vehicle to avoid the proximate object.

14. The method of claim 8 including generating the 3D position and velocity of the proximate object even when intermittent, errant, or unstable image data or distance data is received.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive input object data from a subsystem of an autonomous vehicle, the input object data including image data from an image generating device and distance data from a distance measuring device, the distance measuring device being one or more light imaging, detection, and ranging (LIDAR) sensors;

determine a two-dimensional (2D) position of a proximate object near the autonomous vehicle using the image data received from the image generating device and semantic segmentation processing of the image data;

track a three-dimensional (3D) position of the proximate object using the distance data received from the distance measuring device over a plurality of cycles and generate tracking data;

correlate the proximate object identified from the image data with the proximate object identified and tracked from the distance data, the correlation being configured to match the 2D position of the proximate object detected in the image data with the 3D position of the same proximate object detected in the distance data;

determine a 3D position of the proximate object using the 2D position, the distance data received from the distance measuring device, and the tracking data;

determine a velocity of the proximate object using the 3D position and the tracking data; and

output the 3D position and velocity of the proximate object relative to the autonomous vehicle.

16. The non-transitory machine-useable storage medium of claim 15 wherein the proximate object position and velocity estimation operation being configured to generate the 3D position and velocity of the proximate object even when intermittent, errant, or unstable image data or distance data is received.

17. The non-transitory machine-useable storage medium of claim 15 wherein the 3D position and velocity of the proximate object is output to a vehicle control subsystem causing the autonomous vehicle to avoid the proximate object.