Autonomous vehicle: object-level fusion

• US 10,599,150 B2
• Filed: 09/29/2016
• Issued: 03/24/2020
• Est. Priority Date: 09/29/2016
• Status: Active Grant

• Current Assignee: Charles Stark Draper Laboratory Incorporated, Arriver Software LLC (Qualcomm, Inc.)
• Original Assignee: Charles Stark Draper Laboratory Incorporated
• Inventors: Graham, Matthew, Jones, Troy, Horne, Kyra, Lennox, Scott, Demerly, Jon D., Strong, Stephen Sheldon, Naik, Afrah Faiz
• Primary Examiner(s): Hsieh, Ping Y
• Application Number:
  US15/280,291
• Publication Number:
  US 20180089538A1
• Time in Patent Office: 1,272 Days
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/932 using own vehicle data, e.g. ground speed, steering wheel direction
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G05D 1/021 specially adapted to land vehicles
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

Previous self-driving car systems can detect objects separately with either vision systems, RADAR systems or LIDAR systems. In an embodiment of the present invention, an object fusion module normalizes sensor output from vision, RADAR, and LIDAR systems into a common format. Then, the system fuses the object-level sensor data across all systems by associating all objects detected and predicting tracks for all objects. The present system improves over previous systems by using the data from all sensors combined to develop a single set of knowledge about the objects around the self-driving car, instead of each sensor operating separately.

Claims

1. A method comprising:

converting sensor data of a plurality of detected objects to a common coordinate frame, the sensor data of each detected object collected from a given sensor of a plurality of heterogeneous sensors at a current measurement time, each detected object including at least one of kinematic information, geometric information, and object classification information based on the converted sensor data, the plurality of heterogeneous sensors being mounted on a highly-automated vehicle;

predicting position, velocity, orientation and bounding boxes of existing object tracks at the current measurement time, the predicting resulting in a given predicted object track associated with a given existing object track, the given predicted object track including at least one of kinematic information, geometric information, and object classification information;

associating the detected objects to existing object tracks by determining a similarity of a given detected object and a given predicted object track, the information of the given detected object being a different type than the information of the given predicted object track;

updating the kinematic, geometric and object classification information for existing object tracks by updating the given existing object track with the information of the given detected object determined to be similar to the predicted object track; and

reporting a fused object list having a resulting set of updated object tracks.

2. The method of claim 1, further comprising initiating new object tracks for detected objects that are unassociated to existing object tracks.

3. The method of claim 1, further comprising deleting object tracks that are outside of a field-of-view of the one or more heterogeneous sensors view of the autonomous vehicle.

4. The method of claim 3, wherein deleting the object tracks further includes exempting object tracks from deletion if the object tracks are within a blind spot of the one or more heterogeneous sensors.

5. The method of claim 1, wherein associating detected objects to existing object tracks further determines a similarity of at least three of kinematic information, geometric information, and object classification information.

6. The method of claim 1, further comprising:

associating detected feature tracks to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information.

7. The method of claim 1, wherein kinematic information includes position information, velocity, information, and orientation information, geometric information includes bounding boxes and object contours, and object classification information includes an object type.

8. A system comprising:

a preprocessing module configured to convert sensor data of a plurality of detected objects to a common coordinate frame, the sensor data of each detected object collected from a given sensor of a plurality of heterogeneous sensors at a current measurement time, each detected object including at least one of kinematic information, geometric information, and object classification information based on the converted sensor data, the plurality of heterogeneous sensors being mounted on a highly-automated vehicle;

a track prediction module configured to predict position, velocity, orientation and bounding boxes of existing object tracks at the current measurement time, the predicting resulting in a given predicted object track associated with a given existing object track, the given predicted object track including at least one of kinematic information, geometric information, and object classification information;

a data association module configured to associate the detected objects to existing object tracks by determining a similarity of a given detected object and a given predicted object track, the information of the given detected object being a different type than the information of the given predicted object track;

a track update module configured to update the kinematic, geometric and object classification information for existing object tracks by updating the given existing object track with the information of the given detected object determined to be similar to the predicted object track; and

a reporting module configured to report a fused object list having a resulting set of updated object tracks.

9. The system of claim 8, further comprising a track management module configured to initiate new object tracks for detected objects that are unassociated to existing object tracks.

10. The system of claim 8, wherein the track update module is further configured to delete object tracks that are outside of a field-of-view of the one or more heterogeneous sensors view of the autonomous vehicle.

11. The system of claim 10, wherein the track update module is further configured to delete the object tracks by exempting object tracks from deletion if the object tracks are within a blind spot of the one or more heterogeneous sensors.

12. The system of claim 8, wherein the data association module is further configured to determine a similarity of at least three of kinematic information, geometric information, and object classification information.

13. The system of claim 8, wherein the data association module is further configured to associate detected feature tracks to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information.

14. The system of claim 8, wherein kinematic information includes position information, velocity, information, and orientation information, geometric information includes bounding boxes and object contours, and object classification information includes an object type.

15. A non-transitory computer-readable medium configured to store instructions for operating an autonomous vehicle, the instructions, when loaded and executed by a processor, causes the processor to:

convert sensor data of a plurality of detected objects to a common coordinate frame, the sensor data of each detected object collected from a given sensor of a plurality of heterogeneous sensors at a current measurement time, each detected object including at least one of kinematic information, geometric information, and object classification information based on the converted sensor data, the plurality of heterogeneous sensors being mounted on a highly-automated vehicle;

predict position, velocity, orientation and bounding boxes of existing object tracks at the current measurement time, the predicting resulting in a given predicted object track associated with a given existing object track, the given predicted object track including at least one of kinematic information, geometric information, and object classification information;

associate the detected objects to existing object tracks by determining a similarity of a given detected object and a given predicted object track, the information of the given detected object being a different type than the information of the given predicted object track; and

update the kinematic, geometric and object classification information for existing object tracks by updating the given existing object track with the information of the given detected object determined to be similar to the predicted object track;

report a fused object list having a resulting set of updated object tracks.

16. The non-transitory computer-readable medium of claim 15, wherein the instructions are further configured to cause the processor to initiate new object tracks for detected objects that are unassociated to existing object tracks.

17. The non-transitory computer-readable medium of claim 15, wherein the instructions are further configured to cause the processor to delete object tracks that are outside of a field-of-view of the one or more heterogeneous sensors view of the autonomous vehicle.

18. The non-transitory computer-readable medium of claim 17, wherein deleting the object tracks further includes exempting object tracks from deletion if the object tracks are within a blind spot of the one or more heterogeneous sensors.

19. The non-transitory computer-readable medium of claim 15, wherein associating detected objects to existing object tracks further determines a similarity of at least three of kinematic information, geometric information, and object classification information.

20. The non-transitory computer-readable medium of claim 15, wherein the instructions are further configured to cause the processor to associate detected feature tracks to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information.