AUTONOMOUS VEHICLE: OBJECT-LEVEL FUSION

• US 20180089538A1
• Filed: 09/29/2016
• Published: 03/29/2018
• Est. Priority Date: 09/29/2016
• Status: Active Grant

• Current Assignee: Arriver Software LLC, Charles Stark Draper Laboratory Incorporated
• Original Assignee: Autoliv ASP Incorporated (Autoliv Incorporated), Charles Stark Draper Laboratory Incorporated
• Inventors: Graham, Matthew, Jones, Troy, Horne, Kyra, Lennox, Scott, Demerly, Jon D., Strong, Stephen Sheldon, Naik, Afrah Faiz
• Granted Patent: US 10,599,150 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/932 using own vehicle data, e.g. ground speed, steering wheel direction
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G05D 1/021 specially adapted to land vehicles
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

Previous self-driving car systems can detect objects separately with either vision systems, RADAR systems or LIDAR systems. In an embodiment of the present invention, an object fusion module normalizes sensor output from vision, RADAR, and LIDAR systems into a common format. Then, the system fuses the object-level sensor data across all systems by associating all objects detected and predicting tracks for all objects. The present system improves over previous systems by using the data from all sensors combined to develop a single set of knowledge about the objects around the self-driving car, instead of each sensor operating separately.

Claims

1. A method comprising:

converting sensor data of detected objects from a plurality of heterogeneous sensors to a common coordinate frame;

predicting position, velocity, orientation and bounding boxes of existing object tracks at a current measurement time;

associating detected objects to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information based on the converted sensor data; and

updating the kinematic, geometric and object classification information for object tracks that are associated to detected objects;

reporting a fused object list having a resulting set of updated object tracks.

2. The method of claim 1, further comprising initiating new object tracks for detected objects that are unassociated to existing object tracks.

3. The method of claim 1, further comprising deleting object tracks that are outside of a field-of-view of the one or more heterogeneous sensors view of the autonomous vehicle.

4. The method of claim 3, wherein deleting the object tracks further includes exempting object tracks from deletion if the object tracks are within a blind spot of the one or more heterogeneous sensors.

5. The method of claim 1, wherein associating detected objects to existing object tracks further determines a similarity of at least three of kinematic information, geometric information, and object classification information.

6. The method of claim 1, further comprising:

associating detected feature tracks to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information.

7. The method of claim 1, wherein kinematic information includes position information, velocity, information, and orientation information, geometric information includes bounding boxes and object contours, and object classification information includes an object type.

8. A system comprising:

a preprocessing module configured to convert sensor data of detected objects from a plurality of heterogeneous sensors to a common coordinate frame;

a track prediction module configured to predict position, velocity, orientation and bounding boxes of existing object tracks at a current measurement time;

a data association module configured to associate detected objects to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information using the converted sensor data;

a track update module configured to update the kinematic, geometric and object classification information for object tracks that are associated to detected objects; and

a reporting module configured to report a fused object list having a resulting set of updated object tracks.

9. The system of claim 8, further comprising a track management module configured to initiate new object tracks for detected objects that are unassociated to existing object tracks.

10. The system of claim 8, wherein the track update module is further configured to delete object tracks that are outside of a field-of-view of the one or more heterogeneous sensors view of the autonomous vehicle

11. The system of claim 10, wherein the track update module is further configured to delete the object tracks by exempting object tracks from deletion if the object tracks are within a blind spot of the one or more heterogeneous sensors.

12. The system of claim 8, wherein the data association module is further configured to determine a similarity of at least three of kinematic information, geometric information, and object classification information.

13. The system of claim 8, wherein the data association module is further configured to associate detected feature tracks to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information.

14. The system of claim 8, wherein kinematic information includes position information, velocity, information, and orientation information, geometric information includes bounding boxes and object contours, and object classification information includes an object type.

15. A non-transitory computer-readable medium configured to store instructions for operating an autonomous vehicle, the instructions, when loaded and executed by a processor, causes the processor to:

convert sensor data of detected objects from a plurality of heterogeneous sensors to a common coordinate frame;

predict position, velocity, orientation and bounding boxes of existing object tracks at a current measurement time;

associate detected objects to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information based on the converted sensor data; and

update the kinematic, geometric and object classification information for object tracks that are associated to detected objects;

report a fused object list having a resulting set of updated object tracks.

16. The non-transitory computer-readable medium of claim 15, wherein the instructions are further configured to cause the processor to initiate new object tracks for detected objects that are unassociated to existing object tracks.

17. The non-transitory computer-readable medium of claim 15, wherein the instructions are further configured to cause the processor to delete object tracks that are outside of a field-of-view of the one or more heterogeneous sensors view of the autonomous vehicle.

18. The non-transitory computer-readable medium of claim 17, wherein deleting the object tracks further includes exempting object tracks from deletion if the object tracks are within a blind spot of the one or more heterogeneous sensors.

19. The non-transitory computer-readable medium of claim 15, wherein associating detected objects to existing object tracks further determines a similarity of at least three of kinematic information, geometric information, and object classification information.

20. The non-transitory computer-readable medium of claim 15, wherein the instructions are further configured to cause the processor to associate detected feature tracks to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information.