Systems and methods for multi-sensor fusion using permutation matrix track association

• US 10,466,361 B2
• Filed: 05/22/2017
• Issued: 11/05/2019
• Est. Priority Date: 03/14/2017
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Research Institute, Inc. (Toyota Motor Corporation)
• Inventors: Lee, Kuan-Hui, Kanzawa, Yusuke, Derry, Matthew O., James, Michael R.
• Primary Examiner(s): Antonucci, Anne M
• Assistant Examiner(s): LaRose, Renee
• Application Number:
  US15/601,433
• Publication Number:
  US 20180267544A1
• Time in Patent Office: 897 Days
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/87 Combinations of radar systems, e.g. primary radar and secondary radar
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9318 Controlling the steering
• CPC Class Codes: G01S 2013/93185 Controlling the brakes
• CPC Class Codes: G01S 2013/9319 Controlling the accelerator
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G01S 2013/9324 Alternative operation using ultrasonic waves
• CPC Class Codes: G01S 5/0263 by combining or switching between positions derived from two or more separate positioning systems
• CPC Class Codes: G01S 5/0278 involving statistical or probabilistic considerations G01S5/0252, G01S5/0294 take precedence

Abstract

System, methods, and other embodiments described herein relate to associating disparate tracks from multiple sensor inputs for observed objects. In one embodiment, a method includes, in response to receiving a first input from a first sensor and a second input from a second sensor, generating the disparate tracks including first sensor tracks and second sensor tracks for the observed objects that correspond to the first input and the second input. The method includes identifying correlations between the first sensor tracks and the second sensor tracks by computing association likelihoods between the first tracks and the second tracks within a permutation matrix according to an objective cost function. The method includes controlling a vehicle according to the correlations.

Claims

1. A fusion tracking system for associating disparate tracks from multiple sensor inputs for observed objects, comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing;

a sensor module including instructions that when executed by the one or more processors cause the one or more processors to, in response to receiving a first input from a first sensor and a second input from a second sensor, generate the disparate tracks including first sensor tracks and second sensor tracks for the observed objects that correspond to the first input and the second input; and

a fusion module including instructions that when executed by the one or more processors cause the one or more processors to identify correlations between the first sensor tracks and the second sensor tracks by computing association likelihoods between the first sensor tracks and the second sensor tracks within a permutation matrix according to an objective cost function,

wherein the fusion module further includes instructions to control a vehicle according to the correlations by generating unified tracks from the first sensor tracks and the second sensor tracks using the correlations, and wherein the unified tracks include trajectories of the observed objects that are comprised of inputs from separate sensors.

2. The fusion tracking system of claim 1, wherein the first sensor tracks and the second sensor tracks are trajectories of the observed objects as produced from separate trackers associated with the first sensor and the second sensor,

wherein the fusion module further includes instructions to compute the association likelihoods using the objective cost function to quantize the correlations between the disparate tracks of the observed objects, wherein the objective cost functions is comprised of a spatial cost, a temporal cost, a mismatched cost, and an entropy cost, and

wherein the first sensor tracks and the second sensor tracks include separate tracks for respective ones of the observed objects.

3. The fusion tracking system of claim 2, wherein the fusion module further includes instructions to compute the spatial cost to identify a spatial association likelihood between the disparate tracks by quantizing differences in a euclidian distance between the disparate tracks, wherein the spatial cost includes components for object position, object velocity, and object forwarding direction, and

wherein the fusion module further includes instructions to compute the spatial association likelihood based, at least in part, on a radial basis function.

4. The fusion tracking system of claim 2, wherein the fusion module further includes instructions to compute the temporal cost to identify a temporal association likelihood between the disparate tracks by identifying whether the disparate tracks from a current observation correlate with previous tracks from a previous observation, and

wherein the fusion module further includes instructions to compute the mismatched cost to penalize the association likelihoods for ones of the disparate tracks in the permutation matrix that do not correlate.

5. The fusion tracking system of claim 2, wherein the fusion module further includes instructions to compute the entropy cost to vary the association likelihoods within indices of the permutation matrix over iterations of the association likelihoods until converging on the correlations, and wherein the association likelihoods are soft-decisions within indices of the permutation matrix that identify how closely the disparate tracks correspond.

6. The fusion tracking system of claim 1, wherein the fusion module further includes instructions to identify the correlations between the first sensor tracks and the second sensor tracks by initially populating the permutation matrix according to the objective cost function using soft-decision values as the association likelihoods, wherein the association likelihoods indicate a likelihood that two tracks of the disparate tracks correlate, wherein the permutation matrix is a binary permutation matrix, and

wherein the fusion module further includes instructions to identify the correlations by normalizing the association likelihoods within the permutation matrix into hard decisions that are the correlations upon the objective cost function converging.

7. The fusion tracking system of claim 1, wherein the first sensor tracks and the second sensor tracks include separate individual tracks for respective ones of the observed objects, wherein the first input and the second input are observations of an environment by the first sensor and the second sensor of the vehicle, respectively.

8. A non-transitory computer-readable medium storing instructions for associating disparate tracks from multiple sensor inputs for observed objects and that when executed by one or more processors cause the one or more processors to:

in response to receiving a first input from a first sensor and a second input from a second sensor, generate the disparate tracks including first sensor tracks and second sensor tracks for the observed objects that correspond to the first input and the second input;

identify correlations between the first sensor tracks and the second sensor tracks by computing association likelihoods between the first sensor tracks and the second sensor tracks within a permutation matrix according to an objective cost function; and

control a vehicle according to the correlations including generating unified tracks from the first sensor tracks and the second sensor tracks using the correlations, and wherein the unified tracks include trajectories of the observed objects that are comprised of inputs from separate sensors of the vehicle.

9. The non-transitory computer-readable medium of claim 8, wherein the first sensor tracks and the second sensor tracks are trajectories of the observed objects as produced from separate trackers associated with the first sensor and the second sensor,

wherein the instructions to compute the association likelihoods using the objective cost function quantize the correlations between the disparate tracks of the observed objects, wherein the objective cost functions is comprised of a spatial cost, a temporal cost, a mismatched cost, and an entropy cost, and

wherein the first sensor tracks and the second sensor tracks include separate tracks for respective ones of the observed objects.

10. The non-transitory computer-readable medium of claim 9, wherein the instructions further include instructions to compute the spatial cost to identify a spatial association likelihood between the disparate tracks by quantizing differences in a euclidian distance between the disparate tracks, wherein the spatial cost includes components for object position, object velocity, and object forwarding direction, and

wherein the instructions further include instructions to compute the spatial association likelihood based, at least in part, on a radial basis function.

11. The non-transitory computer-readable medium of claim 8, wherein the instructions to identify the correlations between the first sensor tracks and the second sensor tracks further include instructions to initially populate the permutation matrix according to the objective cost function using soft-decision values as the association likelihoods, wherein the association likelihoods indicate a likelihood that two tracks of the disparate tracks correlate, wherein the permutation matrix is a binary permutation matrix, and

wherein the instructions to identify the correlations include instructions to normalize the association likelihoods within the permutation matrix into hard decisions that are the correlations upon the objective cost function converging.

12. A method of associating disparate tracks from multiple sensor inputs for observed objects, comprising:

in response to receiving a first input from a first sensor and a second input from a second sensor, generating the disparate tracks including first sensor tracks and second sensor tracks for the observed objects that correspond to the first input and the second input;

identifying correlations between the first sensor tracks and the second sensor tracks by computing association likelihoods between the first sensor tracks and the second sensor tracks within a permutation matrix according to an objective cost function; and

controlling a vehicle according to the correlations including generating unified tracks from the first sensor tracks and the second sensor tracks using the correlations, and wherein the unified tracks include trajectories of the observed objects that are comprised of inputs from separate sensors of the vehicle.

13. The method of claim 12, wherein the first sensor tracks and the second sensor tracks are trajectories of the objects as produced from separate trackers associated with the first sensor and the second sensor,

wherein computing the association likelihoods using the objective cost function quantizes the correlations between the disparate tracks of the observed objects and is comprised of a spatial cost, a temporal cost, a mismatched cost, and an entropy cost, and

wherein the first sensor tracks and the second sensor tracks include separate tracks for respective ones of the observed objects.

14. The method of claim 13, wherein the spatial cost identifies a spatial association likelihood between the disparate tracks by quantizing differences in a euclidian distance between the disparate tracks, wherein the spatial cost includes components for object position, object velocity, and object forwarding direction, and

wherein the spatial association likelihood is based, at least in part, on a radial basis function.

15. The method of claim 13, wherein the temporal cost identifies a temporal association likelihood between the disparate tracks by identifying whether the disparate tracks from a current observation correlate with previous tracks from a previous observation, and

wherein the mismatched cost penalizes the association likelihoods for ones of the disparate tracks in the permutation matrix that do not correlate.

16. The method of claim 13, wherein computing the entropy cost varies the association likelihoods within indices of the permutation matrix over iterations of computing the association likelihoods until converging on the correlations, and wherein the association likelihoods are soft-decisions within indices of the permutation matrix that identify how closely the disparate tracks correspond.

17. The method of claim 12, wherein identifying the correlations between the first sensor tracks and the second sensor tracks includes initially populating the permutation matrix according to the objective cost function using soft-decision values as the association likelihoods, wherein the association likelihoods indicate a likelihood that two tracks of the disparate tracks correlate, wherein the permutation matrix is a binary permutation matrix, and

wherein identifying the correlations includes normalizing the association likelihoods within the permutation matrix into hard decisions that are the correlations upon the objective cost function converging.

18. The method of claim 12, wherein the first sensor tracks and the second sensor tracks include separate individual tracks for respective ones of the objects, and wherein the first input and the second input are observations of an environment by the first sensor and the second sensor of the vehicle, respectively.