Localizing traffic situation using multi-vehicle collaboration

• US 10,586,118 B2
• Filed: 01/13/2018
• Issued: 03/10/2020
• Est. Priority Date: 01/13/2018
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Guo, Rui, Pillai, Preeti, Oguchi, Kentaro
• Primary Examiner(s): Tsai, Tsung Yin
• Application Number:
  US15/870,870
• Publication Number:
  US 20190220678A1
• Time in Patent Office: 787 Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0141 for traffic information dissemination
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/091 Traffic information broadcasting broadcasting communication H04H
• CPC Class Codes: G08G 1/096775 where the origin of the information is a central station

Abstract

A method receives situation data including images from vehicles; clusters, into an image cluster, the images included in the situation data of vehicle(s) located in a geographic region from among the vehicles; locates related situation object(s) in image(s) of the image cluster; matches images from different vehicles in the image cluster, the matched images having corresponding feature(s) of the related situation object(s); determines three-dimensional (3D) sensor coordinates of the related situation object(s) relative to a sensor position of a target vehicle associated with at least one matched image, using the corresponding feature(s) of the related situation object(s) in the matched images; converts the 3D sensor coordinates of the related situation object(s) to geolocation coordinates of the related situation object(s) using geolocation data of the different vehicles associated with the matched images; and determines a coverage area of a traffic situation based on the geolocation coordinates of the related situation object(s).

Claims

1. A method comprising:

receiving situation data including images from vehicles;

clustering, into an image cluster, the images included in the situation data of one or more vehicles located in a geographic region from among the vehicles;

locating one or more related situation objects in one or more images of the image cluster;

matching images from different vehicles in the image cluster, the matched images having one or more corresponding features of the one or more related situation objects;

determining three-dimensional (3D) sensor coordinates of the one or more related situation objects relative to a sensor position of a target vehicle associated with at least one of the matched images, using the one or more corresponding features of the one or more related situation objects in the matched images;

converting the 3D sensor coordinates of the one or more related situation objects to geolocation coordinates of the one or more related situation objects using geolocation data of the different vehicles associated with the matched images; and

determining a coverage area of a traffic situation based on the geolocation coordinates of the one or more related situation objects.

2. The method of claim 1, wherein the matched images from the different vehicles are captured by images sensors having different sensor configurations.

3. The method of claim 1, wherein converting the 3D sensor coordinates of the one or more related situation objects to the geolocation coordinates of the one or more related situation objects includes:

determining geolocation coordinates of sensor positions of the different vehicles associated with the matched images;

determining view angles from the sensor positions of the different vehicles associated with the matched images to a first related situation object among the one or more related situation objects using the 3D sensor coordinates of the first related situation object; and

determining the geolocation coordinates of the first related situation object based on the geolocation coordinates of the sensor positions of the different vehicles and the view angles from the sensor positions of the different vehicles to the first related situation object using triangulation computation.

4. The method of claim 1, wherein determining the coverage area of the traffic situation includes:

positioning the one or more related situation objects on a geographical map associated with the geographic region using the geolocation coordinates of the one or more related situation objects; and

determining a convex geographical area encompassing the one or more related situation objects on the geographical map to be the coverage area of the traffic situation.

5. The method of claim 1, further comprising:

determining a first coverage area of the traffic situation at a first time;

determining a second coverage area of the traffic situation at a second time;

determining a localization difference between the first coverage area and the second coverage area;

determining that the localization difference between the first coverage area and the second coverage area satisfies a difference threshold; and

responsive to determining that the localization difference between the first coverage area and the second coverage area satisfies the difference threshold, updating a geographical map associated with the geographic region to include the second coverage area of the traffic situation.

6. The method of claim 5, wherein determining that the localization difference between the first coverage area and the second coverage area satisfies the difference threshold includes:

determining one or more first lanes associated with the first coverage area;

determining one or more second lanes associated with the second coverage area; and

determining that the one or more second lanes are different from the one or more first lanes.

7. The method of claim 5, further comprising:

determining one or more proximate vehicles associated with the geographic region; and

transmitting a notification of situation update to the one or more proximate vehicles, the notification of situation update including the second coverage area of the traffic situation.

8. The method of claim 1, wherein the geographic region is a predefined road segment.

9. The method of claim 1, wherein clustering the images included in the situation data of the one or more vehicles located in the geographic region includes:

clustering the images based on image timestamps of the images.

10. The method of claim 1, wherein the situation data received from a first vehicle among the vehicles includes one or more of a first image captured by an image sensor of the first vehicle, image data associated with the first image, a sensor configuration of the image sensor, and the geolocation data of the first vehicle associated with the first image.

11. A system comprising:

one or more processors;

one or more memories storing instructions that, when executed by the one or more processors, cause the system to;

receive situation data including images from vehicles;

cluster, into an image cluster, the images included in the situation data of one or more vehicles located in a geographic region from among the vehicles;

locate one or more related situation objects in one or more images of the image cluster;

match images from different vehicles in the image cluster, the matched images having one or more corresponding features of the one or more related situation objects;

determine three-dimensional (3D) sensor coordinates of the one or more related situation objects relative to a sensor position of a target vehicle associated with at least one of the matched images, using the one or more corresponding features of the one or more related situation objects in the matched images;

convert the 3D sensor coordinates of the one or more related situation objects to geolocation coordinates of the one or more related situation objects using geolocation data of the different vehicles associated with the matched images; and

determine a coverage area of a traffic situation based on the geolocation coordinates of the one or more related situation objects.

12. The system of claim 11, wherein the matched images from the different vehicles are captured by images sensors having different sensor configurations.

13. The system of claim 11, wherein to convert the 3D sensor coordinates of the one or more related situation objects to the geolocation coordinates of the one or more related situation objects includes:

determining geolocation coordinates of sensor positions of the different vehicles associated with the matched images;

determining view angles from the sensor positions of the different vehicles associated with the matched images to a first related situation object among the one or more related situation objects using the 3D sensor coordinates of the first related situation object; and

determining the geolocation coordinates of the first related situation object based on the geolocation coordinates of the sensor positions of the different vehicles and the view angles from the sensor positions of the different vehicles to the first related situation object using triangulation computation.

14. The system of claim 11, wherein to determine the coverage area of the traffic situation includes:

positioning the one or more related situation objects on a geographical map associated with the geographic region using the geolocation coordinates of the one or more related situation objects; and

determining a convex geographical area encompassing the one or more related situation objects on the geographical map to be the coverage area of the traffic situation.

15. The system of claim 11, wherein the instructions, when executed by the one or more processors, further cause the system to:

determine a first coverage area of the traffic situation at a first time;

determine a second coverage area of the traffic situation at a second time;

determine a localization difference between the first coverage area and the second coverage area;

determine that the localization difference between the first coverage area and the second coverage area satisfies a difference threshold; and

responsive to determining that the localization difference between the first coverage area and the second coverage area satisfies the difference threshold, update a geographical map associated with the geographic region to include the second coverage area of the traffic situation.

16. The system of claim 15, wherein to determine that the localization difference between the first coverage area and the second coverage area satisfies the difference threshold includes:

determining one or more first lanes associated with the first coverage area;

determining one or more second lanes associated with the second coverage area; and

determining that the one or more second lanes are different from the one or more first lanes.

17. The system of claim 15, wherein the instructions, when executed by the one or more processors, further cause the system to:

determine one or more proximate vehicles associated with the geographic region; and

transmit a notification of situation update to the one or more proximate vehicles, the notification of situation update including the second coverage area of the traffic situation.

18. The system of claim 11, wherein the geographic region is a predefined road segment.

19. The system of claim 11, wherein to cluster the images included in the situation data of the one or more vehicles located in the geographic region includes:

clustering the images based on image timestamps of the images.

20. The system of claim 11, wherein the situation data received from a first vehicle among the vehicles includes one or more of a first image captured by an image sensor of the first vehicle, image data associated with the first image, a sensor configuration of the image sensor, and the geolocation data of the first vehicle associated with the first image.

21. A method comprising:

receiving situation data including images from vehicles;

clustering, into an image cluster, the images included in the situation data of one or more vehicles located in a geographic region from among the vehicles;

locating one or more related situation objects in one or more images of the image cluster;

matching images from different vehicles in the image cluster, the matched images having one or more corresponding features of the one or more related situation objects;

determining three-dimensional (3D) sensor coordinates of the one or more related situation objects relative to a sensor position of a target vehicle associated with at least one of the matched images, using the one or more corresponding features of the one or more related situation objects in the matched images;

converting the 3D sensor coordinates of the one or more related situation objects to geolocation coordinates of the one or more related situation objects using geolocation data of the different vehicles associated with the matched images;

determining a coverage area of a traffic situation based on the geolocation coordinates of the one or more related situation objects; and

determining a navigation path associated with the geographical region based on the coverage area of the traffic situation.