Autonomous vehicle localization using passive image data

• US 10,474,162 B2
• Filed: 06/30/2017
• Issued: 11/12/2019
• Est. Priority Date: 07/01/2016
• Status: Active Grant

• Current Assignee: Amgen Inc., Aurora Operations Incorporated (Aurora Innovation, Inc.)
• Original Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Inventors: Browning, Brett, Hansen, Peter, Eade, Ethan, Prasser, David, LaRose, David, Zlot, Robert
• Primary Examiner(s): Harandi, Siamak
• Application Number:
  US15/640,334
• Publication Number:
  US 20180005407A1
• Time in Patent Office: 865 Days
• CPC Class Codes: B60W 30/095 Predicting travel path or likelihood of collision
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3811 Point data, e.g. Point of Interest [POI]
• CPC Class Codes: G01C 21/3815 Road data
• CPC Class Codes: G01C 21/3819 Road shape data, e.g. outline of a route
• CPC Class Codes: G01C 21/3822 Road feature data, e.g. slope data
• CPC Class Codes: G01C 21/3841 Data obtained from two or more sources, e.g. probe vehicles
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G01C 21/3878 Hierarchical structures, e.g. layering
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0276 using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/33 : using feature-based methods
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/52 : specially adapted for the location of the user terminal

Abstract

A system to use submaps to control operation of a vehicle is disclosed. A storage system may be provided with a vehicle to store a collection of submaps that represent a geographic area where the vehicle may be driven. A programmatic interface may be provided to receive submaps and submap updates independently of other submaps.

Claims

1. A method for performing localization for a vehicle in motion, the method being implemented by one or more processors and comprising:

obtaining a set of sensor data captured by the vehicle at a given instance as the vehicle traverses a given area of a road network, the sensor data including passive image data;

selecting a feature from a known feature set for the given area of the road network based on one or more conditions that affect an appearance of individual features of the known feature set;

matching an image subset of the passive image data to the feature from the known feature set for the given area of the road network; and

determining a location of the vehicle within the given area based on reference location information previously associated with the known feature set.

2. The method of claim 1, wherein matching the image subset includes detecting an object depicted by the image subset as corresponding to the feature of the known feature set.

3. The method of claim 2, wherein determining the location is based on determining a difference between a size of the object depicted by the image subset and a dimension associated with the feature when the feature is perceived from a given location.

4. The method of claim 2, wherein determining the location is based on determining a difference in perspective between the object depicted by the image subset and a perspective associated with the feature when the feature is perceived from a given location.

5. The method of claim 2, wherein matching the image subset is based on a manipulation of the image subset so that the object depicted by the image subset more closely matches the feature of the known feature set.

6. The method of claim 1, further comprising determining an orientation of the vehicle relative to a direction of travel as the vehicle progresses through the given area.

7. The method of claim 1, wherein the one or more conditions are based on at least one of weather, season, or time of day.

8. The method of claim 7, wherein the one or more conditions include at least one of (i) a presence of precipitation, or (ii) a presence of snow.

9. The method of claim 1, further comprising identifying a geometry of individual features in the set of features, and wherein selecting the feature is based at least in part on the geometry of the feature.

10. The method of claim 9, wherein selecting the feature includes selecting features that have a vertical geometry when the one or more conditions correspond to a presence of precipitation or snow.

11. The method of claim 1, further comprising:

determining a submap from a collection of submaps that corresponds to the area of the road network where the vehicle is located, wherein the submap includes one or more data layers which include the known feature set.

12. The method of claim 11, wherein the one or more data layers provide a three-dimensional cloud of points, in which individual points of the cloud are associated with imagelets representing one or more features of the known feature set.

13. The method of claim 12, wherein individual points of the cloud of points are associated with reference information indicating at least one of a location or perspective of a sensor device used to capture an associated imagelet at a previous instance in time.

14. The method of claim 12, further comprising selecting a set of points with associated imagelets from a larger number of points of the three-dimensional cloud of points with associated imagelets based on information associated with individual points in the larger number of points, the associated imagelets of the selected set of points corresponding to the known feature set, and wherein the information associated with individual points in the larger number of points include at least one of a semantic label, geometric information and/or contextual information.

15. A method for performing localization for a vehicle in motion, the method being implemented by one or more processors and comprising:

obtaining a set of sensor data captured by the vehicle at a given instance as the vehicle traverses a given area of a road network, the sensor data including passive image data;

matching an image subset of the passive image data to a feature from a known feature set for the given area of the road network, wherein matching the image subset includes implementing one or more rules based on a presence of one or more conditions that affect an appearance of individual features of the known feature set; and

determining a location of the vehicle within the given area based on reference location information previously associated with the known feature set.

16. The method of claim 15, wherein implementing the one or more rules includes avoiding using white pixels of the passive image set for the image subset when snow is present in the given area.

17. The method of claim 15, wherein implementing the one or more rules includes avoiding using image data of the passive image set which is below a horizon of a camera of the vehicle used to capture the passive image set when the snow is present in the given area.

18. A computer system comprising:

a memory to store a set of instructions;

one or more processors to use the set of instructions to;

obtain a set of sensor data captured by a vehicle at a given instance as the vehicle traverses a given area of a road network, the sensor data including passive image data;

select a feature from a known feature set for the given area of the road network based on one or more conditions that affect an appearance of individual features of the known feature set;

match an image subset of the passive image data to the feature from the known feature set for the given area of the road network; and

determines a location of the vehicle within the given area based on reference location information previously associated with the known feature set.

19. The computer system of claim 18, wherein the computer system is provided on an autonomous vehicle.

20. The computer system of claim 18, wherein the computer system communicates with an autonomous vehicle over a network.

21. A computer system comprising:

a memory to store a set of instructions;

one or more processors to use the set of instructions to;

obtain a set of sensor data captured by a vehicle at a given instance as the vehicle traverses a given area of a road network, the sensor data including passive image data;

match an image subset of the passive image data to the feature from a known feature set for the given area of the road network;

determine a location of the vehicle within the given area based on reference location information previously associated with the known feature set;

determine a submap from a collection of submaps that corresponds to the area of the road network where the vehicle is located, wherein the submap includes one or more data layers which include the known feature set, wherein the one or more data layers provide a three-dimensional pointcloud, in which individual points of the pointcloud are associated with imagelets representing one or more features of the known feature set; and

select a set of points with associated imagelets from a larger number of points of the pointcloud with associated imagelets based on information associated with individual points in the larger number of points, the associated imagelets of the selected set of points corresponding to the known feature set, and wherein the information associated with individual points in the larger number of points include at least one of a semantic label, geometric information and/or contextual information.