Autonomous vehicle localization using image analysis and manipulation

• US 10,678,262 B2
• Filed: 06/30/2017
• Issued: 06/09/2020
• Est. Priority Date: 07/01/2016
• Status: Active Grant

• Current Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Original Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Inventors: Browning, Brett, Hansen, Peter, Eade, Ethan, Prasser, David, LaRose, David, Zlot, Robert
• Primary Examiner(s): Khatib, Rami
• Application Number:
  US15/640,340
• Publication Number:
  US 20180005050A1
• Time in Patent Office: 1,075 Days
• Field of Search: 701445
• CPC Class Codes: B60W 30/095 Predicting travel path or likelihood of collision
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3811 Point data, e.g. Point of Interest [POI]
• CPC Class Codes: G01C 21/3815 Road data
• CPC Class Codes: G01C 21/3819 Road shape data, e.g. outline of a route
• CPC Class Codes: G01C 21/3822 Road feature data, e.g. slope data
• CPC Class Codes: G01C 21/3841 Data obtained from two or more sources, e.g. probe vehicles
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G01C 21/3878 Hierarchical structures, e.g. layering
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0276 using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/33 : using feature-based methods
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/52 : specially adapted for the location of the user terminal

Abstract

A system to use submaps to control operation of a vehicle is disclosed. A storage system may be provided with a vehicle to store a collection of submaps that represent a geographic area where the vehicle may be driven. A programmatic interface may be provided to receive submaps and submap updates independently of other submaps.

Claims

1. A method for performing localization for a vehicle in motion, the method being implemented by one or more processors and comprising:

accessing a collection of imagelets for a geographic region that encompasses multiple predefined areas of a road network, each imagelet depicting at least one feature present in a corresponding area of the multiple predefined areas, wherein each imagelet is stored with reference location information identifying a reference location of a prior sensor device determined when the prior sensor device captured prior sensor data for the imagelet;

obtaining current sensor data from one or more sensor devices of the vehicle, the current sensor data including (i) image data captured by one or more cameras of the vehicle, (ii) data that identifies a current location of the vehicle, and (iii) data that identifies current environmental conditions;

selecting a set of imagelets from the collection of imagelets based at least in part on the current location of the vehicle;

for at least one imagelet in the selected set of imagelets, upon detecting a disparity between the current environmental conditions and prior environmental conditions that were present when the prior sensor data for the at least one imagelet was captured, applying a transformation to at least one of the image data or the at least one imagelet to account for the disparity;

matching, to the image data, one of the features depicted in the at least one imagelet of the selected set of imagelets; and

updating the current location for a control system of the vehicle based at least in part on the reference location of the prior sensor device that captured the prior sensor data for the at least one imagelet.

2. The method of claim 1, wherein accessing the collection of imagelets includes accessing a collection of submaps for the geographic region, each submap corresponding to an area of the geographic region, wherein each of the submaps of the collection include one or more of the imagelets of the collection of imagelets.

3. The method of claim 2, wherein each submap includes at least one data layer that provides a pointcloud of imagelets captured by a sensor device that previously traversed the area of that submap.

4. The method of claim 2, wherein selecting the set of one or more imagelets includes selecting a subset of imagelets provided with one or more pointclouds of imagelets in an individual submap of the collection of submaps.

5. The method of claim 1, wherein the current environmental conditions and the prior environmental conditions include lighting conditions.

6. The method of claim 1, wherein the current environmental conditions and the prior environmental conditions include variance in a time of day, a day of season, or weather condition.

7. The method of claim 1, wherein applying the transformation includes determining a model for the transformation and applying the transformation based on the model, wherein the model is trained on sets of prior sensor data previously collected from one or more vehicles that obtained the sets of prior sensor data at the current location of the vehicle.

8. The method of claim 1, wherein applying the transformation includes determining a model for the transformation and applying the transformation based on the model, wherein the model is trained on sets of prior sensor data previously collected from one or more vehicles that obtained the sets of prior sensor data on a road or sub-region that contains the current location of the vehicle.

9. The method of claim 1, wherein applying the transformation includes determining a model for the transformation and applying the transformation based on the model, wherein the model is trained on sets of prior sensor data previously collected from one or more vehicles that obtained the sets of prior sensor data on a different road than the road that includes the current location of the vehicle.

10. The method of claim 9, wherein the different road is intersecting or adjacent to the road that includes the current location of the vehicle.

11. The method of claim 1, further comprising determining an orientation of the vehicle relative to a direction of travel as the vehicle progresses through a given area.

12. The method of claim 1, wherein updating the current location of the vehicle includes determining a lane or lateral position of the vehicle on a road segment.

13. A computer system comprising:

a memory to store a set of instructions;

one or more processors to use the set of instructions to;

access a collection of imagelets for a geographic region that encompasses multiple predefined areas of a road network, each imagelet depicting at least one feature present in a corresponding area of the multiple predefined areas, wherein each imagelet is stored with reference location information identifying a reference location of a prior sensor device determined when the prior sensor device captured prior sensor data for the imagelet;

obtain current sensor data from one or more sensor devices of a vehicle, the current sensor data including (i) image data captured by one or more cameras of the vehicle, (ii) data that identifies a current location of the vehicle, and (iii) data that identifies current environmental conditions;

select a set of imagelets from the collection of imagelets based at least in part on the current location of the vehicle;

for at least one imagelet in the selected set of imagelets, upon detecting a disparity between the current environmental conditions and prior environmental conditions that were present when the prior sensor data for the at least one imagelet was captured, apply a transformation to at least one of the image data or the at least one imagelet to account for the disparity;

match, to the image data, one of the features depicted in the at least one imagelet of the selected set of imagelets; and

update the current location for a control system of the vehicle based at least in part on the reference location of the prior sensor device that captured the prior sensor data for the at least one imagelet.

14. The computer system of claim 13, wherein the vehicle is an autonomous vehicle, and the computer system is provided on the autonomous vehicle.

15. The computer system of claim 13, wherein the vehicle is an autonomous vehicle, and the computer system communicates with the autonomous vehicle over a network.