Adaptive mapping to navigate autonomous vehicles responsive to physical environment changes

• US 9,612,123 B1
• Filed: 11/04/2015
• Issued: 04/04/2017
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Levinson, Jesse Sol, Sibley, Gabriel Thurston
• Primary Examiner(s): LIN, ABBY YEE
• Application Number:
  US14/932,963
• Time in Patent Office: 517 Days
• CPC Class Codes: B60L 2240/622 by satellite navigation
• CPC Class Codes: B60L 2260/32 Auto pilot mode
• CPC Class Codes: B60L 3/0015 Prevention of collisions
• CPC Class Codes: B60Q 1/26 the devices being primarily intended to indicate the vehicle, or parts thereof, or to give signals, to other traffic
• CPC Class Codes: B60Q 1/503 using luminous text or symbol displays in or on the vehicle, e.g. static text
• CPC Class Codes: B60Q 1/507 specific to autonomous vehicles
• CPC Class Codes: B60Q 1/508 specific to vehicles driving in fleets or convoys
• CPC Class Codes: B60Q 1/549 for expressing greetings, gratitude or emotions
• CPC Class Codes: B60Q 5/006 indicating risk of collision between vehicles or with pedestrians
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 2021/01272 Belt tensioners
• CPC Class Codes: B60R 21/01 Electrical circuits for triggering passive safety arrangements, e.g. airbags, safety belt tighteners, in case of vehicle accidents or impending vehicle accidents electrical circuits for transmission of signals between steering wheel and the vehicle itself B60R16/027; for electrically actuating belt retractor locking means B60R22/343
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/406 Traffic density
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 60/0011 involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles
• CPC Class Codes: B60W 60/0027 : using trajectory prediction for other traffic participants
• CPC Class Codes: G01C 21/26 : specially adapted for navigation in a road network
• CPC Class Codes: G01C 21/3446 : Details of route searching algorithms, e.g. Dijkstra, A*, arc-flags, using precalculated routes
• CPC Class Codes: G01C 21/3635 : Guidance using 3D or perspective road maps
• CPC Class Codes: G01C 21/3848 : Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3867 : Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G01C 21/3881 : Tile-based structures
• CPC Class Codes: G01S 13/865 : Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 13/87 : Combinations of radar systems, e.g. primary radar and secondary radar
• CPC Class Codes: G01S 17/87 : Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/875 : for determining attitude
• CPC Class Codes: G01S 2013/9316 : combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/9322 : using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 5/0018 : Transmission from mobile station to base station
• CPC Class Codes: G01S 5/16 : using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 7/4972 : Alignment of sensor
• CPC Class Codes: G05D 1/0038 : by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 : by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0248 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0274 : using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G05D 1/0297 : by controlling means in a control room
• CPC Class Codes: G05D 1/2246 : displaying a map of the environment
• CPC Class Codes: G05D 1/2249 : using augmented reality
• CPC Class Codes: G05D 1/226 : Communication links with the remote-control arrangements
• CPC Class Codes: G05D 1/6987 : by centralised control off-board any of the vehicles
• CPC Class Codes: G05D 2105/22 : of humans
• CPC Class Codes: G05D 2107/13 : Spaces reserved for vehicle traffic, e.g. roads, regulated airspace or regulated waters
• CPC Class Codes: G05D 2109/10 : Land vehicles
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/20 : Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles
• CPC Class Codes: G08G 1/205 : Indicating the location of the monitored vehicles as destination, e.g. accidents, stolen, rental
• CPC Class Codes: G10K 11/26 : Sound-focusing or directing, e.g. scanning

Abstract

Various embodiments relate generally to autonomous vehicles and associated mechanical, electrical and electronic hardware, computer software and systems, and wired and wireless network communications to provide map data for autonomous vehicles. In particular, a method may include accessing subsets of multiple types of sensor data, aligning subsets of sensor data relative to a global coordinate system based on the multiple types of sensor data to form aligned sensor data, and generating datasets of three-dimensional map data. The method further includes detecting a change in data relative to at least two datasets of the three-dimensional map data and applying the change in data to form updated three-dimensional map data. The change in data may be representative of a state change of an environment at which the sensor data is sensed. The state change of the environment may be related to the presence or absences of an object located therein.

Claims

1. A method comprising:

receiving, at a computing system, a first type of sensor data from multiple driverless vehicles in a fleet of autonomous driverless vehicles, wherein the first type of sensor data is acquired by a first type of sensors on the driverless vehicles to sense objects in environments encountered while the driverless vehicles are driving along roads;

receiving, at the computing system, a second type of sensor data from the multiple driverless vehicles in the fleet of autonomous driverless vehicles, wherein the second type of sensor data is acquired by a second type of sensors on the driverless vehicles to sense objects in the environments encountered while the driverless vehicles are driving along the roads;

storing, at the computing system, the first type of sensor data and the second type of sensor data;

accessing, by the computing system and for a particular driverless vehicle, subsets of the first type of sensor data and subsets of the second type of sensor data;

aligning, by the computing system and for the particular driverless vehicle, the subsets of the first type of sensor data with the subsets of the second type of sensor data to provide aligned sensor data pertaining to objects in an environment encountered by the particular driverless vehicle;

aligning, by the computing system and for the particular driverless vehicle, the aligned sensor data relative to positioning map data pertaining to a global coordinate system to localize the aligned sensor data to a location of the particular driverless vehicle;

generating, by the computing system, data sets of three-dimensional map data based on the aligned sensor data and the positioning map data;

detecting a change between the generated data sets of three-dimensional map data and a stored data set of three-dimensional map data, the change being representative of a state change in the environment surrounding the particular driverless vehicle;

updating, at the computing system, the stored data set of three-dimensional map data to generate an updated data set of three-dimensional map data reflecting the state change in the environment; and

transmitting the updated data set of three-dimensional map data to the multiple driverless vehicles in the fleet of autonomous driverless vehicles,

wherein at least one of the driverless vehicles is controlled based at least in part on the updated data set of three-dimensional map data.

2. The method of claim 1, wherein updating, at the computing system, the stored data set of three-dimensional map data to reflect the state change further comprises:

associating the change between at least two data sets to a map portion;

forming an updated map portion to include the change; and

incorporating the updated map portion into a reference data store in the particular driverless vehicle,

wherein implementation of the updated map portion for localization of the particular driverless vehicle invokes a teleoperator request.

3. The method of claim 1, further comprising:

localizing the particular driverless vehicle relative to a coordinate of the global coordinate system associated with the stored data set of three-dimensional map data;

detecting that data representing one or more map portions of the stored data set of three-dimensional map data vary from a local map maintained at the particular driverless vehicle; and

generating a teleoperator request.

4. The method of claim 1, further comprising:

localizing the particular driverless vehicle relative to a coordinate of the global coordinate system associated with the stored data set of three-dimensional map data; and

detecting that data representing one or more map portions of the stored data set of three-dimensional map data vary from a local map maintained at the particular driverless vehicle.

5. The method of claim 4, further comprising localizing the particular driverless vehicle against hybrid map data from the local map and the stored data set of three-dimensional map data.

6. The method of claim 1, wherein accessing subsets of the first type of sensor data and subsets of the second type of sensor data comprises receiving subsets of laser return data and receiving subsets of image data, respectively, wherein each subset of laser return data is associated with a Lidar sensor and each subset of image data is associated with an image capture device.

7. The method of claim 6, further comprising receiving subsets of radar data into a data store originating from a group of radar sensors, wherein each subset radar data is associated with a radar sensor.

8. The method of claim 6, wherein aligning the subsets of the first type of sensor data with the subsets of the second type of sensor data and aligning the aligned sensor data comprises:

forming a global pose graph relative to the global coordinate system based on the subsets of the first type of sensor data and the subsets of the second type of sensor data; and

aligning the subsets of laser return data and the subsets of image data relative to a location relative to a coordinate of the global coordinate system.

9. The method of claim 8, wherein aligning the subsets of laser return data and the subsets of image data relative to a location relative to a coordinate of the global coordinate system comprises receiving trajectory data representing position data to identify the coordinate of the global coordinate system.

10. The method of claim 8, wherein forming the global pose graph comprises performing one or more loop-closure processes to identify a closed loop.

11. The method of claim 6, wherein generating the data sets of three-dimensional map data further comprises integrating at least two types of sensor data including the subsets of laser return data and the subsets of image data.

12. The method of claim 11, wherein integrating the at least two types of sensor data comprises fusing the subsets of laser return data and the subsets of image data to form the generated data sets of three-dimensional map data to include three-dimensional tile data.

13. The method of claim 11, wherein integrating the at least two types of sensor data comprises correlating pixel data of the subsets of image data to laser data of the subsets of laser return data.

14. The method of claim 13, further comprising associating the pixel data of one or more pixels to the laser data of one or more laser returns, the laser data being associated with a portion of a surface in three-dimensional tile data, wherein the pixel data specifies one or more surface characteristics including texture, color, reflectivity, or transparency.

15. The method of claim 1, wherein:

receiving the first type of sensor data and receiving the second type of sensor data comprises receiving the first and second types of sensor data via a network; and

logging the first type of sensor data and the second type of sensor data into a database.

16. The method of claim 4, further comprising:

characterizing a difference between the one or more map portions of the stored data set of three-dimensional map data and the local map; and

using at least part of the local map and at least part of the one or more map portions to update the stored data set of three-dimensional map data.

17. The method of claim 1, further comprising:

exporting the updated data set of three-dimensional map data to a simulator computing device; and

simulating performance of a portion of the fleet of autonomous vehicles in a simulated environment based on the updated data set of three-dimensional map data.

18. A method comprising:

receiving, at a computing system from individual autonomous vehicles in a fleet of autonomous vehicles, multiple types of sensor data indicative of driving conditions and objects present in environments surrounding the autonomous vehicles in the fleet;

storing the multiple types of sensor data in a data store;

accessing, from the data store, subsets of a first type of sensor data received from a first autonomous vehicle in a first location, the subsets of the first type of sensor data originating from one or more first sensors on the first autonomous vehicle;

accessing, from the data store, subsets of a second type of sensor data received from the first autonomous vehicle in the first location, the subsets of the second type of sensor data originating from one or more second sensors on the first autonomous vehicle;

aligning the subsets of the first type of sensor data and the subsets of the second type of sensor data to form aligned sensor data;

determining, at the computing system, whether the aligned sensor data manifests a variation between a stored map of the first location and an environment surrounding the first autonomous vehicle as sensed while the vehicle is in the first location;

updating the stored map to reflect the variation; and

transmitting the updated map from the computing system to the individual autonomous vehicles in a fleet of autonomous vehicles,

wherein at least one of the autonomous vehicles is controlled based at least in part on the updated map.

19. The method of claim 18, wherein determining whether the aligned sensor data manifests a variation comprises:

localizing the first autonomous vehicle relative to a global coordinate system associated with map data; and

detecting a difference between one or more map portions of the map data and a local map maintained at the first autonomous vehicle.

20. The method of claim 19, further comprising:

characterizing the difference between the one or more map portions of the map data and the local map; and

using at least part of the local map and at least part of the one or more map portions to update the stored map.

21. The method of claim 18, further comprising:

exporting the updated map to a simulator computing device; and

simulating, at the simulator computing device, performance of a portion of the fleet of autonomous vehicles in a simulated environment based on the updated map.

22. The method of claim 18, further comprising aligning the aligned sensor data relative to a global coordinate system.