Automated extraction of semantic information to enhance incremental mapping modifications for robotic vehicles

• US 9,734,455 B2
• Filed: 11/04/2015
• Issued: 08/15/2017
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Levinson, Jesse Sol, Sibley, Gabriel Thurston, Rege, Ashutosh Gajanan
• Primary Examiner(s): Pulliam, Christyann
• Assistant Examiner(s): Tran, Mai T
• Application Number:
  US14/932,940
• Publication Number:
  US 20170124476A1
• Time in Patent Office: 650 Days
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4026 Antenna boresight
• CPC Class Codes: G01S 7/4802 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Systems, methods and apparatus may be configured to implement automatic semantic classification of a detected object(s) disposed in a region of an environment external to an autonomous vehicle. The automatic semantic classification may include analyzing over a time period, patterns in a predicted behavior of the detected object(s) to infer a semantic classification of the detected object(s). Analysis may include processing of sensor data from the autonomous vehicle to generate heat maps indicative of a location of the detected object(s) in the region during the time period. Probabilistic statistical analysis may be applied to the sensor data to determine a confidence level in the inferred semantic classification. The inferred semantic classification may be applied to the detected object(s) when the confidence level exceeds a predetermined threshold value (e.g., greater than 50%).

Claims

1. A method, comprising:

receiving, at a computing system, first data sensed at a first time by a sensor system of an autonomous vehicle, the first data being representative of a first object of a plurality of objects on or proximate a road surface in a region in an environment the autonomous vehicle has autonomously navigated;

comparing the first data with reference data associated with a plurality of reference semantic classifications;

based at least in part on the comparing, determining that the first object does not match any of the plurality of reference semantic classifications;

identifying additional objects having additional object data similar to the first data, the first object and the additional objects comprising a subset of the objects;

receiving, at the computing system, second data sensed subsequent to the first time, the second data representing a behavior of at least one of the additional objects;

based at least in part on determining that the first object does not match any of the plurality of reference semantic classifications, determining, based on the first data and the second data, a probability that the subset of the objects conforms to a behavior;

generating, at the computing system, an inferred semantic classification associated with the subset of the objects when the probability indicates a pattern of objects in the subset of the objects conforming to the behavior;

associating the inferred semantic classification with the plurality of reference semantic classifications, the inferred semantic classification being different from each of the plurality of reference semantic classifications;

updating, at the computing system, map data associated with the environment to include information about the inferred semantic classification; and

transmitting the updated map data to the autonomous vehicle and at least one additional autonomous vehicle.

2. The method of claim 1, wherein the determining the probability includes analyzing a heat map generated from sensor data included in at least one of the first data or the second.

3. The method of claim 1, wherein the objects in the subset of the objects includes data representing a pedestrian object type.

4. The method of claim 3, wherein the inferred semantic classification associated with the subset of the objects comprises an un-marked pedestrian crossing associated with a road network in the region.

5. The method of claim 1, wherein the first data comprises route data.

6. The method of claim 5, wherein the route data comprises a route network definition file.

7. The method of claim 1, wherein the first data comprises map data.

8. The method of claim 7, wherein the map data comprises at least one map tile.

9. The method of claim 1 further comprising:

updating the reference semantic classifications to include the inferred semantic classification.

10. The method of claim 1, wherein the determining the probability comprises analyzing data representing heat maps simultaneously generated from frames of sensor data included in the first data from the autonomous vehicle, the frames of sensor data being generated by a plurality of different types of sensors in the sensor system.

11. The method of claim 10, wherein the plurality of different types of sensors includes a light detection and ranging sensor and the frames of sensor data generated by the light detection and ranging sensor include data representing laser depth information associated with the subset of the objects.

12. The method of claim 10, wherein the plurality of different types of sensors includes a multispectral image capture sensor and the frames of sensor data generated by the multispectral image capture sensor include data representing near infrared wavelengths of light associated with the subset of the objects.

13. The method of claim 10, wherein the plurality of different types of sensors includes an image capture sensor and the frames of sensor data generated by the image capture sensor include data representing color intensity of light associated with the subset of the objects.

14. The method of claim 1, wherein the determining the probability comprises analyzing map data based on simultaneous localization and mapping, the map data being generated by a localizer of the autonomous vehicle, the map data including temporal information associated with the period of time.

15. The method of claim 14, wherein the generating the data comprising the inferred semantic classification is based on the temporal information.

16. A system comprising:

a bi-directional autonomous vehicle configured to drive forward in a first direction or drive forward in a substantially opposite second direction without turning around the bi-directional autonomous vehicle, the autonomous vehicle configured to drive autonomously on a roadway;

a plurality of sensors on the bi-directional autonomous vehicle configured to sense a plurality of objects on or proximate a roadway in an environment surrounding the bi-directional autonomous vehicle; and

a computing system communicatively coupled to the bi-directional autonomous vehicle to receive data from the plurality of sensors, the computing system being programmed to;

determine first data for a first object of the plurality of objects, the first object being a moving object at a location in the environment;

compare the first data to reference semantic classifications data indicating object types and object behaviors at locations in the environment, the reference semantic classifications data being associated with one or more reference semantic classifications;

determine, based on the comparison, that the first object does not conform to any of the one or more reference semantic classifications;

determine, based at least in part on data acquired at different times, a pattern of behavior of additional moving objects at the location, each of the additional moving objects at the location having additional moving object data similar to the first data;

based at least in part on the determination that the first object does not conform to any of the one or more reference semantic classifications and based at least in part on the pattern of behavior of the additional moving objects at the location, associate an inferred semantic classification with moving objects at the location;

update the reference semantic classifications data to include the inferred semantic classification as an additional reference semantic classification; and

update route data used to navigate the roadway based on the inferred semantic classification.

17. The system of claim 16, wherein the moving object is a pedestrian, the additional moving objects are additional pedestrians, and the route data is updated to indicate an un-marked pedestrian crossing the roadway.

18. The system of claim 17, wherein the system is further programmed to generate navigation instructions responsive to the updated route data.

19. The system of claim 18, wherein the navigation instructions include instructions to navigate the un-marked pedestrian crossing according to predetermined instructions for navigating a marked pedestrian crossing.

20. The system of claim 16, wherein the location in the environment comprises a position proximate a stationary object of interest.

21. The system of claim 16, wherein the computing system is further programmed to transmit the updated route data to a plurality of autonomous vehicles comprising a fleet of autonomous vehicles.

22. The system of claim 16, wherein the pattern of behavior comprises temporal information including at least one of a time of day or a day of the week.

23. The system of claim 16, wherein the computing system determines the pattern of behavior of additional moving objects at the location by determining a probability that the additional moving objects at the location conform to the behavior and determining that the probability indicates a pattern of the additional moving objects conforming to the behavior.

24. A system comprising:

a fleet of bi-directional autonomous vehicles, each of the bi-directional autonomous vehicles configured to drive forward in a first direction or drive forward in a substantially opposite second direction without turning around the autonomous vehicle;

a plurality of sensors on the bi-directional autonomous vehicles configured to sense a plurality of objects on or proximate a roadway upon which the bi-directional autonomous vehicles travel, in an environment surrounding the bi-directional autonomous vehicles;

one or more data stores storing a reference semantic classification, the reference semantic classification having reference data including data indicating an object type, data indicating an object behavior, and data indicating a location in the environment;

one or more data stores storing mapping information used by the fleet of bi-directional autonomous vehicles to navigate the environment; and

a computing system communicatively coupled to the fleet of bi-directional autonomous vehicles to receive data from the plurality of sensors, the computing system being programmed to;

determine first object data associated with a first of the plurality of objects, the first object data including data indicating a first object type, data indicating a first object behavior, and data indicating a first object location;

compare the first object data to the reference data;

determine, based on the comparison, a difference between the reference data and the first object data, the difference comprising at least one of a difference between the data indicating the object type and the data indicating the first object type, a difference between the data indicating the object behavior and the data indicating the first object behavior, or a difference between the data indicating the location and the data indicating the first object location;

determine, based at least in part on first sensor data from a first of the bi-directional autonomous vehicles in the fleet of bi-directional autonomous vehicles and at least in part on second sensor data from a second of the bi-directional autonomous vehicles in the fleet of bi-directional autonomous vehicles, a pattern of behavior of additional moving objects at the first object location;

based at least in part on determining the difference between the reference data and the first object data and based at least in part on the pattern of behavior, associate an inferred semantic classification with the first object location or with objects at the first object location; and

update the mapping information using the inferred semantic classification to indicate an object at the location.

25. The system of claim 24, wherein the system is further programmed to generate navigation instructions responsive to the updated mapping information.

26. The system of claim 25, wherein the navigation instructions include instructions to navigate the location according to predetermined instructions for navigating a stationary object.

27. The system of claim 24, wherein the location in the environment comprises a position proximate a stationary object of interest.

28. The system of claim 24, wherein the computing system is further programmed to transmit the updated mapping data to the fleet of autonomous vehicles.

29. The system of claim 24, wherein the pattern of behavior comprises temporal information including at least one of a time of day or a day of the week.

30. The system of claim 24, wherein the computing system determines the pattern of behavior by determining a probability that the additional moving objects at the location conform to the behavior and determining that the probability indicates a pattern of the additional moving objects conforming to the behavior.