Inferring state of traffic signal and other aspects of a vehicle's environment based on surrogate data

• US 8,793,046 B2
• Filed: 06/01/2012
• Issued: 07/29/2014
• Est. Priority Date: 06/01/2012
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lombrozo, Peter, Teller, Eric, Templeton, Bradley
• Primary Examiner(s): Zanelli, Michael J
• Application Number:
  US13/486,886
• Publication Number:
  US 20130325241A1
• Time in Patent Office: 788 Days
• Field of Search: None
• US Class Current: 701/23
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/406 Traffic density
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 30/18159 Traversing an intersection
• CPC Class Codes: B60W 40/00 Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
• CPC Class Codes: B60W 60/001 Planning or execution of driving tasks
• CPC Class Codes: B60W 60/0017 of other traffic participants
• CPC Class Codes: B60W 60/0027 using trajectory prediction for other traffic participants
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G01C 21/30 Map- or contour-matching
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 15/88 Sonar systems specially adapted for specific applications seismic or acoustic prospecting or detecting G01V1/00
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

A vehicle configured to operate in an autonomous mode can obtain sensor data from one or more sensors observing one or more aspects of an environment of the vehicle. At least one aspect of the environment of the vehicle that is not observed by the one or more sensors could be inferred based on the sensor data. The vehicle could be controlled in the autonomous mode based on the at least one inferred aspect of the environment of the vehicle.

Claims

1. A method, comprising:

obtaining sensor data from one or more sensors observing one or more aspects of an environment of a vehicle, wherein the one or more sensors are operationally associated with the vehicle, and wherein the vehicle is configured to operate in an autonomous mode;

using an inference system to infer, based on the sensor data, at least one aspect of the environment of the vehicle that is not observed by the one or more sensors;

determining a confidence level based on the at least one inferred aspect of the environment of the vehicle, at least one previous inference, and at least one previous outcome; and

controlling the vehicle in the autonomous mode based on the at least one inferred aspect of the environment of the vehicle and the confidence level.

2. The method of claim 1, wherein the one or more sensors comprises at least two sensors, and wherein the at least one inferred aspect of the environment of the vehicle is based on sensor data from the at least two sensors.

3. The method of claim 1, wherein obtaining sensor data comprises using at least one of a camera, a radar system, an acoustic-sensing system, an ultrasonic-sensing system, and a lidar system.

4. The method of claim 1, wherein the at least one aspect of the environment of the vehicle that is not observed by the one or more sensors comprises at least one of a traffic light, a speed limit, a road condition, an obstacle, a behavior of another vehicle, and a road route.

5. The method of claim 1, further comprising transmitting the at least one inferred aspect of the environment of the vehicle and the confidence level to at least one other vehicle, wherein the at least one other vehicle is configured to operate in an autonomous mode.

6. The method of claim 1, wherein using an inference system comprises using at least one of a Bayesian network, a hidden Markov model, and a decision tree.

7. The method of claim 1, wherein controlling the vehicle comprises at least one of controlling the vehicle to accelerate, controlling the vehicle to decelerate, controlling the vehicle to change heading, controlling the vehicle to change lanes, controlling the vehicle to shift within a current lane and controlling the vehicle to provide a warning notification.

8. The method of claim 7, wherein the warning notification comprises at least one of a horn signal, a light signal, and a vehicle-to-vehicle communication message.

9. A vehicle, comprising:

one or more sensors, wherein the one or more sensors are configured to acquire sensor data, wherein the sensor data relates to one or more aspects of an environment of a vehicle observed by the one or more sensors, and wherein the vehicle is configured to operate in an autonomous mode;

an inference system, wherein the inference system is configured to infer, based on the sensor data, at least one aspect of the environment of the vehicle that is not observed by the one or more sensors, wherein the inference system is further configured to determine a confidence level based on the at least one inferred aspect of the environment of the vehicle, at least one previous inference, and at least one previous outcome; and

a control system, wherein the control system is configured to control the vehicle in the autonomous mode based on the at least one inferred aspect of the environment of the vehicle and the confidence level.

10. The vehicle of claim 9, wherein the one or more sensors comprise at least one of a camera, a radar system, an acoustic-sensing system, an ultrasonic-sensing system, and a lidar system.

11. The vehicle of claim 9, wherein the at least one aspect of the environment of the vehicle that is not observed by the one or more sensors comprises at least one of a traffic light, a speed limit, a road condition, an obstacle, a behavior of another vehicle, and a road route.

12. The vehicle of claim 9, wherein the inference system is further configured to transmit the at least one inferred aspect of the environment of the vehicle and the confidence level to at least one other vehicle, wherein the at least one other vehicle is configured to operate in an autonomous mode.

13. The vehicle of claim 9, wherein the inference system is configured to use at least one of a Bayesian network, a hidden Markov model, and a decision tree.

14. The vehicle of claim 9, wherein the control system is further configured to perform at least one of controlling the vehicle to accelerate, controlling the vehicle to decelerate, controlling the vehicle to change heading, controlling the vehicle to change lanes, controlling the vehicle to shift within a current lane and controlling the vehicle to provide a warning notification.

15. The vehicle of claim 14, wherein the warning notification comprises at least one of a horn signal, a light signal, and a vehicle-to-vehicle communication message.

16. A non-transitory computer readable medium having stored therein instructions executable by a computer system to cause the computer system to perform functions, the functions comprising:

obtaining sensor data from one or more sensors observing one or more aspects of an environment of a vehicle, wherein the one or more sensors are operationally associated with the vehicle, and wherein the vehicle is configured to operate in an autonomous mode;

inferring, based on the sensor data, at least one aspect of the environment of the vehicle that is not observed by the one or more sensors;

determining a confidence level based on the at least one inferred aspect of the environment of the vehicle, at least one previous inference, and at least one previous outcome; and

controlling the vehicle in the autonomous mode based on the at least one inferred aspect of the environment of the vehicle and the confidence level.

17. The non-transitory computer readable medium of claim 16, wherein obtaining sensor data comprises using at least one of a camera, a radar system, an acoustic-sensing system, an ultrasonic-sensing system, and a lidar system.

18. The non-transitory computer readable medium of claim 16, wherein the at least one aspect of the environment of the vehicle that is not observed by the one or more sensors comprises at least one of a traffic light, a speed limit, a road condition, an obstacle, a behavior of another vehicle, and a road route.

19. The non-transitory computer readable medium of claim 16, wherein the functions further comprise transmitting the at least one inferred aspect of the environment of the vehicle and the confidence level to at least one other vehicle, wherein the at least one other vehicle is configured to operate in an autonomous mode.

20. The non-transitory computer readable medium of claim 16, wherein inferring at least one aspect of the environment of the vehicle that is not observed by the one or more sensors comprises using at least one of a Bayesian network, a hidden Markov model, and a decision tree.

21. The non-transitory computer readable medium of claim 16, wherein controlling the vehicle comprises at least one of controlling the vehicle to accelerate, controlling the vehicle to decelerate, controlling the vehicle to change heading, controlling the vehicle to change lanes, controlling the vehicle to shift within a current lane and controlling the vehicle to provide a warning notification.

22. The non-transitory computer readable medium of claim 21, wherein the warning notification comprises at least one of a horn signal, a light signal, and a vehicle-to-vehicle communication message.

23. A method, comprising:

obtaining sensor data from one or more sensors observing at least one light source in an environment of a vehicle, wherein the one or more sensors are operationally associated with the vehicle, wherein a state of a controlling traffic signal is not directly observable using the one or more sensors, wherein the vehicle is configured to operate in an autonomous mode;

using an inference system to infer, based on the sensor data, an inferred state of the controlling traffic signal; and

controlling the vehicle in the autonomous mode based on the inferred state of the controlling traffic signal.

24. The method of claim 23, wherein the at least one light source comprises at least one brake light of another vehicle.

25. The method of claim 23, wherein obtaining sensor data comprises using at least one of a camera, a radar system, an acoustic-sensing system, an ultrasonic-sensing system, and a lidar system.