Inferring state of traffic signal and other aspects of a vehicle's environment based on surrogate data

• US 10,331,133 B2
• Filed: 10/04/2017
• Issued: 06/25/2019
• Est. Priority Date: 06/01/2012
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Lombrozo, Peter, Teller, Eric, Templeton, Bradley
• Primary Examiner(s): Zanelli, Michael J
• Application Number:
  US15/724,428
• Publication Number:
  US 20180032076A1
• Time in Patent Office: 629 Days
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/406 Traffic density
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 30/18159 Traversing an intersection
• CPC Class Codes: B60W 40/00 Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
• CPC Class Codes: B60W 60/001 Planning or execution of driving tasks
• CPC Class Codes: B60W 60/0017 of other traffic participants
• CPC Class Codes: B60W 60/0027 using trajectory prediction for other traffic participants
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G01C 21/30 Map- or contour-matching
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 15/88 Sonar systems specially adapted for specific applications seismic or acoustic prospecting or detecting G01V1/00
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

A vehicle configured to operate in an autonomous mode can obtain sensor data from one or more sensors observing one or more aspects of an environment of the vehicle. At least one aspect of the environment of the vehicle that is not observed by the one or more sensors could be inferred based on the sensor data. The vehicle could be controlled in the autonomous mode based on the at least one inferred aspect of the environment of the vehicle.

Claims

1. A method, comprising:

obtaining, using one or more sensors, first sensor data regarding an environment of a vehicle and second sensor data regarding the environment of the vehicle, wherein the one or more sensors are operationally associated with the vehicle, wherein the vehicle is configured to operate in an autonomous mode, wherein the first sensor data is data regarding an aspect of the environment of the vehicle, and wherein the second sensor data is data regarding one or more other aspects of the environment of the vehicle;

making a determination whether the first sensor data and the second sensor data agree regarding the aspect of the environment of the vehicle, wherein making the determination whether the first sensor data and the second sensor data agree regarding the aspect of the environment of the vehicle, comprises inferring the aspect of the environment of the vehicle based on the second sensor data and determining whether the first sensor data regarding the aspect of the environment of the vehicle agrees with the inferred aspect of the environment of the vehicle; and

controlling the vehicle in the autonomous mode based on the determination.

2. The method of claim 1, wherein the first sensor data is image data regarding the aspect of the environment of the vehicle.

3. The method of claim 1, wherein the aspect of the environment is a state of a traffic signal determined from the first sensor data, wherein the one or more other aspects of the environment of the vehicle comprise a status of at least one other vehicle in the environment determined from the second sensor data, and wherein the inferred aspect of the environment of the vehicle is an inferred state of the traffic signal based on the status of the at least one other vehicle in the environment.

4. The method of claim 3, wherein the traffic signal is at an intersection of a first roadway and a second roadway, wherein the vehicle is on the first roadway, and wherein controlling the vehicle in the autonomous mode based on the determination comprises:

if the state of the traffic signal determined from the first sensor data is green and the inferred state of the traffic signal based on the status of the at least one other vehicle determined from the second sensor data is green, controlling the vehicle to proceed through the intersection; and

if the state of the traffic signal determined from the first sensor data is green but the inferred state of the traffic signal based on the status of the at least one other vehicle determined from the second sensor data is not green, controlling the vehicle to stop at the intersection.

5. The method of claim 4, wherein the status of the at least one other vehicle in the environment comprises at least one vehicle on the first roadway that is stopped at the intersection, and wherein the inferred state of the traffic signal is red based at least in part on the at least one vehicle on the first roadway that is stopped at the intersection.

6. The method of claim 4, wherein the status of the at least one other vehicle in the environment comprises at least one vehicle on the second roadway proceeding through the intersection, and wherein the inferred state of the traffic signal is red based at least in part on the at least one vehicle on the second roadway proceeding through the intersection.

7. The method of claim 1, wherein the one or more sensors includes a camera and at least one other sensor, wherein the first sensor data is obtained using the camera and the second sensor data is obtained using the at least one other sensor.

8. A non-transitory computer readable medium having stored therein instructions executable by a computer system to cause the computer system to perform functions, the functions comprising:

obtaining, using one or more sensors, first sensor data regarding an environment of a vehicle and second sensor data regarding the environment of the vehicle, wherein the one or more sensors are operationally associated with the vehicle, wherein the vehicle is configured to operate in an autonomous mode, wherein the first sensor data is data regarding an aspect of the environment of the vehicle, and wherein the second sensor data is data regarding one or more other aspects of the environment of the vehicle;

making a determination whether the first sensor data and the second sensor data agree regarding the aspect of the environment of the vehicle, wherein making the determination whether the first sensor data and the second sensor data agree regarding the aspect of the environment of the vehicle, comprises inferring the aspect of the environment of the vehicle based on the second sensor data and determining whether the first sensor data regarding the aspect of the environment of the vehicle agrees with the inferred aspect of the environment of the vehicle; and

controlling the vehicle in the autonomous mode based on the determination.

9. The non-transitory computer readable medium of claim 8, wherein the first sensor data is image data regarding the aspect of the environment of the vehicle.

10. The non-transitory computer readable medium of claim 8, wherein the aspect of the environment is a state of a traffic signal determined from the first sensor data, wherein the one or more other aspects of the environment of the vehicle comprise a status of at least one other vehicle in the environment determined from the second sensor data, and wherein the inferred aspect of the environment of the vehicle is an inferred state of the traffic signal based on the status of the at least one other vehicle in the environment.

11. The non-transitory computer readable medium of claim 10, wherein the traffic signal is at an intersection of a first roadway and a second roadway, wherein the vehicle is on the first roadway, and wherein controlling the vehicle in the autonomous mode based on the determination comprises:

if the state of the traffic signal determined from the first sensor data is green and the inferred state of the traffic signal based on the status of the at least one other vehicle determined from the second sensor data is green, controlling the vehicle to proceed through the intersection; and

if the state of the traffic signal determined from the first sensor data is green but the inferred state of the traffic signal based on the status of the at least one other vehicle determined from the second sensor data is not green, controlling the vehicle to stop at the intersection.

12. The non-transitory computer readable medium of claim 11, wherein the status of the at least one other vehicle in the environment comprises at least one vehicle on the first roadway that is stopped at the intersection, and wherein the inferred state of the traffic signal is red based at least in part on the at least one vehicle on the first roadway that is stopped at the intersection.

13. The non-transitory computer readable medium of claim 11, wherein the status of the at least one other vehicle in the environment comprises at least one vehicle on the second roadway proceeding through the intersection, and wherein the inferred state of the traffic signal is red based at least in part on the at least one vehicle on the second roadway proceeding through the intersection.

14. The non-transitory computer readable medium of claim 8, wherein the one or more sensors includes a camera and at least one other sensor, wherein the first sensor data is obtained using the camera and the second sensor data is obtained using the at least one other sensor.

15. A vehicle, comprising:

one or more sensors, wherein the one or more sensors are configured to acquire sensor data regarding an environment of the vehicle, wherein the vehicle is configured to operate in an autonomous mode; and

a control system, wherein the control system is configured to;

use the one or more sensors to obtain first sensor data regarding the environment of the vehicle and second sensor data regarding the environment of the vehicle, wherein the first sensor data is data regarding an aspect of the environment of the vehicle, and wherein the second sensor data is data regarding one or more other aspects of the environment of the vehicle;

make a determination whether the first sensor data and the second sensor data agree regarding the aspect of the environment of the vehicle, wherein the determination comprises inferring the aspect of the environment of the vehicle based on the second sensor data and determining whether the first sensor data regarding the aspect of the environment of the vehicle agrees with the inferred aspect of the environment of the vehicle; and

control the vehicle in the autonomous mode based on the determination.

16. The vehicle of claim 15, wherein the first sensor data is image data regarding the aspect of the environment of the vehicle.

17. The vehicle of claim 15, wherein the aspect of the environment is a state of a traffic signal determined from the first sensor data, wherein the one or more other aspects of the environment of the vehicle comprise a status of at least one other vehicle in the environment determined from the second sensor data, and wherein the inferred aspect of the environment of the vehicle is an inferred state of the traffic signal based on the status of the at least one other vehicle in the environment.