TRAFFIC LIGHT DETECTION AND LANE STATE RECOGNITION FOR AUTONOMOUS VEHICLES

• US 20200135030A1
• Filed: 10/24/2018
• Published: 04/30/2020
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Krivokon, Maxim, Ogale, Abhijit S., Hsiao, Edward, Wendel, Andreas
• Granted Patent: US 11,056,005 B2
• Time in Patent Office: Days
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0145 for active traffic flow control
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/095 Traffic lights
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

Aspects of the disclosure relate to training and using a model for determine states of lanes of interest. For instance, image data including an image and an associated label identifying at least one traffic light, a state of the at least one traffic light, and a lane controlled by the at least one traffic light may be received and used to train the mode such that the model is configured to, in response to receiving an image and a lane of interest included in the image, output a lane state for the lane of interest. This model may then be used by a vehicle in order to determine a state of a lane of interest. This state may then be used to control the vehicle in an autonomous driving mode based on the state of the lane of interest.

Claims

1. A method of training a model for determining states of lanes of interest, the method comprising:

receiving, by one or more server computing devices, image data including an image and an associated label identifying at least one traffic light, a state of the at least one traffic light, and a lane controlled by the at least one traffic light; and

training, by the one or more server computing devices, the model using the image data such that the model is configured to, in response to receiving an image and a lane of interest included in the image, output a lane state for the lane of interest.

2. The method of claim 1, further comprising, prior to the training, projecting a road segment corresponding to the lane into the image.

3. The method of claim 1, further comprising, prior to the training, generating a second image using the image and the projection, the second image highlighting an area of the projection in the image, and wherein training the model is further based on the second image.

4. The method of claim 1, further comprising, generating the associated label by projecting a three-dimensional location of the at least one traffic light into the image.

5. The method of claim 4, further comprising, determining the state by processing the image to identify a blob of color within an area of the projection.

6. The method of claim 1, wherein the lane state identifies whether a vehicle in that lane is required to go, stop, or use caution.

7. The method of claim 1, further comprising, training the model to identify stop lines in images relevant to the lane of interest.

8. The method of claim 7, wherein the training further includes using a label identifying a location of a stop line in the image.

9. The method of claim 1, wherein the image data further includes a second image and a second associated label identifying the at least one traffic light, a second state of the at least one traffic light in the second image, the second state being different from the state of the at least one traffic light such that the training includes using images of the at least one traffic light captured at different times in different states.

10. A method of using a model to determine states of lanes of interest, the method comprising:

receiving, by one or more processors, an image generated by a perception system of a vehicle;

identifying, by the one or more processors, a lane of interest;

using, by the one or more processors, the image and the lane of interest as input into the model to output a state of the lane of interest according to a state of a traffic light in the image; and

controlling, by the one or more processors, the vehicle in an autonomous driving mode based on the state of the lane of interest.

11. The method of claim 10, wherein identifying the lane of interest includes projecting a road segment corresponding to a lane in which the vehicle is currently driving into the image.

12. The method of claim 10, further comprising, comparing the state of the lane of interest to a determined state of the traffic light in the image, and wherein controlling the vehicle is further based on the comparing.

13. The method of claim 12, wherein the lane state identifies whether a vehicle in that lane is required to go, stop, or use caution.

14. The method of claim 10, wherein the model further outputs a location of a stop line in the image that is relevant to the lane of interest, and controlling the vehicle is further based on the location of the stop line.

15. The method of claim 10, further comprising, prior to capturing the image, controlling the vehicle in the autonomous driving mode based on pre-stored map information, and the image is input into the model when the vehicle is located in an area that is not included in the map information.

16. The method of claim 10, further comprising, prior to capturing the image, controlling the vehicle in the autonomous driving mode based on pre-stored map information, and the image is input into the model when the vehicle is located in an area that is not up to date in the map information.

17. The method of claim 10, further comprising, comparing the state of the lane with a state of the lane determined based on a state of a second traffic light, and wherein controlling the vehicle is further based on the comparing.

19. The system of claim 10, wherein the one or more processors are further configured to label the image with the lane of interest includes by projecting a road segment corresponding to a lane in which the vehicle is currently driving into the image.

18. A system for using a model to determine states of lanes of interest, the system comprising one or more processors configured to:

receive an image generated by a perception system of a vehicle;

identify a lane of interest;

use the image and the lane of interest as input into the model to output a state of the lane of interest according to a state of a traffic light in the image; and

control the vehicle in an autonomous driving mode based on the state of the lane of interest.

20. The system of claim 18, further comprising the vehicle.