Use of detected objects for image processing

• US 10,509,402 B1
• Filed: 09/22/2017
• Issued: 12/17/2019
• Est. Priority Date: 04/17/2013
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Ferguson, Dave, Ogale, Abhijit
• Primary Examiner(s): Brushaber, Frederick M
• Application Number:
  US15/713,263
• Time in Patent Office: 816 Days
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 30/10 Path keeping cruise control for automatically following a preceding vehicle B60W30/165
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: B60W 40/06 Road conditions
• CPC Class Codes: G01S 13/02 Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
• CPC Class Codes: G01S 17/02 Systems using the reflection of electromagnetic waves other than radio waves G01S17/66 takes precedence
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Methods and systems for the use of detected objects for image processing are described. A computing device autonomously controlling a vehicle may receive images of the environment surrounding the vehicle from an image-capture device coupled to the vehicle. In order to process the images, the computing device may receive information indicating characteristics of objects in the images from one or more sources coupled to the vehicle. Examples of sources may include RADAR, LIDAR, a map, sensors, a global positioning system (GPS), or other cameras. The computing device may use the information indicating characteristics of the objects to process received images, including determining the approximate locations of objects within the images. Further, while processing the image, the computing device may use information from sources to determine portions of the image to focus upon that may allow the computing device to determine a control strategy based on portions of the image.

Claims

1. A method comprising:

receiving, at a computing device from an image-capture device coupled to a vehicle, an image depicting an environment of the vehicle;

receiving, at the computing device from a sensor coupled to the vehicle, sensor data indicative of a traffic signal in the environment of the vehicle, wherein the sensor captures the sensor data as the vehicle navigates the environment;

determining, using the sensor data, a position of the traffic signal relative to one or more objects in the environment of the vehicle;

mapping a subset of the sensor data representing the position of the traffic signal relative to the one or more objects in the environment to a set of points in the image to identify a portion of a plurality of portions of the image that includes the traffic signal;

determining, using the identified portion of the image, information corresponding to the traffic signal; and

modifying one or more systems associated with the vehicle to control the vehicle based on the information corresponding to the traffic signal.

2. The method of claim 1, wherein receiving sensor data indicative of the traffic signal in the environment of the vehicle comprises:

receiving sensor data indicative of a stationary traffic light positioned in the environment of the vehicle.

3. The method of claim 1, wherein receiving sensor data indicative of the traffic signal in the environment of the vehicle comprises:

receiving sensor data from a LIDAR sensor, wherein the sensor data is indicative of a stationary traffic sign positioned proximate a road of travel of the vehicle.

4. The method of claim 3, further comprising:

determining, based on the sensor data, that a portion of the stationary traffic sign is partially occluded; and

wherein determining, using the identified portion of the image, information corresponding to the traffic signal comprises;

estimating information for the portion of the stationary traffic sign that is partially occluded based on a visible portion of the stationary traffic sign depicted in the identified portion of the image.

5. The method of claim 1, wherein receiving the image depicting the environment of the vehicle comprises:

receiving a plurality of images depicting the environment from a plurality of orientations of the vehicle; and

wherein based on the determined position of the traffic signal relative to one or more objects in the environment, identifying the portion of the plurality of portions of the image depicting the environment that includes the traffic signal comprises;

determining a given image of the plurality of images that includes the traffic signal and the one or more objects; and

based on determining the given image that includes the traffic signal and the one or more objects, identifying a portion of a plurality of portions of the given image that includes the traffic signal.

6. The method of claim 1, wherein receiving sensor data indicative of the traffic signal in the environment of the vehicle comprises:

receiving the sensor data from a RADAR sensor coupled to vehicle.

7. The method of claim 1, wherein determining, using the identified portion of the image, information corresponding to the traffic signal comprises:

determining that the traffic signal corresponds to a traffic light; and

determining a current state of the traffic light.

8. The method of claim 7, wherein determining the current state of the traffic light comprises:

determining that the traffic light is illuminating a green light; and

wherein modifying the one or more systems associated with the vehicle to control the vehicle based on the information corresponding to the traffic signal comprises;

modifying the one or more systems to cause the vehicle to navigate a path.

9. The method of claim 7, wherein determining the current state of the traffic light comprises:

determining that the traffic light is illuminating a red light; and

wherein modifying the one or more systems associated with the vehicle to control the vehicle based on the information corresponding to the traffic signal comprises;

modifying the one or more systems to cause the vehicle to stop navigation at a position relative to the traffic light.

10. The method of claim 1, further comprising:

receiving, at the computing device, a status of the traffic signal from the traffic signal via wireless communication; and

wherein modifying the one or more systems associated with the vehicle to control the vehicle based on the information corresponding to the traffic signal is further based on the status of the traffic signal.

11. The method of claim 1, wherein identifying the portion of the plurality of portions of the image that includes the traffic signal comprises:

partitioning the image into a plurality of portions;

determining a given portion of the plurality of portions that depicts at least the traffic signal or an object of the one or more objects in the environment of the vehicle; and

based on determining the given portion of the plurality of portions, identifying the portion of the plurality of portions of the image that includes the traffic signal.

12. A non-transitory computer readable medium having stored thereon executable instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

receiving, from an image-capture device coupled to a vehicle, an image depicting an environment of the vehicle;

receiving, from a sensor coupled to the vehicle, sensor data indicative of a traffic signal in the environment of the vehicle, wherein the sensor captures the sensor data as the vehicle navigates the environment;

determining, using the sensor data, a position of the traffic signal relative to one or more objects in the environment of the vehicle;

mapping a subset of the sensor data representing the position of the traffic signal relative to the one or more objects in the environment to a set of points in the image to identify a portion of a plurality of portions of the image that includes the traffic signal;

determining, using the identified portion of the image, information corresponding to the traffic signal; and

modifying one or more systems associated with the vehicle to control the vehicle based on the information corresponding to the traffic signal.

13. The non-transitory computer readable medium of claim 12, wherein receiving sensor data indicative of the traffic signal in the environment of the vehicle comprises:

receiving sensor data indicative of a stationary traffic light positioned in the environment of the vehicle.

14. The non-transitory computer readable medium of claim 12, wherein receiving sensor data indicative of the traffic signal in the environment of the vehicle comprises:

receiving sensor data from a LIDAR sensor, wherein the sensor data is indicative of a stationary traffic sign positioned proximate a road of travel of the vehicle.

15. The non-transitory computer readable medium of claim 14, further comprising:

determining, based on the sensor data, that a portion of the stationary traffic sign is partially occluded; and

wherein determining, using the identified portion of the image, information corresponding to the traffic signal comprises;

estimating information for the portion of the stationary traffic sign that is partially occluded based on a visible portion of the stationary traffic sign depicted in the identified portion of the image.

16. The non-transitory computer readable medium of claim 12, wherein determining, using the identified portion of the image, information corresponding to the traffic signal comprises:

determining that the traffic signal corresponds to a traffic light; and

determining a current state of the traffic light.

17. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the system to perform operations comprising;

receiving, from an image-capture device coupled to a vehicle, an image depicting an environment of the vehicle;

receiving, from a sensor coupled to the vehicle, sensor data indicative of a traffic signal in the environment of the vehicle, wherein the sensor captures the sensor data as the vehicle navigates the environment;

determining, using the sensor data, a position of the traffic signal relative to one or more objects in the environment of the vehicle;

mapping a subset of the sensor data representing the position of the traffic signal relative to the one or more objects in the environment to a set of points in the image to identify a portion of a plurality of portions of the image that includes the traffic signal;

determining, using the identified portion of the image, information corresponding to the traffic signal; and

modifying one or more systems associated with the vehicle to control the vehicle based on the information corresponding to the traffic signal.

18. The system of claim 17, wherein the operations further comprise:

receiving a status of the traffic signal from the traffic signal via wireless communication; and

wherein the operation of modifying the one or more systems associated with the vehicle to control the vehicle based on the information corresponding to the traffic signal is further based on the status of the traffic signal.

19. The system of claim 17, wherein the operation of identifying the portion of the plurality of portions of the image that includes the traffic signal comprises:

partitioning the image into a plurality of portions;

determining a given portion of the plurality of portions that depicts at least the traffic signal or an object of the one or more objects in the environment of the vehicle; and

based on determining the given portion of the plurality of portions, identifying the portion of the plurality of portions of the image that includes the traffic signal.

20. The system of claim 17, wherein the operation of receiving sensor data indicative of the traffic signal in the environment of the vehicle comprises:

receiving sensor data from a LIDAR sensor, wherein the sensor data is indicative of a stationary traffic sign positioned proximate a road of travel of the vehicle.