Use of detected objects for image processing

• US 9,804,597 B1
• Filed: 09/15/2015
• Issued: 10/31/2017
• Est. Priority Date: 04/17/2013
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Ferguson, Dave, Ogale, Abhijit
• Primary Examiner(s): Brushaber, Frederick
• Application Number:
  US14/855,009
• Time in Patent Office: 777 Days
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 30/10 Path keeping cruise control for automatically following a preceding vehicle B60W30/165
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: B60W 40/06 Road conditions
• CPC Class Codes: G01S 13/02 Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
• CPC Class Codes: G01S 17/02 Systems using the reflection of electromagnetic waves other than radio waves G01S17/66 takes precedence
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/247 using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
• CPC Class Codes: G05D 1/249 from positioning sensors located off-board the vehicle, e.g. from cameras
• CPC Class Codes: G05D 1/81 Handing over between on-board automatic and on-board manual control
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Methods and systems for the use of detected objects for image processing are described. A computing device autonomously controlling a vehicle may receive images of the environment surrounding the vehicle from an image-capture device coupled to the vehicle. In order to process the images, the computing device may receive information indicating characteristics of objects in the images from one or more sources coupled to the vehicle. Examples of sources may include RADAR, LIDAR, a map, sensors, a global positioning system (GPS), or other cameras. The computing device may use the information indicating characteristics of the objects to process received images, including determining the approximate locations of objects within the images. Further, while processing the image, the computing device may use information from sources to determine portions of the image to focus upon that may allow the computing device to determine a control strategy based on portions of the image.

Claims

1. A method comprising:

receiving, at a computing device from an image-capture device coupled to a vehicle, an image depicting an object in an environment of the vehicle;

receiving, at the computing device via one or more sources coupled to the vehicle, information indicating a plurality of characteristics of the object;

determining by a computing device, using the information indicating the plurality of characteristics of the object, a particular portion of a plurality of portions of the image depicting the object;

based on the plurality of characteristics of the object, determining a location of the object within the particular portion of the image;

identifying by the computing device, using the determined location of the object and the plurality of characteristics of the object, that the object is at least partially occluded in the particular portion of the image;

processing the particular portion of the image to determine a control strategy for the vehicle based on one or more of the plurality of characteristics of the object and the identification that the object is at least partially occluded; and

modifying one or more systems associated with the vehicle to control the vehicle based on the determined control strategy.

2. The method of claim 1, wherein identifying by the computing device, using the determined location of the object and the plurality of characteristics of the object, that the object is at least partially occluded in the particular portion of the image comprises:

determining one or more intersections in the particular portion of the image depicting the object, wherein a given intersection corresponds to an area of the image that includes another object occluding at least a portion of the object; and

determining that the object is present at the one or more intersections based on information indicating one or more characteristics of the object even though the object is at least partially occluded in the image at the one or more intersections.

3. The method of claim 2, wherein processing the particular portion of the image to determine the control strategy for the vehicle based on the characteristics of the object and the identification that the object is at least partially occluded includes:

determining the control strategy based on determining that the object is present at the one or more intersections.

4. The method of claim 1, wherein identifying by the computing device, using the determined location of the object and the plurality of characteristics of the object, that the object is at least partially occluded in the particular portion of the image comprises:

identifying a second object that is at least partially occluding the object; and

determining respective characteristics associated with the second object that is at least partially occluding the object.

5. The method of claim 4, wherein processing the particular portion of the image to determine the control strategy for the vehicle based on the characteristics of the object and the identification that the object is at least partially occluded further comprises:

processing the particular portion of the image to determine the control strategy for the vehicle based also on the respective characteristics associated the second object that is at least partially occluding the object.

6. The method of claim 1, wherein identifying by the computing device, using the determined location of the object and the plurality of characteristics of the object, that the object is at least partially occluded in the particular portion of the image includes:

determining a size of one or more occlusions corresponding to the object that is at least partially occluded.

7. The method of claim 1, wherein the one or more sources coupled to the vehicle include one or more of a laser, radio detection and ranging (RADAR), laser rangefinder and light detection and ranging (LIDAR), a map, sensors, global positioning system (GPS), or a camera.

8. The method of claim 1, wherein the information indicating the plurality of characteristics of the object further include information indicating whether the object is moving or stationary, a longitudinal speed of the object, a lateral speed of the object, a direction of motion of the object, a size of the object, a respective position of the object on a road on which the vehicle is traveling, or a type of the object.

9. The method of claim 1, wherein the computing device is configured to control the vehicle in an autonomous operation mode.

10. The method of claim 1, further comprising:

determining, based on the particular portion of the image depicting the object, one or more features of the object, wherein determining the one or more features of the object includes determining one or more of a brake signal, a turn signal, color of the object, or a license plate number.

11. A non-transitory computer readable medium having stored thereon executable instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

receiving, from an image-capture device coupled to a vehicle, an image depicting an object in an environment of the vehicle;

receiving, via one or more sources coupled to the vehicle, information indicating a plurality of characteristics of the object;

determining, using the information indicating the plurality of characteristics of the object, a particular portion of a plurality of portions of the image depicting the object;

based on the plurality of characteristics of the object, determining a location of the object within the particular portion of the image;

identifying, using the determined location of the object and the plurality of characteristics of the object, that the object is at least partially occluded in the particular portion of the image;

processing the particular portion of the image to determine a control strategy for the vehicle based on one or more of the plurality of characteristics of the object and the identification that the object is at least partially occluded; and

modifying one or more systems associated with the vehicle to control the vehicle based on the determined control strategy.

12. The non-transitory computer readable medium of claim 11, wherein the information indicating the plurality of characteristics of the object further include information indicating whether the object is moving or stationary, a longitudinal speed of the object, a lateral speed of the object, a direction of motion of the object, a size of the object, a respective position of the object on a road on which the vehicle is traveling, or a type of the object.

13. The non-transitory computer readable medium of claim 11, wherein identifying, using the determined location of the object and the plurality of characteristics of the object, that the object is at least partially occluded in the particular portion of the image comprises:

determining one or more intersections in the particular portion of the image depicting the object, wherein a given intersection corresponds to an area of the image that includes another object occluding at least a portion of the object; and

determining that the object is present at the one or more intersections based on information indicating one or more characteristics of the object even though the object is at least partially occluded in the image at the one or more intersections.

14. The non-transitory computer readable medium of claim 11, wherein the image-capture device corresponds to a camera system of the vehicle and the one or more sources includes a LIDAR system.

15. The non-transitory computer readable medium of claim 11, wherein the control strategy for the vehicle based on the characteristics of the object and the identification that the object is at least partially occluded depends on a size of respective occlusions that at least partially occlude the object.

16. The non-transitory computer readable medium of claim 11, wherein the control strategy includes an indication for the one or more systems of the vehicle to reduce speed based on a position of the object is at least partially occluded.

17. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the control system to perform functions comprising;

receiving, from an image-capture device coupled to a vehicle, an image depicting an object in an environment of the vehicle;

receiving, via one or more sources coupled to the vehicle, information indicating a plurality of characteristics of the object;

determining, using the information indicating the plurality of characteristics of the object, a particular portion of a plurality of portions of the image depicting the object;

based on the plurality of characteristics of the object, determining a location of the object within the particular portion of the image;

identifying, using the determined location of the object and the plurality of characteristics of the object, that the object is at least partially occluded in the particular portion of the image;

processing the particular portion of the image to determine a control strategy for the vehicle based on one or more of the plurality of characteristics of the object and the identification that the object is at least partially occluded; and

modifying one or more systems associated with the vehicle to control the vehicle based on the determined control strategy.

18. The system of claim 17, wherein processing the particular portion of the image to determine the control strategy for the vehicle based on the characteristics of the object and the identification that the object is at least partially occluded comprises:

determining the control strategy for the vehicle based on respective positions of one or more occlusions that at least partially occlude the object.

19. The system of claim 17, wherein identifying, using the determined location of the object and the plurality of characteristics of the object, that the object is at least partially occluded in the particular portion of the image includes:

identifying a second object that is at least partially occluding the object; and

determining respective characteristics associated with the second object that is at least partially occluding the object.

20. The system of claim 19, wherein processing the particular portion of the image to determine the control strategy for the vehicle based on the characteristics of the object and the identification that the object is at least partially occluded further comprises:

processing the particular portion of the image to determine the control strategy for the vehicle based also on the respective characteristics associated the second object that is at least partially occluding the object.