Combining Multiple Estimates Of An Environment Into A Consolidated Estimate For An Autonomous Vehicle

• US 20170228604A1
• Filed: 04/27/2017
• Published: 08/10/2017
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Ferguson, David Ian Franklin, Silver, David Harrison
• Granted Patent: US 10,181,084 B2
• Time in Patent Office: Days
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2556/20 Data confidence level
• CPC Class Codes: B60W 2556/35 Data fusion
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/80 Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level multimodal speaker identification or verification G10L17/10
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A vehicle is provided that may combine multiple estimates of an environment into a consolidated estimate. The vehicle may receive first data indicative of the region of interest in an environment from a sensor of the vehicle. The first data may include a first accuracy value and a first estimate of the region of interest. The vehicle may also receive second data indicative of the region of interest in the environment, and the second data may include a second accuracy value and a second estimate of the region of interest. Based on the first data and the second data, the vehicle may combine the first estimate of the region of interest and the second estimate of the region of interest.

Claims

1. A method performed by a computing device comprising one or more processors, the method comprising:

obtaining first data including a first estimate of a location of a region of interest of a road, the first data having been acquired at a first time, wherein the first data comprises a predetermined map of the road, and wherein an autonomous vehicle is operating on the road;

based on the first data, determining a first variance in the first estimate of the location;

receiving, via at least one sensor coupled to the autonomous vehicle, second data including a second estimate of the location, the second data having been acquired at a second time different from the first time;

based on the second data, determining a second variance in the second estimate of the location; and

based on the first estimate, the first variance, the second estimate, and the second variance, determining third data including a combined estimate of the location and further including a combined variance in the combined estimate.

2. The method of claim 1, further comprising:

based on the third data, providing instructions to control the autonomous vehicle on the road.

3. The method of claim 1, wherein the first estimate comprises a first plurality of coordinate points corresponding to the location, and

wherein the second estimate comprises a second plurality of coordinate points corresponding to the location.

4. The method of claim 3, wherein the second variance is determined further based on one or more of (i) an operational state of the at least one sensor, (ii) a distance between the second plurality of coordinate points and the at least one sensor, and (iii) sensor data indicative of environmental conditions present at the second time.

5. The method of claim 1, wherein the first estimate comprises a first image portion, and

wherein the second estimate comprises a second image portion.

6. The method of claim 1, wherein the first estimate comprises a first plurality of coordinate points corresponding to the location,

wherein the second estimate comprises a second plurality of coordinate points corresponding to the location,

wherein the combined estimate comprises a third plurality of coordinate points corresponding to the location,

wherein the first variance comprises, for each coordinate point of the first plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the location,

wherein the second variance comprises, for each coordinate point of the second plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the location, and

wherein the combined variance comprises, for each coordinate point of the third plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the location.

7. The method of claim 6, wherein one or more of the first, second, and third pluralities of coordinate points is a plurality of two-dimensional (2D) coordinate points.

8. An autonomous vehicle comprising:

at least one sensor configured to sense a road on which the autonomous vehicle is operating;

at least one processor; and

data storage comprising instructions executable by the at least one processor to cause the system to perform operations comprising;

obtaining first data including a first estimate of a location of a region of interest of the road, the first data having been acquired at a first time, wherein the first data comprises a predetermined map of the road;

based on the first data, determining a first variance in the first estimate of the location;

receiving, via the at least one sensor, second data including a second estimate of the location, the second data having been acquired at a second time different from the first time;

based on the second data, determining a second variance in the second estimate of the location; and

based on the first estimate, the first variance, the second estimate, and the second variance, determining third data including a combined estimate of the location and further including a combined variance in the combined estimate.

9. The vehicle of claim 8, wherein the operations further comprise:

based on the third data, providing instructions to control the autonomous vehicle on the road.

10. The vehicle of claim 8, wherein the first estimate comprises a first plurality of coordinate points corresponding to the location,

wherein the second estimate comprises a second plurality of coordinate points corresponding to the location, and

wherein the second variance is determined further based on one or more of (i) an operational state of the at least one sensor, (ii) a distance between the second plurality of coordinate points and the at least one sensor, and (iii) sensor data indicative of environmental conditions present at the second time.

11. The vehicle of claim 8, wherein the first time is before the second time.

12. The vehicle of claim 8,

wherein the first estimate comprises a first plurality of coordinate points corresponding to the location,

wherein the second estimate comprises a second plurality of coordinate points corresponding to the location,

wherein the combined estimate comprises a third plurality of coordinate points corresponding to the location,

wherein the first variance comprises, for each coordinate point of the first plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the location,

wherein the second variance comprises, for each coordinate point of the second plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the location, and

wherein the combined variance comprises, for each coordinate point of the third plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the location.

13. The vehicle of claim 8, wherein determining the third data based on the first estimate, the first variance, the second estimate, and the second variance comprises:

making a comparison between one or both of the first and second variances and a predetermined threshold;

in response to the comparison indicating that one or both of the first and second variances is below the predetermined threshold, (i) causing the at least one sensor to acquire additional data including another estimate of the location and (ii) based on the additional data, determining another variance in the other estimate of the location;

determining that the other variance is above the predetermined threshold; and

in response to determining that the other variance is above the predetermined threshold, determining the third data based at least in part on the other estimate and the other variance.

14. The vehicle of claim 13, wherein the at least one sensor includes at least one first sensor and further includes at least one second sensor different from the at least one first sensor,

wherein the second data is received via the at least one first sensor, and

wherein causing the at least one sensor to acquire the additional data including the other estimate of the location comprises causing the at least one second sensor to acquire the additional data including the other estimate of the location.

15. A non-transitory computer readable medium having stored therein instructions that when executed by a computing system cause the computing system to perform operations comprising:

obtaining first data including a first estimate of a location of a region of interest of a road, the first data having been acquired at a first time, wherein the first data comprises a predetermined map of the road, and wherein an autonomous vehicle is operating on the road;

based on the first data, determining a first variance in the first estimate of the location;

receiving, via at least one sensor coupled to the autonomous vehicle, second data including a second estimate of the location, the second data having been acquired at a second time different from the first time;

based on the second data, determining a second variance in the second estimate of the location; and

based on the first estimate, the first variance, the second estimate, and the second variance, determining third data including a combined estimate of the location and further including a combined variance in the combined estimate.

16. The non-transitory computer readable medium of claim 15, wherein the operations further comprise:

based on the third data, providing instructions to control the autonomous vehicle on the road.

17. The non-transitory computer readable medium of claim 15, wherein determining the third data based on the first estimate, the first variance, the second estimate, and the second variance comprises:

making a comparison between one or both of the first and second variances and a predetermined threshold;

in response to the comparison indicating that one or both of the first and second variances is below the predetermined threshold, (i) causing the at least one sensor to acquire additional data including another estimate of the location and (ii) based on the additional data, determining another variance in the other estimate of the location;

determining that the other variance is above the predetermined threshold; and

in response to determining that the other variance is above the predetermined threshold, determining the third data based at least in part on the other estimate and the other variance.

18. The non-transitory computer readable medium of claim 15, wherein the first estimate comprises a first plurality of coordinate points, each coordinate point of the first plurality corresponding to a respective lane marker that makes up a distinct portion of the region of interest of the road,

wherein the second estimate comprises a second plurality of coordinate points, each coordinate point of the second plurality corresponding to a respective lane marker that makes up a distinct portion of the region of interest of the road, and

wherein the combined estimate comprises a third plurality of coordinate points, each coordinate point of the third plurality corresponding to a respective lane marker that makes up a distinct portion of the region of interest of the road.

19. The non-transitory computer readable medium of claim 18, wherein the first variance comprises, for each coordinate point of the first plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the respective lane marker,

wherein the second variance comprises, for each coordinate point of the second plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the respective lane marker, and

wherein the combined variance comprises, for each coordinate point of the third plurality of coordinate points, an associated respective variance value that indicates how accurately the coordinate point corresponds to the respective lane marker.

20. The non-transitory computer readable medium of claim 18, wherein one or more of the first, second, and third pluralities of coordinate points is a plurality of three-dimensional (3D) coordinate points.