Combining multiple estimates of an environment into a consolidated estimate for an autonomous vehicle

• US 9,224,053 B1
• Filed: 07/31/2013
• Issued: 12/29/2015
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ferguson, David Ian Franklin, Silver, David Harrison
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Shin, Soo
• Application Number:
  US13/955,970
• Time in Patent Office: 881 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2420/42 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2556/20 Data confidence level
• CPC Class Codes: B60W 2556/35 Data fusion
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/80 Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level multimodal speaker identification or verification G10L17/10
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A vehicle is provided that may combine multiple estimates of an environment into a consolidated estimate. The vehicle may receive first data indicative of the region of interest in an environment from a sensor of the vehicle. The first data may include a first accuracy value and a first estimate of the region of interest. The vehicle may also receive second data indicative of the region of interest in the environment, and the second data may include a second accuracy value and a second estimate of the region of interest. Based on the first data and the second data, the vehicle may combine the first estimate of the region of interest and the second estimate of the region of interest.

Claims

1. A method comprising:

receiving, using a processor, first data indicative of a lane of travel in an environment from at least one sensor of a vehicle that is substantially in the lane of travel, the first data having been acquired by the at least one sensor at a first time, wherein the first data comprises a first accuracy value and a first estimate of a location of a boundary of the lane of travel, wherein the first estimate comprises a first plurality of coordinate points corresponding to the location of the boundary, wherein the first accuracy value is representative of how accurately the first plurality of coordinate points corresponds to the location of the boundary, and wherein the vehicle is configured to operate in an autonomous mode in the environment;

receiving second data that is different from the first data and indicative of the lane of travel, the second data having been acquired by the at least one sensor at a second time different from the first time, wherein the second data comprises a second accuracy value and a second estimate of the location of the boundary of the lane of travel, wherein the second estimate comprises a second plurality of coordinate points corresponding to the location of the boundary, and wherein the second accuracy value is representative of how accurately the second plurality of coordinate points corresponds to the location of the boundary; and

based on the first data and the second data, combining the first estimate and the second estimate to determine third data comprising a third accuracy value and further comprising a combined estimate of the location of the boundary of the lane of travel, wherein the combined estimate comprises a third plurality of coordinate points corresponding to the location of the boundary, and wherein the third accuracy value is representative of how accurately the third plurality of coordinate points corresponds to the location of the boundary.

2. The method of claim 1, further comprising based on the third accuracy value, providing instructions to control the autonomous vehicle in the lane of travel.

3. The method of claim 1,

wherein the first accuracy value is determined further based on one or more of (i) an operational state of the at least one sensor, (ii) a distance between the first plurality of coordinate points and the at least one sensor, and (iii) first sensor data indicative of environmental conditions present at a time at which the first data was received, and

wherein the second accuracy value is determined further based on one or more of (i) an operational state of the at least one sensor, (ii) a distance between the second plurality of coordinate points and the at least one sensor, and (iii) second sensor data indicative of environmental conditions present at a time at which the second data was received.

4. The method of claim 1,

wherein the first estimate of the location of the boundary of the lane of travel comprises a first image portion, and

wherein the second estimate of the location of the boundary of the lane of travel comprises a second image portion.

5. The method of claim 1,

wherein the first plurality of coordinate points is a first plurality of two-dimensional (2D) coordinate points,

wherein the second plurality of coordinate points is a second plurality of 2D coordinate points,

wherein a given coordinate point of the first plurality of 2D coordinate points is associated with a respective first variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary, and

wherein a given coordinate point of the second plurality of 2D coordinate points is associated with a respective second variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary.

6. The method of claim 5,

wherein the third plurality of coordinate points is a third plurality of two-dimensional (2D) coordinate points, and

wherein a given coordinate point of the third plurality of 2D coordinate points is associated with a respective third variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary.

7. A vehicle comprising:

a sensor configured to sense a lane of travel in an environment of the vehicle, wherein the vehicle is configured to operate in an autonomous mode in the environment, and wherein the vehicle is substantially in the lane of travel in the environment; and

a computer system, wherein the computer system is configured to;

receive first data indicative of the lane of travel in the environment from the sensor, the first data having been acquired by the sensor at a first time, wherein the first data comprises a first accuracy value and a first estimate of a location of a boundary of the lane of travel, wherein the first estimate comprises a first plurality of coordinate points corresponding to the location of the boundary, and wherein the first accuracy value is representative of how accurately the first plurality of coordinate points corresponds to the location of the boundary;

receive second data that is different from the first data and indicative of the lane of travel, the second data having been acquired by the sensor at a second time different from the first time, wherein the second data comprises a second accuracy value and a second estimate of the location of the boundary of the lane of travel, wherein the second estimate comprises a second plurality of coordinate points corresponding to the location of the boundary, and wherein the second accuracy value is representative of how accurately the second plurality of coordinate points corresponds to the location of the boundary; and

based on the first data and the second data, combine the first estimate and the second estimate to determine third data comprising a third accuracy value and further comprising a combined estimate of the location of the boundary of the lane of travel, wherein the combined estimate comprises a third plurality of coordinate points corresponding to the location of the boundary, and wherein the third accuracy value is representative of how accurately the third plurality of coordinate points corresponds to the location of the boundary.

8. The vehicle of claim 7, wherein the computer system is further configured to provide instructions to control the autonomous vehicle in the lane of travel.

9. The vehicle of claim 7, wherein the sensor is a first sensor, wherein the second data is received from a second sensor of the vehicle, wherein the second sensor is different than the first sensor.

10. The vehicle of claim 7,

wherein the first estimate of the location of the boundary of the lane of travel comprises a first image portion, and

wherein the second estimate of the location of the boundary of the lane of travel comprises a second image portion.

11. The vehicle of claim 7,

wherein the first plurality of coordinate points is a first plurality of two-dimensional (2D) coordinate points,

wherein the second plurality of coordinate points is a second plurality of 2D coordinate points,

wherein a given coordinate point of the first plurality of 2D coordinate points is associated with a respective first variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary, and

wherein a given coordinate point of the second plurality of 2D coordinate points is associated with a respective second variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary.

12. The vehicle of claim 11,

wherein the third plurality of coordinate points is a third plurality of two-dimensional (2D) coordinate points, and

wherein a given coordinate point of the third plurality of 2D coordinate points is associated with a respective third variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary.

13. A non-transitory computer readable medium having stored therein instructions that when executed by a computer system in a vehicle cause the computer system to perform functions comprising:

receiving first data indicative of a lane of travel in an environment from at least one sensor of the vehicle, the first data having been acquired by the at least one sensor at a first time, wherein the first data comprises a first accuracy value and a first estimate of a location of a boundary of the lane of travel, wherein the first estimate comprises a first plurality of coordinate points corresponding to the location of the boundary, wherein the first accuracy value is representative of how accurately the first plurality of coordinate points corresponds to the location of the boundary, wherein the vehicle is configured to operate in an autonomous mode in the environment, and wherein the vehicle is substantially in the lane of travel;

receiving second data that is different from the first data and indicative of the lane of travel, the second data having been acquired by the at least one sensor at a second time different from the first time, wherein the second data comprises a second accuracy value and a second estimate of the location of the boundary of the lane of travel, wherein the second estimate comprises a second plurality of coordinate points corresponding to the location of the boundary, and wherein the second accuracy value is representative of how accurately the second plurality of coordinate points corresponds to the location of the boundary; and

based on the first data and the second data, combining the first estimate and the second estimate to determine third data comprising a third accuracy value and further comprising a combined estimate of the location of the boundary of the lane of travel, wherein the combined estimate comprises a third plurality of coordinate points corresponding to the location of the boundary, and wherein the third accuracy value is representative of how accurately the third plurality of coordinate points corresponds to the location of the boundary.

14. The non-transitory computer readable medium of claim 13, wherein the instructions are further executable by the computer system in the vehicle to cause the computer system to perform functions comprising:

based on the third accuracy value, providing instructions to control the autonomous vehicle in the lane of travel.

15. The non-transitory computer readable medium of claim 13, wherein the at least one sensor is at least one first sensor, wherein the second data is received from at least one second sensor of the vehicle, wherein the at least one second sensor is different than the at least one first sensor.

16. The non-transitory computer readable medium of claim 13,

wherein the first plurality of coordinate points is a first plurality of two-dimensional (2D) coordinate points,

wherein the second plurality of coordinate points is a second plurality of 2D coordinate points,

wherein a given coordinate point of the first plurality of 2D coordinate points is associated with a respective first variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary, and

wherein a given coordinate point of the second plurality of 2D coordinate points is associated with a respective second variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary.

17. The non-transitory computer readable medium of claim 16,

wherein the third plurality of coordinate points is a third plurality of two-dimensional (2D) coordinate points, and

wherein a given coordinate point of the third plurality of 2D coordinate points is associated with a respective third variance value that indicates how accurately the given coordinate point corresponds to the location of the boundary.