Teleassistance data encoding for self-driving vehicles

• US 10,202,126 B2
• Filed: 03/07/2017
• Issued: 02/12/2019
• Est. Priority Date: 03/07/2017
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Uber Technologies, Inc.
• Inventors: Kroop, Benjamin, Ross, William, Heine, Andrew
• Primary Examiner(s): Black, Thomas G
• Assistant Examiner(s): Smith-Stewart, Demetra R
• Application Number:
  US15/452,456
• Publication Number:
  US 20180257661A1
• Time in Patent Office: 707 Days
• Field of Search: None
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2530/20 Tyre data
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4026 Cycles
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/4041 Position
• CPC Class Codes: B60W 2554/802 Longitudinal distance
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/20 Steering systems
• CPC Class Codes: B60W 2720/10 Longitudinal speed
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/18109 Braking
• CPC Class Codes: B60W 30/18181 Propulsion control with common controlling member for different functions
• CPC Class Codes: B60W 50/0098 : Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: G01S 13/862 : Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/865 : Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 15/931 : of land vehicles
• CPC Class Codes: G01S 17/86 : Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G01S 2013/9318 : Controlling the steering
• CPC Class Codes: G01S 2013/93185 : Controlling the brakes
• CPC Class Codes: G01S 2013/9319 : Controlling the accelerator
• CPC Class Codes: G01S 2013/9321 : Velocity regulation, e.g. cruise control
• CPC Class Codes: G01S 2013/9323 : Alternative operation using light waves
• CPC Class Codes: G05D 1/0027 : involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0282 : generated in a local control room
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/165 : for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

A self-driving vehicle (SDV) can analyze a live sensor view to autonomously operate acceleration, braking, and steering systems of the SDV along a current route. The SDV can identify an indeterminate object in the live sensor view, and encoding sensor data identifying the indeterminate object for transmission to a backend transport system over one or more networks. The SDV may then receive a resolution response from the backend transport system to resolve the indeterminate object, and cause the SDV to proceed in accordance with the resolution response.

Claims

1. A self-driving vehicle (SDV) comprising:

a plurality of communication modules to communicate with a backend transport system;

a sensor system generating sensor data providing a live sensor view of a situational environment of the SDV;

acceleration, braking, and steering systems; and

a control system to execute instructions that cause the control system to;

monitor bandwidth of each of the plurality of communication modules to determine a highest bandwidth communication module;

analyze the live sensor view to autonomously operate the acceleration, braking, and steering systems along a current route;

identify an indeterminate object in the live sensor view by determining that the indeterminate object does not meet a certainty threshold;

in response to identifying the indeterminate object, specify the indeterminate object in a bounding box;

encode sensor data identifying the indeterminate object within the bounding box for transmission over one or more networks by (i) encoding the bounding box at a first bitrate, and (ii) encoding a remainder of the sensor data at a second bitrate;

transmit the encoded sensor data to the backend transport system using the highest bandwidth communication module;

receive a resolution response from the backend transport system to resolve the indeterminate object; and

cause the SDV to proceed in accordance with the resolution response.

2. The SDV of claim 1, wherein the resolution response classifies the indeterminate object.

3. The SDV of claim 1, wherein the sensor system comprises at least a LIDAR system and a camera system that combine to generate the live sensor view.

4. The SDV of claim 3, wherein the encoded sensor data comprises image data from at least one camera sensor of the camera system.

5. The SDV of claim 1, wherein the executed instructions further cause the control system to:

detect, in the live sensor view, objects of interest along the current route; and

attempt to classify each of the objects of interest in accordance with the certainty threshold.

6. The SDV of claim 1, wherein the resolution response is determined by one or more human teleassistance operators.

7. The SDV of claim 1, wherein the executed instructions further cause the control system to:

duplicate transmission of the of the encoded sensor data by transmitting the encoded sensor data to the backend transport system using a second communication module of the plurality of communication modules.

8. A non-transitory computer readable medium storing instructions that, when executed by a control system of a self-driving vehicle (SDV), cause the control system to:

monitor bandwidth of each of a plurality of communication modules of the SDV to determine a highest bandwidth communication module;

analyze a live sensor view of the SDV to autonomously operate acceleration, braking, and steering systems of the SDV along a current route;

identify an indeterminate object in the live sensor view by determining that the indeterminate object does not meet a certainty threshold;

in response to identifying the indeterminate object, specify the indeterminate object in a bounding box;

encode sensor data identifying the indeterminate object within the bounding box for transmission over one or more networks by (i) encoding the bounding box at a first bitrate, and (ii) encoding a remainder of the sensor data at a second bitrate;

transmit the encoded sensor data to a backend transport system over the one or more networks using the highest bandwidth communication module;

receive, over the one or more networks, a resolution response from the backend transport system to resolve the indeterminate object; and

cause the SDV to proceed in accordance with the resolution response.

9. The non-transitory computer readable medium of claim 8, wherein the executed instructions further cause the control system to:

duplicate transmission of the of the encoded sensor data by transmitting the encoded sensor data to the backend transport system using a second communication module of the plurality of communication modules.

10. The non-transitory computer readable medium of claim 9, wherein the resolution response classifies the indeterminate object.

11. The non-transitory computer readable medium of claim 9, wherein the SDV comprises at least a LIDAR system and a camera system that combine to generate the live sensor view.

12. The non-transitory computer readable medium of claim 11, wherein the encoded sensor data comprises image data from at least one camera sensor of the camera system.

13. The non-transitory computer readable medium of claim 9, wherein the executed instructions further cause the control system to:

detect, in the live sensor view, objects of interest along the current route; and

attempt to classify each of the objects of interest in accordance with the certainty threshold.

14. The non-transitory computer readable medium of claim 9, wherein the resolution response is determined by human teleassistance operators.

15. A computer-implemented method of operating a self-driving vehicle (SDV), the method being performed by one or more processors and comprising:

monitoring bandwidth of each of a plurality of communication modules of the SDV to determine a highest bandwidth communication module;

analyzing a live sensor view of the SDV to autonomously operate acceleration, braking, and steering systems of the SDV along a current route;

identifying an indeterminate object in the live sensor view by determining that the indeterminate object does not meet a certainty threshold;

in response to identifying the indeterminate object, specify the indeterminate object in a bounding box;

encoding sensor data identifying the indeterminate object within the bounding box for transmission over one or more networks by (i) encoding the bounding box at a first bitrate, and (ii) encoding a remainder of the sensor data at a second bitrate;

transmitting the encoded sensor data to a backend transport system over the one or more networks using the highest bandwidth communication module;

receiving, over the one or more networks, a resolution response from the backend transport system to resolve the indeterminate object; and

causing the SDV to proceed in accordance with the resolution response.

16. The method of claim 15, wherein the resolution response classifies the indeterminate object.

17. The method of claim 15, wherein the SDV comprises at least a LIDAR system and a camera system that combine to generate the live sensor view.

18. The method of claim 17, wherein the encoded sensor data comprises image data from at least one camera sensor of the camera system.

19. The method of claim 15, further comprising:

detecting, in the live sensor view, objects of interest along the current route; and

attempting to classify each of the objects of interest in accordance with a certainty threshold.

20. The method of claim 15, further comprising:

duplicating transmission of the of the encoded sensor data by transmitting the encoded sensor data to the backend transport system using a second communication module of the plurality of communication modules.