Teleassistance data prioritization for self-driving vehicles

• US 10,293,818 B2
• Filed: 03/07/2017
• Issued: 05/21/2019
• Est. Priority Date: 03/07/2017
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Uber Technologies, Inc.
• Inventors: Kroop, Benjamin, Ross, William, Heine, Andrew
• Primary Examiner(s): Badii, Behrang
• Assistant Examiner(s): Testardi, David A
• Application Number:
  US15/452,431
• Publication Number:
  US 20180257643A1
• Time in Patent Office: 805 Days
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2050/0215 Sensor drifts or sensor failures
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2552/45 Pedestrian sidewalk
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4026 Cycles
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/802 Longitudinal distance
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/20 Steering systems
• CPC Class Codes: B60W 2720/106 Longitudinal acceleration
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 50/029 Adapting to failures or work around with other constraints, e.g. circumvention by avoiding use of failed parts
• CPC Class Codes: B60W 60/0015 specially adapted for safety
• CPC Class Codes: B60W 60/00253 Taxi operations
• CPC Class Codes: B60W 60/0027 using trajectory prediction for other traffic participants
• CPC Class Codes: B60W 60/00276 : for two or more other traffic participants
• CPC Class Codes: G05D 1/0022 : characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0038 : by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0061 : for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G05D 1/0231 : using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G07C 5/008 : communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: H04W 84/04 : Large scale networks; Deep hierarchical networks
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]

Abstract

A self-driving vehicle can operate by analyzing a live sensor view to autonomously operate acceleration, braking, and steering systems of the SDV along a current route. Based on a detected anomaly associated with the live sensor view, the SDV can transmit a teleassistance inquiry to a backend transport system in accordance with a data prioritization scheme. The SDV can receive a resolution response from the backend transport system based on the teleassistance inquiry, and proceed in accordance with the resolution response.

Claims

1. A self-driving vehicle (SDV) comprising:

a communication interface;

a sensor system comprising one or more LIDAR sensors generating LIDAR data and one or more cameras generating image data, the LIDAR data and image data providing a live sensor view of a situational environment of the SDV;

acceleration, braking, and steering systems; and

a control system to execute instructions that cause the control system to;

analyze the live sensor view in accordance with a safety threshold to (i) detect objects of interest along a current route, (ii) classify each of the detected objects of interest in accordance with a certainty threshold, and (iii) autonomously operate the acceleration, braking, and steering systems along the current route;

when the safety threshold is not met, transmit a first teleassistance inquiry to a backend transport system, the first teleassistance inquiry comprising LIDAR data;

when the certainty threshold for an indeterminate object is not met, transmit a second teleassistance inquiry to the backend transport system, the second teleassistance inquiry comprising image data that includes the indeterminate object;

for each of the first and the second teleassistance inquiries, receive a resolution response from the backend transport system; and

autonomously operate the acceleration, braking, and steering systems to proceed in accordance with the resolution response.

2. The SDV of claim 1, wherein the safety threshold not being met corresponds to an obstruction in the live sensor view.

3. The SDV of claim 1, wherein the executed instructions cause the control system to transmit LIDAR data and not image data when the safety threshold is not met.

4. The SDV of claim 1, wherein the executed instructions cause the control system to transmit image data and not LIDAR data when the certainty threshold is not met.

5. The SDV of claim 1, wherein the communication interface comprises a plurality of wireless communication modules enabling communication with the backend transport system via a plurality of wireless communication providers, and wherein the executed instructions further cause the control system to:

monitor bandwidth of each of the plurality of wireless communication modules to determine a highest bandwidth communication module;

wherein the executed instructions cause the control system to transmit each of the first and the second teleassistance inquiries to the backend transport system using the highest bandwidth communication module.

6. The SDV of claim 5, wherein the executed instructions further cause the control system to:

duplicate transmission of each of the first and the second teleassistance inquiries by transmitting the first and the second teleassistance inquiries over a second communication module from the plurality of wireless communication modules.

7. The SDV of claim 6, wherein the highest bandwidth communication module and the second communication module both comprise Long Term Evolution (LTE) communications modules.

8. The SDV of claim 6, wherein the highest bandwidth communication module comprises a LTE communication module and the second communication module comprises a Wi-Fi module.

9. The SDV of claim 1, wherein the first and the second teleassistance inquiries include telemetry data for the SDV.

10. The SDV of claim 1, wherein the first and the second teleassistance inquiries include localization information for the SDV.

11. The SDV of claim 1, wherein the first and the second teleassistance inquiries include diagnostics data for the SDV.

12. A non-transitory computer readable medium storing instructions that, when executed by a control system of a self-driving vehicle (SDV), cause the control system to:

analyze a live sensor view of the SDV in accordance with a safety threshold to (i) detect objects of interest along a current route, (ii) classify each of the detected objects of interest in accordance with a certainty threshold, and (iii) autonomously operate acceleration, braking, and steering systems of the SDV along the current route, wherein the live sensor view comprises LIDAR data from one or more LIDAR sensors and image data from one or more cameras of the SDV;

when the safety threshold is not met, transmit a first teleassistance inquiry to a backend transport system, the first teleassistance inquiry comprising LIDAR data;

when the certainty threshold for an indeterminate object is not met, transmit a second teleassistance inquiry to the backend transport system, the second teleassistance inquiry comprising image data that includes the indeterminate object;

for each of the first and the second teleassistance inquiries, receive a resolution response from the backend transport system; and

autonomously operate the acceleration, braking, and steering systems to proceed in accordance with the resolution response.

13. The non-transitory computer readable medium of claim 12, wherein the safety threshold not being met corresponds to an obstruction in the live sensor view.

14. The non-transitory computer readable medium of claim 12, wherein the executed instructions cause the control system to transmit LIDAR data and not image data when the safety threshold is not met.

15. The non-transitory computer readable medium of claim 12, wherein the SDV comprises a plurality of wireless communication modules enabling communication with the backend transport system via a plurality of wireless communication providers, and wherein the executed instructions further cause the control system to:

monitor bandwidth of each of the plurality of wireless communication modules to determine a highest bandwidth communication module;

wherein the executed instructions cause the control system to transmit each of the first and the second teleassistance inquiries to the backend transport system using the highest bandwidth communication module.

16. A computer-implemented method of operating a self-driving vehicle (SDV), the method being performed by a control system of the SDV and comprising:

analyzing a live sensor view of the SDV in accordance with a safety threshold to (i) detect objects of interest along a current route, (ii) classify each of the detected objects of interest in accordance with a certainty threshold, and (iii) autonomously operate acceleration, braking, and steering systems of the SDV along the current route, wherein the live sensor view comprises LIDAR data from one or more LIDAR sensors and image data from one or more cameras of the SDV;

in response to the safety threshold not being met, transmit a first teleassistance inquiry to a backend transport system, the first teleassistance inquiry comprising LIDAR data;

in response to the certainty threshold for an indeterminate object not being met, transmit a second teleassistance inquiry to the backend transport system, the second teleassistance inquiry comprising image data that includes the indeterminate object;

for each of the first and the second teleassistance inquiries, receive a resolution response from the backend transport system; and

autonomously operate the acceleration, braking, and steering systems to proceed in accordance with the resolution response.