Intervention in operation of a vehicle having autonomous driving capabilities

• US 10,514,692 B2
• Filed: 06/16/2017
• Issued: 12/24/2019
• Est. Priority Date: 06/16/2017
• Status: Active Grant

• Current Assignee: Motional AD, LLC (Aptiv PLC)
• Original Assignee: nuTonomy, Inc. (BorgWarner Incorporated)
• Inventors: Liu, Shih-Yuan, Ravichandran, Harshavardhan, Iagnemma, Karl, Chang, Hsun-Hsien
• Primary Examiner(s): Ismail, Mahmoud S
• Application Number:
  US15/624,802
• Publication Number:
  US 20180364700A1
• Time in Patent Office: 921 Days
• Field of Search: 701 2, 701 23
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0055 with safety arrangements
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0816 Indicating performance data, e.g. occurrence of a malfunction

Abstract

Among other things, a determination is made that intervention in an operation of one or more autonomous driving capabilities of a vehicle is appropriate. Based on the determination, a person is enabled to provide information for an intervention. The intervention is caused in the operation of the one or more autonomous driving capabilities of the vehicle.

Claims

1. A method comprising:

determining, using a monitoring module of a vehicle, that a motion planning module of the vehicle is unable to generate a trajectory that includes traversable road segments, the trajectory for operating the vehicle;

responsive to the determining, generating, using the monitoring module, a teleoperation event;

generating, using a teleoperation event handling module of the vehicle, an intervention request, based on the teleoperation event, to a teleoperation server for an intervention, the intervention request comprising one or more un-traversable road segments;

receiving, from the teleoperation server, the intervention, wherein the intervention comprises a trajectory generated by the teleoperation server for operating the vehicle;

and

operating, using a teleoperation command handling module of the vehicle, the vehicle in accordance with the trajectory generated by the teleoperation server.

2. The method of claim 1, further comprising receiving or generating or analyzing information about a status or environment of the vehicle.

3. The method of claim 2, in which the information about the status or the environment of the vehicle comprises a functionality of a hardware component or software of the vehicle.

4. The method of claim 2, in which the information about the status or the environment of the vehicle comprises a signal from a hardware component or software of the vehicle.

5. The method of claim 2, in which the information about the status or the environment of the vehicle comprises presence of unexpected data or absence of expected data.

6. The method of claim 2, in which the information about the status or the environment of the vehicle comprises a mismatch between a measured quantity and a model-estimated quantity for a hardware component or software of the vehicle.

7. The method of claim 2, in which analyzing the information comprises using pattern recognition to evaluate an abnormal pattern in the information.

8. The method of claim 7, in which the abnormal pattern is learned by a machine learning algorithm.

9. The method of claim 2, in which analyzing the information comprises inferring a malfunction in the hardware component or the software.

10. The method of claim 2, in which analyzing the information comprises detecting an unknown object present in the environment of the vehicle.

11. The method of claim 2, in which analyzing the information comprises inferring an event that is or will be happening in the environment of the vehicle.

12. The method of claim 1, in which the intervention request further comprises data associated with status or environment of a vehicle or a related AV system.

13. The method of claim 1, in which the intervention request further comprises one or more signals from one or more hardware components or one or more software processes of the vehicle or a related AV system.

14. The method of claim 1, further comprising maintaining a queue of one or more intervention requests.

15. The method of claim 14, in which maintaining the queue comprises prioritizing an intervention request based on one or more of the following:

a decision tree, a combinatorial optimization, a machine algorithm, and a past intervention.

16. The method of claim 1, further comprising allocating a person to interact with the vehicle based on availability of the person, and one or more of:

(a) time, (b) knowledge of the vehicle, (c) knowledge of the environment of the vehicle, or (d) a language.

17. The method of claim 1, further comprising presenting an interactive interface including a field of view or a bird'"'"'s-eye of a vision sensor of the vehicle.

18. The method of claim 1, further comprising presenting an interactive interface including current or past or both perception information.

19. The method of claim 1, further comprising presenting an interactive interface including a current or a past or both trajectories.

20. The method of claim 1, further comprising presenting an interactive interface including current or past or both motion planning information.

21. The method of claim 1, further comprising presenting an interactive interface including a system diagram of the vehicle, the system diagram comprising one or more hardware components, or one or more software processes, or both.

22. The method of claim 1, in which the intervention further comprises a current location of the vehicle identified by a person, and the operating comprises treating the current location identified by the person as prior knowledge and using an inference algorithm to update the current location.

23. The method of claim 1, in which the intervention further comprises a goal location identified by a person, and the operating comprises treating the goal location identified as prior knowledge and using an inference algorithm to update the goal location.

24. The method of claim 1, in which the intervention further comprises a trajectory identified by a person, and the operating comprises treating the trajectory identified by the person as prior knowledge and using an inference algorithm to update the trajectory.

25. The method of claim 1, in which the intervention further comprises one or more trajectory sampling points identified by a person, and the operating comprises inferring a trajectory or a trajectory segment based on the one or more trajectory sampling points.

26. The method of claim 25, in which inferring a trajectory or a trajectory segment is based on one or more trajectory primitives.

27. The method of claim 25, in which the intervention further comprises concatenating two trajectory segments, the concatenating of the two trajectory segments comprising smoothing the trajectory segments and smoothing speed profiles across the trajectory segments.

28. The method of claim 1, in which the intervention further comprises setting a speed profile, and the intervention comprises treating the speed profile as prior knowledge and using an inference algorithm to update the speed profile.

29. The method of claim 1, in which the intervention is based on inferring a speed profile by a learning algorithm.

30. The method of claim 1, in which the intervention is based on inferring a steering angle by a learning algorithm.

31. The method of claim 1, in which the operating comprises enabling, editing or disabling a hardware component or a software process.

32. The method of claim 31, in which the operating further comprises enabling, editing or disabling a subcomponent of a hardware component or a processing step of a software process.

33. The method of claim 1, in which the intervention further comprises overwriting a travel preference or a travel rule.

34. The method of claim 1, in which the intervention further comprises editing data, the data comprising one or more of the following:

a map, sensor data in the vehicle, trajectory data in the vehicle, vision data in the vehicle, or any past data in the vehicle.