INTERVENTION IN OPERATION OF A VEHICLE HAVING AUTONOMOUS DRIVING CAPABILITIES

• US 20180365908A1
• Filed: 06/16/2017
• Published: 12/20/2018
• Est. Priority Date: 06/16/2017
• Status: Active Grant

• Current Assignee: Motional AD, LLC (Aptiv PLC)
• Original Assignee: nuTonomy, Inc. (Aptiv PLC)
• Inventors: Liu, Shih-Yuan, Ravichandran, Harshavardhan, Iagnemma, Karl, Chang, Hsun-Hsien
• Granted Patent: US 10,740,988 B2
• Time in Patent Office: Days
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/005 by providing the operator with signals other than visual, e.g. acoustic, haptic
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/224 Output arrangements on the remote controller, e.g. displays, haptics or speakers
• CPC Class Codes: G05D 1/225 operated by off-board computers
• CPC Class Codes: G05D 1/226 Communication links with the remote-control arrangements
• CPC Class Codes: G05D 1/227 Handing over between remote control and on-board control; Handing over between remote control arrangements
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G07C 5/006 Indicating maintenance
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C

Abstract

Among other things, a determination is made that intervention in an operation of one or more autonomous driving capabilities of a vehicle is appropriate. Based on the determination, a person is enabled to provide information for an intervention. The intervention is caused in the operation of the one or more autonomous driving capabilities of the vehicle.

Claims

1. A method comprising:

determining that intervention in an operation of one or more autonomous driving capabilities of a vehicle is appropriate;

based on the determination, enabling a person to provide information for an intervention; and

causing the intervention in the operation of the one or more autonomous driving capabilities of the vehicle.

2. The method of claim 1, in which determining that intervention is appropriate comprises receiving a request for intervention.

3. The method of claim 1, in which determining that intervention is appropriate comprises receiving information about a status or environment of the vehicle or a related AV system.

4. The method of claim 3, in which the status or the environment of the vehicle comprises a functionality of a hardware component or software of the vehicle or the AV system.

5. The method of claim 3, in which the information about the status or the environment of the vehicle or the AV system comprises a signal from a hardware component or software of the vehicle or the AV system.

6. The method of claim 5, in which determining that intervention is appropriate comprises analyzing the signal to detect presence of unexpected data or absence of expected data.

7. The method of claim 6, in which analyzing the signal comprises evaluating a mismatch between a measured quantity and a model-estimated quantity for the hardware component or software.

8. The method of claim 6, in which analyzing the signal comprises using pattern recognition to evaluate an abnormal pattern in the signal.

9. The method of claim 8, in which the abnormal pattern is learned by a machine learning algorithm.

10. The method of claim 6, in which analyzing the signal comprises inferring a malfunction in the hardware component or the software.

11. The method of claim 6, in which analyzing the signal comprises detecting an unknown object present in the environment of the vehicle or a related AV system.

12. The method of claim 6, in which analyzing the signal comprises inferring an event that is or will be happening in the environment of the vehicle or a related AV system.

13. The method of claim 2, in which the request comprises a request initiated over a wireless communication channel by a remote operator.

14. The method of claim 2, in which the request comprises data associated with a status or environment of the vehicle or a related AV system.

15. The method of claim 2, in which the request comprises one or more signals from one or more hardware components or one or more software processes of the vehicle or a related AV system.

16. The method of claim 1, comprising, based on the determination, causing a fallback intervention in the operation of the one or more autonomous driving capabilities of the vehicle.

17. The method of claim 16, in which the fallback intervention comprises causing the vehicle or a related AV system to enter a fully autonomous driving mode, a semi-autonomous driving mode, or a fully manual driving mode.

18. The method of claim 16, in which the fallback intervention comprises causing the vehicle to operate at a reduced velocity.

19. The method of claim 16, in which the fallback intervention comprises identifying a safe-to-stop location.

20. The method of claim 19, in which the fallback intervention comprises generating a new trajectory to the safe-to-stop location.

21. The method of claim 16, in which the fallback intervention comprises invoking a backup hardware component or a backup software process.

22. The method of claim 16, in which the fallback intervention comprises evaluating a functional hardware component or a functional software processes required to operate the vehicle.

23. The method of claim 1, in which determining that intervention is appropriate comprises evaluating one or more active events associated with the vehicle or a related AV system, or associated with the environment of the vehicle or a related AV system.

24. The method of claim 23, in which evaluating one or more events comprises merging two or more active events.

25. The method of claim 1, in which enabling the person to provide information for an intervention comprises maintaining a queue based on one or more determinations that intervention is appropriate.

26. The method of claim 25, in which maintaining the queue comprises prioritizing an intervention based on one or more of the following:

a decision tree, a combinatorial optimization, a machine algorithm, and a past intervention.

27. The method of claim 1, in which enabling the person to provide information for an intervention comprises allocating the person to provide the information based on availability of the person, and one or more of:

(a) time, (b) knowledge of the vehicle, (c) knowledge of the environment of the vehicle, or (d) a language.

28. The method of claim 1, in which enabling the person to provide information for an intervention comprises presenting an interactive interface.

29. The method of claim 28, in which presenting an interactive interface comprises presenting a field of view or a bird'"'"'s-eye of a vision sensor of the vehicle.

30. The method of claim 28, in which presenting an interactive interface comprises presenting current or past or both perception information.

31. The method of claim 28, in which presenting an interactive interface comprises presenting current or past or both trajectories.

32. The method of claim 28, in which presenting an interactive interface comprises presenting current or past or both motion planning information.

33. The method of claim 28, in which presenting an interactive interface comprises presenting a system diagram of the vehicle, the system diagram comprising one or more hardware components, or one or more software processes, or both.

34. The method of claim 1, in which the information for the intervention comprises a current location of the vehicle determined by the person, and the intervention comprises treating the current location as prior knowledge and using an inference algorithm to update the current location.

35. The method of claim 1, in which the intervention is based on the person identifying a goal location of the vehicle, and the intervention comprises treating the goal location as prior knowledge and using an inference algorithm to update the goal location.

36. The method of claim 1, in which the intervention comprises a trajectory to be found by the person, and the intervention comprises treating the trajectory as prior knowledge and using an inference algorithm to update the trajectory.

37. The method of claim 1, in which the intervention comprises one or more trajectory sampling points identified by the person, and the intervention comprises inferring a trajectory or a trajectory segment based on the one or more trajectory sampling points.

38. The method of claim 37, in which inferring a trajectory or a trajectory segment is based on one or more trajectory primitives.

39. The method of claim 37, in which the intervention comprises concatenating two trajectory segments.

40. The method of claim 39, in which concatenating two trajectory segments comprises smoothing the trajectory segments and smoothing speed profiles across the trajectory segments.

41. The method of claim 1, in which the intervention comprises specifying one or more un-traversable road segments.

42. The method of claim 1, in which the intervention comprises setting a speed profile by the person, and the intervention comprises treating the speed profile as prior knowledge and using an inference algorithm to update the speed profile.

43. The method of claim 1, in which the intervention is based on inferring a speed profile by a learning algorithm.

44. The method of claim 1, in which the intervention is based on inferring a steering angle by a learning algorithm.

45. The method of claim 1, in which the intervention comprises enabling, editing or disabling a hardware component or a software process.

46. The method of claim 45, in which the intervention comprises enabling, editing or disabling a subcomponent of a hardware component or a processing step of a software process.

47. The method of claim 1, in which the intervention comprises overwriting a travel preference or a travel rule.

48. The method of claim 1, in which the intervention comprises editing data, the data comprising one or more of the following:

a map, sensor data in the vehicle or a related AV system, trajectory data in the vehicle or a related AV system, vision data in the vehicle or a related AV system, or any past data in the vehicle or a related AV system.

49. The method of claim 1, comprising configuring the vehicle or a related AV system based on a command, the configuration comprising treating the command as prior knowledge and using an inference algorithm to update the command.