Autonomous system operator cognitive state detection and alerting

• US 10,576,994 B1
• Filed: 08/22/2018
• Issued: 03/03/2020
• Est. Priority Date: 08/22/2018
• Status: Active Grant

• Current Assignee: NIO Technology Company Limited (Shanghai Anbin Technology Co., Ltd.)
• Original Assignee: Nio Usa Inc.
• Inventors: Higgins, James S., Singhal, Abhishek
• Primary Examiner(s): Nguyen, Hung T
• Application Number:
  US16/108,327
• Publication Number:
  US 20200062275A1
• Time in Patent Office: 559 Days
• Field of Search: 3404255, 340575, 340576, 4554261, 702 19, 702 94
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness

Abstract

Autonomous systems, such as vehicles being operated in a self-driving mode, may allow for human occupants to be inattentive to the operation of the system. However, events may occur that require a human user to perform an action. Systems and method are provided that enable an autonomous system to determine whether the system requires, or will require, a user action and whether the user is or is not attentive. If an action is required and the human is not attentive, an alerting protocol is executed to cause the system to perform actions that enable the user to be alerted, such as to enable the, now attentive, user to be able to perform the action.

Claims

1. An alerting component for an autonomous system, comprising:

a user sensor configured to sense a cognition-indicating attribute of a user utilizing the autonomous system and the user sensor is further configured to output a user sensor signal in accord with the sensed cognition-indicating attribute;

a system state input configured to receive a system state indicating whether the autonomous system requires the attention of the user;

a processor, comprising a memory for storing instructions for execution by the processor, and receiving the user sensor signal and the system state sensor; and

the processor, upon determining that each of the system state indicates the autonomous system is in a state that requires an action from the user and the user sensor signal indicates the user is presently inattentive, executes an alerting protocol; and

wherein the processor, as an output of the execution of the alerting protocol, signals the autonomous system to perform an alerting action; and

wherein the processor, having at least initiated execution of the alerting protocol, fails to receive an acknowledgement input to a device configured to receive an input from the user, provides a subsequent alerting signal to the autonomous system to perform an alerting action.

2. The alerting component for the autonomous system of claim 1, wherein the cognition-indicating attribute comprises a signal of a state of a component of the autonomous system.

3. The alerting component for the autonomous system of claim 1, wherein the component of the autonomous system is operable to indicate a usage by the user determinative, at least in part, of attentiveness.

4. The alerting component for the autonomous system of claim 1, wherein the processor signals the autonomous system to perform an alerting action, further comprising signaling an output component of the autonomous system to perform the alerting action.

5. The alerting component for an autonomous system of claim 1, wherein the processor signals the autonomous system to perform an alerting action, further comprising signaling a configuration component of the autonomous system to perform the alerting action comprising a reconfiguration of the configuration component.

6. The alerting component for the autonomous system of claim 1, wherein the processor, having at least initiated execution of the alerting protocol, receives either an updated value from the system state input indicating the system state does not require the action from the user or the user sensor signal indicating the user is not presently inattentive, terminates the alerting protocol.

7. The alerting component for the autonomous system of claim 6, wherein the processor determines the system state indicates the autonomous system is not in the state that requires the action from the user, when the system state indicates the autonomous system is currently being operated by the user or not in an autonomous mode.

8. The alerting component for the autonomous system of claim 1, wherein the processor executes the alerting protocol modified by configuration preferences and provides signals the autonomous system to perform an alerting action in accord with the configuration preferences.

9. The alerting component of the autonomous system of claim 1, wherein the alerting component comprises a portion of the autonomous system and the autonomous system is vehicle for conveying the user.

10. A method for alerting a user utilizing an autonomous system, comprising:

sensing a cognition-indicating attribute of the user;

sensing an attention requiring attribute of the autonomous system;

upon determining that each of cognitive-indicating attribute of the user indicates inattentiveness and attention requiring attribute of the autonomous system indicates the autonomous system is in a state that requires an action from the user,

executing an alerting protocol comprising outputting signals to the autonomous system to cause the autonomous system to perform an alerting action; and

upon having at least initiated execution of the alerting protocol, and failing to receive an acknowledgement input to a device configured to receive an input from the user, providing a subsequent alerting signal to the autonomous system to perform an alerting action.

11. The method of claim 10, wherein the cognition-indicating attribute comprises a signal of a state of the component of the autonomous system.

12. The method of claim 10, wherein outputting signals to the autonomous system to cause the autonomous system to perform the alerting action, further comprises outputting signals to the autonomous system to cause a component of the autonomous system to perform the alerting action.

13. The method of claim 10, wherein the alerting action, further comprises an alerting action signaling a configuration component of the autonomous system to perform the alerting action comprising a reconfiguration of the configuration component.

14. The method of claim 10, further comprising, having at least initiated execution of the alerting protocol, receiving either an updated value from the system state input indicating the system state does not require the action from the user or the user sensor signal indicating the user is not presently inattentive, and in response thereto, terminating the alerting protocol.

15. The method of claim 10, wherein receiving the updated value from the system state indicating the autonomous system is not in the state that requires the action from the user, comprising the system state indicates the autonomous system is currently being operated by the user or not in an autonomous mode.

16. The method of claim 10, wherein the alerting protocol is determined, at least in part, in accord with configured preferences.

17. The method of claim 10, wherein the alerting component comprises a portion of the autonomous system and the autonomous system is vehicle for conveying the user.

18. An autonomous vehicle, comprising:

a processor configured to operate the vehicle in an autonomous mode and a non-autonomous mode, comprising providing all control inputs to vehicle control components and operate the vehicle in a non-autonomous mode wherein a user input to a user input component provides at least one of the all control inputs to at least one of the vehicle control components;

a user sensor configured to sense a cognition-indicating attribute of the user utilizing the vehicle and, in response thereto, output a user sensor signal to the processor;

the processor, comprising a memory for storing instructions for execution by the processor, and receiving the user sensor signal; and

the processor, upon determining that the vehicle is in the autonomous mode and the user sensor signal indicates the user is presently inattentive, executes an alerting protocol; and

wherein the processor, as an output of the execution of the alerting protocol, signals the vehicle to perform an alerting action; and

wherein the processor, having at least initiated execution of the alerting protocol, fails to receive an acknowledgement input to a device configured to receive an input from the user, provides a subsequent alerting signal to the autonomous system to perform an alerting action.

19. The autonomous vehicle of claim 18, wherein:

the user sensor further comprises a configuration sensor to output the sensor signal indicating a configuration of a user-configured component of the vehicle; and

the processor, upon determining the sensor signal indicates the user-configured component of the vehicle is configured in a manner that hinders the user from being attentive, further determining the user sensor indicates the user is presently inattentive.