System and method of conversational adjustment based on user's cognitive state and/or situational state

• US 10,431,215 B2
• Filed: 12/06/2016
• Issued: 10/01/2019
• Est. Priority Date: 12/06/2015
• Status: Active Grant

• Current Assignee: Cerence Inc.
• Original Assignee: VoiceBox Technologies Corporation (Microsoft Corporation)
• Inventors: Cohen, Philip R.
• Primary Examiner(s): Kim, Jonathan C
• Application Number:
  US15/370,268
• Publication Number:
  US 20170162197A1
• Time in Patent Office: 1,029 Days
• Field of Search: None
• CPC Class Codes: B60R 16/0373 Voice control in general G10L
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

A system and method is provided for adjusting natural language conversations between a human user and a computer based on the human user'"'"'s cognitive state and/or situational state, particularly when the user is operating a vehicle. The system may disengage in conversation with the user (e.g., the driver) or take other actions based on various situational and/or user states. For example, the system may disengage conversation when the system detects that the driving situation is complex (e.g., car merging onto a highway, turning right with multiple pedestrians trying to cross, etc.). The system may (in addition or instead) sense the user'"'"'s cognitive load and disengage conversation based on the cognitive load. The system may alter its personality (e.g. by engaging in mentally non-taxing conversations such as telling jokes based on situational and/or user states.

Claims

1. A computer implemented method of adjusting natural language conversations between a user and a computer based on the user'"'"'s cognitive state while operating a vehicle, the method being implemented in the computer having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer to perform the method, the method comprising:

identifying, by the computer, a user of the vehicle;

receiving, by the computer, a first utterance from the user during a conversation with the user;

obtaining, by the computer, sensor information relating to at least a first physiological characteristic of the user;

determining, by the computer, a current cognitive state of the user based on the sensor information;

obtaining, by the computer, a baseline cognitive state of the user relating to the first physiological characteristic of the user, the baseline cognitive state of the user being representative of an average cognitive state of the user based on previously obtained, over the course of a plurality of different driving trips, sensor information corresponding to the first physiological characteristic;

comparing, by the computer, the current cognitive state of the user with the baseline cognitive state of the user;

determining, by the computer, that the current cognitive state of the user deviates from the baseline cognitive state of the user beyond a predetermined threshold based on the comparison;

delaying, by the computer, a response to the first utterance based on the determination that the current cognitive state of the user deviates from the baseline cognitive state of the user beyond the predetermined threshold;

placing, by the computer, said first utterance on a conversation stack;

receiving, by the computer, a second utterance from the user while the computer is delaying the response;

adding, by the computer, the second utterance to the conversation stack so that the conversation stack includes the first utterance and the second utterance, thus also delaying a response to the second utterance, and avoiding distracting the user; and

processing, by the computer, the second utterance to determine its intent in relation to the first utterance.

2. The method of claim 1, the method further comprising:

obtaining, by the computer, second sensor information relating to at least a second physiological characteristic of the user;

determining, by the computer, a second cognitive state of the user based on the sensor information;

determining, by the computer, that the conversation should be re-engaged with the user based on the second cognitive state of the user; and

providing, by the computer, the response to the first utterance in response to determining that the conversation should be re-engaged with the user.

3. The method of claim 1, the method further comprising:

altering, by the computer, a behavior of the computer, other than delaying the response, based on the cognitive state.

4. The method of claim 3, wherein altering the behavior comprises:

providing, by the computer, a joke instead of the response based on the current cognitive state.

5. The method of claim 1, the method further comprising:

obtaining, by the computer, information relating to the vehicle or its surroundings; and

determining, by the computer, a situational state based on the information relating to the vehicle or its surroundings,

wherein delaying the response is based further on the situational state.

6. The method of claim 5, wherein the information relating to the vehicle or its surroundings comprises global positioning system (GPS) information that indicates a current location of the vehicle.

7. The method of claim 6, wherein determining the situational state comprises:

obtaining location information comprising one or more coordinates that define a location that requires heightened driver attention; and

determining that the current location of the vehicle corresponds to the location based on the location information, wherein the situational state is based further on the determination that the current location of the vehicle corresponds to the locations.

8. The method of claim 5, the method further comprising:

providing, by the computer, a natural language alert to the user based on the cognitive state of the user or based on the current situational state.

9. The method of claim 5, wherein delaying the response to the first utterance further comprises:

obtaining, by the computer, a first predefined threshold relating to the situational state, the first predefined threshold corresponding to a threshold that if deviated from by an amount will trigger the delaying;

altering, by the computer, the first predefined threshold to generate a second predefined threshold responsive to the determination that the current cognitive state of the user deviates from the baseline cognitive state of the user beyond a predetermined threshold;

comparing, by the computer, the situational state with the second predefined threshold; and

determining, by the computer, that the situational state deviates from the second predefined threshold by at least the amount, wherein delaying the response is based on the determination that the situational state deviates from the second predefined threshold by at least the amount.

10. The method of claim 1, wherein the vehicle comprises an automobile and wherein the computer is integrated with the automobile.

11. The method of claim 1, wherein identifying the user of the vehicle occurs in association with a startup of the vehicle and includes:

identifying, by the computer, a key fob used to access the vehicle; and

associating, by the computer, an identity of the user with the key fob.

12. The method of claim 1, wherein identifying the user of the vehicle occurs in association with a startup of the vehicle and includes:

obtaining, by the computer, pre-stored biometric information associated with the user;

obtaining, by the computer, sensor information indicating biometric information of the user; and

comparing, by the computer, the pre-stored biometric information with the biometric information, wherein the identification of the user is based on the comparison of the pre-stored biometric information with the biometric information.

13. The method of claim 1, wherein identifying the user of the vehicle occurs in association with a startup of the vehicle and includes:

identifying, by the computer, a user device communicably coupled to the vehicle; and

associating, by the computer, an identity of the user with the user device.

14. The method of claim 1, the method further comprising:

updating, by the computer, the baseline cognitive state of the user based on the sensor information relating to the first physiological characteristic of the user.

15. The method of claim 1, wherein the first physiological characteristic of the user comprises at least one of a pupil position of the user or a speech of the user.

16. The method of claim 1, wherein delaying the response to the first utterance comprises:

obtaining, by the computer, one or more adjustment rules that each specify actions to take in response to a corresponding cognitive state of the user; and

selecting, by the computer, an adjustment rule based on the current cognitive state of the user, wherein the selected adjustment rule specifies that responses to user utterances should be delayed.

17. A system of adjusting natural language conversations between a user and a computer based on the user'"'"'s cognitive state while operating a vehicle, the system comprising:

a computer comprising one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer to;

identify a user of the vehicle;

receive a first utterance from the user during a conversation with the user;

obtain sensor information relating to at least a first physiological characteristic of the user;

determine a current cognitive state of the user based on the sensor information;

obtain a baseline cognitive state of the user relating to the first physiological characteristic of the user, the baseline cognitive state of the user being representative of an average cognitive state of the user based on previously obtained sensor information, over the course of a plurality of different driving trips, corresponding to the first physiological characteristic;

compare the current cognitive state of the user with the baseline cognitive state of the user;

determine that the current cognitive state of the user deviates from the baseline cognitive state of the user beyond a predetermined threshold based on the comparison;

delay a response to the first utterance based on the determination that the current cognitive state of the user deviates from the baseline cognitive state of the user beyond the predetermined threshold;

place said first utterance on a conversation stack;

receive a second utterance from the user while the response is delayed by the computer;

add the second utterance to the conversation stack so that the conversation stack includes the first utterance and the second utterance, and thus also delays a response to the second utterance, and avoids distracting the user; and

process the second utterance to determine its intent in relation to the first utterance.

18. The system of claim 17, wherein the computer is further programmed to:

obtain second sensor information relating to at least a second physiological characteristic of the user;

determine a second cognitive state of the user based on the sensor information;

determine that the conversation should be re-engaged with the user based on the second cognitive state of the user; and

provide the response to the first utterance in response to the determination that the conversation should be re-engaged with the user.

19. The system of claim 17, wherein the computer is further programmed to:

alter a behavior of the computer, other than delay the response, based on the cognitive state.

20. The system of claim 19, wherein to alter the behavior, the computer is further programmed to:

provide a joke instead of the response based on the current cognitive state.

21. The system of claim 17, wherein the computer is further programmed to:

obtain information relating to the vehicle or its surroundings; and

determine a situational state based on the information relating to the vehicle or its surroundings,

wherein to delay the response is based further on the situational state.

22. The system of claim 21, wherein the information relating to the vehicle or its surroundings comprises global positioning system (GPS) information that indicates relating to a current location or heading indicating heightened driver attention is required of the vehicle.

23. The system of claim 22, wherein to determine the situational state, the computer is further programmed to:

obtain location information comprising one or more coordinates that define a location that requires heightened driver attention, and

determine that the current location of the vehicle corresponds to the location based on the location information, wherein the situational state is based further on the determination that the current location of the vehicle corresponds to the locations.

24. The system of claim 21, wherein the computer is further programmed to:

provide a natural language alert to the user based on the current cognitive state of the user or based on the situational state.

25. The system of claim 17, wherein the vehicle comprises an automobile and wherein the computer is integrated with the automobile.