Vehicle personal assistant

• US 9,085,303 B2
• Filed: 11/15/2012
• Issued: 07/21/2015
• Est. Priority Date: 11/15/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Wolverton, Michael J., Mark, William S., Bratt, Harry, Bercow, Douglas A.
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): MYHRE, KEVIN C
• Application Number:
  US13/678,213
• Publication Number:
  US 20140136013A1
• Time in Patent Office: 978 Days
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/161 Explanation of functions, e.g. instructions
• CPC Class Codes: B60K 2360/165 Videos and animations
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60W 2040/089 Driver voice
• CPC Class Codes: B60W 50/085 Changing the parameters of the control units, e.g. changing limit values, working points by control input
• CPC Class Codes: B60W 50/10 Interpretation of driver requests or demands
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A vehicle personal assistant to engage a user in a conversational dialog about vehicle-related topics, such as those commonly found in a vehicle owner'"'"'s manual, includes modules to interpret spoken natural language input, search a vehicle knowledge base and/or other data sources for pertinent information, and respond to the user'"'"'s input in a conversational fashion. The dialog may be initiated by the user or more proactively by the vehicle personal assistant based on events that may be currently happening in relation to the vehicle. The vehicle personal assistant may use real-time inputs obtained from the vehicle and/or non-verbal inputs from the user to enhance its understanding of the dialog and assist the user in a variety of ways.

Claims

1. A vehicle personal assistant embodied in one or more non-transitory machine readable storage media, the vehicle personal assistant executable by a computing system to:

receive a plurality of different vehicle-related real-time sensor inputs indicative of a current state of a vehicle;

use one or more of the vehicle-related real-time sensor inputs to determine a current context of human activity in relation to the vehicle;

in a dialog involving a person and the vehicle personal assistant, receive a human-generated spoken natural language phrase comprising a plurality of words;

execute a semantic interpretation process on the human-generated spoken natural language phrase to derive, from the plurality of words, a trigger condition, a search action, and a search parameter to be used in executing the search action if the trigger condition is met;

use a first vehicle-related real-time sensor input to determine if the trigger condition is met;

determine a second vehicle-related real-time sensor input in response to the trigger condition being met;

with the second real-time sensor input and the search parameter derived from the human-generated spoken natural language phrase, execute the search action; and

determine a style of presentation of output of the executing of the search action based on the current context.

2. The vehicle personal assistant of claim 1, executable by a computing system to determine the current context by comparing the real-time sensor inputs to a vehicle-specific knowledge base.

3. The vehicle personal assistant of claim 1, wherein the real-time sensor inputs comprise one or more of accelerating, braking and steering inputs.

4. The vehicle personal assistant of claim 3, executable by a computing system to determine the current context by comparing the real-time sensor inputs to a stored template.

5. The vehicle personal assistant of claim 4, executable by a computing system to compare the current context to a stored template and, based on the comparison, determine an aspect of driving behavior.

6. The vehicle personal assistant of claim 5, executable by a computing system to determine the output based on the aspect of driving behavior.

7. The vehicle personal assistant of claim 5, executable by a computing system to determine the presentation style of the output based on the aspect of driving behavior.

8. The vehicle personal assistant of claim 7, executable by a computing system to determine the presentation style by comparing the real-time sensor inputs to a stored rule corresponding to the aspect of driving behavior.

9. The vehicle personal assistant of claim 1, wherein determining the style of the presentation of output comprises selecting one or more of conversational spoken natural language, text, and recorded media.

10. The vehicle personal assistant of claim 1, executable by the computing system to present the output using the style of presentation.

11. A vehicle personal assistant embodied in one or more non-transitory machine readable storage media, the vehicle personal assistant executable by a computing system to:

monitor a plurality of real-time sensor inputs relating to the vehicle;

in a dialog involving natural language speech supplied by a person and natural language output by the vehicle personal assistant, receive a human-generated spoken natural language phrase comprising a plurality of words;

execute a semantic interpretation process on the human-generated spoken natural language phrase to derive, from the plurality of words, a vehicle-related trigger condition, a search action, and a search parameter to be used in executing the search action if the trigger condition is met, the trigger condition comprising a data value;

store the data value in computer memory;

compare a real-time vehicle-related sensor input to the stored data value to determine whether the vehicle-related trigger condition has occurred; and

in response to determining that the vehicle-related trigger condition has occurred, execute the search action using (i) a different real-time sensor input than the real-time sensor input used to determine whether the vehicle-related trigger condition has occurred and (ii) the search parameter derived from the human-generated spoken natural language phrase; and

present system-generated output resulting from the executing of the search action.

12. The vehicle personal assistant of claim 11, executable by the computing system to present machine-generated spoken natural language output at a speaker of the vehicle in response to determining that the vehicle-related condition has occurred.

13. The vehicle personal assistant of claim 11, executable by the computing system to display one or more of text, images, graphics, and video on a display of the computing system in response to determining that the vehicle-related condition has occurred.

14. The vehicle personal assistant of claim 11, executable by the computing system to present a suggested action to be performed by the person in relation to the vehicle in response to determining that the vehicle-related condition has occurred.

15. The vehicle personal assistant of claim 11, executable by the computing system to communicate data relating to the vehicle-related condition to another computing system in response to determining that the vehicle-related condition has occurred.

16. The vehicle personal assistant of claim 11, executable by the computing system to receive the real-time sensor inputs from a vehicle network of the vehicle.

17. The vehicle personal assistant of claim 11, wherein the stored data comprises a value associated with the occurrence of the vehicle-related condition, and the vehicle personal assistant is executable by the computing system to compare the real-time sensor inputs to the value.

18. The vehicle personal assistant of claim 11, wherein the stored data relates to a user-specified driving situation associated with the occurrence of the vehicle-related condition, and the vehicle personal assistant is executable by the computing system to compare the real-time sensor inputs to the user-specified driving situation.

19. The vehicle personal assistant of claim 11, executable by the computing system to receive human-generated input in response to the system-generated output and present further system-generated output relating to the conversation in response to the human-generated input.

20. The vehicle personal assistant of claim 11, executable by the computing system to receive human-generated input in response to the system-generated output and continue the conversation based on the human-generated input.

21. The vehicle personal assistant of claim 11, executable by the computing system to monitor the real-time sensor inputs and compare the real-time sensor inputs to the stored data while the vehicle is turned on.

22. The vehicle personal assistant of claim 11, executable by the computing system to access one or more third-party data sources to determine whether the real-time sensor inputs indicate that the vehicle-related condition has occurred.

23. A vehicle personal assistant embodied in one or more non-transitory machine readable storage media, the vehicle personal assistant executable by a computing system to:

receive a human-generated spoken natural language phrase;

execute a semantic interpretation process on the human-generated spoken natural language phrase to derive, from the human-generated spoken natural language phrase, a trigger condition, a search action, and a search parameter to be used in executing the search action if the trigger condition is met, the trigger condition comprising a data value;

monitor one or more vehicle-related real-time sensor inputs;

compare a first vehicle-related real-time sensor input to the data value to determine whether the vehicle-related trigger condition has occurred; and

in response to determining that the trigger condition has occurred;

determine a current value of a second real-time input associated with the occurrence of the trigger condition; and

execute the search action using (i) the current value of the second real-time input and (ii) the search parameter derived from the human-generated spoken natural language phrase.

24. The vehicle personal assistant of claim 23, executable by the computing system to apply a vehicle-specific knowledge base to the human-generated input to interpret the vehicle-related triggering condition.

25. The vehicle personal assistant of claim 23, executable by the computing system to determine a current context based on the vehicle-related real-time sensor inputs and, in response to determining that the vehicle-related condition has occurred, present a system-generated output using a presentation style based on the current context.

26. The vehicle personal assistant of claim 25, executable by the computing system to determine the current context by comparing the vehicle-related real-time sensor inputs to a stored template.

27. The vehicle personal assistant of claim 23, executable by the computing system to compare the vehicle-related real-time sensor inputs to the vehicle-related condition while the vehicle is in operation.

28. The vehicle personal assistant of claim 23, executable by the computing system to receive the vehicle-related real-time sensor inputs from a vehicle network of the vehicle.

29. The vehicle personal assistant of claim 23, executable by the computing system to, in response to determining that the vehicle-related triggering condition has occurred, present machine-generated conversational spoken language output relating to the vehicle-related triggering condition.

30. The vehicle personal assistant of claim 25, executable by the computing system to receive further human-generated input in response to the system-generated output and determine whether to modify the vehicle-related triggering condition based on the further human-generated input.

31. The vehicle personal assistant of claim 30, wherein the further human-generated input comprises one or more of conversational spoken language input and non-verbal input.

32. The vehicle personal assistant of claim 23, executable by the computing system to present further system-generated output in response to the further human-generated input.