Vehicle personal assistant that interprets spoken natural language input based upon vehicle context

• US 9,798,799 B2
• Filed: 11/15/2012
• Issued: 10/24/2017
• Est. Priority Date: 11/15/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Wolverton, Michael J., Mark, William S., Bratt, Harry, Bercow, Douglas A.
• Primary Examiner(s): WOZNIAK, JAMES S
• Application Number:
  US13/678,209
• Publication Number:
  US 20140136187A1
• Time in Patent Office: 1,804 Days
• Field of Search: 704251, 704257-258, 704270, 7042701, 704275, 701 36, 715863
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A vehicle personal assistant to engage a user in a conversational dialog about vehicle-related topics, such as those commonly found in a vehicle owner'"'"'s manual, includes modules to interpret spoken natural language input, search a vehicle knowledge base and/or other data sources for pertinent information, and respond to the user'"'"'s input in a conversational fashion. The dialog may be initiated by the user or more proactively by the vehicle personal assistant based on events that may be currently happening in relation to the vehicle. The vehicle personal assistant may use real-time inputs obtained from the vehicle and/or non-verbal inputs from the user to enhance its understanding of the dialog and assist the user in a variety of ways.

Claims

1. A computer-implemented method, comprising:

receiving, by a vehicle computing system, natural language input interpreted from audio input, wherein the natural language input is associated with a vehicle, and wherein the natural language input is received after an interval of time has passed since a previous input was received;

receiving visual input, wherein the visual input is associated with the vehicle, and wherein the visual input includes an image of a person when the person is in proximity to the vehicle;

determining a current context of the vehicle, wherein determining includes using vehicle-related inputs;

determining that the natural language input is associated with a previous dialog, wherein the previous dialog includes the previous input, wherein determining includes using a vehicle-specific conversation model, wherein the vehicle-specific conversation model includes one or more rules for determining when a conversation has started or ended, and wherein the vehicle-specific conversation model associates the interval of time with a pause that is associated with an operation related to the vehicle;

determining a query, wherein the query is determined using the natural language input, the visual input, the previous dialog, and the current context of the vehicle;

using the query to determine information about the vehicle, wherein the information is determined using a vehicle user guide; and

presenting the information using a component of the vehicle.

2. The computer-implemented method of claim 1, wherein the information about the vehicle includes information for troubleshooting a component of the vehicle.

3. The computer-implemented method of claim 1, wherein determining that the natural language input is associated with the previous dialog includes determining that the previous input was incomplete.

4. The computer-implemented method of claim 1, wherein determining that the natural language input is associated with the previous dialog includes determining content associated with the natural language input and determining that the previous dialog includes the associated content.

5. The computer-implemented method of claim 1, wherein the vehicle-specific conversation model incudes one or more rules for determining whether the natural language input is associated with other inputs, and wherein the other inputs include the vehicle-related inputs.

6. The computer-implemented method of claim 1, wherein determining the query includes using touch input generated by one or more touch sensors in the vehicle.

7. The computer-implemented method of claim 1, wherein the current context of the vehicle includes a current operational state of a component the vehicle.

8. The computer-implemented method of claim 1, wherein determining the current context of the vehicle includes using historical information to determine a particular driving situation for the vehicle, wherein a driving situation describes a behavior related to the vehicle, and wherein the historical information records one or more recent operational states of the vehicle.

9. The computer-implemented method of claim 1, further comprising:

using the visual input to determine a geospatial reference, wherein a geospatial reference identifies a point in space related to the vehicle.

10. The computer-implemented method of claim 1, further comprising:

determining that additional information is needed to determine the query; and

executing an action to solicit the additional information.

11. The computer-implemented method of claim 1, further comprising:

modifying the query, wherein the query is modified using the current context of the vehicle.

12. The computer-implemented method of claim 1, wherein the information about the vehicle is determined using an external data source, wherein the external data source is accessible using a network interface of the vehicle.

13. A vehicle computing system, comprising:

one or more processors; and

a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including;

receiving natural language input interpreted from audio input, wherein the natural language input is associated with a vehicle, and wherein the natural language input is received after an interval of time has passed since a previous input was received;

receiving visual input, wherein the visual input is associated with the vehicle, and wherein the visual input includes an image of a person when the person is in proximity to the vehicle;

determining a current context of the vehicle, wherein determining includes using vehicle-related inputs;

determining that the natural language input is associated with a previous dialog, wherein the previous dialog includes the previous input, wherein determining includes using a vehicle-specific conversation model, wherein the vehicle-specific conversation model includes one or more rules for determining when a conversation has started or ended, and wherein the vehicle-specific conversation model associates the interval of time with a pause that is associated with an operation related to the vehicle;

determining a query, wherein the query is determined using the natural language input, the visual input, the previous dialog, and the current context of the vehicle;

using the query to determine information about the vehicle, wherein the information is determined using a vehicle user guide; and

presenting the information using a component of the vehicle.

14. The vehicle computing system of claim 13, wherein the information about the vehicle includes information for troubleshooting a component of the vehicle.

15. The vehicle computing system of claim 13, wherein the instructions for determining that the natural language input is associated with the previous dialog include instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

determining that the previous input was incomplete.

16. The vehicle computing system of claim 13, wherein the instructions for determining that the natural language input is associated with the previous dialog include instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

determining content associated with the natural language input and determining that the previous dialog includes the associated content.

17. The vehicle computing system of claim 13, wherein the vehicle-specific conversation model includes one or more rules for determining whether the natural language input is associated with other inputs, and wherein the other inputs include the vehicle-related inputs.

18. The vehicle computing system of claim 13, wherein the instructions for determining the query include instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

using touch input generated by one or more touch sensors in the vehicle.

19. The vehicle computing system of claim 13, wherein the current context of the vehicle includes a current operational state of a component the vehicle.

20. The vehicle computing system of claim 13, wherein the instructions for determining the current context of the vehicle include instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

using historical information to determine a particular driving situation for the vehicle, wherein a driving situation describes a behavior related to the vehicle, and wherein the historical information records one or more recent operational states of the vehicle.

21. The vehicle computing system of claim 13, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

using the visual input to determine a geospatial reference, wherein a geospatial reference identifies a point in space related to the vehicle.

22. The vehicle computing system of claim 13, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

determining that additional information is needed to determine the query; and

executing an action to solicit the additional information.

23. The vehicle computing system of claim 13, wherein the non-transitory computer-readable medium further includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:

modifying the query, wherein the query is modified using the current context of the vehicle.

24. The vehicle computing system of claim 13, wherein the information about the vehicle is determined using an external data source, wherein the external data source is accessible using a network interface of the vehicle.

25. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions that, when executed by one or more processors, cause the one or more processors to:

receive natural language input interpreted from audio input, wherein the natural language input is associated with a vehicle, and wherein the natural language input is received after an interval of time has passed since a previous input was received;

receive visual input, wherein the visual input is associated with the vehicle, and wherein the visual input includes an image of a person when the person is in proximity to the vehicle;

determine a current context of the vehicle, wherein determining includes using vehicle-related inputs;

determine that the natural language input is associated with a previous dialog, wherein the previous dialog includes the previous input, wherein determining includes using a vehicle-specific conversation model, wherein the vehicle-specific conversation model includes one or more rules for determining when a conversation has started or ended, and wherein the vehicle-specific conversation model associates the interval of time with a pause that is associated with an operation related to the vehicle;

determine a query, wherein the query is determined using the natural language input, the visual input, the previous dialog, and the current context of the vehicle;

use the query to determine information about the vehicle, wherein the information is determined using a vehicle user guide; and

present the information using a component of the vehicle.

26. The computer-program product of claim 25, wherein the information about the vehicle includes information for troubleshooting a component of the vehicle.

27. The computer-program product of claim 25, wherein the instructions for determining that the natural language input is associated with the previous dialog include instructions that, when executed by the one or more processors, cause the one or more processors to:

determine that the previous input was incomplete.

28. The computer-program product of claim 25, wherein the instructions for determining that the natural language input is associated with the previous dialog include instructions that, when executed by the one or more processors, cause the one or more processors to:

determine content associated with the natural language input and determining that the previous dialog includes the associated content.

29. The computer-program product of claim 25, wherein the vehicle-specific conversation model includes one or more rules for determining whether the natural language input is associated with other inputs, and wherein the other inputs include the vehicle-related inputs.

30. The computer-program product of claim 25, wherein the instructions for determining the query include instructions that, when executed by the one or more processors, cause the one or more processors to:

use touch input generated by one or more touch sensors in the vehicle.

31. The computer-program product of claim 25, wherein the current context of the vehicle includes a current operational state of a component the vehicle.

32. The computer-program product of claim 25, wherein the instructions for determining the current context of the vehicle include instructions that, when executed by the one or more processors, cause the one or more processors to:

use historical information to determine a particular driving situation for the vehicle, wherein a driving situation describes a behavior related to the vehicle, and wherein the historical information records one or more recent operational states of the vehicle.

33. The computer-program product of claim 25, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

use the visual input to determine a geospatial reference, wherein a geospatial reference identifies a point in space related to the vehicle.

34. The computer-program product of claim 25, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

determine that additional information is needed to determine the query; and

execute an action to solicit the additional information.

35. The computer-program product of claim 25, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

modify the query, wherein the query is modified using the current context of the vehicle.

36. The computer-program product of claim 25, wherein the information about the vehicle is determined using an external data source, wherein the external data source is accessible using a network interface of the vehicle.