Mobile systems and methods for responding to natural language speech utterance

• US 9,031,845 B2
• Filed: 02/12/2010
• Issued: 05/12/2015
• Est. Priority Date: 07/15/2002
• Status: Expired due to Term

• Current Assignee: Dialect, LLC
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Kennewick, Robert A., Locke, David, Kennewick, Michael R. Sr., Kennewick, Michael R. Jr., Kennewick, Richard, Freeman, Tom, Elston, Stephen F.
• Primary Examiner(s): Shah, Paras D
• Assistant Examiner(s): Sirjani, Fariba
• Application Number:
  US12/705,311
• Publication Number:
  US 20100145700A1
• Time in Patent Office: 1,915 Days
• Field of Search: 704/270, 704/275
• US Class Current: 704/270
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G10L 15/18 using natural language modelling

Abstract

Mobile systems and methods that overcomes the deficiencies of prior art speech-based interfaces for telematics applications through the use of a complete speech-based information query, retrieval, presentation and local or remote command environment. This environment makes significant use of context, prior information, domain knowledge, and user specific profile data to achieve a natural environment for one or more users making queries or commands in multiple domains. Through this integrated approach, a complete speech-based natural language query and response environment can be created. The invention creates, stores and uses extensive personal profile information for each user, thereby improving the reliability of determining the context and presenting the expected results for a particular question or command. The invention may organize domain specific behavior and information into agents, that are distributable or updateable over a wide area network. The invention can be used in dynamic environments such as those of mobile vehicles to control and communicate with both vehicle systems and remote systems and devices.

Claims

1. A mobile system for processing natural language utterances, comprising:

one or more physical processors at a vehicle that are programmed to execute one or more computer program instructions which, when executed, cause the one or more physical processors to;

receive a natural language utterance associated with a user;

perform speech recognition on the natural language utterance;

parse and interpret the speech recognized natural language utterance;

determine a domain and a context that are associated with the parsed and interpreted natural language utterance;

formulate a command or query based on the domain and the context;

determine whether the command or query is to be executed on-board or off-board the vehicle;

execute the command or query at the vehicle in response to a determination that the command or query is to be executed on-board the vehicle; and

invoke a device that communicates wirelessly over a wide area network to process the command or query such that the command or query is executed off-board the vehicle in response to a determination that the command or query is to be executed off-board the vehicle.

2. The mobile system of claim 1, wherein executing the command or query at the vehicle comprises executing the command or query at a device built into the vehicle.

3. The mobile system of claim 1, wherein executing the command or query at the vehicle comprises executing the command or query at a device docked to the vehicle.

4. The mobile system of claim 1, wherein the instructions cause the one or more physical processors to:

determine whether executing the command or query will create a hazardous condition for the vehicle;

provide interactive guidance to resolve the hazardous condition via an output device connected to the vehicle based on a determination that executing the command or query will create the hazardous condition; and

receive an input that manually overrides the hazardous condition determination, wherein the command or query is executed based on the manual override.

5. The mobile system of claim 1, wherein the instructions cause the one or more physical processors to:

associate one or more emergency words or phrases with the user; and

identify the one or more emergency words or phrases in one or more recognized words or phrases of the natural language utterance,

wherein the device is invoked to process the command or query to summon emergency help further based on the identification of the one or more emergency words or phrases.

6. The mobile system of claim 1, wherein the instructions cause the one or more physical processors to:

detect an accident situation from information received from one or more sensors connected to the vehicle;

provide one or more interactive announcements or warnings via an output device connected to the vehicle in response to detecting the accident situation;

receive an input that is in response to the one or more interactive announcements or warnings, the input indicating a nature of the accident situation or a condition of a person in the vehicle; and

communicate the nature of the accident situation or the condition of the person in the vehicle to emergency personnel over a communications link.

7. The mobile system of claim 1, wherein executing the command or query at the vehicle comprises executing the command or query at a device that is in or on the vehicle and that is in wireless communication with the vehicle.

8. The mobile system of claim 1, wherein the command or query includes a search query that is to be executed on-board the vehicle by searching one or more data sources that are on-board the vehicle using at least one recognized word or phrase of the natural language utterance, and wherein the search query relates to an aspect of the vehicle.

9. The mobile system of claim 8, wherein the search query relates to status information associated with one or more on-board systems of the vehicle.

10. The mobile system of claim 8, wherein the instructions cause the one or more physical processors to:

provide a natural language speech response to the natural language utterance based on one or more results from the search of the one or more data sources.

11. The mobile system of claim 1, wherein the instructions cause the one or more physical processors to:

detect that the vehicle is approaching or has exceeded a travel limit for the vehicle;

provide interactive guidance for returning to the travel limit via an output device connected to the vehicle in response to detecting that the vehicle is approaching or has exceeded the travel limit; and

determine whether to extend the travel limit.

12. The mobile system of claim 1, wherein the command or query is to be executed on-board the vehicle, and wherein the command or query includes a command to control the vehicle.

13. The mobile system of claim 1, wherein the instructions cause the one or more physical processors to:

identify a second command or query based on the natural language utterance;

determine, based on the natural language utterance, whether the second command or query is to be executed on-board or off-board the vehicle;

execute the second command or query at the vehicle in response to a determination that the second command or query is to be executed on-board the vehicle; and

invoke a device that communicates wirelessly over a wide area network to process the second command or query such that the second command or query is executed off-board the vehicle in response to a determination that the second command or query is to be executed off-board the vehicle.

14. The mobile system of claim 13, wherein the command or query is executed at the vehicle and the second command or query is executed off-board the vehicle, or wherein the command or query is executed off-board the vehicle and the second command or query is executed on-board the vehicle.

15. The mobile system of claim 1, wherein the instructions cause the one or more physical processors to:

identify a context associated with the natural language utterance based on a current or prior dialog between the system and the user,

wherein the determination of whether the command or query is to be executed on-board or off-board the vehicle is further based on the context.

16. The mobile system of claim 1, wherein the system has access to domain-specific agents, and wherein the instructions cause the one or more physical processors to:

identify a first domain-specific agent to execute the command or query based on the domain, wherein the command or query is executed at the vehicle or executed off-board the vehicle further based on whether the first domain-specific agent is at the vehicle or on a system remote from the vehicle;

identify a second command or query based on the natural language utterance;

determine a second domain associated with natural language utterance;

identify a second domain-specific agent to execute the second command or query based on the second domain;

determine whether the second command or query is to be executed on-board or off-board the vehicle based on whether the second domain-specific agent is at the vehicle or on a system remote from the vehicle;

execute the second command or query at the vehicle in response to a determination that the second command or query is to be executed on-board the vehicle; and

invoke a device that communicates wirelessly over a wide area network to process the second command or query such that the second command or query is executed off-board the vehicle in response to a determination that the second command or query is to be executed off-board the vehicle.

17. The mobile system of claim 1, wherein the command or query includes a command that is to be executed at the vehicle, and wherein the instructions cause the one or more physical processors to:

identify a query based on the natural language utterance;

invoke a device that communicates wirelessly over a wide area network to process the query such that the query is executed off-board the vehicle based on a determination that the query is to be executed off-board the vehicle; and

receive, from a device off-board the vehicle, a response to the query, wherein the command is executed at the vehicle based on the query response.

18. A computer-implemented method for processing natural language utterances, the method being implemented by a computer system that includes one or more physical processors at a vehicle executing one or more computer program instructions which, when executed, perform the method, the method comprising:

receiving, at the one or more physical processors, a natural language utterance associated with a user;

performing, by the one or more physical processors, speech recognition on the natural language utterance;

parsing and interpreting, by the one or more physical processors, the speech recognized natural language utterance;

determining, by the one or more physical processors, a domain and a context that are associated with the parsed and interpreted natural language utterance;

formulating, by the one or more physical processors, a command or query based on the domain and the context;

determining, by the one or more physical processors, whether the command or query is to be executed on-board or off-board the vehicle;

executing, by the one or more physical processors, the command or query at the vehicle in response to a determination that the command or query is to be executed on-board the vehicle; and

invoking, by the one or more physical processors, a device that communicates wirelessly over a wide area network to process the command or query such that the command or query is executed off-board the vehicle in response to a determination that the command or query is to be executed off-board the vehicle.

19. The method of claim 18, wherein executing the command or query at the vehicle comprises executing the command or query at a device built into the vehicle.

20. The method of claim 18, wherein executing the command or query at the vehicle comprises executing the command or query at a device docked to the vehicle.

21. The method of claim 18, further comprising:

determining, by the one or more physical processors, whether executing the command or query will create a hazardous condition for the vehicle;

providing, by the one or more physical processors, interactive guidance to resolve the hazardous condition via an output device connected to the vehicle based on a determination that executing the command or query will create the hazardous condition; and

receiving, at the one or more physical processors, an input that manually overrides the hazardous condition determination, wherein the command or query is executed based on the manual override.

22. The method of claim 18, further comprising:

associating, by the one or more physical processors, one or more emergency words or phrases with the user; and

identifying, by the one or more physical processors, the one or more emergency words or phrases in one or more recognized words or phrases of the natural language utterance,

wherein the device is invoked to process the command or query to summon emergency help further based on the identification of the one or more emergency words or phrases.

23. The method of claim 18, further comprising:

detecting, by the one or more physical processors, an accident situation from information received from one or more sensors connected to the vehicle;

providing, by the one or more physical processors, one or more interactive announcements or warnings via an output device connected to the vehicle in response to detecting the accident situation;

receiving, at the one or more physical processors, an input that is in response to the one or more interactive announcements or warnings, the input indicating a nature of the accident situation or a condition of a person in the vehicle; and

communicating, by the one or more physical processors, the nature of the accident situation or the condition of the person in the vehicle to emergency personnel over a communications link.

24. The method of claim 18, wherein executing the command or query at the vehicle comprises executing the command or query at a device that is in or on the vehicle and that is in wireless communication with the vehicle.

25. The method of claim 18, wherein the command or query includes a search query that is to be executed on-board the vehicle by searching one or more data sources that are on-board the vehicle using at least one recognized word or phrase of the natural language utterance, and wherein the search query relates to an aspect of the vehicle.

26. The method of claim 25, wherein the search query relates to status information associated with one or more on-board systems of the vehicle.

27. The method of claim 25, further comprising:

providing, by the one or more physical processors, a natural language speech response to the natural language utterance based on one or more results from the search of the one or more data sources.

28. The method of claim 18, further comprising:

detecting, by the one or more physical processors, that the vehicle is approaching or has exceeded a travel limit for the vehicle;

providing, by the one or more physical processors, interactive guidance for returning to the travel limit via an output device connected to the vehicle in response to detecting that the vehicle is approaching or has exceeded the travel limit; and

determining, by the one or more physical processors, whether to extend the travel limit.

29. The method of claim 18, wherein the command or query is to be executed on-board the vehicle, and wherein the command or query includes a command to control the vehicle.

30. The method of claim 18, further comprising:

identifying, by the one or more physical processors, a second command or query based on the natural language utterance;

determining, by the one or more physical processors, based on the natural language utterance, whether the second command or query is to be executed on-board or off-board the vehicle;

executing, by the one or more physical processors, the second command or query at the vehicle in response to a determination that the second command or query is to be executed on-board the vehicle; and

invoking, by the one or more physical processors, a device that communicates wirelessly over a wide area network to process the second command or query such that the second command or query is executed off-board the vehicle in response to a determination that the second command or query is to be executed off-board the vehicle.

31. The method of claim 30, wherein the command or query is executed at the vehicle and the second command or query is executed off-board the vehicle, or wherein the command or query is executed off-board the vehicle and the second command or query is executed on-board the vehicle.

32. A computer-implemented method of processing natural language utterances, the method being implemented by a computer system that includes one or more physical processors at a vehicle executing one or more computer program instructions which, when executed, perform the method, the method comprising:

receiving, at the one or more physical processors, a natural language utterance associated with a user;

performing, by the one or more physical processors, speech recognition on the natural language utterance;

parsing and interpreting, by the one or more physical processors, the speech recognized natural language utterance;

determining, by the one or more physical processors, a domain and a context that are associated with the parsed and interpreted natural language utterance;

formulating, by the one or more physical processors, a search query based on the domain and the context wherein the search query includes at least one recognized word or phrase of the natural language utterance;

determining, by the one or more physical processors, whether the search query is to be executed on-board or off-board a vehicle;

executing, by the one or more physical processors, the search query at the vehicle in response to a determination that the search query is to be executed on-board the vehicle; and

invoking, by the one or more physical processors, a device that communicates wirelessly over a wide area network to process the search query such that the search query is executed off-board the vehicle in response to a determination that the search query is to be executed off-board the vehicle.

33. The method of claim 32, further comprising:

identifying, by the one or more physical processors, a command or query based on the natural language utterance, wherein the command or query is different than the search query;

determining, by the one or more physical processors, based on the natural language utterance, whether the command or query is to be executed on-board or off-board the vehicle;

executing, by the one or more physical processors, the command or query at the vehicle in response to a determination that the command or query is to be executed on-board the vehicle; and

invoking, by the one or more physical processors, a device that communicates wirelessly over a wide area network to process the command or query such that the command or query is executed off-board the vehicle in response to a determination that the command or query is to be executed off-board the vehicle.

34. The method of claim 32, wherein the search query is to be executed on-board the vehicle by searching one or more data sources that are on-board the vehicle using the at least one recognized word or phrase, and wherein the search query relates to an aspect of the vehicle.

35. The method of claim 34, wherein the search query relates to status information associated with one or more on-board systems of the vehicle.

36. The method of claim 34, further comprising:

providing, by the computer system, a natural language speech response to the natural language utterance based on one or more results from the search of the one or more data sources.