System and method for hybrid processing in a natural language voice services environment

• US 9,171,541 B2
• Filed: 02/09/2010
• Issued: 10/27/2015
• Est. Priority Date: 11/10/2009
• Status: Active Grant

• Current Assignee: Cerence Inc.
• Original Assignee: VoiceBox Technologies Corporation (Microsoft Corporation)
• Inventors: Kennewick, Robert A., Armstrong, Lynn Elise
• Primary Examiner(s): SPOONER, LAMONT M
• Application Number:
  US12/703,032
• Publication Number:
  US 20110112827A1
• Time in Patent Office: 2,086 Days
• Field of Search: 704/1, 704/9, 704/10, 704/251, 704/257, 704/275
• US Class Current: 1/1
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

A system and method for hybrid processing in a natural language voice services environment that includes a plurality of multi-modal devices may be provided. In particular, the hybrid processing may generally include the plurality of multi-modal devices cooperatively interpreting and processing one or more natural language utterances included in one or more multi-modal requests. For example, a virtual router may receive various messages that include encoded audio corresponding to a natural language utterance contained in a multi-modal interaction provided to one or more of the devices. The virtual router may then analyze the encoded audio to select a cleanest sample of the natural language utterance and communicate with one or more other devices in the environment to determine an intent of the multi-modal interaction. The virtual router may then coordinate resolving the multi-modal interaction based on the intent of the multi-modal interaction.

Claims

1. A method of natural language utterance processing, the method being implemented in a computer system that includes one or more physical processors executing one or more computer program instructions which, when executed, perform the method, the method comprising:

receiving, at the computer system, from a first user device, a natural language utterance;

performing, at the computer system, speech recognition to determine one or more words of the natural language utterance;

receiving, at the computer system, from a second user device, a non-voice user input that relates to the natural language utterance, wherein the first user device and the second user device are associated with a user and are independently operable of one another and of the computer system;

processing, at the computer system, the non-voice user input to determine context information for the one or more words;

determining, at the computer system, one or more interpretations of the one or more words based on the context information;

generating, at the computer system, a user request related to the natural language utterance based on the one or more interpretations;

selecting, at the computer system, at least one user processing device to process the user request such that the selection of the user processing device is based on a determination that content related to the user request resides at the user processing device, wherein the user processing device comprises the first user device, the second user device, or a third user device independently operable of the computer system; and

transmitting, by the computer system, the user request to the user processing device to invoke the user processing device to process the user request.

2. The method of claim 1, further comprising:

receive, at the computer system, information related to local timing mechanisms of a plurality of user devices, wherein the local timing mechanisms comprises a first local timing mechanism of the first user device and a second local timing mechanism of the second user device;

determining, at the computer system, based on the first local timing mechanism, first timing information relating to a time of receipt of the natural language utterance at the first user device;

determining, at the computer system, based on the second local timing mechanism, second timing information relating to a time of receipt of the non-voice user input at the second user device; and

determining, at the computer system, whether the natural language utterance and the non-voice user input relate to one another based on the first and second timing information,

wherein determining the one or more interpretations comprises determining the one or more interpretations using the context information based on a determination that the natural language utterance and the non-voice user input relate to one another.

3. The method of claim 2, further comprising:

determining, at the computer system, an order sequence among the natural language utterance and the non-voice user input based on the first and second timing information,

wherein generating the user request comprises generating the user request based on the order sequence.

4. The method of claim 1, further comprising:

receiving, at the computer system, another natural language utterance from a device different than the first user device, wherein the natural language utterance and the other natural language utterance are received in response to a capturing of an utterance from the user while the user is in proximity to the first user device and the device different than the first user device,

wherein the speech recognition is performed on the natural language utterance based on a determination that the natural language utterance is a cleaner sample of the utterance than the other natural language utterance.

5. The method of claim 1, wherein the first user device and the second user device are mobile devices.

6. The method of claim 1, further comprising:

obtaining, at the computer system, one or more personalized cognitive models associated with the user, wherein the one or more personalized cognitive models are based on one or more prior user inputs associated with the user, and wherein the one or more interpretations are determined further based on the one or more personalized cognitive models.

7. The method of claim 1, wherein the user request comprises a query, and wherein the user processing device is selected based on a determination that content related to the query resides at the user processing device.

8. The method of claim 1, wherein the one or more words include a reference word, wherein the context information indicates context for the reference word, and wherein determining the one or more interpretations comprises:

identifying the reference word;

determining, based on the reference word context, an item to which the reference word refers; and

determining the one or more interpretations based on the determined item.

9. The method of claim 1, wherein the non-voice user input identifies text selected by the user, and the context information is determined based on the text.

10. The method of claim 1, wherein the non-voice user input identifies a point of focus on a user interface, and the context information is determined based on the point of focus.

11. The method of claim 1, further comprising:

determining, at the computer system, additional context information based on one or more of the natural language utterance or the non-voice user input,

wherein the selection of the user processing device is based on the additional context information.

12. The method of claim 1, further comprising:

receiving, at the computer system, from the first user device, a transcription of at least part of the natural language utterance, wherein the transcription comprises one or more other words of the natural language utterance,

wherein determining the one or more interpretations comprises determining, based on the context information, an interpretation of the one or more words and the one or more other words in the transcription, and

wherein the user request related to the natural language utterance is generated based on the interpretation of the one or more words and the one or more other words in the transcription.

13. The method of claim 1, wherein the computer system comprises one or more servers and a virtual router that is independently operable of the one or more servers,

wherein receiving the natural language utterance comprises receiving, at the virtual router, the natural language utterance from the first user device, and

wherein receiving the non-voice input comprises receiving, at the virtual router, the non-voice user input from the second user device, the method further comprising:

transmitting, by the virtual router, the natural language utterance and the non-voice user input to the one or more servers; and

receiving, at the virtual router, the user request from the one or more servers,

wherein selecting the user processing device comprises selecting, at the virtual router, from among a plurality of user processing devices, the user processing device to process the user request, the selection at the virtual router being based on the determination that content related to the user request resides at the user processing device, and

wherein transmitting the user request comprises transmitting, by the virtual router, the user request to the selected user processing device to invoke the selected user processing device to process the user request.

14. The method of claim 13, further comprising:

selecting, at the virtual router, from among the one or more servers and the plurality of user processing devices, the one or more servers to determine the one or more words and the context information,

wherein the natural language utterance and the non-voice user input are transmitted to the one or more servers based on the selection of the one or more servers.

15. The method of claim 13, further comprising:

receiving, at the virtual router, a second natural language utterance;

selecting, at the virtual router, based on context information associated with the second natural language utterance, at least one of the user processing device or another user processing device to determine one or more interpretations of the second natural language utterance;

transmitting, by the virtual router, the second natural language utterance to the selected at least one of the user processing devices to invoke the selected at least one of the user processing devices to determine the one or more interpretations of the second natural language utterance;

receiving, at the virtual router, from the selected at least one of the user processing devices, the one or more interpretations of the second natural language utterance; and

transmitting, by the virtual router, the one or more interpretations of the second natural language utterance to one or more of the plurality of user processing devices.

16. A system for natural language utterance processing, the system comprising:

one or more physical processors programmed to execute one or more computer program instructions which, when executed, cause the one or more physical processors to;

receive, from a first user device, a natural language utterance;

perform speech recognition to determine one or more words of the natural language utterance;

receive, from a second user device, a non-voice user input that relates to the natural language utterance, wherein the first user device and the second user device are associated with a user and are independently operable of one another and of the computer system;

process the non-voice user input to determine context information for the one or more words;

determine one or more interpretations of the one or more words based on the context information;

generate a user request related to the natural language utterance based on the one or more interpretations;

select at least one user processing device to process the user request such that the selection of the user processing device is based on a determination that content related to the user request resides at the user processing device, wherein the user processing device comprises the first user device, the second user device, or a third user device independently operable of the computer system; and

transmit the user request to the user processing device to invoke the user processing device to process the user request.

17. The system of claim 16, wherein the one or more physical processors are further caused to:

receive, at the computer system, information related to local timing mechanisms of a plurality of user devices, wherein the local timing mechanisms comprises a first local timing mechanism of the first user device and a second local timing mechanism of the second user device;

determine, based on the first local timing mechanism, first timing information relating to a time of receipt of the natural language utterance at the first user device;

determine, based on the second local timing mechanism, second timing information relating to a time of receipt of the non-voice user input at the second user device; and

determine whether the natural language utterance and the non-voice user input relate to one another based on the first and second timing information,

wherein determining the one or more interpretations comprises determining the one or more interpretations using the context information based on a determination that the natural language utterance and the non-voice user input relate to one another.

18. The system of claim 17, wherein the one or more physical processors are further caused to:

determine an order sequence among the natural language utterance and the non-voice user input based on the first and second timing information,

wherein generating the user request comprises generating the user request based on the order sequence.

19. The system of claim 16, wherein the one or more physical processors are further caused to:

receive another natural language utterance from a device different than the first user device, wherein the natural language utterance and the other natural language utterance are received in response to a capturing of an utterance from the user while the user is in proximity to the first user device and the device different than the first user device,

wherein the speech recognition is performed on the natural language utterance based on a determination that the natural language utterance is a cleaner sample of the utterance than the other natural language utterance.

20. The system of claim 16, wherein the one or more physical processors are further caused to:

obtain one or more personalized cognitive models associated with the user, wherein the one or more personalized cognitive models are based on one or more prior user inputs associated with the user, and wherein the one or more interpretations are determined further based on the one or more personalized cognitive models.

21. The system of claim 16, wherein the one or more words include a reference word, wherein the context information indicates context for the reference word, and wherein determining the one or more interpretations comprises:

identifying the reference word;

determining, based on the reference word context, an item to which the reference word refers; and

determining the one or more interpretations based on the determined item.

22. The system of claim 16, wherein the non-voice user input identifies text selected by the user, and the context information is determined based on the text.

23. The system of claim 16, wherein the non-voice user input identifies a point of focus on a user interface, and the context information is determined based on the point of focus.

24. The system of claim 16, wherein the one or more physical processors are further caused to:

determine additional context information based on one or more of the natural language utterance or the non-voice user input,

wherein the selection of the user processing device is based on the additional context information.

25. A method of natural language utterance processing, the method being implemented in a computer system that includes one or more physical processors executing one or more computer program instructions which, when executed, perform the method, the method comprising:

receiving, at the computer system, from a first user device, a natural language utterance;

processing, at the computer system, the natural language utterance to determine one or more words of the natural language utterance;

receiving, at the computer system, from a second user device, a non-voice user input that relates to the natural language utterance, wherein the first user device and the second user device are associated with a user and are independently operable of one another and of the computer system;

processing, at the computer system, the non-voice user input to determine context information for the one or more words; and

transmit, by the computer system, the one or more words and the context information to a user processing device to invoke the user processing device to determine one or more interpretations of the natural language utterance based on the one or more words and the context information, wherein the user processing device comprises the first user device, the second user device, or a third user device independently operable of the computer system.

26. The method of claim 25, wherein the natural language utterance is received at the first user device before the non-voice user input is received at the second user device.

27. The method of claim 25, wherein the natural language utterance is received at the first user device after the non-voice user input is received at the second user device.

28. The method of claim 25, further comprising:

determining, at the computer system, whether to invoke a peer-to-peer mode; and

selecting, at the computer system, based on a determination to invoke the peer-to-peer mode, the user processing device to determine the one or more interpretations of the natural language utterance based on the one or more words and the context information,

wherein the one or more words and the context information are transmitted to the user processing device based on the selection of the user processing device.

29. The method of claim 25, wherein the computer system comprises one or more servers and a virtual router that is independently operable of the one or more servers,

wherein receiving the natural language utterance comprises receiving, at the virtual router, the natural language utterance from the first user device, and

wherein receiving the non-voice input comprises receiving, at the virtual router, the non-voice user input from the second user device, the method further comprising:

transmitting, by the virtual router, the natural language utterance and the non-voice user input to the one or more servers;

receiving, at the virtual router, the one or more words and the context information from the one or more servers; and

selecting, at the virtual router, from among a plurality of user processing devices, the user processing device to determine the one or more interpretations of the natural language utterance,

wherein transmitting the one or more words and the context information comprises transmitting, by the virtual router, the one or more words and the context information to the selected user processing device to invoke the selected user processing device to determine the one or more interpretations of the natural language utterance based on the one or more words and the context information.

30. The method of claim 29, further comprising:

selecting, at the virtual router, from among the one or more servers and the plurality of user processing devices, the one or more servers to process the natural language utterance and the non-voice user input,

wherein the natural language utterance and the non-voice user input are transmitted to the one or more servers based on the selection of the one or more servers.

31. The method of claim 29, further comprising:

receiving, at the virtual router, a second natural language utterance;

selecting, at the virtual router, based on context information associated with the second natural language utterance, at least one of the user processing device or another user processing device to determine one or more interpretations of the second natural language utterance;

transmitting, by the virtual router, the second natural language utterance to the selected at least one of the user processing devices to invoke the selected at least one of the user processing devices to determine the one or more interpretations of the second natural language utterance;

receiving, at the virtual router, from the selected at least one of the user processing devices, the one or more interpretations of the second natural language utterance; and

transmitting, by the virtual router, the one or more interpretations of the second natural language utterance to one or more of the plurality of user processing devices.