Speech controlled computer user interface

• US 6,311,159 B1
• Filed: 10/05/1999
• Issued: 10/30/2001
• Est. Priority Date: 10/05/1998
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Lernout & Hauspie Speech Products NV (Intel Corporation)
• Inventors: Van Tichelen, Luc, Gallopyn, Guido
• Primary Examiner(s): {haeck over (S)}mits, Ta̅livaldis I.
• Assistant Examiner(s): Nolan, Daniel A.
• Application Number:
  US09/412,251
• Time in Patent Office: 756 Days
• Field of Search: 704/260, 704/275, 704/257, 704/251
• US Class Current: 704/275
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A speech controlled computer user interface communicates between a user and at least one application program. The user interface has a speech layer, an utterance layer, and a discourse layer. The speech layer is in communication with the user and converts between speech messages and text messages. The utterance layer is in communication with the speech layer, and converts between text messages and semantic meaning messages. The discourse layer is in communication with the utterance layer and the at least one application program, and processes messages from the user and the at least one application program, and generates responsive messages to the user and the at least one application program.

Claims

1. A speech controlled computer user interface for communicating between a user and at least one application program, the user interface comprising:

a speech layer in communication with the user that converts between speech messages and text messages;

an utterance layer in communication with the speech layer that converts between text messages and semantic meaning messages;

a discourse layer in communication with the utterance layer and the at least one application program that processes messages from the user and the at least one application program and generates responsive messages to the user and the at least one application program; and

a resource manager in communication with the discourse layer that manages use of system resources by the user interface.

2. A user interface according to claim 1, wherein the speech layer includes at least one of:

a DTMF module that converts Dial Tone Multi-Frequency (DTMF) tones into representative text-based codes;

an ASR module that converts speech signals into representative text using Automatic Speech Recognition (ASR) techniques;

an SMC module that converts acoustic signals into digitally encoded speech signals using Speech/Music Compression (SMC) techniques;

a concatenation module that converts text messages into electronic speech representative signals; and

a TTS (Text-to-Speech) module that converts text messages into representative acoustic speech signals.

3. A user interface according to claim 1, wherein the utterance layer includes a natural language understanding module that converts text messages from the speech layer into representative semantic meaning messages for the discourse layer.

4. A user interface according to claim 1, wherein the utterance layer includes a message generator module that converts semantic meaning messages from the discourse layer into representative text messages for the speech layer.

5. A user interface according to claim 1, wherein the discourse layer includes a dialogue manager based on a conversational agent model that analyzes internal beliefs, intentions, and desires that are associated with the user and the at least one application, updates the beliefs, and generates new intentions.

17. A user interface according to claim 1, further comprising:

a set of development tools that allow an application developer to integrate the user interface with an application program.

6. A speech controlled computer user interface for communicating between a user and at least one application program, the user interface comprising:

a speech layer in communication with the user that converts between speech messages and text messages;

an utterance layer in communication with the speech layer that converts between text messages and semantic meaning messages; and

a discourse layer in communication with the utterance layer and the at least one application program that processes messages from the user and the at least one application program and generates responsive messages to the user and the at least one application program, wherein the discourse layer includes;

(i) a dialogue manager based on a conversational agent model that analyzes internal beliefs, intentions, and desires that are associated with the user and the at least one application, updates the beliefs, and generates new intentions, and

(ii) an application perception module that converts application messages from the at least one application program into representative beliefs for the dialogue manager.

7. A speech controlled computer user interface for communicating between a user and at least one application program, the user interface comprising:

a speech layer in communication with the user that converts between speech messages and text messages;

an utterance layer in communication with the speech layer that conerts between text messages and semantic meaning messages; and

a discourse layer in communication with the utterance layer and the at least one application program that processes messages from the user and the at least one application program and generates responsive messages to the user and the at least one application program, wherein the discourse layer includes;

(i) a dialogue manager based on a conversational agent model that analyzes internal beliefs, intentions, and desires that are associated with the user and the at least one application, updates the beliefs, and generates new intentions, and

ii) an application action module that converts intentions from the dialogue manager into representative application messages for the at least one application program.

8. A speech controlled computer user interface for communicating between a user and at least one application program, the user interface comprising:

a speech layer in communication with the user that converts between speech messages and text messages;

an utterance layer in communication with the speech layer that conerts between text messages and semantic meaning messages; and

a discourse layer in communication with the utterance layer and the at least one application program that processes messages from the user and the at least one application program and generates responsive messages to the user and the at least one application program, wherein the discourse layer includes;

(i) a dialogue manager based on a conversational agent model that analyzes internal beliefs, intentions, and desires that are associated with the user and the at least one application, updates the beliefs, and generates new intentions, and

(ii) a speech perception module that converts semantic meaning messages from the utterance layer into representative beliefs for the dialogue manager.

9. A speech controlled computer user interface for communicating between a user and at least one application program, the user interface comprising:

a speech layer in communication with the user that converts between speech messages and text messages;

an utterance layer in communication with the speech layer that conerts between text messages and semantic meaning messages; and

a discourse layer in communication with the utterance layer and the at least one application program that processes messages from the user and the at least one application program and generates responsive messages to the user and the at least one application program, wherein the discourse layer includes;

(i) a dialogue manager based on a conversational agent model that analyzes internal beliefs, intentions, and desires that are associated with the user and the at least one application, updates the beliefs, and generates new intentions, and

(ii) a speech action module that converts intentions from the dialogue manager into representative semantic meaning messages for the utterance layer.

10. A speech controlled computer user interface for communicating between a user and at least one application program, the user interface comprising:

a speech layer in communication with the user that converts between speech messages and text messages;

an utterance layer in communication with the speech layer that conerts between text messages and semantic meaning messages; and

a discourse layer in communication with the utterance layer and the at least one application program that processes messages from the user and the at least one application program and generates responsive messages to the user and the at least one application program, wherein the discourse layer includes a dialogue manager based on a conversational agent model that analyzes internal beliefs, intentions, and desires that are associated with the user and the at least one application, updates the beliefs, and generates new intentions, wherein the dialogue manager uses a perception process that receives information from the user and the at least one application program and generates beliefs representative of current states of the user and the at least one application program.

11. A user interface according to claim 10, wherein the discourse layer includes a beliefs knowledge base in communication with the perception process that contains past and current beliefs for use by the dialogue manager.

12. A user interface according to claim 11, wherein the dialogue manager uses a planning process in communication with the beliefs knowledge base that determines how to change a current state to attain another possible state.

13. A user interface according to claim 12, wherein the discourse layer includes a desires knowledge base that contains goals for the dialogue manager to determine a desirability of alternate possible states.

14. A user interface according to claim 13, wherein the dialogue manager uses a commitment process in communication with the beliefs knowledge base and the desires knowledge base that compares the desirability of selected possible states and determines a desired policy based on the current state and the desirability of the selected possible states.

15. A user interface according to claim 14, wherein the discourse layer includes an intentions knowledge base in communication with the commitment process that maintains intentions representative of the desired policy.

16. A user interface according to claim 15, wherein the dialogue manager uses an acting process in communications with the intentions knowledge base that converts the intentions into information for the user and the at least one application program to accomplish the desired policy.

18. A method of communicating via a speech controlled computer user interface between a user and at least one application program, the method comprising:

converting between speech messages and text messages with a speech layer in communication with the user;

converting between text messages and semantic meaning messages with an utterance layer in communication with the speech layer;

processing messages from the user and the at least one application program with a discourse layer in communication with the utterance layer and the at least one application program, and generating responsive messages to the user and the at least one application program; and

managing, with a resource manager in communication with the discourse layer, use of system resources by the user interface.

19. A method according to claim 18, wherein the converting between speech messages and text messages includes at least one of:

converting Dial Tone Multi-Frequency (DTMF) tones into representative text-based codes with a DTMF module;

converting speech signals into representative text using Automatic Speech Recognition (ASR) techniques with an ASR module;

converting acoustic signals into digitally encoded speech signals using Speech/Music Compression (SMC) techniques with an SMC module;

converting text messages into electronic speech representative signals with a concatenation module;

converting text messages into representative acoustic speech signals with a Text-to-Speech (TTS) module; and

managing, with a resource manager in communication with the discourse layer, use of system resources by the user interface.

20. A method according to claim 18, wherein converting between text messages and semantic meaning messages includes converting, with a natural language understanding module, text messages from the speech layer into representative semantic meaning messages for the discourse layer.

21. A method according to claim 18, wherein converting between text messages and semantic meaning messages includes converting, with a message generator module, semantic meaning messages from the discourse layer into representative text messages for the speech layer.

22. A method according to claim 18, wherein processing messages includes analyzing, with a dialogue manager based on a conversational agent model, internal beliefs, intentions, and desires that are associated with the user and the at least one application, updating the beliefs, and generating new intentions.

34. A method according to claim 18, further comprising:

allowing, with a set of development tools, an application developer to integrate the user interface with an application program.

23. A method of communicating via a speech controlled computer user interface between a user and at least one application program, the method comprising:

converting between speech messages and text messages with a speech layer in communication with the user;

converting between text messages and semantic meaning messages with an utterance layer in communication with the speech layer;

processing messages from the user and the at least one application program with a discourse layer in communication with the utterance layer and the at least one application program, and generating responsive messages to the user and the at least one application program, the processing including analyzing, with a dialogue manager based on a conversational agent model, internal beliefs, intentions, and desires that are associated with the user and the at least one application, updating the beliefs, and generating new intentions;

wherein the analyzing with a dialogue manager includes converting, with an application perception module, application messages from the at least one application program into representative beliefs for the dialogue manager.

24. A method of communicating via a speech controlled computer user interface between a user and at least one application program, the method comprising:

converting between speech messages and text messages with a speech layer in communication with the user;

converting between text messages and semantic meaning messages with an utterance layer in communication with the speech layer;

processing messages from the user and the at least one application program with a discourse layer in communication with the utterance layer and the at least one application program, and generating responsive messages to the user and the at least one application program, the processing including analyzing, with a dialogue manager based on a conversational agent model, internal beliefs, intentions, and desires that are associated with the user and the at least one application, updating the beliefs, and generating new intentions;

wherein the analyzing with a dialogue manager includes converting, with an application action module, intentions from the dialogue manager into representative application messages for the at least one application program.

25. A method of communicating via a speech controlled computer user interface between a user and at least one application program, the method comprising:

converting between speech messages and text messages with a speech layer in communication with the user;

converting between text messages and semantic meaning messages with an utterance layer in communication with the speech layer;

processing messages from the user and the at least one application program with a discourse layer in communication with the utterance layer and the at least one application program, and generating responsive messages to the user and the at least one application program, the processing including analyzing, with a dialogue manager based on a conversational agent model, internal beliefs, intentions, and desires that are associated with the user and the at least one application, updating the beliefs, and generating new intentions;

wherein the analyzing with a dialogue manager includes converting, with a speech perception module, semantic meaning messages from the utterance layer into representative beliefs for the dialogue manager.

26. A method of communicating via a speech controlled computer user interface between a user and at least one application program, the method comprising:

converting between speech messages and text messages with a speech layer in communication with the user;

converting between text messages and semantic meaning messages with an utterance layer in communication with the speech layer;

processing messages from the user and the at least one application program with a discourse layer in communication with the utterance layer and the at least one application program, and generating responsive messages to the user and the at least one application program, the processing including analyzing, with a dialogue manager based on a conversational agent model, internal beliefs, intentions, and desires that are associated with the user and the at least one application, updating the beliefs, and generating new intentions;

wherein the analyzing with a dialogue manager includes converting, with a speech action module, intentions from the dialogue manager into representative semantic meaning messages for the utterance layer.

27. A method of communicating via a speech controlled computer user interface between a user and at least one application program, the method comprising:

converting between speech messages and text messages with a speech layer in communication with the user;

converting between text messages and semantic meaning messages with an utterance layer in communication with the speech layer;

processing messages from the user and the at least one application program with a discourse layer in communication with the utterance layer and the at least one application program, and generating responsive messages to the user and the at least one application program, the processing including analyzing, with a dialogue manager based on a conversational agent model, internal beliefs, intentions, and desires that are associated with the user and the at least one application, updating the beliefs, and generating new intentions;

wherein the analyzing with a dialogue manager includes receiving, with a perception process, information from the user and the at least one application program and generating beliefs representative of current states of the user and the at least one application program.

28. A method according to claim 27, wherein analyzing with a dialogue manager includes containing, in a beliefs knowledge base in communication with the perception process, past and current beliefs for use by the dialogue manager.

29. A method according to claim 28, wherein analyzing with a dialogue manager includes determining, with a planning process in communication with the beliefs knowledge base, how to change a current state to attain another possible state.

30. A method according to claim 29, wherein analyzing with a dialogue manager includes containing, in a desires knowledge base, goals for the dialogue manager to determine a desirability of alternate possible states.

31. A method according to claim 30, wherein analyzing with a dialogue manager includes comparing, with a commitment process in communication with the beliefs knowledge base and the desires knowledge base, the desirability of selected possible states, and determining a desired policy based on the current state and the desirability of the selected possible states.

32. A method according to claim 31, wherein analyzing with a dialogue manager includes maintaining, in an intentions knowledge base in communication with the commitment process, intentions representative of the desired policy.

33. A method according to claim 32, wherein analyzing with a dialogue manager includes converting, with an acting process in communications with the intentions knowledge base, the intentions into information for the user and the at least one application program to accomplish the desired policy.

35. A speech controlled computer user interface for communicating between a user and at least one application program, the interface comprising:

a perception process that receives information from the user and the at least one application program and generates beliefs representative of current states of the user and the at least one application program;

a beliefs knowledge base in communication with the perception process that contains past and current beliefs;

a planning process in communication with the beliefs knowledge base that determines how to change the current states;

a desires knowledge base that contains goals to determine a desirability of alternate possible states;

a commitment process in communication with the beliefs knowledge base and the desires knowledge base that compares desirability of selected possible states and determines a desired policy based on the current state and the desirability of the selected possible states;

an intentions knowledge base in communication with the commitment process that maintains intentions representative of the desired policy; and

an acting process in communications with the intentions knowledge base that converts the intentions into information for the user and the at least one application program to accomplish the desired policy.

36. A user interface as in claim 35, wherein the information received by the perception process from the user is provided by at least one of a speech music compression (SMC) process, an automatic speech recognition process (ASR), and a Dial Tone Multi-Frequency (DTMF) process.

37. A user interface as in claim 35, wherein the information received by the perception process from the user is in semantic meaning form from a natural language understanding process.

38. A user interface as in claim 35, wherein the information received by the perception process from the at least one application program includes at least one of keystrokes from a keyboard and selections from an application-associated menu.

39. A user interface as in claim 35, wherein the beliefs knowledge base uses data frames to model a conversation.

40. A user interface as in claim 35, wherein the acting process includes using artificial speech.

41. A user interface as in claim 35, wherein the acting process includes an acting queue that sequences the information provided by the acting process.

42. A user interface as in claim 35, wherein the intentions knowledge base is further in communication with an expectations process that defines a grammar to use for a speech-related process.

43. A user interface as in claim 42, wherein the speech-related processes include automatic speech recognition.

44. A user interface as in claim 42, wherein the speech-related processes include natural language understanding.

45. A method of communicating via a speech controlled computer user interface between a user and at least one application program, the method comprising:

receiving information from the user and the at least one application program with a perception process, and generating beliefs representative of current states of the user and the at least one application program;

containing past and current beliefs in a beliefs knowledge base in communication with the perception process;

determining how to change the current states with a planning process in communication with the beliefs knowledge base;

containing goals to determine a desirability of alternate possible states in a desires knowledge base;

comparing desirability of selected possible states with a commitment process in communication with the beliefs knowledge base and the desires knowledge base, and determining a desired policy based on the current state and the desirability of the selected possible states;

maintaining intentions representative of the desired policy in an intentions knowledge base in communication with the commitment process; and

converting, with an acting process in communications with the intentions knowledge base, the intentions into information for the user and the at least one application program to accomplish the desired policy.

46. A method as in claim 45, wherein the information received from the user is provided by at least one of a speech music compression (SMC) process, an automatic speech recognition process (ASR), and a Dial Tone Multi-Frequency (DTMF) process.

47. A method as in claim 45, wherein the information received from the user is in semantic meaning form from a natural language understanding process.

48. A method as in claim 45, wherein the information received by the perception process from the at least one application program includes at least one of keystrokes from a keyboard and selections from an application-associated menu.

49. A method as in claim 45, wherein containing past and current beliefs includes using data frames to model a conversation.

50. A method as in claim 45, wherein converting with an acting process includes using artificial speech.

51. A method as in claim 45, wherein converting with an acting process includes sequencing the information provided by the acting process with an acting queue.

52. A method as in claim 45, wherein maintaining intentions includes defining, with an expectations process, a grammar to use for a speech-related process.

53. A method as in claim 52, wherein the speech-related process includes an automatic speech recognition process.

54. A method as in claim 52, wherein the speech-related process includes a natural language understanding process.

55. A method for a user to use a spoken message to control at least one application program, the method comprising:

converting the spoken message to a semantic meaning message;

processing the semantic meaning message to generate a set of commands to control the at least one application program; and

managing, with a resource manager, use of system resources by the user interface.

56. A method according to claim 55, wherein the at least one application program is other than a word processing program.

57. A method according to claim 55, wherein converting the spoken message to a semantic meaning message further comprises converting the spoken message to a text message, then converting the text message to a semantic meaning message.

58. A method according to claim 57, wherein the converting between speech messages and text messages includes at least one of:

converting Dial Tone Multi-Frequency (DTMF) tones into representative text-based codes with a DTMF module;

converting speech signals into representative text using Automatic Speech Recognition (ASR) techniques with an ASR module;

converting acoustic signals into digitally encoded speech signals using Speech/Music Compression (SMC) techniques with an SMC module;

converting text messages into electronic speech representative signals with a concatenation module; and

converting text messages into representative acoustic speech signals with a Text-to-Speech (TTS) module.

59. A method according to claim 57, wherein converting between text messages and semantic meaning messages includes converting, with a natural language understanding module, text messages into representative semantic meaning messages.

60. A method according to claim 57, wherein converting between text messages and semantic meaning messages includes converting, with a message generator module, semantic meaning messages into representative text messages.

61. A method according to claim 57, wherein processing messages includes analyzing, with a dialogue manager based on a conversational agent model, internal beliefs, intentions, and desires that are associated with the user and the at least one application, updating the beliefs, and generating new intentions.

62. A method according to claim 57, further comprising:

allowing, with a set of development tools, an application developer to integrate the user interface with an application program.