Speech user interface for portable personal devices

• US 20020055844A1
• Filed: 02/26/2001
• Published: 05/09/2002
• Est. Priority Date: 02/25/2000
• Status: Abandoned Application

• Current Assignee: ScanSoft, Inc. n/k/a Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: ScanSoft, Inc. n/k/a Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Goslin, Mark, Verhoeve, Piet, Smolders, Johan, Hemenway, Erin, Niblack, Eric, Schell, Alan, LʼEsperance, Lauren
• Application Number:
  US09/793,377
• Publication Number:
  US 20020055844A1
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04M 1/271 controlled by voice recognition
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 2250/56 including a user help function

Abstract

A handheld electronic device such as a personal digital assistant (PDA) has multiple application processes. A speech recognition process takes input speech from a user and produces a recognition output representative of the input speech. A text-to-speech process takes output text and produces a representative speech output. A speech manager interface allows the speech recognition process and the text-to-speech process to be accessed by other application processes.

Claims

1. A handheld electronic device having automatic speech recognition, the device comprising:

a. a speech pre-processor that receives input speech and produces a target signal representative of the input speech;

b. a database of speaker independent acoustic models, each acoustic model representing a word or subword unit in a recognition vocabulary, each acoustic model being representative of its associated word or subword unit as spoken in a plurality of acoustic environments; and

c. a speech recognizer that compares the target signal to the acoustic models and generates a recognition output of at least one word or subword unit in the recognition vocabulary representative of the input speech.

2. A handheld electronic device according to claim 1, further including:

d. a language model that characterizes context-dependent probability relationships of words in the recognition vocabulary, wherein the speech recognizer compares the target signal to the acoustic models and the language model to generate the recognition output.

3. A handheld electronic device according to claim 1, wherein the plurality of acoustic environments includes a first acoustic environment and a second acoustic environment, the second acoustic environment having more background noise than the first acoustic environment.

4. A handheld electronic device according to claim 3, wherein the second acoustic environment is the passenger compartment of an automobile or airplane.

5. A handheld electronic device according to claim 1, wherein the database further includes acoustic models for non-word sounds, and no recognition output is generated when the speech recognizer determines that a non-word sound acoustic model represents the input speech.

6. A handheld electronic device according to claim 1, wherein the device further includes a user interface output module that communicates information to a user.

7. A handheld electronic device according to claim 6, wherein the user interface output module includes a user interface display that displays text to the user, the text being representative of the recognition output.

8. A handheld electronic device according to claim 6, wherein the user interface output module includes an audio output module that generates audio output for communication to the user.

9. A handheld electronic device according to claim 8, wherein the audio output module generates an audio cue output for the user each time the speech recognizer generates a recognition output.

10. A handheld electronic device according to claim 8, further comprising:

e. a text-to-speech application that processes output text, and produces a representative speech output to the audio output module.

11. A handheld electronic device according to claim 10, further comprising:

f. a speech manager interface that allows the speech recognizer and the text-to-speech application to be accessed by other applications, so as to prevent more than one instantiation of the speech recognizer and one instantiation of the text-to-speech application at any given time.

12. A handheld electronic device according to claim 11, wherein the speech manager interface further includes a dialog manager that manages a plurality of speech recognition grammars, each grammar being associated with at least one application, the dialog manager selecting a current recognition grammar for the speech recognizer, the current recognition grammar being associated with a currently selected application.

13. A handheld electronic device according to claim 11, further comprising:

g. a speech tips module in communication with the speech recognizer and the user interface output module, the speech tips module using the output module to indicate to the user commands currently available to the user.

14. A handheld electronic device according to claim 13, wherein the speech tips module indicates to the user all commands currently available to the user.

15. A handheld electronic device according to claim 13, wherein the speech tips module operates responsive to a user input signal.

16. A handheld electronic device according to claim 15, wherein the user input signal is operation of a microphone on/off button on the device.

17. A handheld electronic device according to claim 13, wherein the commands currently available to the user are indicated to the user for a predetermined length of time.

18. A handheld electronic device according to claim 17, wherein the predetermined length of time is selectable by the user.

19. A handheld electronic device according to claim 13, wherein when the device is in a selected state, the speech tips module automatically indicates commands currently available.

20. A handheld electronic device according to claim 19, wherein the selected state is a microphone enabled condition.

21. A handheld electronic device according to claim 13, wherein the speech tips module uses a first perceptually distinctive characteristic to indicate global commands that are always available, and a second perceptually distinctive characteristic to indicate context-dependent commands that are currently available.

22. A handheld electronic device according to claim 21, wherein the speech tips module uses a visual display to indicate commands currently available, and wherein the first perceptually distinctive characteristic is a first distinctive text appearance and the second perceptually distinctive characteristic is a second text appearance.

23. A handheld electronic device according to claim 21, wherein the speech tips module uses an audio indication to indicate commands currently available, and wherein the first perceptually distinctive characteristic is a first voice, and the second perceptually distinctive characteristic is a second voice.

24. A handheld electronic device according to claim 1, further comprising:

h. an audio processor including;

i. a microphone module that generates an electrical signal representative of a spoken input from the user, and provides the electrical signal to the speech pre-processor, and

ii. an output module that generates sound intended for the user; and

i. an audio duplexing module responsive to a current state of the device, the duplexing module enabling one module in the processor to operate and disabling the other module from operation.

25. A handheld electronic device according to claim 24, wherein the audio duplexing module is further responsive to a user input signal.

26. A handheld electronic device according to claim 25, wherein the user input signal is operation of a microphone on/off button on the device.

27. A handheld electronic device according to claim 24, wherein the device further indicates to a user which module is currently enabled by the audio duplexing module.

28. A handheld electronic device according to claim 24, wherein the audio duplexer is further responsive to a user command.

29. A handheld electronic device according to claim 1, wherein the recognition output represents a command to control operation of the device.

30. A handheld electronic device according to claim 1, wherein the device is a personal digital assistant (PDA) device having a personal information manager (PIM) application.

31. A handheld electronic device comprising:

a. a plurality of application processes available for interaction with a user, including;

i. a speech recognition process that processes input speech from a user, and produces a recognition output representative of the input speech,

ii. a text-to-speech process that processes output text, and produces a representative speech output, and

iii. an audio recorder process that processes input audio, and produces a representative audio recording output;

b. an audio processor including

i. a microphone module that generates an electrical signal representative of a spoken input from the user, and

ii. an output module that generates sound intended for the user; and

c. an audio duplexing module responsive to a current state of the device, the duplexing module enabling one module in the processor to operate and disabling the other module from operation.

32. A handheld electronic device according to claim 31, wherein the duplexing module is further responsive to operation of a microphone on/off button on the device.

33. A handheld electronic device according to claim 31, further comprising:

d. a user interface display that displays visual information to the user, and wherein the duplexing module is further responsive to selection of a microphone icon on the display.

34. A handheld electronic device according to claim 33, wherein the user interface display displays text to the user, the text being representative of the recognition output.

35. A handheld electronic device according to claim 31, wherein the audio output module generates an audio cue output for the user each time the speech recognizer generates a recognition output.

36. A handheld electronic device according to claim 31, further comprising:

e. a speech manager interface that allows the speech recognition process and the text-to-speech process to be accessed by other processes, so as to prevent more than one instantiation of the speech recognition process and one instantiation of the text-to-speech process at any given time.

37. A handheld electronic device according to claim 36, wherein the speech manager interface further includes a dialog manager that manages a plurality of speech recognition grammars, each grammar being associated with at least one process, the dialog manager selecting a current recognition grammar for the speech recognition process, the current recognition grammar being associated with a currently selected process.

38. A handheld electronic device according to claim 31, further comprising:

f. a speech tips module in communication with the speech recognition that indicates to the user commands currently available to the user.

39. A handheld electronic device according to claim 38, wherein the speech tips module indicates to the user all commands currently available to the user.

40. A handheld electronic device according to claim 38, wherein the speech tips module operates responsive to a user input signal.

41. A handheld electronic device according to claim 40, wherein the user input signal is operation of a microphone on/off button on the device.

42. A handheld electronic device according to claim 38, wherein the commands currently available to the user are indicated to the user for a predetermined length of time.

43. A handheld electronic device according to claim 42, wherein the predetermined length of time is selectable by the user.

44. A handheld electronic device according to claim 38, wherein when the device is in a selected state, the speech tips module automatically indicates commands currently available.

45. A handheld electronic device according to claim 44, wherein the selected state is a microphone enabled condition.

46. A handheld electronic device according to claim 38, wherein the speech tips module uses a first distinctive voice to indicate global commands that are always available, and a second distinctive voice to indicate context-dependent commands that are currently available.

47. A handheld electronic device according to claim 38, wherein the device further includes a user interface display that displays visual information to the user, and wherein the first perceptually distinctive characteristic is a first distinctive text appearance and the second perceptually distinctive characteristic is a second text appearance.

48. A handheld electronic device according to claim 31, wherein the speech recognition process uses a database of acoustic models, each acoustic model representing a word or subword unit in a recognition vocabulary, each acoustic model being representative of its associated word or subword unit as spoken in a plurality of acoustic environments.

49. A handheld electronic device according to claim 35, wherein the database further includes acoustic models for non-word sounds, and no recognition output is generated when the speech recognition process determines that a non-word sound acoustic model represents the input speech.

50. A handheld electronic device according to claim 31, wherein the recognition output represents a command to control operation of the device.

51. A handheld electronic device according to claim 31, wherein the recognition output represents a command to one of the plurality of application processes

52. A handheld electronic device according to claim 31, wherein the device is a personal digital assistant (PDA) device having a personal information manager (PIM) application process.

53. A handheld electronic device having a plurality of application processes, the device comprising:

a. a speech recognition process that processes input speech from a user, and produces a recognition output representative of the input speech;

b. a text-to-speech process that processes output text, and produces a representative speech output;

c. a speech manager interface that allows the speech recognition process and the text-to-speech process to be accessed by other processes, so as to prevent more than one instantiation of the speech recognition process and one instantiation of the text-to-speech process at any given time.

54. A handheld electronic device according to claim 53, wherein the speech manger interface further includes a dialog manager that manages a plurality of speech recognition grammars, each grammar being associated with at least one process, the dialog manager selecting a current recognition grammar for the speech recognition process, the current recognition grammar being associated with a current process.

55. A handheld electronic device according to claim 54, wherein the speech recognition application process is speaker independent.

56. A handheld electronic device according to claim 54, wherein the recognition output represents a command to one of the plurality of application processes.

57. A handheld electronic device according to claim 54, wherein the recognition output represents a command to control operation of the device.

58. A handheld electronic device according to claim 53, wherein the device is a personal digital assistant (PDA) device having a personal information manager (PIM) application process.