Display indications of speech processing states in speech recognition system

• US 5,819,225 A
• Filed: 05/30/1996
• Issued: 10/06/1998
• Est. Priority Date: 05/30/1996
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Happ, Alan J., Milenkovic, Maria, Eastwood, Peter Rowland, Klein, Alice G., Kruse, Daniel William
• Primary Examiner(s): Hudspeth, David R.
• Assistant Examiner(s): OPSASNICK, MICHAEL N
• Application Number:
  US08/655,541
• Time in Patent Office: 859 Days
• Field of Search: 704/270, 704/275, 395/155, 395/159, 395/161, 345/326, 345/348
• US Class Current: 704/275
• CPC Class Codes: G06F 11/32 with visual or acoustical indication of the functioning of the machine
• CPC Class Codes: G06F 11/324 Display of status information
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/225 Feedback of the input speech

Abstract

A visual feedback aid, for a computer system performing speech recognition functions, provides indications on a display monitor of the system representing the current state of operation of a system microphone, the current mode of operation of the system in respect to speech, and a string of text representing the system'"'"'s recognition (correct or incorrect) of commands instantly spoken into the microphone. The indications preferably are located in a reserved area of a display window associated with a currently active application involving speech recognition. The reserved area preferably would be a prominent one, such as the application title bar.

Claims

1. In a computer system adapted for speech recognition, said system having a speech transducer and display monitor attached thereto, said system having n (>

2) different modes of speech recognition usage, said transducer having m (>

2) different states of operation associated with individual said modes of usage, and said display monitor being adapted by said system to display windows pertaining to program applications currently active in said system, each said window having an area reserved for displaying information pertinent to speech recognition functions currently being performed in said system, display apparatus comprising;

means for adapting said monitor, while a said window is being used relative to a program application and while said system is instantaneously operating in one of said n speech recognition modes relative to said program application, to display first and second symbols in said reserved area of said window;

said first symbol uniquely reflecting said instantaneous speech recognition mode of said system, and said second symbol uniquely reflecting the instantaneous state of operation of said speech transducer relative to said program application;

said first symbol constituting a unique one of n mode indicating symbols corresponding to said n speech recognition modes of said system; and

said second symbol constituting a unique one of m state indicating symbols associated with said m states of operation of said speech transducer; and

means responsive to commands spoken by a user of said system for selectively varying said displayed first and second symbols, as said system varies its speech recognition mode of operation and as the operating state of said speech transducer is varied, in order to maintain correspondence between said indicated first and second symbols and instantaneous speech recognition modes and speech transducer states pertaining to the program application in said window.

2. Display apparatus in accordance with claim 1 wherein individual said windows contain a title bar and said reserved area in which said first and second symbols are displayed is in said title bar in the respective window.

3. Display apparatus in accordance with claim 1 wherein said speech transducer is a microphone, and wherein said displayed second symbol is:

a unique one of m icons for indicating states of operation of said microphone.

4. Display apparatus in accordance with claim 3 wherein said m microphone state icons include two icons for respectively indicating on and off states of said microphone.

5. Display apparatus in accordance with claim 3 wherein said m microphone state icons include at least 3 icons having the appearance of a microphone and a 4^th icon indicative of a "busy" state of operation of said system relative to said microphone;

each said symbol having the appearance of a microphone having one of the following orientations;

horizontal, vertical and diagonal;

said symbols having the appearance of a microphone including symbols individually indicating the following microphone states;

an Off state in which the microphone is disabled, an On state in which the microphone is enabled, a Listening state indicating that the microphone is instantaneously receiving voice input; and

a sleeping state indicating that the microphone is instantaneously but the speech recognition function of the unavailable;

said 4^th icon indicative of said busy state having the appearance of an hourglass and being displayed while said computer system is busy and therefore unavailable to handle speech recognition functions.

6. Display apparatus in accordance with claim 3 wherein said first symbol is an icon depicting the instantaneous mode of usage of said system in respect to speech recognition.

7. Display apparatus in accordance with claim 6 wherein said first symbol is an icon indicating a unique one of the following speech recognition modes:

a navigation mode in which a user of said system can control operations of said system using spoken commands;

a conversation mode in which said user can effectively converse with a program application in said system and thereby for instance receive helpful instructions from said application; and

a dictation mode in which said system operates to convert speech of a said user into displayed alphanumeric characters representing printable text.

8. Display apparatus in accordance with claim 7 wherein said icon indicating said navigation mode is displayed in conjunction with a string of alphanumeric characters representing a command recognized by said system as having been spoken by a said user of said system;

said string of displayed characters allowing a user of said system to verify that a command just spoken by said user has been correctly recognized by said system.

9. A computer program product for providing visual feedback during interaction between a computer system and a speaking user of said system, said user communicating with said system by speaking into a microphone, said system having n (>

2) different modes of speech recognition usage and a graphic user interface displaying windows to said user, said microphone having m (>

2) different operating states relative to said system, said product being deliverable on computer-readable media and comprising;

means for instructing said computer system to create a display window having a specific area reserved for providing visual feedback cues to said user; and

means for further instructing said computer system to selectively display visual feedback cues in said reserved area, said cues specifically reflecting both the instantaneous mode of speech recognition usage of said system and the instantaneous state of operation of said microphone, and also indicating strings of text representing system recognition of terms instantly spoken into said microphone by said user.

10. A computer program product in accordance with claim 9 wherein said reserved area is in a portion of a title bar shown in said window.

11. A computer program product in accordance with claim 9 wherein said cues reflecting instantaneous states of said microphone are icons pictorially representing respective microphone states.

12. A computer program product in accordance with claim 11 wherein said icons pictorially representing said instantaneous microphone states are displayed against a highlighted background having a predetermined color.

13. A computer program product in accordance with claim 11 wherein said icons representing said instantaneous microphone states consist of a plurality of symbols having the appearance of a microphone and another symbol having the appearance of an hourglass;

each of said symbols when highlighted having a selected background of a predetermined color;

said plurality of symbols having the appearance of a microphone each depicting a microphone having one of the following orientations;

horizontal, vertical and diagonal;

said plurality of symbols having the appearance of a microphone including symbols individually defining the following microphone states;

an Off state in which the microphone is disabled, an On state in which the microphone is enabled, a Listening state while the microphone is actively transducing speech, and a sleeping state used to indicate that the microphone is currently unavailable;

said another symbol having the appearance of an hourglass serving to indicate a Busy state in which the computer system is busy and therefore unavailable to handle speech recognition functions.

14. A computer program product in accordance with claim 9 wherein said n speech recognition modes include:

a Navigation mode in which said system performs actions in response to spoken commands, a Conversation mode in which said system interacts audibly with a speaking user, said system, while so interacting, producing audible speech representing responses of a virtual assistant to spoken inquiries made by said user, and a Dictation mode in which said system converts user speech into corresponding displayed strings of text; and

wherein each said cue reflecting a said instantaneous speech recognition mode of operation includes an icon pictorially representing the respective mode of operation.

15. A computer program product in accordance with claim 14 wherein said cues reflecting said speech recognition modes include displayed strings of alphanumeric characters and words representing speech recognized by said system, either correctly or incorrectly, as having just been spoken into said microphone.