Method and apparatus for displaying speech recognition progress

• US 7,292,986 B1
• Filed: 10/20/1999
• Issued: 11/06/2007
• Est. Priority Date: 10/20/1999
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Venolia, Daniel S., Quinn, Scott D.
• Primary Examiner(s): ARMSTRONG, ANGELA A
• Application Number:
  US09/421,710
• Time in Patent Office: 2,939 Days
• Field of Search: 704/270, 704/275, 704/276, 345/835, 345/700, 345/808, 345/844, 345/859, 345/978, 345/772
• US Class Current: 704/276
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A graphical user interface provides a graphical volume meter indicating the volume of the user'"'"'s speech and a speech recognition meter showing the progress of a speech recognizer. The graphical volume meter and recognition meter are both located near each other on the display such that the user can focus on both meters at the same time. One aspect of the present invention is that a speech recognition meter is placed on the display near the insertion point where the user intends their speech to take effect. Thus, the user does not have to divert their view from the insertion point in order to check the progress of the speech recognizer.

Claims

1. A method in a computer system for generating images on a display device, the method comprising:

displaying an insertion marker at an insertion area on a display, the insertion area representing the location at which the user desires to provide input; and

displaying a progress meter so that at least a portion of the progress meter is less than the height of one text line away from the insertion area based on the location of the insertion marker, the progress meter quantitatively indicative of the amount of progress in decoding a speech input.

2. The method of claim 1 further comprising:

receiving input from the user indicating that a microphone is to be activated;

activating the microphone; and

displaying an indication that the microphone is active near the progress meter.

3. The method of claim 1 further comprising:

converting a user'"'"'s speech input into an analog speech signal;

converting the analog speech signal into at least one digital speech value; and

transforming the at least one digital speech value into coordinates for at least one shape on the display positioned near the progress meter.

4. The method of claim 3 wherein transforming the at least one digital speech value into coordinates comprises:

applying a mathematical function to the at least one digital speech value to produce a transform value, the range between the lowest possible transform value and the highest possible transform value being less than the range between the lowest possible digital speech value and the highest possible speech value; and

using the transform value to identify coordinates for the at least one shape on the display.

5. The method of claim 4 wherein applying a mathematical function comprises taking the logarithm of at least one digital value.

6. The method of claim 4 wherein applying a mathematical function comprises taking the square-root of at least one digital value.

7. The method of claim 4 wherein the shape is a base rectangle and wherein using the transform value to determine the coordinates of the base rectangle comprises:

determining a base point for the base rectangle on the display;

accessing a stored rectangle width;

accessing a maximum transform value;

dividing the transform value by the maximum transform value to produce a transform ratio;

determining a calculated height based in part on the transform ratio; and

calculating the coordinates of the base rectangle based on the base point, the stored rectangle width and the calculated height.

8. The method of claim 7 wherein determining a calculated height comprises determining if the transform ratio is greater than a maximum height ratio for the base rectangle and if it is, performing a further step of multiplying the maximum height ratio for the base rectangle by a full meter height to produce the calculated height.

9. The method of claim 8 further comprising steps of:

subtracting the maximum height ratio for the base rectangle from the transform ratio to produce an excess ratio;

determining a second rectangle height based in part on the excess ratio; and

calculating the coordinates of a second rectangle based on the coordinates of the base rectangle, the stored rectangle width and the second rectangle height, the coordinates of the base rectangle and the second rectangle such that the second rectangle appears connected to a top edge of the base rectangle on the display.

10. The method of claim 9 wherein determining a second rectangle height comprises determining if the excess ratio is greater than a maximum height ratio for the second rectangle and if it is, performing a further step of multiplying the maximum height ratio for the second rectangle by the full meter height to produce the second rectangle height.

11. The method of claim 10 further comprising steps of:

subtracting the maximum height ratio for the second rectangle from the excess ratio to produce a remainder ratio;

determining a third rectangle height by multiplying the remainder ratio by the full meter height; and

calculating the coordinates of a third rectangle based on the coordinates of the second rectangle, the stored rectangle width and the third rectangle height, the coordinates of the second rectangle and the third rectangle such that the third rectangle appears connected to a top edge of the second rectangle on the display.

12. The method of claim 7 further comprising calculating the coordinates of a background rectangle, the background rectangle appearing between the base rectangle and a point at a full meter height above a bottom edge of the base rectangle.

13. The method of claim 12 further comprising:

dividing the speech input into frames;

decoding at least one of the frames of speech into a sub-word unit;

dividing a frame number of the last frame to be decoded by the total number of frames to produce a decode ratio; and

displaying the progress meter based on the decode ratio by changing the color of at least one background rectangle.

14. The method of claim 13 wherein displaying the progress meter comprises:

multiplying the decode ratio by a full meter width to produce a progress width;

dividing the progress width by a rectangle width that is indicative of the width of each background rectangle, the division producing a rectangle count; and

changing the color of a number of background rectangles, the number of background rectangles being equal to the rectangle count.

15. The method of claim 1 wherein displaying a progress meter further comprises:

dividing the speech input into frames;

decoding at least one of the frames of speech into a sub-word unit;

dividing a frame number of the last frame to be decoded by the total number of frames to produce a decode ratio; and

displaying the progress meter based on the decode ratio.

16. The method of claim 15 wherein displaying the progress meter further comprises:

multiplying the decode ratio by a full meter width to determine a progress width; and

calculating the coordinates of a progress rectangle based on the progress width, a stored meter height and a base point on the display.

17. A computer-readable medium having stored thereon computer-executable instructions comprising:

insertion point marker instructions capable of maintaining the coordinates of an insertion point on a display, the insertion point representing a location on the display where a user desires to provide input;

speech recognition instructions capable of decoding a speech signal; and

meter generation instructions capable of displaying a meter within one text line of the insertion point based on the insertion point marker, the meter being indicative of an amount of a speech signal that has been decoded by the speech recognition instructions.

18. The computer-readable medium of claim 17 wherein the meter generation instructions further comprise:

a microphone state variable having a value that is indicative of whether a microphone is active; and

active microphone display instructions, capable of displaying an indication that the microphone is active near the insertion point.

19. The computer-readable medium of claim 17 wherein the meter generation instructions further comprise transform instructions capable of transforming a digital value into a set of coordinates for a shape on the display, the digital value being indicative of the magnitude of a portion of a speech signal.

20. The computer-readable medium of claim 17 wherein the speech recognition instructions are capable of decoding a speech signal into a set of sub-words.

21. A method in a computer system for generating images on a display device, the method comprising:

receiving a speech input signal that is indicative of human speech;

displaying a volume meter that is indicative of the magnitude of at least a portion of the speech input signal; and

displaying a progress meter on the display so that a user can perceive both the progress meter and the volume meter without moving their eyes, the progress meter quantitatively indicating the amount of progress of a speech recognition system in decoding the speech input signal.

22. The method of claim 21 wherein displaying a volume meter comprises:

storing digital values representing the magnitudes of different respective portions of the speech signal;

accessing the stored digital values;

displaying a separate token for each separate digital value that is accessed.

23. The method of claim 22 wherein displaying a separate token comprises:

displaying a meter portion of the token, the meter portion'"'"'s size being positively related to the magnitude of the speech signal such that higher magnitude portions of the speech signal have larger meter portions; and

displaying a background portion of the token, the background portion'"'"'s size being negatively related to the magnitude of the speech signal such that higher magnitude portions of the speech signal have smaller background portions.

24. The method of claim 23 wherein displaying the meter portion comprises:

transforming a digital value representing the magnitude of a portion of the speech signal to produce a transform value, the range between the smallest and largest transform value being less than the range between the smallest and largest digital value;

dividing the transform value by a maximum meter value to produce a meter ratio; and

determining the height of at least a portion of the meter portion using the meter ratio and a full meter height.

25. The method of claim 24 wherein determining the height of at least a portion of the meter portion comprises multiplying the meter ratio by the full meter height.

26. The method of claim 24 wherein determining the height of at least a portion of the meter portion comprises:

determining if the meter ratio is greater than a base ratio and if the meter ratio is greater than the base ratio performing steps comprising;

multiplying the base ratio by the full meter height to determine the height of a base block of the meter portion;

subtracting the base ratio from the meter ratio to produce an excess ratio;

using the excess ratio and the full meter height to determine a height of a second block of the meter portion.

27. The method of claim 26 wherein using the excess ratio and the full meter height to determine a height of a second block comprises:

determining if the excess ratio exceeds an intermediate ratio and if the excess ratio exceeds the intermediate ratio performing steps comprising;

multiplying the intermediate ratio by the full meter height to produce the height of the second block;

subtracting the intermediate ratio from the excess ratio to produce a remainder ratio; and

multiplying the remainder ratio by the full meter height to produce a height for a top block of the meter portion.

28. The method of claim 24 wherein the speech recognizer decodes the speech input signal by converting frames of the speech input signal into sub-words and wherein displaying a token comprises:

dividing the number of the last frame decoded by the speech recognizer by a total number of frames that form the speech input signal to produce a progress ratio;

multiplying the progress ratio by a full meter width to produce a progress width;

dividing the progress width by a token width to produce an affected number of tokens; and

for each of the affected number of tokens, setting the color of at least a portion of each token so that it is different from the color of other tokens.

29. A computer-readable medium having stored thereon computer-executable instructions designed to be executed in a computer system having a display, the computer-executable instructions comprising:

a volume meter portion capable of displaying a volume meter on the display that is indicative of the volume of a human speech signal;

a speech recognition portion that is capable of converting the human speech signal into a set of sub-words; and

a progress meter portion capable of displaying a progress meter on the display proximate the volume meter, the progress meter being quantitatively indicative of the amount of progress of the speech recognition portion in converting the human speech signal.

30. The computer-readable medium of claim 29 wherein the volume meter portion comprises:

meter size computer-executable instructions capable of determining a maximum dimension for the volume meter;

volume ratio computer-executable instructions capable of calculating a volume ratio that is defined as a magnitude value associated with the human speech signal over a maximum possible magnitude value; and

volume token computer-executable instructions capable of generating a volume token on the display that has a size that is determined from the volume ratio and the maximum dimension for the volume meter.

31. The computer-readable medium of claim 30 wherein the volume token computer-executable instructions comprise:

positive relation computer-executable instructions capable of generating a foreground portion of the volume token that is larger for higher magnitude values associated with the human speech signal; and

negative relation computer-executable instructions capable of generating a background portion of the volume token that is smaller for higher magnitude values associated with the human speech signal.

32. The computer-readable medium of claim 31 wherein the progress meter portion comprises:

progress ratio computer-executable instructions capable of dividing a frame number representing the last frame of the human speech signal converted by the speech recognition system by a total frame number representing the total number of frames found in the human speech signal to produce a progress ratio;

meter dimension computer-executable instructions capable of determining a maximum dimension for the progress meter; and

progress dimension computer-executable instructions capable of multiplying the progress ratio by the maximum dimension for the progress meter to produce a progress dimension.

33. The computer-readable medium of claim 32 wherein the volume token computer-executable instructions generate a volume token based in part on the progress dimension.