Quality evaluation tool for dynamic voice portals

• US 8,050,918 B2
• Filed: 12/11/2003
• Issued: 11/01/2011
• Est. Priority Date: 12/11/2003
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Ghasemi, Reza, Haenel, Walter
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): Godbold; Douglas
• Application Number:
  US10/733,995
• Publication Number:
  US 20050131676A1
• Time in Patent Office: 2,882 Days
• Field of Search: 704/231, 704/239, 704/244, 704/243, 379/88.01
• US Class Current: 704/244
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

A method and system for evaluating the quality of voice input recognition by a voice portal is provided. An analysis interface extracts a set of current grammars from the voice portal. A test pattern generator generates a test input for each current grammar. The test input includes a test pattern and a set of active grammars corresponding to each current grammar. The system further includes a text-to-speech engine for entering each test pattern into the voice server. A results collector analyzes each test pattern entered into the voice server with the speech recognition engine against the set of active grammars corresponding to the current grammar for said test pattern. A results analyzer derives a set of statistics of a quality of recognition of each current grammar.

Claims

1. A method of evaluating grammars associated with a voice portal on a portal server, said method comprising:

generating, for a current grammar of the voice portal representing a valid input for a first menu of the voice portal, a test input, the test input for the current grammar including a test pattern;

providing the test input to the voice portal on the portal server using a voice server;

receiving at least one measure of how distinguishable the current grammar is from other grammars of a set of active grammars that are active when the current grammar is active, the set of active grammars including the current grammar and at least one grammar from a second menu of the voice portal, the at least one measure based at least in part on analysis of the test pattern with respect to the set of active grammars, the at least one measure comprising at least one measure of how distinguishable the current grammar is from the at least one grammar from the second menu of the voice portal; and

determining, based at least in part on the at least one measure, whether to modify the current grammar from the first menu to be distinguishable from the at least one grammar from the second menu.

2. The method of claim 1, wherein the at least one measure of how distinguishable the current grammar is from other grammars of the set of active grammars includes a confidence level and a set of n-best results for the test input, and wherein the method further comprises comparing the confidence level and set of n-best results for the test input with an expected value to assess the at least one measure of how distinguishable the current grammar is from other grammars of the set of active grammars.

3. The method of claim 1, further comprising modifying the current grammar to create a modified grammar if the at least one measure indicates that the current grammar is not sufficiently distinguishable from the other grammars of the set of active grammars.

4. The method of claim 3, further comprising:

(i) generating a test input for the modified grammar, the test input including a test pattern for the modified grammar;

(ii) providing the test input for the modified grammar to the voice portal on the portal server using the voice server;

(iii) receiving at least one measure of how distinguishable the modified grammar is from other grammars of a set of active grammars that are active when the modified grammar is active, the modified grammar being one grammar of the set of active grammars that are active when the modified grammar is active; and

(iv) modifying the modified grammar and repeating steps (i) through (iii) until the at least one measure of how distinguishable the modified grammar is from other grammars of the set of active grammars that are active when the modified grammar is active indicates that the modified grammar is sufficiently distinguishable from the other grammars of the set of active grammars that are active when the modified grammar is active.

5. The method of claim 1, further comprising modifying the test pattern to emulate one or more user voices prior to providing the test input to the voice portal.

6. The method of claim 1, further comprising modifying the test pattern to emulate the influence of one or more communications network qualities prior to providing the test input to the voice portal.

7. A tangible computer-readable storage device encoded with instructions which, when executed by a computer, cause the computer to perform a method of evaluating grammars associated with a voice portal, the method comprising:

generating, for a current grammar of the voice portal representing a valid input for a first menu of the voice portal, a test input, the test input for the current grammar including a test pattern;

providing the test input to the voice portal;

receiving at least one measure of how distinguishable the current grammar is from other grammars of a set of active grammars that are active when the current grammar is active, the set of active grammars including the current grammar and at least one grammar from a second menu of the voice portal, the at least one measure based at least in part on analysis of the test pattern with respect to the set of active grammars, the at least one measure comprising at least one measure of how distinguishable the current grammar is from the at least one grammar from the second menu of the voice portal; and

determining, based at least in part on the at least one measure, whether to modify the current grammar from the first menu to be distinguishable from the at least one grammar from the second menu.

8. The tangible computer-readable storage device of claim 7, wherein the at least one measure of how distinguishable the current grammar is from other grammars of the set of active grammars includes a confidence level and a set of n-best results for the test input, and wherein the method further comprises comparing the confidence level and set of n-best results for the test input with an expected value to assess the at least one measure of how distinguishable the current grammar is from other grammars of the set of active grammars.

9. The tangible computer-readable storage device of claim 7, wherein the method further comprises modifying the current grammar to create a modified grammar if the at least one measure indicates that the current grammar is not sufficiently distinguishable from the other grammars of the set of active grammars.

10. The tangible computer-readable storage device of claim 9, wherein the method further comprises:

(i) generating a test input for the modified grammar, the test input including a test pattern for the modified grammar;

(ii) providing the test input for the modified grammar to the voice portal;

(iii) receiving at least one measure of how distinguishable the modified grammar is from other grammars of a set of active grammars that are active when the modified grammar is active, the modified grammar being one grammar of the set of active grammars that are active when the modified grammar is active; and

(iv) modifying the modified grammar and repeating steps (i) through (iii) until the at least one measure of how distinguishable the modified grammar is from other grammars of the set of active grammars that are active when the modified grammar is active indicates that the modified grammar is sufficiently distinguishable from the other grammars of the set of active grammars that are active when the modified grammar is active.

11. The tangible computer-readable storage device of claim 7, wherein the method further comprises modifying the test pattern to emulate one or more user voices prior to providing the test input to the voice portal.

12. The tangible computer-readable storage device of claim 7, wherein the method further comprises modifying the test pattern to emulate the influence of one or more communications network qualities prior to providing the test input to the voice portal.

13. A system for evaluating grammars of a voice portal executing on a portal server, the system comprising:

an analysis interface to extract a current grammar from a set of active grammars of the voice portal, the current grammar representing a valid input for a first menu of the voice portal and being one grammar of the set of active grammars, the set of active grammars including at least one grammar from a second menu of the voice portal;

a test pattern generator to generate a test input for the current grammar, the test input including a test pattern;

a text-to-speech engine on a computer to enter the test input into the voice portal;

a results collector to analyze the test input entered into the voice portal against the set of active grammars; and

a results analyzer to derive a set of statistics indicative of how distinguishable the current grammar is from other grammars of the set of active grammars, the set of statistics being indicative of how distinguishable the current grammar is from the at least one grammar from the second menu of the voice portal, and to determine, based at least in part on the at least one measure, whether to modify the current grammar from the first menu to be distinguishable from the at least one grammar from the second menu.

14. The system of claim 13, wherein the set of statistics includes a confidence level and a set of n-best results for the test input, and wherein the results analyzer is configured to compare the confidence level and set of n-best results for the test input with an expected value to assess how distinguishable the current grammar is from other grammars of the set of active grammars.

15. The system of claim 13, wherein the test pattern generator is configured to modify the test pattern to emulate one or more user voices prior to entering the test input into the voice portal.

16. The system of claim 13, wherein the test pattern generator is configured to modify the test pattern to emulate the influence of one or more communications network qualities prior to entering the test input into the voice portal.

17. The system of claim 13, further comprising at least one processor programmed to act as the analysis interface, the test pattern generator, the text-to-speech engine, and the results analyzer.