Method and system for evaluating automatic speech recognition telephone services

• US 20030187639A1
• Filed: 03/26/2002
• Published: 10/02/2003
• Est. Priority Date: 03/26/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: SBC Technology Resources, Inc. (AT&T, Inc.)
• Inventors: Mills, Scott H.
• Granted Patent: US 7,016,842 B2
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

A method and system for evaluating telephone services provided by speech recognition interfaces an evaluation engine with a voice recognition service over a telephone system to submit speech utterance samples to the voice recognition service, receive the response of the voice recognition service to the sample utterances, and determine error and recognition of the sample utterances by the voice recognition service by comparing actual voice recognition service responses to expected responses. The evaluation engine permits evaluation of a voice recognition service for plural glossaries in different contexts, such as through predetermined nodes of a voice recognition service menu having plural glossaries.

Claims

1. A method for evaluating telephone services provided by speech recognition, the method comprising:

interfacing over a telephone network with a speech recognition service, the speech recognition service having plural nodes and plural glossaries, each glossary applicable to one or more nodes of the service;

submitting speech utterance samples to the speech recognition service, the speech utterance samples associated with a node having plural glossaries; and

determining the response of the speech recognition service to each speech utterance sample.

2. The method of claim 1 wherein at least one glossary is a global glossary applicable to all nodes of the speech recognition service, submitting speech utterance samples further comprising submitting one or more speech utterance samples associated with the global directory to plural nodes of the speech recognition service.

3. The method of claim 2 wherein the utterance sample comprises a command to go to a main menu node.

4. The method of claim 1 further comprising:

defining a test configuration for the telephone service, the test configuration having plural utterance samples for submission to one or more nodes of the telephone service.

5. The method of claim 4 wherein the test configuration navigates the telephone service menu to evaluate recognition of the utterance samples.

6. The method of claim 4 further comprising:

providing the test configuration to the telephone service; and

navigating the telephone service by coordinating submission of the utterance samples to predetermined telephone service nodes according to the test configuration.

7. The method of claim 4 further comprising:

introducing simulated noise to one or more utterance samples.

8. The method of claim 7 wherein the simulated noise comprises simulated road noise.

9. The method of claim 7 wherein the simulated noise comprises simulated static noise.

10. The method of claim 7 wherein the simulated noise comprises simulated crowd noise.

11. A system for evaluating a telephone service provided by speech recognition, the service having plural nodes, each node having one or more glossaries, the system comprising:

a library of sample speech utterances;

a map of one or more of the telephone service nodes;

a telephone system interface operable to communicate with the service through a telephone network; and

an evaluation engine interfaced with the library, the map, and the telephone interface, the evaluation engine operable to submit speech utterance samples to the service at one or more nodes and to determine the response of the service.

12. The system of claim 11 further comprising:

a configuration engine associated with the evaluation engine; and

a user interface in communication with the configuration engine, the user interface providing selections of utterance samples for one or more nodes of the telephone service.

13. The system of claim 12 wherein the configuration engine defines a test configuration to navigate one or more nodes of the telephone service with one or more utterance samples.

14. The system of claim 13 further comprising an evaluation engine module associated with the telephone service, the telephone system interface operable to provide the evaluation engine module with the test configuration, the evaluation engine module coordinating the navigation of utterance samples through one or more telephone service nodes.

15. The system of claim 11 wherein the telephone system interface comprises an analog telephone line.

16. The system of claim 11 wherein the utterance sample comprises a command associated with a global glossary.

17. A method for evaluating a voice recognition telephone service, the method comprising:

establishing communication with the voice recognition service through a telephone network;

submitting a first sample utterance associated with a first sample speaker to plural predetermined nodes of the telephone service; and

comparing the response of the telephone service for each predetermined node with an expected response to determine recognition error rates.

18. The method of claim 17 wherein one or more of the predetermined nodes has plural voice recognition glossaries.

19. The method of claim 17 wherein submitting a first sample utterance further comprises automatically submitting the sample utterance to the plural nodes according to a test configuration.

20. The method of claim 19 further comprising:

synchronizing the sample utterances with predetermined nodes of the voice recognition service menu by coordinating the menu nodes with the test configuration.