Method and system for evaluating automatic speech recognition telephone services

• US 20060106613A1
• Filed: 12/28/2005
• Published: 05/18/2006
• Est. Priority Date: 03/26/2002
• Status: Abandoned Application

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: SBC Technology Resources, Inc. (AT&T, Inc.)
• Inventors: Mills, Scott H.
• Application Number:
  US11/319,989
• Publication Number:
  US 20060106613A1
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

A method and system for evaluating telephone services provided by speech recognition interfaces an evaluation engine with a voice recognition service over a telephone system to submit speech utterance samples to the voice recognition service, receive the response of the voice recognition service to the sample utterances, and determine error and recognition of the sample utterances by the voice recognition service by comparing actual voice recognition service responses to expected responses. The evaluation engine permits evaluation of a voice recognition service for plural glossaries in different contexts, such as through predetermined nodes of a voice recognition service menu having plural glossaries.

Claims

1-20. -20. (canceled)

21. A system for evaluating a voice recognition service, comprising:

an evaluation graphical user interface (GUI) operable to facilitate the generation of a test configuration for evaluating a voice recognition service, the evaluation GUI including;

a node selection interface allowing the user to select from a plurality of voice recognition menu nodes a particular node to be evaluated; and

a utterance selection interface allowing the user to select one or more sample utterances for the evaluation of the voice recognition service; and

an evaluation engine communicatively coupled to the evaluation GUI and configured to cooperate with the evaluation GUI to generate the test configuration, the evaluation engine including;

a voice recognition service interface for communicating to the voice recognition service the one or more selected sample utterances and for receiving from the voice recognition service one or more actual responses to the one or more selected sample utterances; and

an assessment engine for comparing the one or more actual responses to one or more expected responses to determine recognition rates of the voice recognition service.

22. A system according to claim 21, wherein:

the node selection interface comprises a map interface configured to display a map of the plurality of voice recognition menu nodes indicating relationships between particular nodes; and

the utterance selection interface comprises a library interface configured to display a library of sample utterances from which the user can select the one or more sample utterances.

23. A system according to claim 21, wherein the node selection interface comprises a map interface configured to display a map of the plurality of voice recognition menu nodes indicating the relationships between particular nodes and to allow the user to use the map to select the particular node to be evaluated.

24. A system according to claim 21, wherein the evaluation GUI further includes a speaker selection interface allowing the user to select a particular speaker from a plurality of different speakers;

and

wherein the utterance selection interface comprises a library interface configured to display a library of sample utterances generated by the selected particular speaker and to allow the user to select the one or more sample utterances from the displayed library.

25. A system according to claim 21, wherein the evaluation GUI further includes a noise selection interface allowing the user to select one or more simulated background noises.

26. A system according to claim 25, wherein the one or more simulated background noises comprise at least one of static noise, road noise, and crowd noise.

27. A system according to claim 21, further comprising a telephony interface for communicating the one or more selected sample utterances to the voice recognition service.

28. A method for evaluating a voice recognition service, comprising:

receiving a user selection of a particular voice recognition menu node to be evaluated, the particular node selected from a plurality of voice recognition menu nodes;

receiving a user selection of one or more sample utterances for an evaluation of the particular node;

communicating the one or more selected sample utterances to a voice recognition service;

receiving one or more actual responses to the one or more selected sample utterances from the voice recognition service; and

comparing the one or more actual responses to one or more expected responses to determine recognition rates of the voice recognition service.

29. A method according to claim 28, wherein the user selection of the particular node and the one or more sample utterances are provided by an evaluation graphical user interface.

30. A method according to claim 28, further comprising:

displaying a map of the plurality of voice recognition menu nodes indicating relationships between particular nodes; and

displaying a library of sample utterances from which the user can select the one or more sample utterances.

31. A method according to claim 28, further comprising:

generating an evaluation graphical user interface (GUI) that displays a map of the plurality of voice recognition menu nodes indicating the relationships between particular nodes; and

receiving the user selection of the particular node to be evaluated via the evaluation GUI.

32. A method according to claim 28, further comprising generating an evaluation graphical user interface (GUI), the evaluation GUI:

allowing a user to select a particular speaker from a plurality of different speakers;

displaying a library of sample utterances generated by the selected particular speaker; and

allowing the user to select from the displayed library the one or more sample utterances.

33. A method according to claim 28, further comprising receiving a user selection of one or more simulated background noises.

34. A method according to claim 33, wherein the one or more simulated background noises comprise at least one of static noise, road noise, and crowd noise.

35. A method according to claim 28, wherein the one or more selected sample utterances are communicated to the voice recognition service through a telephony interface.

36. Computer instructions encoded in computer-readable media and executable by a processor, comprising:

logic for receiving a user selection of a particular voice recognition menu node to be evaluated, the particular node selected from a plurality of voice recognition menu nodes;

logic for receiving a user selection of one or more sample utterances for an evaluation of the particular node;

logic for communicating to a voice recognition service the one or more selected sample utterances;

logic for receiving from the voice recognition service one or more actual responses to the one or more selected sample utterances; and

logic for comparing the one or more actual responses to one or more expected responses to determine recognition rates of the voice recognition service.

37. Computer logic according to claim 36, further comprising:

logic for displaying a map of the plurality of voice recognition menu nodes indicating relationships between particular nodes; and

logic for displaying a library of sample utterances from which the user can select the one or more sample utterances.

38. Computer logic according to claim 36, further comprising:

logic for generating an evaluation graphical user interface (GUI) that displays a map of the plurality of voice recognition menu nodes indicating the relationships between particular nodes; and

logic for receiving the user selection of the particular node to be evaluated via the evaluation GUI.

39. Computer logic according to claim 36, further comprising logic for generating an evaluation graphical user interface (GUI) including:

logic for allowing the user to select a particular speaker from a plurality of different speakers;

logic for displaying a library of sample utterances generated by the selected particular speaker; and

logic for allowing the user to select the one or more sample utterances from the displayed library.

40. Computer logic according to claim 36, further comprising logic for communicating the one or more selected sample utterances to the voice recognition service through a telephony interface.