Method of automatically ranking speech dialog states and transitions to aid in performance analysis in speech applications

• US 7,643,995 B2
• Filed: 02/09/2005
• Issued: 01/05/2010
• Est. Priority Date: 02/09/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Yu, Dong, Acero, Alejandro
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): Saint Cyr; Leonard
• Application Number:
  US11/054,096
• Publication Number:
  US 20060178883A1
• Time in Patent Office: 1,791 Days
• Field of Search: None
• US Class Current: 704/251
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/083 Recognition networks G10L15/142, G10L15/16 take precedence

Abstract

A method of identifying problems in a speech recognition application is provided and includes the step of obtaining a speech application call log containing log data on question-answer (QA) states and transitions. Then, in accordance with the method, for each of a multiple transitions between states, a parameter is generated which is indicative of a gain in a success rate of the speech recognition application if all calls passing through the transition passed instead through other transitions. In exemplary embodiments, the parameter is an Arc Cut Gain in Success Rate (ACGSR) parameter. Methods of generating the ACGSR, as well as systems and tools for aiding developers are also disclosed.

Claims

1. A method of identifying problems in a speech recognition application, the method comprising:

obtaining from a computer storage medium a speech application call log containing log data on question-answer (QA) states and transitions;

generating from the speech application call log, using a processor coupled to the computer storage medium, for each of a plurality of transitions between states in the speech application call log, a parameter which is indicative of a gain in a success rate of the speech recognition application if all calls passing through the transition passed instead through others of the plurality of transitions, wherein generating the parameter for each of the plurality of transitions comprises generating the parameter, for each of the plurality of transitions between the states, indicative of the gain in the success rate of the speech recognition application if all calls passing through the transition were instead evenly distributed between others of the plurality of transitions;

ranking, using the processor, the plurality of transitions as a function of their corresponding generated parameters, indicative of a gain in the success rate of the speech recognition application if all calls passing through the transition passed instead through others of the plurality of transitions, to identify transitions which will most improve performance of the speech recognition application; and

displaying on a display device the plurality of transitions ordered as a function of rank for use by a user in identifying problems in the speech recognition application in order to improve performance of the speech recognition application.

2. The method of claim 1, wherein generating the parameter for each of the plurality of transitions comprises:

generating the parameter, for each of the plurality of transitions between the states, such that it is indicative of the gain in the success rate of the speech recognition application if all calls passing through the transition passed instead through others of the plurality of transitions at a same success rate as other calls originally passing through the others of the plurality of transitions.

3. The method of claim 1, wherein the transitions correspond to arcs between states in a call transition diagram, and wherein the parameter generated for each of the plurality of transitions is an Arc Cut Gain in Success Rate (ACGSR).

4. The method of claim 3, and further comprising:

determining for each of the plurality of arcs or transitions which calls passing through the arc or transition are sunken calls; and

wherein generating the parameter for each of the plurality of arcs or transitions further comprises generating the ACGSR for each of the arcs or transitions using only sunken calls for that arc or transition.

5. The method of claim 4, wherein determining for each of the plurality of arcs or transitions which calls passing through the arc or transition are sunken calls comprises:

determining which calls passing through the arc or transition are sunken calls based upon a timestamp.

6. The method of claim 4, wherein generating the ACGSR for each of the arcs or transitions comprises generating the ACGSR for each of the arcs or transitions in accordance with a relationship representable by the equation:

$\begin{array}{c}\mathrm{ACGSR}<br><br>\left(a_{\mathrm{ij}}\right)=\left[\frac{s_i-s_{\mathrm{ij}}}{t_i-t_{\mathrm{ij}}}·<br><br>t_{\mathrm{ij}}-s_{\mathrm{ij}}\right]/N\\ =\left(s_i·<br><br>t_{\mathrm{ij}}-s_{\mathrm{ij}}·<br><br>t_{\mathrm{ij}}-t_i·<br><br>s_{\mathrm{ij}}+t_{\mathrm{ij}}·<br><br>s_{\mathrm{ij}}\right)/\left[N<br><br>\left(t_i-t_{\mathrm{ij}}\right)\right]\\ =\left(s_i·<br><br>t_{\mathrm{ij}}-t_i·<br><br>s_{\mathrm{ij}}\right)/\left[N<br><br>\left(t_i-t_{\mathrm{ij}}\right)\right].\end{array}$

where, a_ij is the arc or transition, t_i is the total number of calls passing through a state q_i, s_i as the total number of successful calls passing through the state q_i, T_ij is used to represent a set of calls associated with the arc a_ij, t_ij=|T_ij| is used to represent the number of calls in the set, s_ij=|S_ij| is used to represent the number of successful calls associated with the arc, and N is a total number of calls placed.

7. The method of claim 4, wherein generating the ACGSR for each of the arcs or transitions comprises generating the ACGSR for each of the arcs or transitions in accordance with a relationship representable by the equation:

$\begin{array}{c}\mathrm{ACGSR}<br><br>\left(a_{\mathrm{ij}}\right)=\left[\frac{s_i-s_{\mathrm{ij}}}{t_i-t_{\mathrm{ij}}}·<br><br>t_{\mathrm{ij}}-s_{\mathrm{ij}}\right]/N\\ =\left(s_i·<br><br>t_{\mathrm{ij}}-s_{\mathrm{ij}}·<br><br>t_{\mathrm{ij}}-t_i·<br><br>s_{\mathrm{ij}}+t_{\mathrm{ij}}·<br><br>s_{\mathrm{ij}}\right)/\left[N<br><br>\left(t_i-t_{\mathrm{ij}}\right)\right]\\ =\left(s_i·<br><br>t_{\mathrm{ij}}-t_i·<br><br>s_{\mathrm{ij}}\right)/\left[N<br><br>\left(t_i-t_{\mathrm{ij}}\right)\right].\end{array}$

where, a_ij is the arc or transition, t_i, is the total number of calls passing through a state q_i, s_i as the total number of successful calls passing through the state q_i, T_ij is the set of sunken calls of the arc a_ij, t_{ij =|Tij}| is used to represent the number of calls in the set, s_ij=|T_ij| is used to represent the number of successful sunken calls associated with the arc, and N is a total number of calls placed.

8. A computer storage medium having stored thereon computer-executable instructions for implementing speech recognition application problem identifying steps comprising:

accessing a speech application call log containing log data on question-answer (QA) states and transitions, wherein the transitions correspond to arcs between states in a call transition diagram;

determining for each of the plurality of arcs or transitions which calls passing through the arc or transition are sunken calls; and

generating, for each of a plurality of transitions between states, a parameter which is indicative of a gain in a success rate of the speech recognition application if all calls passing through the transition were routed instead through others of the plurality of transitions using a processor of the computer, wherein the parameter generated for each of the plurality of transitions is an Arc Cut Gain in Success Rate (ACGSR), and wherein generating the parameter for each of the plurality of arcs or transitions further comprises generating the ACGSR for each of the arcs or transitions using only sunken calls for that arc or transition.

9. The computer storage medium of claim 8, and having further stored thereon computer-executable instructions for implementing further speech recognition application problem identifying steps comprising:

ranking the plurality of transitions as a function of their corresponding generated parameters; and

displaying the plurality of transitions as a function of rank.

10. The computer storage medium of claim 9, wherein generating the parameter for each of the plurality of transitions comprises:

generating the parameter, for each of the plurality of transitions between the states, such that it is indicative of the gain in the success rate of the speech recognition application if all calls passing through the transition passed instead through others of the plurality of transitions at a same success rate as other calls originally passing through the others of the plurality of transitions.

11. The computer storage medium of claim 9, wherein generating the parameter for each of the plurality of transitions comprises:

generating the parameter, for each of the plurality of transitions between the states, indicative of the gain in the success rate of the speech recognition application if all calls passing through the transition were instead evenly distributed between others of the plurality of transitions.

12. The computer storage medium of claim 8, wherein determining for each of the plurality of arcs or transitions which calls passing through the arc or transition are sunken calls comprises:

determining which calls passing through the arc or transition are sunken calls based upon a number associated an event.

13. The computer storage medium of claim 8, wherein generating the ACGSR for each of the arcs or transitions comprises generating the ACGSR for each of the arcs or transitions in accordance with a relationship representable by the equation:

$\begin{array}{c}\mathrm{ACGSR}<br><br>\left(a_{\mathrm{ij}}\right)=\left[\frac{s_i-s_{\mathrm{ij}}}{t_i-t_{\mathrm{ij}}}·<br><br>t_{\mathrm{ij}}-s_{\mathrm{ij}}\right]/N\\ =\left(s_i·<br><br>t_{\mathrm{ij}}-s_{\mathrm{ij}}·<br><br>t_{\mathrm{ij}}-t_i·<br><br>s_{\mathrm{ij}}+t_{\mathrm{ij}}·<br><br>s_{\mathrm{ij}}\right)/\left[N<br><br>\left(t_i-t_{\mathrm{ij}}\right)\right]\\ =\left(s_i·<br><br>t_{\mathrm{ij}}-t_i·<br><br>s_{\mathrm{ij}}\right)/\left[N<br><br>\left(t_i-t_{\mathrm{ij}}\right)\right].\end{array}$

where, a_ij is the arc or transition, t_i is the total number of calls passing through a state q_i, s_i as the total number of successful calls passing through the state q_i, T_ij is used to represent a set of calls associated with the arc a_ij, t_ij=|T_ij| is used to represent the number of calls in the set, s_ij=|S_ij| is used to represent the number of successful calls associated with the arc, and N is a total number of calls placed.

14. The computer storage medium of claim 8, wherein generating the ACGSR for each of the arcs or transitions comprises generating the ACGSR for each of the arcs or transitions in accordance with a relationship representable by the equation:

$\begin{array}{c}\mathrm{ACGSR}<br><br>\left(a_{\mathrm{ij}}\right)=\left[\frac{s_i-s_{\mathrm{ij}}}{t_i-t_{\mathrm{ij}}}·<br><br>t_{\mathrm{ij}}-s_{\mathrm{ij}}\right]/N\\ =\left(s_i·<br><br>t_{\mathrm{ij}}-s_{\mathrm{ij}}·<br><br>t_{\mathrm{ij}}-t_i·<br><br>s_{\mathrm{ij}}+t_{\mathrm{ij}}·<br><br>s_{\mathrm{ij}}\right)/\left[N<br><br>\left(t_i-t_{\mathrm{ij}}\right)\right]\\ =\left(s_i·<br><br>t_{\mathrm{ij}}-t_i·<br><br>s_{\mathrm{ij}}\right)/\left[N<br><br>\left(t_i-t_{\mathrm{ij}}\right)\right].\end{array}$

where, a_ij is the arc or transition, t_i is the total number of calls passing through a state q_i, s_i as the total number of successful calls passing through the state q_i, T_ij is the set of sunken calls of the arc a_ij, t_ij=|T_ij| is used to represent the number of calls in the set, s_ij=|S_ij| is used to represent the number of successful sunken calls associated with the arc, and N is a total number of calls placed.