Natural Language System and Method Based on Unisolated Performance Metric

• US 20090030692A1
• Filed: 05/15/2008
• Published: 01/29/2009
• Est. Priority Date: 02/28/2005
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Incorporated (International Business Machines Corporation)
• Inventors: Wu, Cheng, Gao, Yuqing, Kuo, Hong-Kwang, Goel, Vaibhava, Deligne, Sabine
• Granted Patent: US 8,571,869 B2
• Time in Patent Office: Days
• US Class Current: 704/257
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/18 using natural language modelling

Abstract

A natural language business system and method is developed to understand the underlying meaning of a person'"'"'s speech, such as during a transaction with the business system. The system includes a speech recognition engine, and action classification engine, and a control module. The control module causes the system to execute an inventive method wherein the speech recognition and action classification models may be recursively optimized on an unisolated performance metric that is pertinent to the overall performance of the natural language business system, as opposed to the isolated model-specific criteria previously employed

Claims

1. A method of enhancing performance of a natural language system employing an initial speech recognition model and an initial action classification model, said system, during operation, engaging in a plurality of transactions, said method comprising the steps of:

obtaining, during the operation of said system based on the initial speech recognition model and the initial action classification model, for pertinent ones of said transactions, at least audio data and associated transaction classification data; and

modifying at least one of the initial speech recognition model and the initial action classification model to obtain an iterated speech recognition model and an iterated action classification model, said modifying being based at least in part on said audio data and associated transaction classification data, so as to effect a desired change in an unisolated performance metric pertinent to performance of said natural language system

2. The method of claim 1, wherein said modifying step comprises jointly modifying the initial speech recognition model and the initial action classification model so as to effect said desired change in said unisolated performance metric.

3. The method of claim 2, further comprising the additional step of repeating said modifying step, with said iterated speech recognition model and said iterated action classification model substituted for the initial speech recognition model and the initial action classification model, until a desired value of said unisolated performance metric is achieved, whereby said unisolated performance metric is recursively optimized

4. The method of claim 3, further comprising the additional steps of:

subsequent to said repeating of said modifying step, evaluating said iterated speech recognition model and said iterated action classification model against a baseline;

if said iterated speech recognition model and said iterated action classification model compare favorably to said baseline, formatting said iterated speech recognition model and said iterated action classification model for subsequent use in operation of said system; and

if said iterated speech recognition model and said iterated action classification model compare unfavorably to said baseline, repeating said obtaining and modifying steps.

5. The method of claim 3, wherein:

said natural language system is a call routing system and said unisolated performance metric comprises a classification error late indicative of end-to-end routing accuracy of said call routing system

6. The method of claim 3, wherein said unisolated performance metric is given by:

M=f(ZPC),

where;

Z comprises a parameter related to at least one of the following;

raw accuracy;

automation rate; and

concept accuracy;

P is a performance weight parameter;

C is a cost weight parameter; and

f is a function preselected based on business needs.

7. The method of claim 3, wherein:

the initial speech recognition model comprises an initial acoustic model and an initial language model;

said iterated speech recognition model comprises an iterated acoustic model and an iterated language model; and

said modifying of the initial speech recognition model comprises modifying at least one of the initial language model and the initial acoustic model.

8. The method of claim 7, wherein said unisolated performance metric is given by the equation:

9. The method of claim 3, further comprising the additional steps of:

observing a change in said unisolated performance metric of said system;

establishing a first fraction of said transactions to be handled by first selected humans; and

changing said first fraction to obtain a second fraction of said transactions to be handled by second selected humans, said changing being responsive to one of an improvement and a degradation in said unisolated performance metric of said system detected during said observing step.

10. The method of claim 1, wherein:

said plurality of transactions include transactions with a plurality of agents of an enterprise associated with said natural language system; and

said pertinent ones of said transactions include at least some transactions with at least a selected multi-member group of the agents;

whereby knowledge of more than one of said agents is incorporated in said modifying step.

11. The method of claim 1, further comprising the additional steps of:

automatically detecting, from among said plurality of transactions, at least one confusion class;

formatting data indicative of said at least one confusion class in a form suitable for subsequent display; and

formatting a command to update a call flow model of said system to address said at least one confusion class.

12. The method of claim 1, further comprising the additional step of repeating said modifying step, with said iterated speech recognition model and said iterated action classification model substituted for the initial speech recognition model and the initial action classification model, until a desired value of said unisolated performance metric is achieved, whereby said unisolated performance metric is recursively optimized.

13. The method of claim 12, further comprising the additional steps of:

determining data representing a performance measure matrix indicative of performance of said system, said data including at least metric data associated with the optimization of said unisolated performance metric; and

formatting, for subsequent display, said data representing said performance measure matrix

14. The method of claim 12, wherein said unisolated performance metric is given by the equation:

15. The method of claim 12, further comprising the additional steps of:

subsequent to said repeating of said modifying step, evaluating said iterated speech recognition model and said iterated action classification model against a baseline; and

if said iterated speech recognition model and said iterated action classification model compare favorably to said baseline, formatting said iterated speech recognition model and said iterated action classification model for subsequent use in operation of said system.

16. The method of claim 12, further comprising the additional steps of:

subsequent to said repeating of said modifying step, evaluating said iterated speech recognition model and said iterated action classification model against a baseline; and

if said iterated speech recognition model and said iterated action classification model compare unfavorably to said baseline, repeating said obtaining and modifying steps.

17. An article of manufacture for enhancing performance of a natural language system employing an initial speech recognition model and an initial action classification model, said system, during operation, engaging in a plurality of transactions, the article of manufacture comprising:

a computer readable medium containing one or more programs which when executed implement the steps of;

obtaining, during the operation of said system based on the initial speech recognition model and the initial action classification model, for pertinent ones of said transactions, at least audio data and associated transaction classification data; and

modifying at least one of the initial speech recognition model and the initial action classification model to obtain an iterated speech recognition model and an iterated action classification model, said modifying being based at least in part on said audio data and associated transaction classification data, so as to effect a desired change in an unisolated performance metric pertinent to performance of said natural language system

18. The article of manufacture of claim 17, wherein in said method, said modifying step comprises jointly modifying the initial speech recognition model and the initial action classification model so as to effect said desired change in said unisolated performance metric.

19. The article of manufacture of claim 18, wherein said method further comprises the additional step of repeating said modifying step, with said iterated speech recognition model and said iterated action classification model substituted for the initial speech recognition model and the initial action classification model, until a desired value of said unisolated performance metric is achieved, whereby said unisolated performance metric is recursively optimized.

20. The article of manufacture of claim 17, wherein said method further comprises the additional step of repeating said modifying step, with said iterated speech recognition model and said iterated action classification model substituted for the initial speech recognition model and the initial action classification model, until a desired value of said unisolated performance metric is achieved, whereby said unisolated performance metric is recursively optimized.

21. A natural language system for interfacing with:

a processing center that conducts a plurality of verbal transactions, a speech recognition module that has an initial speech recognition model and that transcribes speech data for a first set of the transactions based on the initial speech recognition model, and an action classification module that has an initial action classification model and that classifies each of the first set of the transactions based on the transcribed speech data and the initial action classification model to generate transaction classification data, said system comprising;

a speech recognition interface configured to interface with the speech recognition module;

an action classification interface configured to interface with the action classification module; and

a control module coupled to said speech recognition interface and said action classification interface, said control module causing modification of at least one of the initial speech recognition model and the initial action classification model to obtain an iterated speech recognition model and an iterated action classification model, said modifying being based at least in part on the transcribed speech data and the transaction classification data, so as to effect a desired change in an unisolated performance metric pertinent to performance of the natural language system.

22. The system of claim 21, wherein said control module jointly modifies the initial speech recognition model and the initial action classification model so as to effect said desired change in said unisolated performance metric

23. The system of claim 22, wherein said control module repeats said joint modification, with said iterated speech recognition model and said iterated action classification model substituted for the initial speech recognition model and the initial action classification model, until a desired value of said unisolated performance metric is achieved, whereby said unisolated performance metric is recursively optimized.

24. The system of claim 23, wherein said control module is configured to:

subsequent to said repeating of said joint modification, evaluate said iterated speech recognition model and said iterated action classification model against a baseline; and

if said iterated speech recognition model and said iterated action classification model compare favorably to said baseline, format said iterated speech recognition model and said iterated action classification model for subsequent use in operation of the processing center.

25. The system of claim 23, wherein said control module is configured to:

subsequent to said repeating of said joint modification, evaluate said iterated speech recognition model and said iterated action classification model against a baseline; and

if said iterated speech recognition model and said iterated action classification model compare unfavorably to said baseline, (i) cause the speech recognition module and the action classification module to obtain speech data and generate transaction classification data for an additional set of said transactions, and (ii) repeat said joint modification.

26. The system of claim 23, wherein:

said natural language system comprises a call routing system; and

said unisolated performance metric comprises a classification error rate indicative of end-to-end routing accuracy of said call routing system

27. The system of claim 23, wherein said unisolated performance metric is given by:

M=f(ZPC),

where;

Z comprises a parameter related to at least one of the following;

raw accuracy;

automation rate; and

concept accuracy;

P is a performance weight parameter;

C is a cost weight parameter; and

f is a function preselected based on business needs.

28. The system of claim 23, wherein:

said initial speech recognition model comprises an initial acoustic model and an initial language model;

said modified speech recognition model comprises an iterated acoustic model and an iterated language model; and

said control module modifies said initial speech recognition model by modifying at least one of said initial language model and said initial acoustic model.

29. The system of claim 28, wherein:

said unisolated performance metric is given by the equation;

30. The system of claim 23 wherein said control module:

observes a change in said unisolated performance metric of said system;

establishes a first fraction of said transactions to be handled by first selected humans; and

changes said first fraction to obtain a second fraction of said transactions to be handled by second selected humans, said change being responsive to observing one of an improvement and a degradation in said unisolated performance metric of said system.

31. The system of claim 21, wherein:

said plurality of transactions include transactions with a plurality of agents of an enterprise associated with said natural language system;

said first set of said transactions include at least some transactions with at least a selected multi-member group of the agents; and

said control module incorporates knowledge of more than one of said agents when modifying said initial speech recognition model and said initial action classification model.

32. The system of claim 21, wherein said control module is configured to:

automatically detect, from among said plurality of transactions, at least one confusion class;

format data indicative of said at least one confusion class in a form suitable for subsequent display; and

format a command to update a call flow model of said system to address said at least one confusion class.

33. The system of claim 21 wherein said control module is configured to:

determine data representing a performance measure matrix indicative of performance of said system, said data including at least metric data associated with the optimization of said unisolated performance metric; and

format, for subsequent display, said data representing said performance measure matrix.

34. The system of claim 21, wherein said control module is configured to repeatedly modify said at least one of said initial speech recognition model and the initial action classification model, with said iterated speech recognition model and said iterated action classification model substituted for said initial speech recognition model and the initial action classification model, until a desired value of said unisolated performance metric is achieved, whereby said unisolated performance metric is recursively optimized.

35. The system of claim 34, wherein:

said unisolated performance metric is given by the equation;

36. The system of claim 34, wherein said control module is configured to:

subsequent to said repeating of said modification, evaluate said iterated speech recognition model and said iterated action classification model against a baseline;

if said iterated speech recognition model and said iterated action classification model compare favorably to said baseline, format said iterated speech recognition model and said iterated action classification model for subsequent use in operation of said system; and

if said iterated speech recognition model and said iterated action classification model compare unfavorably to said baseline, (i) cause said speech recognition module and said action classification module to obtain speech data and generate transaction classification data for an additional set of said transactions, and (ii) repeat said modification.

37. A method of gathering data pertinent to performance of a natural language system employing a speech recognition model and an action classification model, said system, during operation, engaging in a plurality of transactions, said method comprising the steps of:

obtaining, during the operation of said system, for pertinent ones of said transactions, audio data;

automatically obtaining, during the operation of said system, for said pertinent ones of said transactions, corresponding action classification data comprising at least one of a call transfer pattern, a department type, or a transaction type; and

formatting said audio data for each of said pertinent ones of said transactions with said corresponding action classification data for each of said pertinent ones of said transactions, in a form to facilitate training of at least one of said speech recognition model and said action classification model