Semi-supervised training of destination map for call handling applications

• US 8,009,819 B2
• Filed: 05/04/2007
• Issued: 08/30/2011
• Est. Priority Date: 03/26/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Stubley, Peter R., Faizakov, Avraham, Peters, S. Douglas
• Primary Examiner(s): ADDY, THJUAN KNOWLIN
• Application Number:
  US11/800,106
• Publication Number:
  US 20080240396A1
• Time in Patent Office: 1,579 Days
• Field of Search: 379/211.02, 379/88.03, 379/212.01, 379/218.01, 379/201.01, 379/88.01, 379/265.02, 704/273
• US Class Current: 379/211.02
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: H04M 2201/12 Counting circuits
• CPC Class Codes: H04M 2201/36 Memories
• CPC Class Codes: H04M 2201/40 using speech recognition
• CPC Class Codes: H04M 3/42204 Arrangements at the exchange for service or number selection by voice at the terminal H04M1/27
• CPC Class Codes: H04M 3/4931 Directory assistance systems
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00

Abstract

A method of semi-supervised synonym inference for a call handling application, such as automated directory assistance or call routing, is described. In one embodiment the method comprises examining a database of caller interaction results from a directory assistance system that includes an automated directory assistance engine, detecting a specified characteristic in the caller interaction results, and using the detected characteristic to automatically train a destination map, where the destination map is for use by the automated directory assistance engine in automatically mapping human speech to a destination. The detecting of the specified characteristic in the caller interaction results may include a statistical analysis of the caller interaction results for each of one or more speech recognition strings.

Claims

1. A method, comprising the steps of:

establishing a destination map for use by an automated call handling engine associated with a first call handling system,

wherein the first call handing system comprises a system database that comprises a plurality of caller interaction results from one or more callers, and

wherein the destination map associates destinations with recognition strings from automatically recognized speech;

examining a plurality of the caller interaction results from the first call handling system;

detecting a specified characteristic in the plurality of caller interaction results, wherein the specified characteristic is associated with a level of confidence in the caller interaction results;

automatically updating the destination map using the detected specified characteristic; and

automatically associating recognized speech from a subsequent caller with a corresponding destination using the updated destination map, in any of the first call handling system or a second call handling system.

2. The method of claim 1, wherein the call handling system comprises an automated directory assistance system.

3. The method of claim 1, wherein the step of detecting the specified characteristic in the caller interaction results comprises the steps of:

examining a distribution of recorded operator-selected destinations that all correspond to a particular recognition string output by an automatic speech recognizer, the particular recognition string representing speech from one or more call handling system users, the distribution representing a plurality of instances in which the automated call handling engine did not correctly map a user'"'"'s speech to a destination; and

determining whether any one or more of the operator-selected destinations is represented disproportionately higher than other destinations in the distribution.

4. The method of claim 3, wherein the step of using the detected characteristic to automatically update a destination map comprises:

updating the destination map for the particular recognition string toward the at least one operator-selected destination determined to be represented disproportionately higher than other destinations in the distribution when at least one of the operator-selected destinations is determined to be represented disproportionately higher than other destinations in the distribution.

5. The method of claim 3, further comprising the step of:

updating the automated directory assistance engine to automatically refer a user to a human directory assistance operator whenever a directory assistance user'"'"'s speech is determined to match the particular recognition string when none of the operator-selected destinations is determined to be represented disproportionately higher than other destinations in the distribution.

6. The method of claim 1, wherein the step of detecting the specified characteristic in the caller interaction results comprises the steps of:

estimating a probability of error, p(ε

|s), for a particular speech recognition string based on a plurality of recorded caller interaction results;

estimating a probability of consistent error, p(c|s,ε

), for the particular speech recognition string based on the plurality of caller interaction results; and

determining whether the quantity (p(ε

|s)*p(c|s,ε

)) is greater than the quantity (1−

p(ε

|s));

wherein the step of automatically updating the destination map comprises mapping the speech recognition string to a highest value destination of a plurality of recorded operator-released destinations corresponding to the speech recognition string, only if the quantity (p(ε

|s)*p(c|s,ε

)) is greater than the quantity (1−

p(ε

|s)).

7. The method of claim 1, wherein the destination map comprises a statistical semantic model.

8. The method of claim 1, further comprising the steps of:

examining a plurality of caller-operator interactions in the call handling system, the plurality of caller-operator interactions corresponding to a particular speech recognition string produced by an automatic speech recognizer, the particular recognition string representing speech from one or more callers; and

automatically creating a disambiguation dialog for the speech recognition string, for disambiguating between a plurality of destinations, based on the plurality of caller-operator interactions.

9. A method of automatically updating a destination map for use by an automated directory assistance engine in mapping speech of a directory assistance user to a destination, the method comprising the steps of:

examining a plurality of caller interaction results from a directory assistance system;

examining a distribution of recorded operator-selected destinations, all of which correspond to a particular recognition string output by an automatic speech recognizer in the automated directory assistance engine, the particular recognition string representing speech from one or more directory assistance users, the distribution representing a plurality of instances in which the automated directory assistance system did not correctly map a user'"'"'s speech to a destination;

determining whether any of the operator-selected destinations is represented disproportionately higher than other destinations in the distribution;

updating the destination map for the particular recognition string toward the at least one operator-selected destination determined to be represented disproportionately higher than other destinations in the distribution when at least one of the operator-selected destinations is determined to be represented disproportionately higher than other destinations in the distribution; and

updating the automated directory assistance engine to automatically refer a user to a human directory assistance operator whenever a directory assistance user'"'"'s speech is determined to match the particular recognition string when none of the operator-selected destinations is determined to be represented disproportionately higher than other destinations in the distribution.

10. The method of claim 9, wherein the destination map comprises a statistical semantic model.

11. The method of claim 9, further comprising the steps of:

examining a plurality of caller-operator interactions, the plurality of caller-operator interactions corresponding to a particular speech recognition string produced by the automatic speech recognizer, the particular recognition string representing speech from one or more users; and

automatically creating a disambiguation dialog for the speech recognition string, based on the plurality of caller-operator interactions.

12. An apparatus, comprising:

a destination map for use by an automated call handling engine associated with a first call handling system,

wherein the first call handing system comprises a system database that comprises a plurality of caller interaction results from one or more callers, and

wherein the destination map associates destinations with recognition strings from automatically recognized speech;

means for examining the plurality of caller interaction results from the first call handling system;

means for detecting a specified characteristic in the plurality of caller interaction results, wherein the specified characteristic is associated with a level of confidence in the caller interaction results;

means for automatically updating the destination map using the detected specified characteristic; and

means for automatically associating recognized speech from a subsequent caller with a corresponding destination using the updated destination map, in any of the first call handling system or a second call handling system.

13. The apparatus of claim 12, wherein the automated call handling engine comprises an automated directory assistance engine.

14. The apparatus of claim 12, wherein the means for detecting the specified characteristic in the caller interaction results comprises:

means for examining a distribution of recorded operator-selected destinations which all correspond to a particular recognition string output by an automatic speech recognizer, the particular recognition string representing speech from one or more call handling system users, the distribution representing a plurality of instances in which said automated call handling engine did not correctly map a user'"'"'s speech to a destination; and

means for determining whether any of the operator-selected destinations is represented disproportionately higher than other destinations in the distribution.

15. The apparatus of claim 14, wherein the means for using the detected characteristic to automatically update the destination map comprises:

means for updating the destination map for the particular recognition string toward one of the operator-selected destinations that is represented disproportionately higher than other destinations in the distribution.

16. The apparatus of claim 14, further comprising:

means for training the automated call handling engine to automatically refer a user to a human operator when a user'"'"'s speech is determined to match said particular recognition string, if none of the operator-selected destinations is determined to be represented disproportionately higher than other destinations in the distribution.

17. The apparatus of claim 12, wherein the destination map comprises a statistical semantic model.

18. The apparatus of claim 12, further comprising:

means for examining a plurality of caller-operator interactions in the call handling system, the plurality of caller-operator interactions corresponding to a particular speech recognition string produced by an automatic speech recognizer, the particular recognition string representing speech from one or more callers; and

means for automatically creating a disambiguation dialog for the speech recognition string, for disambiguating between a plurality of destinations, based on the plurality of caller-operator interactions.

19. A method comprising the steps of:

examining a plurality of caller-operator interactions in a call handling system which includes an automated call handling engine, wherein the plurality of caller-operator interactions correspond to

a particular speech recognition string produced by an automatic speech recognizer, wherein the particular recognition string represents speech from one or more callers, and

destinations that are associated with the particular recognition string which are selected by one or more operators

automatically creating a disambiguation dialog for the particular speech recognition string, based on the plurality of caller-operator interactions; and

using the disambiguation dialog with at least one subsequent caller to disambiguate between candidate destinations.

20. The method of claim 19, further comprising the steps of:

examining a distribution of recorded operator-selected destinations recorded in the call handling system, all of which correspond to the particular recognition string, the distribution representing a plurality of instances in which said automated call handling engine did not correctly map a caller'"'"'s speech to a destination; and

identifying a plurality of operator-selected destinations represented disproportionately higher than the other destinations in the distribution.

21. The method of claim 20, wherein the step of automatically creating the disambiguation dialog comprises the step of:

using the plurality of caller-operator interactions to automatically create the disambiguation dialog, the disambiguation dialog for disambiguating the speech recognition string between the plurality of operator-selected destinations.