Training call routing applications by reusing semantically-labeled data collected for prior applications

• US 8,515,736 B1
• Filed: 09/30/2010
• Issued: 08/20/2013
• Est. Priority Date: 09/30/2010
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Duta, Nicolae
• Primary Examiner(s): JACKSON, JAKIEDA R
• Application Number:
  US12/894,752
• Time in Patent Office: 1,055 Days
• Field of Search: 704/9
• US Class Current: 704/9
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04M 2201/18 Comparators
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 2203/2038 Call context notifications
• CPC Class Codes: H04M 2203/357 Autocues for dialog assistance
• CPC Class Codes: H04M 2203/551 Call history recording a computer conference H04L12/1831
• CPC Class Codes: H04M 2203/558 Databases
• CPC Class Codes: H04M 3/5232 Call distribution algorithms

Abstract

Techniques disclosed herein include systems and methods for reusing semantically-labeled data collected for previous or existing call routing applications. Such reuse of semantically-labeled utterances can be used for automating and accelerating application design as well as data transcription and labeling for new and future call routing applications. Such techniques include using a semantic database containing transcriptions and semantic labels for several call routing applications along with corresponding baseline routers trained for those applications. This semantic database can be used to derive a semantic similarity measure between any pair of utterances, such as transcribed sentences. A mathematical model predicts how semantically related two utterances are, such as by identifying a same user intent to identifying completely unrelated intents. Such a semantic similarity measure can be used for various tasks including semantic-based example selection for language model and router training, and semantic data clustering for semi-automated labeling.

Claims

1. A computer-implemented method of semantic classification, the computer-implemented method comprising:

receiving, at a computerized training manager, a first utterance for use in training a first computerized semantic classifier, the first computerized semantic classifier having a set of manually-created semantic classes for labeling transcribed utterances, the first utterance being a transcribed utterance manually classified in a first semantic class within the set of manually-created semantic classes;

semantically labeling the first utterance by a second computerized semantic classifier, the second computerized semantic classifier being a previously-trained computerized semantic classifier;

semantically labeling existing utterances by the second computerized semantic classifier, the existing utterances accessed from a database of existing utterances, the database of existing utterances containing transcribed utterances used to train previously-trained computerized semantic classifiers;

identifying, by the computerized training manager, existing utterances that are semantically similar to the first utterance based on semantic labeling from the second computerized semantic classifier; and

training, by the computerized training manager, the first computerized semantic classifier using the semantically similar existing utterances.

2. The computer-implemented method of claim 1, further comprising:

semantically labeling the first utterance by multiple computerized semantic classifiers, the multiple computerized semantic classifiers being previously-trained computerized semantic classifiers, wherein each of the multiple computerized semantic classifiers separately labels the first utterance; and

semantically labeling the existing utterances by the multiple computerized semantic classifiers, wherein each of the multiple computerized semantic classifiers separately labels the existing utterances.

3. The computer-implemented method of claim 1, wherein semantically labeling the first utterance and semantically labeling the existing utterances includes associating, by the multiple computerized semantic classifiers, a level of confidence with each applied semantic label, the level of confidence identifying a probability that a given utterance has been correctly labeled.

4. The computer-implemented method of claim 3, wherein identifying existing utterances that are semantically similar includes identifying, by the computerized training manager, existing utterances having a computed level of confidence within a predetermined threshold associated with the first utterance.

5. The computer-implemented method of claim 1, wherein receiving the first utterance for use in training the first computerized semantic classifier includes receiving, at the computerized training manager, the first utterance as part of training for a speech recognition call routing application, the first computerized semantic classifier being part of the speech recognition call routing application.

6. A computer system for semantic classification, the computer system comprising:

a processor; and

a memory coupled to the processor, the memory storing instructions that when executed by the processor cause the system to perform the operations of;

receiving a first utterance for use in training a first automatic semantic classifier, the first automatic semantic classifier having a set of semantic classes for labeling transcribed utterances, the first utterance being a transcribed utterance classified in a first semantic class within the set of semantic classes;

semantically labeling the first utterance using a second automatic semantic classifier, the second automatic semantic classifier being a previously-trained semantic classifier;

semantically labeling existing utterances using the second automatic semantic classifier, the existing utterances accessed from a database of existing utterances, the database of existing utterances containing transcribed utterances used to train previously-trained automatic semantic classifiers;

identifying existing utterances that are semantically similar to the first utterance based on semantic labeling from the second automatic classifier; and

training the first automatic semantic classifier using the semantically similar existing utterances.

7. The computer system of claim 6, wherein the memory stores further instructions that when executed by the processor cause the system to perform the operations of:

semantically labeling the first utterance using multiple automatic semantic classifiers, the multiple automatic semantic classifiers being previously-trained semantic classifiers, wherein each of the multiple automatic semantic classifiers separately labels the first utterance; and

semantically labeling the existing utterances using the multiple automatic semantic classifiers, wherein each of the multiple automatic semantic classifiers separately labels the existing utterances.

8. The computer system of claim 6, wherein semantically labeling the first utterance and semantically labeling the existing utterances includes associating a level of confidence with each applied semantic label, the level of confidence identifying a probability that a given utterance has been correctly labeled.

9. The computer system of claim 8, wherein identifying existing utterances that are semantically similar includes identifying existing utterances having a computed level of confidence within a predetermined threshold associated with the first utterance.

10. The computer system of claim 6, wherein receiving the first utterance for use in training the first automatic semantic classifier includes receiving the first utterance as part of training for a speech recognition call routing application, the first automatic semantic classifier being part of the speech recognition call routing application.

11. A computer-implemented method of semantic classification, the computer-implemented method comprising:

receiving, at a computerized training manager, a set of utterances for training a given computerized semantic classifier, the set of utterances being transcribed utterances;

semantically labeling utterances from the set of utterances by multiple previously-trained computerized semantic classifiers;

identifying, by the computerized training manager, semantically labeled utterances that are semantically similar;

grouping, by the computerized training manager, utterances from the semantically labeled utterances into sets of utterances based on a predetermined measure of semantic similarity between pairs of semantically labeled utterances;

presenting, by the computerized training manager, a given utterance from the set of utterances along with available semantic labels for manual selection of an available semantic label to classify a given set of semantically grouped utterances; and

training, by the computerized training manager, the given computerized semantic classifier using the classified given set of semantically grouped utterances.

12. The computer-implemented method of claim 11, wherein semantically labeling utterances includes associating, by the multiple previously-trained computerized semantic classifiers, a level of confidence with each applied semantic label, the level of confidence identifying a probability that a given utterance has been correctly labeled, wherein each utterance receives pairings of a semantic label and the level of confidence.

13. The computer-implemented method of claim 12, wherein associating the level of confidence with each applied semantic label includes applying, by the multiple previously-trained computerized semantic classifiers, both a level of confidence automatically generated by a respective previously-trained computerized semantic classifier and a level of confidence manually identified and associated with semantic labeling applied by the respective previously-trained computerized semantic classifier.

14. The computer-implemented method of claim 11, wherein grouping utterances from the semantically labeled utterances includes creating, by the computerized training manager, a hierarchy of groups structured for selecting various degrees of overlap among semantically labeled utterances.

15. The computer-implemented method of claim 11, wherein receiving the set of utterances includes receiving, at the computerized training manager, text utterances automatically transcribed by a speech recognizer, the set of utterances being received for use in training a speech recognition call routing application, the given computerized semantic classifier being part of the speech recognition call routing application.

16. A computer system for semantic classification, the computer system comprising:

a processor; and

a memory coupled to the processor, the memory storing instructions that when executed by the processor cause the system to perform the operations of;

receiving a set of utterances for training a given automatic semantic classifier, the set of utterances being transcribed utterances;

semantically labeling utterances from the set of utterances using multiple previously-trained automatic semantic classifiers;

identifying semantically labeled utterances that are semantically similar;

grouping utterances from the semantically labeled utterances into sets of utterances based on a measure of semantic similarity between pairs of semantically labeled utterances;

presenting a given utterance from the set of utterances along with available semantic labels for manual selection of an available semantic label to classify a given set of semantically grouped utterances; and

training the given automatic semantic classifier using the classified given set of semantically grouped utterances.

17. The computer system of claim 16, wherein the memory further stores instructions that when executed by the processor cause the system to perform the operation of:

presenting a set of semantically grouped utterances along with available semantic labels for manual selection of an available semantic label to classify the given set of semantically grouped utterances; and

wherein semantically labeling utterances includes associating a level of confidence with each applied semantic label, the level of confidence identifying a probability that a given utterance has been correctly labeled, wherein each utterance receives pairings of a semantic label and the level of confidence.

18. The computer system of claim 17, wherein associating the level of confidence with each applied semantic label includes applying both a level of confidence automatically generated by a respective previously-trained automatic semantic classifier and a level of confidence manually identified and associated with semantic labeling applied by the respective previously-trained automatic semantic classifier.

19. The computer system of claim 16, wherein grouping utterances from the semantically labeled utterances includes creating a hierarchy of groups structured for selecting various degrees of overlap among semantically labeled utterances.

20. The computer system of claim 16, wherein receiving the set of utterances includes receiving text utterances automatically transcribed by a speech recognizer, the set of utterances being received for use in training a speech recognition call routing application, the given automatic semantic classifier being part of the speech recognition call routing application.

21. A computer-implemented method of semantic classification, the computer-implemented method comprising:

labeling sentences from a first group of sentences by multiple different computerized text classifiers to produce a label and produce a classification confidence corresponding to the produced label, wherein each of the multiple different computerized text classifiers produces a respective label and classification pairing for each respective sentence from the first group of sentences, the first group of sentences including sentences collected from sample sentences used for training existing text classifiers used with speech recognition applications, the label representing a sentence meaning;

labeling sentences from a second group of sentences by the multiple different computerized text classifiers to produce a label and produce a classification confidence corresponding to the produced label, wherein each of the multiple different computerized text classifiers produces a respective label and classification pairing for each respective sentence from the second group of sentences, the second group of sentences including sentences collected from sample training sentences for training a new computerized text classifier used with a speech recognition application;

computing, by a computerized training manager, a semantic similarity for each labeled sentence from the second group of sentences as compared to each labeled sentence from the first group of sentences;

identifying, by the computerized training manager, labeled sentences from the first group of sentences that meet a predetermined measure of similarity to each given labeled sentence from the second group of sentences;

grouping, by the computerized training manager, identified labeled sentences from the first group of sentences with corresponding sentences from the second group of sentences based on the predetermined measure of similarity; and

training, by the computerized training manager, the new text classifier used with the speech recognition application using identified labeled sentences from the first group of sentences and sentences from a second group of sentences, the speech recognition application being a call routing application.

22. A computer-implemented method of semantic classification, the computer-implemented method comprising:

receiving, at a computerized training manager, a first set of semantic labels;

receiving, at a computerized training manager, a first set of sentences that includes at least one sample sentence for each semantic label from the first set of semantic labels;

creating, by the computerized training manager, a database of sentences that includes sentences used as training sentences to train text classifiers used with speech recognition applications;

identifying, by the computerized training manager, multiple different computerized text classifiers trained for classifying text as part of speech recognition applications, each of multiple different computerized text classifiers, when executed with a given input sentence, produces a semantic classification label and corresponding classification confidence of the produced semantic classification label for the given input sentence;

semantically labeling sentences from the database of sentences by the multiple different computerized text classifiers, wherein each of the multiple different computerized text classifiers produces a respective semantic label and classification pairing for each respective sentence from the database of sentences;

semantically labeling sentences from the first set of sentences by the multiple different computerized text classifiers, wherein each of the multiple different computerized text classifiers produces a respective semantic label and classification pairing for each respective sentence from the first set of sentences;

computing, by the computerized training manager, a semantic similarity for each semantically labeled sentence from the first group of sentences as compared to each semantically labeled sentence from the database of sentences;

for each semantically labeled sentence from the first set of sentences, identifying, by the computerized training manager, semantically labeled sentences from the database of sentences that meet a predetermined measure of similarity;

adding, by the computerized training manager, the identified semantically labeled sentences from the database of sentences as sample sentences to the first set of sentences according to the computed semantic similarity and the first set of semantic labels; and

training, by the computerized training manager, a statistical language model according to the first set of semantic labels using the identified semantically labeled sentences from the database of sentences and the first set of sentences.

23. A computer-implemented method of semantic classification, the computer-implemented method comprising:

receiving, at a computerised training manner, a first set of sample training sentences for training a text classifier, the sample training sentences requiring semantic labeling;

identifying, by the computerized training manager, multiple different computerized text classifiers trained for classifying text as part of speech recognition applications, each of multiple different computerized text classifiers, when executed with a given input sentence, produces a semantic classification label and corresponding classification confidence of the produced classification label for the given input sentence;

labeling sentences from the first set of sample training sentences by the multiple different computerized text classifiers, wherein each of the multiple different computerized text classifiers produces a respective semantic label and classification pairing for each respective sentence from the first set of sample training sentences;

computing, by the computerized training manager, a semantic similarity for each labeled sentence from the first set of sample training sentences as compared to other labeled sentences from the first set of sample training sentences;

hierarchically grouping, by the computerized training manager, labeled sentences from the first set of sample training sentences using a complete linkage algorithm based on a predetermined measure of similarity, each group within the hierarchy of labeled sentences corresponding to a semantic label;

presenting, by the computerized training manager, a given sentence from the first set of sample training sentences with at least one available semantic label, the at least one available semantic label corresponding to a group within the hierarchy or labeled sentences; and

training, by the computerized training manager, the text classifier using the group within the hierarchy of labeled sentences.