Apparatus and method for analysis of language model changes

• US 20060212294A1
• Filed: 03/21/2005
• Published: 09/21/2006
• Est. Priority Date: 03/21/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Gorin, Allen Louis, Grothendieck, John, Wright, Jeremy Huntley Greet
• Granted Patent: US 7,805,300 B2
• Time in Patent Office: Days
• US Class Current: 704/245
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

An apparatus, a method, and a machine-readable medium are provided for characterizing differences between two language models. A group of utterances from each of a group of time domains are examined. One of a significant word change or a significant word class change within the plurality of utterances is determined. A first cluster of utterances including a word or a word class corresponding to the one of the significant word change or the significant word class change is generated from the utterances. A second cluster of utterances not including the word or the word class corresponding to the one of the significant word change or the significant word class change is generated from the utterances.

Claims

1. A method of characterizing differences between two language models, comprising:

examining a plurality of utterances from each of a plurality of time domains;

determining one of a significant word change or a significant word class change within the plurality of utterances;

generating, from the utterances, a first cluster of utterances including a word or a word class corresponding to the one of the significant word change or the significant word class change; and

generating, from the utterances, a second cluster of utterances not including the word or the word class corresponding to the one of the significant word change or the significant word class change.

2. The method of claim 1, wherein determining one of a significant word change or a significant word class change within the plurality of utterances further comprises comparing ones of the utterances from a particular group of speakers of one of the plurality of time domains with ones of the utterances from the particular group of speakers from at least one other of the plurality of time domains.

3. The method of claim 1, wherein determining one of a significant word change or a significant word class change within the plurality of utterances further comprises comparing ones of the utterances from a first particular group of speakers with ones of the utterances from a second particular group of speakers.

4. The method of claim 1, wherein determining one of a significant word change or a significant word class change further comprises comparing ones of the utterances from one of the plurality of time domains with ones of the utterances from at least one other of the plurality of time domains.

5. The method of claim 1, further comprising outputting a list of the clusters with analytical information.

6. The method of claim 1, wherein the plurality of time domains are two time domains.

7. The method of claim 1, further comprising iteratively performing:

examining a plurality of utterances from the second cluster of utterances not including the word or the class of words;

determining one of a next significant word change or a next significant word class change;

generating, from the examined utterances, a new first cluster of utterances including a next word or a next word class corresponding to the one of the next significant word change or the next significant word class change; and

generating, from the examined utterances, a new second cluster of utterances not including the next word or the next word class corresponding to the one of the next significant word change or the next significant word class change.

8. The method of claim 5, wherein the analytical information includes a history of splits that produced each of the clusters.

9. The method of claim 1, further comprising:

pooling the plurality of utterances from the plurality of time domains;

assigning each of the utterances to one of a plurality of subpopulations;

generating a language model for each of the subpopulations;

reassigning each of the utterances to one of the subpopulations according to a criterion;

determining whether any of the subpopulations fulfill a splitting criterion; and

splitting ones of the subpopulations that fulfill the splitting criterion, wherein;

examining a plurality of utterances from each of the time domains, determining one of a significant word change or a significant word class change within the plurality of utterances, generating, from the plurality of utterances, a first cluster of utterances including a word or a word class corresponding to the one of the significant word change or the significant word class change, and generating, from the plurality of utterances, a second cluster of utterances not including the word or the word class corresponding to the one of the significant word change or the significant word class change are performed after pooling, assigning, generating a language model, reassigning, determining whether any of the subpopulations fulfill a splitting criterion, and splitting.

10. The method of claim 9, wherein examining, determining one of a significant word change or a significant word class change, generating a first cluster, and generating a second cluster are performed for each of the subpopulations of the utterances.

11. The method of claim 9, wherein:

assigning each of the utterances to one of a plurality of subpopulations comprises assigning each of the utterances to one of two subpopulations,

reassigning each of the utterances to one of the subpopulations according to a criterion comprises reassigning each of the utterances to one of two subpopulations according to the criterion, and

splitting ones of the subpopulations that fulfill the splitting criterion comprises splitting ones of the subpopulations that fulfill the splitting criterion into two subpopulations.

12. The method of claim 9, further comprising:

iteratively performing, until the language models converge;

generating a language model for each of the subpopulations, and

reassigning each of the utterances to one of the subpopulations according to a criterion.

13. The method of claim 9, wherein the criterion comprises a subpopulation that maximizes a probability of an utterance occurring.

14. The method of claim 1, further comprising performing, before performing the acts of claim 1:

for each of the time domains, computing a matrix of associational scores among all words of a list of frequently occurring words;

computing differences in the associational scores of at least two of the matrices to produce a difference matrix;

producing a set of clusters of utterances based on similarity in associational scores of the difference matrix; and

creating a plurality of word classes based on a result of producing a set of clusters of utterances, wherein;

determining determines a significant word class change, and

generating the first cluster includes generating, from the examined utterances, a cluster of utterances including a word class corresponding to the significant word class change.

15. The method of claim 14, further comprising:

prioritizing the produced set of clusters.

16. An apparatus comprising:

at least one processor;

a device for displaying output; and

storage associated with the at least one processor, wherein;

the storage includes instructions for the at least one processor, and

the apparatus is configured to perform;

examining a plurality of utterances from each of a plurality of time domains;

determining one of a significant word change or a significant word class change within the plurality of utterances;

generating, from the utterances, a first cluster of utterances including a word or a word class corresponding to the one of the significant word change or the significant word class change; and

generating, from the utterances, a second cluster of utterances not including the word or the word class corresponding to the one of the significant word change or the significant word class change.

17. The apparatus of claim 16, wherein the apparatus is further configured to determine one of a significant word change or a significant word class change within the plurality of utterances by comparing ones of the utterances from a particular group of speakers of one of the plurality of time domains with ones of the utterances from the particular group of speakers from at least one other of the plurality of time domains.

18. The apparatus of claim 16, wherein the apparatus is further configured to determine one of a significant word change or a significant word class change within the plurality of utterances by comparing ones of the utterances from a first particular group of speakers of with ones of the utterances from a second particular group of speakers.

19. The apparatus of claim 16, wherein the apparatus is further configured to perform the determining by comparing ones of the utterances from one of the plurality of time domains with ones of the utterances from at least one other of the plurality of time domains.

20. The apparatus of claim 16, wherein the apparatus is further configured to perform outputting a list of the clusters with analytical information.

21. The apparatus of claim 16, wherein the plurality of time domains are two time domains.

22. The apparatus of claim 20, wherein the analytical information includes a history of splits that produced each of the clusters.

23. The apparatus of claim 16, wherein the apparatus is further configured to perform:

pooling the plurality of utterances from the plurality of time domains;

assigning each of the utterances to one of a plurality of subpopulations;

generating a language model for each of the subpopulations;

reassigning each of the utterances to one of the subpopulations according to a criterion;

determining whether any of the subpopulations fulfill a splitting criterion; and

splitting ones of the subpopulations that fulfill the splitting criterion, wherein;

examining a plurality of utterances from each of the time domains, determining one of a significant word change or a significant word class change within the plurality of utterances, generating, from the utterances, a first cluster of utterances including a word or a word class corresponding to the one of the significant word change or the significant word class change, and generating, from the utterances, a second cluster of utterances not including the word or the word class corresponding to the one of the significant word change or the significant word class change are performed after the apparatus performs pooling, assigning, generating a language model for each of the subpopulations, reassigning, determining whether any of the subpopulations fulfill a splitting criterion, and splitting.

24. The apparatus of claim 23, wherein the apparatus is further configured to perform:

examining, determining one of a significant word change or a significant word class change, generating a first cluster, and generating a second cluster for each of the subpopulations of the utterances.

25. The apparatus of claim 16, further configured to perform:

for each of the time domains, computing a matrix of associational scores among all words of a list of frequently occurring words;

computing differences in the associational scores of at least two of the matrices to produce a difference matrix;

producing a set of clusters of utterances based on similarity in associational scores of the difference matrix; and

creating a plurality of word classes based on a result of the producing a set of clusters of utterances, wherein;

determining determines a significant word class change, and

generating a first cluster includes generating, from the examined utterances, the first cluster of utterances including a word class corresponding to the significant word class change.

26. A machine-readable medium comprising a plurality of instructions for at least one processor, the machine-readable medium comprising:

instructions for examining a plurality of utterances from each of a plurality of time domains;

instructions for determining one of a significant word change or a significant word class change within the plurality of utterances;

instructions for generating, from the utterances, a first cluster of utterances including a word or a word class corresponding to the one of the significant word change or the significant word class change; and

instructions for generating, from the utterances, a second cluster of utterances not including the word or the word class corresponding to the one of the significant word change or the significant word class change.

27. The machine-readable medium of claim 26, further comprising:

instructions for pooling the plurality of utterances from the plurality of time domains;

instructions for assigning each of the utterances to one of a plurality of subpopulations;

instructions for generating a language model for each of the subpopulations;

instructions for reassigning each of the utterances to one of the subpopulations according to a criterion;

instructions for determining whether any of the subpopulations fulfill a splitting criterion; and

instructions for splitting ones of the subpopulations that fulfill the splitting criterion.

28. The machine-readable medium of claim 26, further comprising:

instructions for performing, for each of the time domains, computing a matrix of associational scores among all words of a list of frequently occurring words;

instructions for computing differences in the associational scores of at least two of the matrices to produce a difference matrix;

instructions for producing a set of clusters of utterances based on similarity in associational scores of the difference matrix; and

instructions for creating a plurality of word classes based on a result of producing a set of clusters of utterances.

29. An apparatus comprising:

means for examining a plurality of utterances from each of a plurality of time domains;

means for determining one of a significant word change or a significant word class change within the plurality of utterances;

means for generating, from the utterances, a first cluster of utterances including a word or a word class corresponding to the one of the significant word change or the significant word class change; and

means for generating, from the utterances, a second cluster of utterances not including the word or the word class corresponding to the one of the significant word change or the significant word class change.