Method for dynamic context scope selection in hybrid N-GRAM+LSA language modeling

• US 7,720,673 B2
• Filed: 02/23/2007
• Issued: 05/18/2010
• Est. Priority Date: 03/10/2000
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bellegarda, Jerome R.
• Primary Examiner(s): Han; Qi
• Application Number:
  US11/710,098
• Publication Number:
  US 20070162276A1
• Time in Patent Office: 1,180 Days
• Field of Search: 704/9, 704/8, 704/231, 704/1, 704/10
• US Class Current: 704/1
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams

Abstract

A method and system for dynamic language modeling of a document are described. In one embodiment, a number of local probabilities of a current document are computed and a vector representation of the current document in a latent semantic analysis (LSA) space is determined. In addition, a number of global probabilities based upon the vector representation of the current document in an LSA space is computed. Further, the local probabilities and the global probabilities are combined to produce the language modeling.

Claims

1. A method of language modeling of a document comprising:

computing, by a computer processor, a plurality of local probabilities of a current document;

determining a vector representation of the current document in a latent semantic analysis (LSA) space at a first time based on a first number of words present in the current document from a second time to the first time, wherein the second time precedes the first time;

computing, by a computer processor, a plurality of global probabilities based upon the vector representation of the current document in an LSA space; and

combining the local probabilities and the global probabilities to produce a language modeling.

2. The method of claim 1 wherein the plurality of local probabilities is based upon an n-gram paradigm.

3. The method of claim 1 wherein the plurality of local probabilities Pr(w_q|H_q^(t)) for a particular word w_q, drawn from a vocabulary V comprising a plurality of words w_i, given a hybrid contextual history H_q^(T) of n−

1 words w<sub>q−

1</sub>, w<sub>q−

2</sub>, . . . w<sub>q−

n+1</sub>, as;

Pr(w_q|H_q^(t))=Pr(w_q|w<sub>q−

1</sub>W<sub>q−

2</sub> . . . w<sub>q−

n+1</sub>).

4. The method of claim 1 wherein the vector representation of the current document in an LSA space is generated from at least one decomposition matrix of a singular value decomposition of a co-occurrence matrix, W, between M words in a vocabulary V and N documents in a text corpus T.

5. The method of claim 4 wherein the first number of words in the current document is a total number of words from the second time to the first time, wherein the second time is a beginning of a session.

6. The method of claim 4 wherein the first number of words is a number of temporally adjacent words.

7. The method of claim 1, wherein the vector representation of the current document in an LSA space, {tilde over (V)}_q, at the first time q, wherein n_q is a total number of words in the current document, i_p is an index of a word observed at time p, ε

_ip is a normalized entropy of the word observed at time p within a text T, μ

_ip is a left singular vector at time p of a singular value decomposition of W, and S is a diagonal matrix of singular values of the singular value decomposition of W, is as;

8. The method of claim 1 wherein the vector representation of the current document in an LSA space, {tilde over (v)}_q, at the first time q, wherein n_q is a total number of words in the current document, i_p is an index of a word observed at time p, ε

_ip is a normalized entropy of the word observed at time p within a text T, P is a number of temporally adjacent words up to the current word, μ

_ip is a left singular vector at time p of a singular value decomposition of W, and S is a diagonal matrix of singular values of the singular value decomposition of W, is as;

9. The method of claim 4 wherein the first number of words is a number of exponentially weighted temporally adjacent words.

10. The method of claim 1 wherein the vector representation of the current document in an LSA space, {tilde over (v)}, at the first time q, wherein n_q is a total number of words in the current document, i_p is an index of a word observed at time p, ε

_ip is a normalized entropy of the word observed at time p within a text T, 0<

λ

≦

1, μ

_ip is a left singular vector at time p of the singular value decomposition of W, and S is a diagonal matrix of singular values of a singular value decomposition of W, is as;

11. The method of claim 1 wherein the plurality of global probabilities is based upon a latent semantic paradigm.

12. The method of claim 1 wherein the plurality of global probabilities Pr(w_q|H_q−

1) for a particular word w_q, for an associated history of the word, H<sub>q−

1</sub>, for the current document {tilde over (d)}<sub>q−

i</sub>, as;

Pr(w_q|H<sub>q−

1</sub>)=Pr(w_q|{tilde over (d)}<sub>q−

i</sub>), based upon the vector representation of the current document in an LSA space.

13. The method of claim 12 wherein combining the local probabilities and the global probabilities is computed as follows:

14. A computer system for language modeling of a document comprising:

means for computing a plurality of local probabilities of a current document;

means for determining a vector representation of the current document in a latent semantic analysis (LSA) space at a first time based on a first number of words present in the current document from a second time to the first time, wherein the second time precedes the first time;

means for computing a plurality of global probabilities based upon the vector representation of the current document in an LSA space; and

means for combining the local probabilities and the global probabilities to produce a language modeling.

15. A non-transitory computer readable storage medium comprising instructions, which when executed on a processor, perform a method for language modeling of a document, comprising:

computing a plurality of local probabilities of a current document;

determining a vector representation of the current document in a latent semantic analysis (LSA) space at a first time based on a first number of words present in the current document from a second time to the first time, wherein the second time precedes the first time;

computing a plurality of global probabilities based upon the vector representation of the current document in an LSA space; and

combining the local probabilities and the global probabilities to produce a language modeling.

16. A system for language modeling of a document comprising:

a digital signal processor;

a hybrid training/recognition processor coupled to the digital signal processor, configured to compute a plurality of local probabilities of a current document, determine a vector representation of the current document in a latent semantic analysis (LSA) space at a first time based on a first number of words present in the current document from a second time to the first time, wherein the second time precedes the first time, compute a plurality of global probabilities based upon the vector representation of the current document in an LSA space, and combine the local probabilities and the global probabilities to produce a language modeling.

17. The system of claim 16 wherein the processor is further configured to generate the plurality of local probabilities based upon an n-gram paradigm.

18. The system of claim 16 wherein the processor is further configured to generate the plurality of local probabilities Pr(w_q|H_q^(t)) for a particular word w_q, drawn from a vocabulary V comprising a plurality of words w_i, given a hybrid contextual history H_q^(T) of n−

1 words w<sub>q−

1</sub>, w<sub>q−

2</sub>, . . . w<sub>q−

n+1</sub>, as;

Pr(w_q|H_q^(t))=Pr(w_q|w<sub>q−

1</sub>W<sub>q−

2</sub> . . . w<sub>q−

n+1</sub>).

19. The system of claim 16 wherein the processor is further configured to generate the vector representation of the current document in an LSA space from at least one decomposition matrix of a singular value decomposition of a co-occurrence matrix, W, between M words in a vocabulary V and N documents in a text corpus T.

20. The system of claim 19 wherein the first number of words is a total number of words from the second time to the first time, wherein the second time is a beginning of a session.

21. The system of claim 19 wherein first number of words is a number of temporally adjacent words.

22. The system of claim 16 wherein the processor is further configured to generate the vector representation of the current document in an LSA space, {tilde over (V)}_q, at the first time q, wherein n_q is a total number of words in the current document, i_p is an index of a word observed at time p, ε

_ip is a normalized entropy of the word observed at time p within a text T, μ

_ip is a left singular vector at time p of a singular value decomposition of W, and S is a diagonal matrix of singular values of the singular value decomposition of W, as;

23. The system of claim 16 wherein the processor is further configured to generate the vector representation of the current document in an LSA space, {tilde over (v)}_q, at the first time q, wherein n_q is a total number of words in the current document, i_p is an index of a word observed at time p, ε

_ip is a normalized entropy of the word observed at time p within a text T, P is a number of temporally adjacent words up to the current word, μ

_ip is a left singular vector at time p of a singular value decomposition of W, and S is a diagonal matrix of singular values of the singular value decomposition of W, as;

24. The system of claim 16 wherein the first number of words is a number of exponentially weighted temporally adjacent words.

25. The system of claim 16 wherein the processor is further configured to generate the plurality of global probabilities based upon a latent semantic paradigm.

26. The system of claim 16 wherein the processor is further configured to generate the plurality of global probabilities Pr(w_q|H_q−

1) for a particular word w_q, for an associated history of the word, H<sub>q−

1</sub>, for the current document {tilde over (d)}<sub>q−

1</sub>, as;

Pr(w_q|H<sub>q−

1</sub>)=Pr(w_q|{tilde over (d)}<sub>q−

i</sub>), based up on the vector representation of the current document in an LSA space.

27. The system of claim 25 wherein the processor is further configured to combine the local probabilities and the global probabilities as follows:

28. A system for language modeling of a document comprising:

a digital signal proceeor;

a hybrid training/recognition processor coupled to the digital signal processor, configured to compute a plurality of local probabilities of a current document, determine a vector representation of the current document in a latent semantic analysis (LSA) space based on a first number of words in the current document, compute a plurality of global probabilities based upon the vector representation of the current document in an LSA space, and combine the local probabilities and the global probabilities to produce a language modeling, wherein the processor is further configured to generate the vector representation of the current document in an LSA space, {tilde over (v)}, at time q, wherein n_q is the total number of words in the current document, i_p is the index of the word observed at time p, ε

_ip is the normalized entropy of the word observed at time p within a text T, 0<

λ

≦

1, μ

_ip is the left singular vector at time p of the singular value decomposition of W, and S is the diagonal matrix of singular values of the singular value decomposition of W, as;