Method for the automatic determination of context-dependent hidden word distributions

• US 20110119050A1
• Filed: 11/18/2010
• Published: 05/19/2011
• Est. Priority Date: 11/18/2009
• Status: Abandoned Application

• Current Assignee: Katholieke Universiteit Leuven
• Original Assignee: Katholieke Universiteit Leuven
• Inventors: Moens, Marie-Francine, Deschacht, Koen
• Application Number:
  US12/927,651
• Publication Number:
  US 20110119050A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Described is method, the Latent Words Language Model (LWLM), that automatically determines context-dependent word distributions (called hidden or latent words) for each word of a text. The probabilistic word distributions reflect the probability that another word of the vocabulary of a language would occur at that position in the text. Furthermore, a method is described to use these word distributions in statistical language processing applications, such as information extraction applications (for example, semantic role labeling, named entity recognition), automatic machine translation, textual entailment, paraphrasing, information retrieval, and speech recognition.

Claims

1. A method for determining a probabilistic, context dependent word distribution for each word in a previously unseen text, the method comprising:

in a training phase, learning for each word of a large corpus of natural language texts a probabilistic context model that describes the context these words typically occur in and learning a hidden-to-observed distribution that that describes words with similar meaning and usage;

storing the context model and the hidden-to-observed distribution on a storage device; and

in an inference phase, retrieving the context model and the hidden-to-observed distribution from the storage device and for each word in the previously unseen text determining the probabilistic, context dependent word distribution utilizing the context model and the hidden-to-observed distribution obtained in the training phase.

2. The method according to claim 1 wherein, in the training phase, the probabilistic context model and the context dependent word distribution are iteratively refined.

3. The method according to claim 1 wherein the training phase comprises:

tokenizing the corpus of natural language texts into individual words;

representing the corpus of natural language text with a Bayesian model with a hidden or latent variable for every word in the corpus, the Bayesian model representing the context dependent set of similar words, and with dependencies between the hidden variable and the hidden variables in its context, the dependencies representing the context model, and with dependencies between the hidden variable and the observed word at that position, the dependencies representing the hidden-to-observed distribution; and

utilizing approximate inference methods to determine a probabilistic distribution of words for the hidden variables, to learn the context model and to learn the hidden-to-observed distribution.

4. The method according to claim 2 wherein the training phase comprises:

tokenizing the corpus of natural language texts into individual words;

representing the corpus of natural language text with a Bayesian model with a hidden or latent variable for every word in the corpus, the Bayesian model representing the context dependent set of similar words, and with dependencies between the hidden variable and the hidden variables in its context, the dependencies representing the context model, and with dependencies between the hidden variable and the observed word at that position, the dependencies representing the hidden-to-observed distribution; and

utilizing approximate inference methods to determine a probabilistic distribution of words for the hidden variables, to learn the context model and to learn the hidden-to-observed distribution.

5. The method according to claim 1 wherein the inference phase comprises:

tokenizing the text into individual words;

representing the text with a Bayesian model with a hidden or hidden variable for every word in the corpus, the Bayesian model representing the context dependent set of similar words, and with dependencies between the hidden variable and the hidden variables in its context and between the hidden variable and the observed word at that position; and

utilizing the context model and the hidden-to-observed distribution learned in the training phase together with approximate inference methods to determine a probabilistic distribution of words for the hidden variables in a previously unseen text.

6. The method according to claim 2 wherein the inference phase comprises:

tokenizing the text into individual words;

representing the text with a Bayesian model with a hidden or hidden variable for every word in the corpus, the Bayesian model representing the context dependent set of similar words, and with dependencies between the hidden variable and the hidden variables in its context and between the hidden variable and the observed word at that position; and

utilizing the context model and the hidden-to-observed distribution learned in the training phase together with approximate inference methods to determine a probabilistic distribution of words for the hidden variables in a previously unseen text.

7. The method according to claim 3 wherein the inference phase comprises:

tokenizing the text into individual words;

representing the text with a Bayesian model with a hidden or hidden variable for every word in the corpus, the Bayesian model representing the context dependent set of similar words, and with dependencies between the hidden variable and the hidden variables in its context and between the hidden variable and the observed word at that position; and

utilizing the context model and the hidden-to-observed distribution learned in the training phase together with approximate inference methods to determine a probabilistic distribution of words for the hidden variables in a previously unseen text.

8. The method according to claim 4 wherein the inference phase comprises:

tokenizing the text into individual words;

representing the text with a Bayesian model with a hidden or hidden variable for every word in the corpus, the Bayesian model representing the context dependent set of similar words, and with dependencies between the hidden variable and the hidden variables in its context and between the hidden variable and the observed word at that position; and

utilizing the context model and the hidden-to-observed distribution learned in the training phase together with approximate inference methods to determine a probabilistic distribution of words for the hidden variables in a previously unseen text.

9. A method for automatic analysis of natural language, the method comprising:

utilizing a probabilistic, context dependent word distribution determined by the method according to claim 1 for each word in a previously unseen text.

10. The method according to claim 9, wherein the automatic analysis is semantic role labeling.

11. A method for automatic analysis of natural language, the method comprising:

utilizing a probabilistic, context dependent word distribution determined by the method according to claim 2 for each word in a previously unseen text.

12. The method according to claim 11, wherein the automatic analysis is semantic role labeling.

13. A method for automatic analysis of natural language, the method comprising:

utilizing a probabilistic, context dependent word distribution determined by the method according to claim 3 for each word in a previously unseen text.

14. The method according to claim 13, wherein the automatic analysis is semantic role labeling.

15. A method for automatic analysis of natural language, the method comprising:

utilizing a probabilistic, context dependent word distribution determined by the method according to claim 4 for each word in a previously unseen text.

16. The method according to claim 15, wherein the automatic analysis is semantic role labeling.

17. A method for automatic analysis of natural language, the method comprising:

utilizing a probabilistic, context dependent word distribution determined by the method according to claim 5 for each word in a previously unseen text.

18. The method according to claim 17, wherein the automatic analysis is semantic role labeling.

19. A method for automatic analysis of natural language, the method comprising:

utilizing a probabilistic, context dependent word distribution determined by the method according to claim 6 for each word in a previously unseen text.

20. The method according to claim 19, wherein the automatic analysis is semantic role labeling.