Cooccurrence and constructions

• US 7,373,102 B2
• Filed: 08/11/2004
• Issued: 05/13/2008
• Est. Priority Date: 08/11/2003
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Deane, Paul
• Primary Examiner(s): MOSSER, KATHLEEN MICHELE
• Application Number:
  US10/915,881
• Publication Number:
  US 20050049867A1
• Time in Patent Office: 1,371 Days
• Field of Search: 704/1, 704/3, 704/9, 704/10, 434/353, 434/323, 434/350
• US Class Current: 434/353
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding

Abstract

A method and system for performing automatic text analysis is described. A local ranking for one or more contexts with respect to a word and a global ranking for one or more contexts are generated. The rankings are based on the frequency with which the contexts appear in a corpus. A statistic may be generated using the local and global rankings, such as a log ratio rank statistic equal to the logarithm of the global rank divided by local rank, to measure the similarity of contexts with respect to words with which they combine. A source matrix of word to context values is then created. Singular value decomposition is used to create sub-matrices from the source matrix. Vectors from the sub-matrices corresponding to context(s) and/or word(s) are then selected to determine term-term or context-context similarity or term-context correspondence.

Claims

1. A system for ranking contexts within which a word in a corpus appears, comprising:

a processor; and

a processor-readable storage medium operably connected to the processor,

wherein the processor-readable storage medium contains one or more programming instructions for performing a method for ranking contexts within which a word in a corpus appears, the method comprising;

for each word in a corpus, determining a local ranking for each of one or more contexts,

wherein each context comprises a word sequence located in a particular arrangement relative to the word, wherein the particular arrangement comprises the relative ordering of the word sequence and the word,

wherein the local ranking comprises an ordering based on the frequency with which each context appears with the word;

for each context, determining a global ranking;

computing a statistic for each context based on one or more of the local ranking and the global ranking, and

ordering the one or more contexts based on the computed statistic for each context.

2. The system of claim 1 wherein the global ranking includes an ordering based on the overall frequency of each context in the corpus.

3. The system of claim 1 wherein the computed statistic comprises a log rank ratio statistic, wherein the log rank ratio statistic for a context with respect to a word may equal the logarithm of the global rank for the context divided by the local rank for the context with the word.

4. A system for ranking contexts within which a word in a corpus appears, comprising:

a processor; and

a processor-readable storage medium operably connected to the processor,

wherein the processor-readable storage medium contains one or more programming instructions for performing a method for ranking contexts within which a word in a corpus appears, the method comprising;

for each word in a corpus, determining a local ranking for each of one or more contexts,

wherein each context comprises a word sequence located in a particular arrangement relative to the word, wherein the particular arrangement comprises the relative ordering of the word sequence and the word,

wherein the local ranking comprises an ordering based on the frequency with which each context appears with the word,

for each context, determining a global ranking,

computing a statistic for each context based on one or more of the local ranking and the global ranking, and

producing a source matrix of words by contexts in which an attribute of each context is used as a value for the context-word combination.

5. The system of claim 4 wherein the global ranking includes an ordering based on the overall frequency of each context in the corpus.

6. The system of claim 4 wherein the computed statistic comprises a log rank ratio statistic, wherein the log rank ratio statistic for a context with respect to a word may equal the logarithm of the global rank for the context divided by the local rank for the context with the word.

7. The system of claim 4 wherein each value comprises a frequency for the context in the corpus.

8. The system of claim 4 wherein the processor-readable storage medium further contains one or more programming instructions for:

generating a term matrix, a factor-weighting matrix and a context matrix from the source matrix.

9. The system of claim 8 wherein the processor-readable storage medium further contains one or more programming instructions for:

computing term-term similarity based on a term vector for each term from the term matrix and a weight vector for each term from the factor-weighting matrix.

10. The system of claim 8 wherein the processor-readable storage medium further contains one or more programming instructions for:

computing context-context similarity based on a context vector for each context from the context matrix and a weight vector for each context from the factor-weighting matrix.

11. The system of claim 8 wherein the processor-readable storage medium further contains one or more programming instructions for:

computing term-context similarity based on a term vector for the term from the term matrix, a context vector for the context from the context matrix, and a weight vector from the factor-weighting matrix.