System and method of generating dictionary entries

• US 7,254,530 B2
• Filed: 09/26/2002
• Issued: 08/07/2007
• Est. Priority Date: 09/26/2001
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Klavans, Judith L., Muresan, Smaranda
• Primary Examiner(s): {hacek over (S)}mits; Talivaldis Ivars
• Assistant Examiner(s): Siedler; Dorothy S
• Application Number:
  US10/398,535
• Publication Number:
  US 20050234709A1
• Time in Patent Office: 1,776 Days
• Field of Search: 704/10, 706/55
• US Class Current: 704/10
• CPC Class Codes: G06F 40/242 Dictionaries

Abstract

A system for automatically generating a dictionary from full text articles extracts <term, definition> pairs from full text articles and stores the <term, definition> pairs as dictionary entries. The system includes a computer readable corpus having a plurality of documents therein. A pattern processing module (120) and a grammar processing module (125) are provided for extracting <term, definition> pairs from the corpus and storing the <term, definition> pairs in a dictionary database (145). A routing processing module selectively routes sentences in the corpus to at least one of the pattern processing module or grammar processing module. In one embodiment, the routing module is incorporated into the pattern processing module which then selectively routes a portion of the sentences to the grammar processing module. A bootstrapping processing module (150) can be used to apply <term, definition> entries against the corpus to identify and extract additional <terms, definition> entries.

Claims

1. A method for automatically generating a dictionary based on a corpus of full text articles, comprising:

applying linguistic pattern analysis to the sentences in the corpus to extract <

term, definition>

pairs and identify sentences with candidate complex <

term, definition>

pairs;

applying grammar analysis to the sentences with candidate complex <

term, definition>

pairs to extract <

term, definition>

pairs;

storing the extracted <

term, definition>

pairs in a dictionary database;

wherein the linguistic pattern analysis further comprises identifying sentences including text markers, and wherein the sentences including text markers are subjected to filtering to remove sentences not likely to include <

term, definition>

pairs, and wherein the filtering includes rules selected from the group including sentences with conjunctions at the beginning of a text marker, sentences having phrases indicative of explanation, and sentences having patterns indicative of enumeration.

2. The method for automatically generating a dictionary according to claim 1, wherein pattern analysis of sentences including text markers identifies noun phrase patterns of the form <

Noun Phrase 1>

text marker <

Noun Phrase 2>

as <

term, definition>

pairs.

3. The method for automatically generating a dictionary according to claim 2, wherein the frequencies of Noun Phrase 1 and Noun Phrase 2 are determined and the noun phrase having the higher frequency is considered the term and the other noun phrase is considered the definition for the term.

4. The method for automatically generating a dictionary according to claim 2, wherein sentences with text markers having complex linguistic patterns are identified as sentences including candidate complex <

term, definition>

pairs.

5. The method for automatically generating a dictionary according to claim 1, wherein sentences which do not include text markers, but include complex linguistic patterns are identified as including candidate complex <

term, definition>

pairs.

6. The method for automatically generating a dictionary according to claim 1, wherein grammar analysis of sentences including candidate complex <

term, definition>

pairs includes at least one of processing of sentences exhibiting apposition and processing of sentences in the form <

term>

is <

definition>

.

7. The method for automatically generating a dictionary according to claim 1, further comprising a bootstrap process which applies the <

term, definition>

pairs in the dictionary to the corpus to extract additional <

term, definition>

pairs.

8. A method for automatically generating a dictionary based on a corpus of full text articles, comprising:

applying linguistic pattern analysis to the sentences in the corpus to extract <

term, definition>

pairs and identify sentences with candidate complex <

term, definition>

pairs;

applying grammar analysis to the sentences with candidate complex <

term, definition>

pairs to extract <

term, definition>

pairs;

storing the extracted <

term, definition>

pairs in a dictionary database;

wherein the linguistic pattern analysis further comprises identifying sentences including cue phrases and wherein sentences including predetermined cue phrases are parsed to identify left hand side context and right hand side context of the cue phrase, and wherein if the left hand side and right hand side contexts are noun phrases, then the left hand side context is considered the term and the right hand side context is considered the definition for the term.

9. A method for automatically generating a dictionary based on a corpus of full text articles, comprising:

applying linguistic pattern analysis to the sentences in the corpus to extract <

term, definition>

pairs and identify sentences with candidate complex <

term, definition>

pairs;

applying grammar analysis to the sentences with candidate complex <

term, definition>

pairs to extract <

term, definition>

pairs;

storing the extracted <

term, definition>

pairs in a dictionary database;

wherein grammar analysis of sentences including candidate complex <

term, definition>

pairs includes processing of sentences exhibiting apposition wherein apposition processing includes;

identifying the left conjunction and right conjunction in the sentence; and

if the right conjunction is a noun phrase and does not include a further conjunction, the right conjunction is considered the definition and the left conjunction is considered the term of a <

term, definition>

pair.

10. A method for automatically generating a dictionary based on a corpus of full text articles, comprising:

applying linguistic pattern analysis to the sentences in the corpus to extract <

term, definition>

pairs and identify sentences with candidate complex <

term, definition>

pairs;

applying grammar analysis to the sentences with candidate complex <

term, definition>

pairs to extract <

term, definition>

pairs;

storing the extracted <

term, definition>

pairs in a dictionary database;

wherein grammar analysis of sentences including candidate complex <

term, definition>

pairs includes processing of sentences in the form <

term>

is <

definition> and

wherein sentences of the form <

term>

is <

definition>

are further processed by the steps;

if the root of the sentence is a present tense for of the verb “

to be” and

the subject of the sentence is a noun then evaluate the predicate of the sentence;

evaluate daughters of subject to determine if daughters are left conjunctions and right conjunctions;

if daughters are not in the form of left conjunctive and right conjunctive, evaluate predicate, and if the predicate of the sentence is a noun, the subject is considered the term and a subtree rooted at the predicate is considered the definition of the <

term, definition>

pair.

11. The method for automatically generating a dictionary according to claim 10, wherein if the daughters are in the form of left conjunction and right conjunction, determine if daughters are both nouns;

if daughters are both nouns, assign left conjunctive and right conjunctive as additional terms; and

if predicate is a noun, then consider subtree rooted at the predicate as the definition for <

term, definition>

pairs for both left conjunction and right conjunction terms.

12. A system for automatically generating a dictionary from full text articles comprising:

a computer readable corpus having a plurality of documents therein;

a computer readable dictionary database;

a pattern processing module for extracting <

term, definition>

pairs from the corpus and storing the <

term, definition>

pairs in the dictionary database;

a grammar processing module for extracting <

term, definition>

pairs from the corpus and storing the <

term, definition>

pairs in the dictionary database; and

a routing processing module which routes sentences in the corpus to the pattern processing module;

wherein the pattern processing module further comprises a filtering module to remove sentences not likely to include <

term, definition>

pairs and wherein the filtering module applies at least one rule selected from the group including sentences with conjunctions at the beginning of a text marker, sentences having phrases indicative of explanation, and sentences having patterns indicative of enumeration.

13. The system for automatically generating a dictionary according to claim 12, further comprising a bootstrap processing module, the bootstrap processing module applying entries in the dictionary database to the corpus and extracting and storing additional <

term, definition>

pairs.

14. The system for automatically generating a dictionary according to claim 12 wherein:

the routing processing module tags sentences which may include <

term, definition>

pairs and routes all tagged sentences to the pattern processing module;

the pattern processing module performs the additional operation of identifying sentences which include candidate complex <

term, definition>

pairs; and

the grammar processing module receives the candidate complex <

term, definition>

pairs from the pattern processing module.

15. The system for automatically generating a dictionary according to claim 12, wherein the pattern processing module identifies sentences including at least one of text markers and cue phrases as <

term, definition>

pairs.

16. The system for automatically generating a dictionary according to claim 15, wherein the pattern processing module processes sentences including text markers by identifying noun phrase patterns of the form <

Noun Phrase 1>

text marker <

Noun Phrase 2>

as <

term, definition>

pairs.

17. The system for automatically generating a dictionary according to claim 16, wherein the pattern processing module determines the frequency of occurrence of each of Noun Phrase 1 and Noun Phrase 2 and assigns the noun phrase having the higher frequency as the term and the other noun phrase as the definition for the term.

18. The system for automatically generating a dictionary according to claim 16, wherein the pattern processing module identifies sentences with text markers having complex linguistic patterns as sentences including candidate complex <

term, definition>

pairs.

19. The system for automatically generating a dictionary according to claim 15, wherein the pattern processing module identifies sentences which do not include text markers, but include complex linguistic patterns as including candidate complex <

term, definition>

pairs.

20. The system for automatically generating a dictionary according to claim 12, wherein the grammar processing module extracts <

term, definition>

pairs from sentences including at least one of sentences exhibiting apposition and sentences in the form <

term>

is <

definition>

.

21. A system for automatically generating a dictionary from full text articles comprising:

a computer readable corpus having a plurality of documents therein;

a computer readable dictionary database;

a pattern processing module for extracting <

term, definition>

pairs from the corpus and storing the <

term, definition>

pairs in the dictionary database;

a grammar processing module for extracting <

term, definition>

pairs from the corpus and storing the <

term, definition>

pairs in the dictionary database; and

a routing processing module which routes sentences in the corpus to the pattern processing module,

wherein the pattern processing module identifies sentences including cue phrases as <

term, definition>

pairs, and wherein the pattern processing module parses sentences including predetermined cue phrases to identify left hand side context and right hand side context of the cue phrase, and wherein if the left hand side and right hand side contexts are noun phrases, then the left hand side context is assigned as the term and the right hand side context is assigned as the definition for the term.

22. Computer readable media encoded with instructions to direct a computer system to generate a dictionary based on a corpus of full text articles, comprising the steps of:

applying linguistic pattern analysis to sentences in a corpus to extract <

term, definition>

pairs and identify sentences with candidate complex <

term, definition>

pairs;

applying grammar analysis to the sentences with candidate complex <

term, definition>

pairs to extract <

term, definition>

pairs; and

storing the extracted <

term, definition>

pairs in a dictionary database;

wherein the linguistic pattern analysis further comprises identifying sentences including text markers and, wherein the sentences including text markers are subjected to filtering to remove sentences not likely to include <

term, definition>

pairs, and wherein the filtering includes rules selected from the group including sentences with conjunctions at the beginning of a text marker, sentences having phrases indicative of explanation, and sentences having patterns indicative of enumeration.