Natural language information retrieval system and method

• US 6,081,774 A
• Filed: 08/22/1997
• Issued: 06/27/2000
• Est. Priority Date: 08/22/1997
• Status: Expired due to Term

• Current Assignee: RPX Corporation
• Original Assignee: Novell Incorporated (Open Text Corporation)
• Inventors: de Bie, Peter, Macpherson, Melissa, Deun, Kurt Van, Platteau, Frank M. J., Govaers, Erik C. E., Akker, David van den, de Hita, Carolina Rubio, Laviolette, Sophie
• Primary Examiner(s): Isen, Forester W.
• Assistant Examiner(s): EDOUARD, PATRICK NESTOR
• Application Number:
  US08/916,628
• Time in Patent Office: 1,040 Days
• Field of Search: 704/1, 704/9, 704/10, 704/530, 704/531, 707/3, 707/10
• US Class Current: 704/9
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

An information retrieval system that represents the content of a language-based database being searched as well as the user'"'"'s natural language query. In accordance with one aspect of the invention, the information retrieval system includes a non-real-time development system for automatically creating a database index having one or more content-based database keywords of the data base; and a real-time retrieval system that, in response to a user'"'"'s natural language query, searches the keyword index for one or more content-based query keywords derived from the natural language query. The development system and the retrieval system morphologically, syntactically and linguistically analyze the data base and the natural language query, respectively, to generate the one or more database keywords and query keywords representing the content of the database and the natural language query, respectively. The development system includes a software development system for creating the database index utilizing a pattern dictionary that includes synonyms and skip words and a morphosyntactic dictionary that includes morphological and syntactic information for words in the natural language of the language-based database and the natural language query. In one embodiment, the retrieval system includes a natural language interface system for creating the one or more query keywords utilizing the pattern dictionary and the morphosyntactic dictionary. In one embodiment, the retrieval system also includes a query-index matcher for matching the one or more query keywords with the one or more database keywords.

Claims

1. An information retrieval system using natural language queries to retrieve information from a language-based database containing one or more files, comprising:

a non-real-time development system for automatically creating a database index having one or more content-based database keywords of the database; and

a real-time retrieval system that, in response to a user'"'"'s natural language queries, searches said database index for one or more content-based query keywords derived from a natural language query,

wherein said development system and said retrieval system morphologically, syntactically and linguistically analyze said language-based database and said natural language query, respectively, to generate said one or more database keywords and query keywords representing the content of said language-based database and said natural language query, respectively.

2. The information retrieval system of claim 1, wherein said development system comprises:

a software development system for creating said database index utilizing a pattern dictionary that includes synonyms and skip words and a morphosyntactic dictionary that includes morphological and syntactic information for words in the natural language of the language-based database and natural language query.

3. The information retrieval system of claim 2, wherein said retrieval system comprises:

a natural language interface system for creating said one or more query keywords utilizing said pattern dictionary and said morphosyntactic dictionary; and

a query-index matcher for matching said one or more query keywords with said one or more database keywords.

4. The information retrieval system of claim 3, wherein said software development system is configured as a software developer'"'"'s kit (SDK).

5. The information retrieval system of claim 3, wherein said pattern dictionary and said morphosyntactic dictionary are customized to more accurately reflect linguistic features of said language-based database and said natural language query.

6. The information retrieval system of claim 3, wherein said software development system comprises:

an index builder that generates said database index; and

an expression list editor that enables a software developer to modify said pattern dictionary to generate a merged pattern dictionary.

7. The information retrieval system of claim 6, wherein said index builder comprises:

a keyword builder for identifying topics most useful in identifying the contents of each of the one or more data base files; and

a linguistic image builder for identifying and representing linguistic relationships most useful in identifying the contents of each sentence of each of the one or more data base files.

8. The information retrieval system of claim 7, wherein said keyword builder comprises:

a linguistic array generator for sequentially processing each sentence unit in each database file to generate a temporary linguistic array that contains tokens associated with selected words and contains relevant morphosyntactic and linguistic attributes of said associated word; and

a topic tree builder that builds a topic tree of linked data structures, including,

topic records for each of said tokens, each said topic record containing roots of said words associated with said tokens and token attributes identifying morphological, syntactical and semantic characteristics of and relations between said associated words,

occurrence records containing location of said associated words and original form of each of said associated words, and

synonym records containing synonyms of said associated words.

9. The information retrieval system of claim 8, wherein said keyword builder further comprises:

a topic prioritizer that prioritizes the normalized topics in said topic tree in accordance with the frequency of their occurrence and other indicators of their importance inferred from said token attributes; and

a keyword processor for selecting topic records based on their assigned priority, and for normalizing said topics contained in said selected topic records, said topics inserted into said keyword list as said content-based data base keywords.

10. The information retrieval system of claim 9, wherein said linguistic array generator comprises:

a token attribute generator for identifying and morphologically and syntactically characterizing said tokens and said sentence units in the said database files and for populating said a linguistic array with identified attributes;

a syntactic tagger for supplementing said linguistic array with morphological, syntactic and linguistic data identifying a relative importance of each token in said array; and

a parse filter for filtering tokens from said array that are likely not to assist in matching said query keywords with said database keywords.

11. The information retrieval system of claim 10, wherein said token attribute generator comprises:

a token identifier for identifying tokens in said data base help files;

a sentence unit identifier for demarcating sentences in said data base files;

a format identifier for identifying a category of text in which said token appears.

12. The information retrieval system of claim 11, wherein said token attribute generator further comprises:

a position identifier for identifying the position of the token in said data base files;

a layout identifier for identifying a location of said sentence unit in said data base files;

a morphosyntactic identifier for populating said array with all morphosyntactic information from said morphosyntactic dictionary for each said token; and

a special token identifier for identifying said tokens related to particular categories of information.

13. The information retrieval system of claim 11, wherein said syntactic tagger comprises:

a morphological identifier for identifying a preferred morphological form of each word associated with each of said tokens; and

a syntactic unit identifier for identifying groups of tokens in syntactic units and identifying dependencies among said tokens within each syntactic unit.

14. The information retrieval system of claim 13, wherein said syntactic tagger further comprises:

a grammatical identifier that identifies the grammatical function of the tokens; and

a synonym identifier that identifies a label representing a group of synonyms in the merged pattern dictionary.

15. The information retrieval system of claim 14, wherein said parse filter comprises:

a syntactic filter for syntactically filtering tokens from said array that are likely not to assist in matching query keywords and database keywords; and

a lexical filter that eliminates tokens associated with predetermined words that are likely not to assist in matching topics of a natural language query to topics of a database text file.

16. The information retrieval system of claim 11, wherein said topic prioritizer comprises:

a token weight calculator for prioritizing the topics in the topic tree in accordance with the frequency of their occurrence and other indicators of their importance inferred from their token attributes; and

a token normalizer that determines the most appropriate morphosyntactic form for the topic based upon the form that occurred most frequently for that topic.