System and method for interpreting document contents

• US 6,772,170 B2
• Filed: 11/16/2002
• Issued: 08/03/2004
• Est. Priority Date: 09/13/1996
• Status: Expired due to Fees

• Current Assignee: Battelle Memorial Institute
• Original Assignee: Battelle Memorial Institute
• Inventors: Pennock, Kelly A., Miller, Nancy E.
• Primary Examiner(s): Pardo, Thuy N.
• Application Number:
  US10/298,361
• Publication Number:
  US 20030097375A1
• Time in Patent Office: 626 Days
• Field of Search: 707/6, 707/5, 707/3, 707/10, 707/101, 707/102, 715/531, 715/500, 715/542, 715/532
• US Class Current: 1/1
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 16/3332 Query translation
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99937 Sorting
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A sequence of word filters are used to eliminate terms in the database which do not discriminate document content, resulting in a filtered word set and a topic word set whose members are highly predictive of content. These two word sets are then formed into a two dimensional matrix with matrix entries calculated as the conditional probability that a document will contain a word in a row given that it contains the word in a column. The matrix representation allows the resultant vectors to be utilized to interpret document contents.

Claims

1. A method, comprising the steps of:

a) semantically filtering a set of documents in a database to extract a set of semantic concepts, to improve an efficiency of a predictive relationship to its content, based on at least one of word frequency, overlap and topicality;

b) defining a topic set, said topic set being characterized as the set of semantic concepts which best discriminate the content of the documents containing them, said topic set being defined based on at least one of word frequency, overlap and topicality;

c) forming a matrix with the semantic concepts contained within the topic set defining one dimension of said matrix and the semantic concepts contained within the filtered set of documents comprising another dimension of said matrix;

d) calculating matrix entries as the conditional probability that a document in the database will contain each semantic concept in the topic set given that it contains each semantic concept in the filtered set of documents; and

e) providing said matrix entries as vectors to interpret the document contents of said database.

2. The method of claims 1, wherein said semantic concepts comprise words.

3. The method of claim 1, wherein said semantic concepts comprise word sets.

4. The method of claim 1, wherein said filtering comprises determining correlations of semantic concepts.

5. The method of claim 1, wherein said semantically filtering results in a set of documents represented by approximately 10% of an original number of words.

6. The method of claim 1, further comprising the step of ranking the filtered semantic concepts according to predetermined criteria.

7. The method of claim 6, further comprising the step of truncating the ranked filtered semantic concepts according to a cut off, to form the topic set.

8. The method of claim 6, wherein the predetermined criteria of said ranking substep comprises weighting words according to their inherent indicia of content.

9. The method of claim 1, wherein said defining step further comprises the substep of ranking the extracted semantic concepts according to their frequency in the database.

10. The method of claim 1, wherein said defining further comprises the substep of ranking the extracted semantic concepts according to their rank in the topicality filter.

11. The method of claim 1, wherein said defining step further comprises the substep of ranking the extracted semantic concepts, and truncating semantic concepts having a relatively lower rank.

12. A computer readable software medium containing computer software for controlling a general purpose computer to perform the method of claim 1.

13. A computer readable software medium containing computer software for controlling a general purpose computer to perform the method of claim 1.

14. A system, comprising:

a) a filter for semantically filtering a set of documents in a database to extract a set of semantic concepts, to improve an efficiency of a predictive relationship to its content;

b) means for defining a topic set, said topic set being characterized as the set of semantic concepts which best discriminate the content of the documents containing them, said topic set being defined based on at least one of word frequency, overlap and topicality;

c) a matrix, with the semantic concepts contained within the topic set defining one dimension of said matrix and the semantic concepts contained within the filtered set of documents comprising another dimension of said matrix, wherein matrix entries are calculated as the conditional probability that a document in the database will contain each semantic concept in the topic set given that it contains each semantic concept in the filtered set of documents; and

e) means for interpreting the document contents of said database based on vectors derived from said matrix.

15. The system of claim 14, wherein said filter comprises a correlator for filtering semantic concepts, said filtered semantic concepts being topically ranked and truncated to form said topic set.