Word importance calculation method, document retrieving interface, word dictionary making method

• US 6,850,937 B1
• Filed: 08/22/2000
• Issued: 02/01/2005
• Est. Priority Date: 08/25/1999
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Niwa, Yoshiki, Hisamitsu, Toru
• Primary Examiner(s): Pardo, Thuy N.
• Application Number:
  US09/642,771
• Time in Patent Office: 1,624 Days
• Field of Search: 707/6, 707/7, 707/101, 707/102, 707/104, 707/3, 707/5, 707/2, 704/2
• US Class Current: 707/750
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: Y10S 707/917 Text
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99937 Sorting
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A known method for selecting words (or word sequences), which is an important aspect of information retrieval, involves the problems of inability to eliminate high-frequency common words and of often arbitrary setting of the threshold value for dividing important and unimportant words. These problems are solved by normalizing the difference between the word distribution in a subset of all documents containing a word to be extracted (or a subset of said document set) and the word distribution in the set of all documents with the number of words in the said subset of all documents containing the word as a parameter, and the accuracy of support information retrieval is thereby enhanced.

Claims

1. A word importance calculation method for calculating the importance of words contained in a document set, whereby the difference between the word distribution in a subset of every document containing a specified word and the word distribution in a set of whole documents including said subset is used to calculate the importance of the word.

2. A word importance calculation method, as claimed in claim 1, wherein:

said difference is determined by comparing the distance d between said subset and said set of whole documents with the distance d′

, or the estimated value of d′

, between another subset of documents which contain substantially the same number of words as said subset of documents and are randomly selected from said subset of whole documents, and said set of whole documents.

3. A word importance calculation method, as claimed in claim 2, wherein:

the distance d between the two document sets is calculated by using the probability of occurrence of each word in each of said document set.

4. A word importance calculation method, as claimed in claim 2, wherein:

in case that the number of documents containing said word is larger than a prescribed number, a preset number of documents are extracted from the said subset of whole documents by random sampling, and the difference between the extracted set of documents and said set of whole documents is used instead of the difference between the original subset of documents and the set of whole documents.

5. A word importance calculation method according to claim 1, said difference being defined by $\sum _{}^{}$

i=in⁢

ki⁢

log⁢

 

⁢

ki#⁢

D⁡

(W)-∑

k=in⁢

ki⁢

log⁢

 

⁢

Ki#⁢

D0

D(W) being said subset,

D0 being the set of the whole document,

k_i being a frequency of an occurrence of the specified word in D(W),

k_i being a frequency of an occurrence of the specified word in D(0).

6. A document retrieval interface having a function to display on a screen words characterizing a document set, wherein the importance of each word occurring in a set of whole documents is calculated using the difference between the word distribution in the subset of every document containing the word and the word distribution in the set of whole documents including said subset, and the importance is brought to bear on the selection, arrangement or coloring of the words displayed on the screen.

7. A document retrieval interface having a function to display on a screen words characterizing a document set, wherein the importance of each word occurring in the document set obtained as a result of retrieval is calculated using the difference between the word distribution in the subset of documents out of the document set obtained as a result of that retrieval containing that word and the word distribution in the document set obtained as a result of that retrieval, and the importance is brought to bear on the selection, arrangement or coloring of the words displayed on the screen.

8. A word dictionary construction method by extracting important words from a document set in accordance with rules given in advance, wherein the importance of each word occurring in a set of whole documents is calculated using the difference between a subset of every document containing the word and the word distribution in the set of whole documents including said subset, and words to be extracted are selected on the basis of that importance.