Method and product for determining salient features for use in information searching

• US 5,924,105 A
• Filed: 01/26/1998
• Issued: 07/13/1999
• Est. Priority Date: 01/27/1997
• Status: Expired due to Term

• Current Assignee: Michigan State University
• Original Assignee: Michigan State University
• Inventors: Goodman, Erik D., Wulfekuhler, Marilyn R., Punch, William F. III
• Primary Examiner(s): Kulik, Paul V.
• Assistant Examiner(s): HOMERE, JEAN RAYMOND
• Application Number:
  US09/013,453
• Time in Patent Office: 533 Days
• Field of Search: 707/5, 707/3, 707/2, 707/513, 707/534, 704/9, 704/7
• US Class Current: 715/234
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A method and product are provided for generating a word set for use in locating a document having a type similar to a type of document in a document collection. The method includes selecting multiple documents from the document collection, each document selected including multiple words, and stemming the words in each document selected to obtain multiple stem words. The method also includes determining a word count for each stem word in each document, and clustering the stem words based on the word count of each stem word in each document to obtain a word set. The product includes a storage medium having programmed instructions recorded thereon for performing the method steps.

Claims

1. A computer method for generating a word set for use in locating a document having a type similar to a type of document in a document collection, the method comprising:

selecting a plurality of documents from the document collection, each document selected including a plurality of words;

stemming the plurality of words in each document selected to obtain a plurality of stem words;

determining a word count for each stem word in each document; and

clustering the plurality of stem words based on the word count of each stem word in each document to obtain a word set.

2. The method of claim 1 further comprising removing stop words from the plurality of words in each document.

3. The method of claim 1 further comprising:

constructing a query from the word set;

providing the query to a search engine; and

activating the search engine to locate a document having a type similar to the type of document in the document collection.

4. The method of claim 3 wherein the word set includes a plurality of words and constructing a query comprises creating a conjunction of the plurality of words in the word set.

5. The method of claim 1 wherein stemming comprises reducing each of the plurality of words in each document to a minimum string of contiguous letters.

6. The method of claim 2 wherein the stop words include "a", "and", "of", "it" and "the".

7. The method of claim 1 wherein clustering comprises performing a K-means partitioning clustering algorithm.

8. The method of claim 1 further comprising normalizing the word count of each stem word in each document selected.

9. The method of claim 1 wherein selecting a plurality of documents from the document collection includes parsing a list.

10. A product for generating a word set for use in locating a document having a type similar to a type of document in a document collection, the product comprising a storage medium having programmed instructions recorded thereon, the programmed instructions operative to select a plurality of documents from the document collection, each document selected including a plurality of words, stem the plurality of words in each document selected to obtain a plurality of stem words, determine a word count for each stem word in each document, and cluster the plurality of stem words based on the word count of each stem word in each document to obtain a word set.

11. The product of claim 10 wherein the programmed instructions are further operative to remove stop words from the plurality of words in each document.

12. The product of claim 10 wherein the programmed instructions are further operative to construct a query from the word set, provide the query to a search engine, and activate the search engine to locate a document having a type similar to the type of document in the document collection.

13. The product of claim 12 wherein the word set includes a plurality of words and, to construct a query, the programmed instructions are operative to create a conjunction of the plurality of words in the word set.

14. The product of claim 10 wherein, to stem the plurality of words in each document selected, the programmed instructions are operative to reduce each of the plurality of words in each document to a minimum string of contiguous letters.

15. The product of claim 11 wherein the stop words include "a", "and", "of", "it" and "the".

16. The product of claim 10 wherein, to cluster the plurality of stem words, the programmed instructions are operative to perform a K-means partitioning clustering algorithm.

17. The product of claim 10 wherein the programmed instructions are further operative to normalize the word count of each stem word in each document selected.

18. The product of claim 10 wherein, to select a plurality of documents from the document collection, the programmed instructions are operative to parse a list.

19. A computer method for generating a feature set for use in locating a pattern having a type similar to a type of pattern in a pattern space, the method comprising:

selecting a plurality of patterns from the pattern space, each pattern selected including a plurality of features;

minimizing the plurality of features in each pattern selected to obtain a plurality of core features;

determining a count for each core feature in each pattern; and

clustering the plurality of core features based on the count of each core feature in each pattern to obtain a feature set.

20. The method of claim 19 further comprising:

constructing a query from the feature set;

providing the query to a search engine; and

activating the search engine to locate a pattern having a type similar to the type of pattern in the pattern set.

21. The method of claim 20 wherein the feature set includes a plurality of features and constructing a query comprises creating a conjunction of the plurality of features in the feature set.

22. The method of claim 19 wherein clustering comprises performing a K-means partitioning clustering algorithm.

23. The method of claim 19 further comprising normalizing the count of each core feature in each pattern selected.