SELF-ORGANIZED CONCEPT SEARCH AND DATA STORAGE METHOD

• US 20060167930A1
• Filed: 01/13/2006
• Published: 07/27/2006
• Est. Priority Date: 10/08/2004
• Status: Abandoned Application

• Current Assignee: George Witwer, Ravi Kumar Kondadadi
• Original Assignee: George Witwer, Ravi Kumar Kondadadi
• Inventors: Witwer, George, Kondadadi, Ravi Kumar
• Application Number:
  US11/275,554
• Publication Number:
  US 20060167930A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 18/23211 with adaptive number of clusters

Abstract

A document search and retrieval system and method stores documents in groups based on content. The documents are self-organized into a hierarchy of conceptual clusters, and branches of the hierarchy are stored separately in distinct physical stores, each having an index. In response to a query, the system finds the concepts (clusters) that best match the search criteria and returns the documents from those content categories. The indexing, clustering, and searching are performed using document themes and/or summaries. Themes are automatically developed by stemming and scoring phrases from the sentences in each document, and clustering the sentences containing the highest-scoring stems. A set of phrases (themes) is taken from each cluster. Document summaries are taken from text segments for each cluster of sentences within a document, then strung together to create a summary.

Claims

1. A system for indexing and retrieving information regarding a plurality of documents, comprising:

a plurality of data stores, each having

an index and

a search engine for finding documents in the data store that meet one or more search criteria;

a plurality of document concepts, each associated with exactly one of the data stores;

a clustering engine that,

for each of the plurality of documents;

associates the document with one or more of the concepts; and

adds information about the document to the index of each data store with which the one or more concepts is associated; and

updates organization of the concepts according to one or more predetermined criteria.

2. The system of claim 1, wherein the programming instructions are further executable by the processor to:

accept a new document for adding to the data stores;

determine one or more concepts to which the new document relates;

adding the new document to the one or more concepts;

if one or more predetermined criteria are met, dividing at least one of the one or more concepts into a plurality of concepts, each being assigned to a data store.

3. The system of claim 1, wherein the programming instructions are further executable by the processor to:

receive a search signal;

search the indexes of each data store as a function of the search signal;

return a result signal as a result of the search.

4. The system of claim 3, wherein:

the search signal comprises keywords, and

the selecting is performed as a function of the presence of the keywords in each indexed document.

5. The system of claim 1, wherein the one or more search criteria include applying a threshold for a similarity value that quantifies similarity of an indexed document to one or more provided search terms.

6. The system of claim 1, wherein at least two of the plurality of data stores are physically within the same computer housing.

7. The system of claim 1, wherein at least two of the plurality of data stores are physically within different computer housings.

8. The system of claim 1, wherein the data stores are connected to the clustering engine via a computer network.

9. A method of self-organizing and storing a plurality of electronic documents in a plurality of physical storage partitions, including:

clustering a plurality of electronic documents so that each document is in at least one of a plurality of concept clusters, the plurality of concept clusters forming a hierarchy and including;

a first concept cluster and

a second concept cluster that is not a super-cluster of the first concept cluster;

for each concept cluster in the plurality of concept clusters, storing each document in the concept cluster in one of the one or more physical storage partitions;

wherein

all documents in the first concept cluster are stored in a first storage partition;

all documents in the second concept cluster are stored in a second storage partition; and

there is no document that is simultaneously

in the second concept cluster,

stored in the first storage partition, and

not in the first concept cluster.

10. The method of claim 9, further comprising:

receiving a new document;

determining a concept cluster in which the new document fits;

adding information about the document to the physical storage partition in which other documents of the fitting concept cluster is stored; and

if one or more predetermined criteria are met as to the fitting concept cluster, that concept cluster being stored in a particular physical storage partition;

splitting the fitting concept cluster into at least two concept clusters;

storing a one of the at least two concept clusters in the particular physical storage partition in which the fitting concept cluster was stored; and

storing a second of the at least two concept clusters in a different physical storage partition from the one in which the fitting concept cluster was stored.

11. The method of claim 9, further comprising:

automatically searching an index of each concept cluster based on a query signal, the query signal including request data, to identify one or more concept clusters that match the request data;

processing each document in the identified concept clusters.

12. The method of claim 9, further comprising independently indexing the documents stored in each physical storage partition.

13. A method of searching electronic documents, comprising:

receiving a query signal that includes one or more search terms;

responsively to receiving the query signal, searching a plurality of concept indexes, each providing an index to a plurality of electronic documents that relate to a common concept, including;

quantifying the relationship between the one or more search terms and each of the concept indexes as a similarity value; and

selecting the concept indexes having a similarity value indicating a relationship closer than a threshold; and

retrieving references to each of the electronic documents in each of the selected concept indexes.

14. The method of claim 13, wherein the retrieving step includes using the references to the electronic documents to retrieve the documents themselves.

15. The method of claim 14, wherein the retrieving step further includes providing the electronic documents in a response signal.

16. The method of claim 14, wherein the retrieving step further includes providing automatically generated summaries of the electronic documents in a response signal.

17. The method of claim 13, wherein the selecting is done as a function of the average of all similarity values from the quantifying step.

18. The method of claim 13, wherein the selecting includes up to a predetermined number of concept clusters that have the best similarity values.

19. The method of claim 13, wherein the selecting includes up to a predetermined number of concept clusters that have the best similarity values, but does not include any concept cluster that has a similarity value that indicates less than a threshold level of similarity.

20. A system for storing and retrieving electronic documents, including:

a search string layer that receives a search query;

one or more physical data stores; and

a concept index layer that includes a plurality of indexes, each index being associated with one of the physical data stores, and each index containing data that relates to a plurality of electronic documents;

wherein the system

quantifies the closeness of the conceptual relationship between each of the indexes and the search query;

based on the quantification, identifies one or more indexes that best match the search query;

identifies the documents indexed by the one or more identified indexes; and

provides a result signal as a function of the identified documents.

21. The system of claim 20, wherein the result signal includes a list of references to the identified documents.

22. The system of claim 21, wherein the list is sorted by similarity of the identified documents to the search query.

23. The system of claim 20, wherein the system also adds documents by:

determining one or more concepts in which a new document fits;

adding information about the new document to the index for each of the one or more concepts;

storing the new document in the physical data store with which the index for each of the one or more concepts is associated.

24. A system for generating a list of one or more themes from an electronic document, comprising a processor and a memory in communication with the processor, the memory being encoded with programming instructions executable by the processor to:

identify sentences in the document;

parse the sentences into tokens;

list all phrases in the document having no more than a predetermined number of tokens;

count the frequency of the phrases;

stem the phrases to a predetermined length;

score each stem as a function of the stem'"'"'s length and the frequency of the corresponding phrases in the document;

cluster the sentences based at least in part on the scores of the stems they contain; and

generate a phrase set containing phrases from those sentences that were clustered into a cluster with at least one other sentence.

25. The system of claim 24, wherein tokens are words.

26. The system of claim 24, wherein the counting for a document occurs simultaneously with the listing for that document.

27. The system of claim 24, wherein the stemming for a document occurs before the counting for that document.

28. The system of claim 24, wherein the stemming for a document occurs after the counting for that document.

29. The system of claim 24, wherein the scoring is also a function of the position of the stem.

30. The system of claim 24, wherein the programming instructions are further executable by the processor to:

determine the part of speech of a token; and

remove tokens from further processing if they are determined to be of one or more predetermined parts of speech.

31. The system of claim 24, wherein the programming instructions are further executable by the processor to remove from further processing any token that is on a predetermined list.

32. The system of claim 24, wherein the predetermined length for stemming is measured in number of characters.

33. A system for generating a summary of an electronic document, comprising a processor and a memory in communication with the processor, the memory being encoded with programming instructions executable by the processor to:

identify coherent segments of text in an electronic document, each sentence being part of at least one coherent segment;

cluster sentences in the document based on their content;

for each cluster of sentences, generate a passage by;

sorting the sentences in the cluster based on their position in the original document;

selecting a first number of sentences from the beginning of the sorted list; and

for each of the first number of sentences, adding to the passage the smallest coherent segment of which the sentence is a part.

34. The system of claim 33, wherein the clustering is performed as a function of one or more themes for each sentence.

35. The system of claim 33, wherein the programming instructions are further executable by the processor to present each passage as a paragraph of human-readable text.

36. The system of claim 33, wherein the first number of sentences is two.