Clustered Information Processing and Searching with Structured-Unstructured Database Bridge

• US 20130268532A1
• Filed: 04/05/2013
• Published: 10/10/2013
• Est. Priority Date: 04/09/2012
• Status: Active Grant

• Current Assignee: Vivek Ventures, LLC
• Original Assignee: Vivek Ventures, LLC
• Inventors: Doshi, Sundeep
• Granted Patent: US 9,092,504 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/3331 Query processing
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods for indexing information and for performing searches are disclosed. In these systems and methods information is “ingested” into the system by clustering the information using a clustering algorithm such as k-means or k-medoids clustering. During the clustering process, a hybrid distance measurement is used that allows the systems and methods to determine similarity across a number of different types of information. Once the information is clustered, it is stored and “mirrored” both in a structured (e.g., relational) data repository and in an unstructured data repository. Methods according to the invention allow the retrieval of both direct search results and search results including related concepts. After clustered information is stored, future searches can be performed by searching the stored results in whichever data repository is most appropriate for the context.

Claims

1. A method for indexing and classifying related information, comprising:

using a computing system, measuring the similarity of or distance between a plurality of individual resources using a hybrid distance measurement;

clustering the plurality of individual resources into a plurality of clusters using the hybrid distance measurement; and

storing the plurality of clusters in both a structured and an unstructured data repository on the computing system or another computing system.

2. The method of claim 1, wherein the plurality of individual resources are of different types.

3. The method of claim 1, wherein said clustering comprises k-means or k-medoids clustering.

4. The method of claim 1, wherein said clustering comprises hierarchical clustering.

5. The method of claim 1, wherein the structured data repository comprises a relational database.

6. The method of claim 1, wherein at least some of the plurality of individual resources are of different types.

7. The method of claim 1, further comprising:

receiving a set of search terms at a search server via a communication network; and

searching a defined set of locations for the set of search terms to locate the plurality of individual resources.

8. The method of claim 1, wherein the hybrid distance measurement comprises a linearly weighted composite distance measurement based, at least in part, on cosine similarity.

9. A method for retrieving data, comprising:

receiving a query at a first computing system;

automatically directing the query to either a structured results repository or an unstructured results repository, the structured and unstructured results repositories being associated with the first computing system or other computing systems and containing essentially the same information clustered into one or more clusters to reflect relationships between data elements in the one or more clusters;

searching the structured or the unstructured results repositories in accordance with said automatically directing; and

returning a result set to the query including at least a portion of the contents of at least one of the clusters.

10. The method of claim 9, wherein said automatically directing the query comprises determining whether searching the structured results repository or searching the unstructured results repository would be faster in view of the query and a measure of available resources.

11. The method of claim 9, wherein the structured results repository comprises a relational database.

12. The method of claim 11, wherein the unstructured results repository comprises an unstructured database.

13. The method of claim 9, further comprising re-clustering the data elements in the one or more clusters in both the structured and the unstructured results repositories to include the query.

14. A system for information storage and retrieval, comprising:

a search and information retrieval computing system, the search and retrieval computing system having associated with it an unstructured results repository and a structured results repository, the unstructured and structured results repositories essentially the same information clustered into one or more clusters to reflect relationships between data elements in the one or more clusters;

an information clustering and storage routine running on the search and information retrieval computing system that processes new data received at the search and information retrieval computing system, the information clustering and storage routine determining a hybrid distance measurement for the new data, clustering the new data into one or more clusters based on the hybrid distance measurement, and storing the one or more clusters in the structured and unstructured results repositories; and

an information retrieval routine running on the searching and information retrieval computing system that processes a query received at the search and information retrieval computing system, determine whether the query is best searched using the structured or the unstructured results repository, execute a search of either the structured or the unstructured results repository based on the determination, and return results including at least a portion of the contents of at least one of the clusters.

15. The system of claim 14, wherein the search and information retrieval computing system is connected to a communications network.

16. The system of claim 14, wherein the structured results repository comprises a relational database.

17. The system of claim 14, wherein the unstructured results repository comprises an unstructured database.

18. A method for indexing, searching, and retrieving information, comprising:

using a first computing system, measuring the similarity of or distance between a plurality of individual resources using a hybrid distance measurement;

clustering the plurality of individual resources into a plurality of clusters using the hybrid distance measurement;

storing the plurality of clusters in both a structured and an unstructured data repository on the computing system or another computing system;

receiving a query at the first computing system or a second computing system in communication with a first computing system;

automatically directing the query to either the structured repository or an unstructured repository;

searching the structured or the unstructured repositories in accordance with said automatically directing; and

returning a result set to the query including at least a portion of the contents of at least one of the clusters.

19. The method of claim 18, wherein the structured repository comprises a relational database.

20. The method of claim 18, wherein the unstructured repository comprises an unstructured database.

21. The method of claim 18, wherein said automatically directing the query comprises determining whether searching the structured repository or searching the unstructured repository would be faster in view of the query and a measure of available resources.

22. The method of claim 18, wherein the hybrid distance measurement comprises a linearly weighted composite distance measurement based, at least in part, on cosine similarity.