Knowledge catalysts

• US 10,127,292 B2
• Filed: 11/25/2013
• Issued: 11/13/2018
• Est. Priority Date: 12/03/2012
• Status: Active Grant

• Current Assignee: UT-Battelle LLC
• Original Assignee: UT-Battelle LLC
• Inventors: Sukumar, Sreenivas R., Ferrell, Regina K., Shankar, Mallikarjun
• Primary Examiner(s): Chbouki, Tarek
• Assistant Examiner(s): Cheung, Eddy
• Application Number:
  US14/089,395
• Publication Number:
  US 20140156591A1
• Time in Patent Office: 1,814 Days
• Field of Search: 707602
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A computer implemented method integrates data from remote disparate data sources by processing a non-transitory media. The non-transitory media stores instructions for detecting data sets in different formats hosted in a plurality of heterogeneous databases that are accessible through a distributed network. The method extracts schema data from the plurality of heterogeneous databases and identifies related fields in two or more of the heterogeneous databases. The method links the related fields in the two or more of the plurality of heterogeneous databases and makes the data accessible through a virtual warehouse. As schemas change, as new data sources and analysis artifacts are created, the computer implemented method and system can act as a meta-data store, a provenance tracking device, and/or a knowledge management service.

Claims

1. A computer implemented method of integrating data from remote disparate data sources comprising a non-transitory media, comprising programming for:

detecting data sets in different formats having a plurality of fields hosted in a plurality of remote heterogeneous databases accessible through infrastructures that are coupled through a distributed network;

extracting schema data of the plurality of remote heterogeneous databases;

modeling each position of selected plurality of fields of the plurality of remote heterogeneous databases as a plurality of polynomials,

identifying related fields in two or more of the plurality of remote heterogeneous databases by automatically hypothesizing data links based on column features that identify the number of distinct data items in each column in the plurality of remote heterogeneous databases and fuzzy logic matching that compares divergence of the plurality of polynomials; and

linking the related fields automatically in the two or more of the plurality of remote heterogeneous databases through a virtual warehouse.

2. The computer implemented method of claim 1 further comprising displaying visual representations of relationship links of the related fields.

3. The computer implemented method of claim 1 further comprising capturing and publishing collaborative discussions about data associated with the related fields.

4. The computer implemented method of claim 1 further comprising scraping the schema data from the plurality of remote heterogeneous databases and generating a metadata repository.

5. The computer implemented method of claim 1 where the identification of the related fields occurs through a fuzzy logic.

6. The computer implemented method of claim 1 further comprising displaying visual representations of relationship links of the related fields and a user interface that allows a user to confirm or reject one or more of the relationship links.

7. The computer implemented method of claim 1 further comprising automatically identifying a plurality of crosswalk keys for identifying common fields in the plurality of remote heterogeneous databases.

8. The computer implemented method of claim 1 further comprising scraping the schema data from the plurality of remote heterogeneous databases and generating a metadata repository and a data level matching.

9. The computer implemented method of claim 1 further comprising generating profiles and histogram distributions of a random sample of data sampled from the plurality of remote heterogeneous databases before identifying the related fields of the databases.

10. The computer implemented method of claim 1 further comprising displaying visual representations of the relationship links of the related fields on a touch sensitive display programmed to accept the linking of the related fields if a user contact is detected that corresponds to a predefined gesture.

11. The computer implemented method of claim 1 further comprising displaying visual representations of the relationship links of the related fields on a touch sensitive display programmed to reject the linking of the related fields if a user contact is detected that corresponds to a predefined gesture.

12. A unified data integration system, comprising:

one or more processors;

a memory; and

one or more programs, where the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for automatically;

detecting data sets hosted in a plurality of heterogeneous databases that are coupled through a distributed network;

extracting schema and relationship data from the plurality heterogeneous databases;

generating polynomial models and histogram distributions of a random sample of data sampled from the plurality of heterogeneous databases;

generating schema-level hypotheses that makes hypothetical content connections between previously unknown data sources accessed from at least two of the plurality heterogeneous databases to build a virtual schema based on the hypothetical content connections, an identified number of distinct data values in the plurality heterogeneous databases and fuzzy logic string matching that compares divergence between the polynomial models;

automatically generating schema crosswalk keys that identify equivalent elements; and

validating at least some of the schema-level hypotheses automatically.

13. The unified data integration system of claim 12 further comprising displaying visual representations of the schema-level hypotheses.

14. The unified data integration system of claim 12 further comprising capturing and publishing collaborative information about the data associated with the related fields.

15. The unified data integration system of claim 12 further comprising instructions for scraping the schema data from the plurality of heterogeneous databases and generating a metadata repository.

16. The unified data integration system of claim 12 where the generating schema-level hypotheses occurs through instructions that execute a fuzzy logic matching process.

17. The unified data integration system of claim 12 further comprising displaying visual representations of the schema-level hypotheses and a gesture control user interface that allows a user to confirm or reject one or more of the schema-level hypotheses in response to a gesture.

18. A method for analyzing a large data set from disparate sources using a computer, the method comprising:

building a virtual schema of the data set in a memory of the computer automatically;

modeling selected fields of a plurality of remote heterogeneous databases as polynomials;

linking data elements automatically by hypothesizing links based between data elements based on naming conventions of remote data elements, identifying the number of remote distinct data elements, and fuzzy logic means that compares divergence between the polynomials; and

displaying the data set visually on a monitor of the computer as an interconnected plurality of local databases and tables automatically.

19. The method of claim 18 and further comprising analyzing the displayed data set to find relationships across databases and tables.

20. The method of claim 18 further comprising generating profiles and histograms distributions of a random sample of data sampled from a plurality of heterogeneous databases.