Systems and methods for generating schemas that represent multiple data sources

• US 9,811,543 B2
• Filed: 03/24/2015
• Issued: 11/07/2017
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Saama Technologies LLC (Saama Technologies, Inc.)
• Original Assignee: Comprehend Systems, Inc. (Saama Technologies, Inc.)
• Inventors: Morrison, Rick, Gardner, Jud, Saffer, Matthew
• Primary Examiner(s): GORTAYO, DANGELINO N
• Application Number:
  US14/667,272
• Publication Number:
  US 20150199389A1
• Time in Patent Office: 959 Days
• Field of Search: 707769, 707756, 707760, 707803, 707804, 707808, 707809
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/2468 Fuzzy queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 16/835 Query processing
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods generating schemas that represent multiple data sources are provided herein. According to some embodiments, methods may include determining a schema for each of the multiple data sources via a computing device communicatively couplable with each of the multiple data sources, each of the multiple data sources including one or more data structures that define how data is stored in the data source, generating a negotiated schema by comparing the schemas of the multiple data sources to one another and interrelating data points of the multiple data sources based upon the schemas, interrelating the negotiated schema with the schema for each of the multiple data sources based upon the interrelation of the data points, and storing the negotiated schema in a storage media by way of the computing device.

Claims

1. A method for generating a negotiated schema for multiple data sources, the method comprising:

interrogating multiple data sources via a computing device communicatively couplable with each of the multiple data sources, each of the multiple data sources including different, unknown types of data structures that define how data is stored in each data source, the interrogation of at least one of the data sources including;

comparing the unknown data structure of the data source to a plurality of schema templates,

determining a fuzzy template match between the unknown data structure and one or more of the plurality of schema templates based on the comparison, and

selecting a schema representing the unknown data structure that is a fuzzy template match between the unknown data structure and a schema template of one or more of the plurality of schema templates;

generating a negotiated schema by;

comparing the selected schemas of the multiple data sources to one another, the selected schemas including at least one schema representing the unknown data structure that is the fuzzy template match; and

interrelating data points of the multiple data sources that correspond to one another;

interrelating the negotiated schema with the selected schema for each of the multiple data sources based upon the interrelation of the data points; and

storing the negotiated schema in a storage media by way of the computing device.

2. The method according to claim 1, wherein the interrogating of the multiple data sources includes:

comparing the different, unknown types of data structures to a plurality of schema templates; and

selecting one of the plurality of schema templates that at least partially matches the different, unknown types of data structures.

3. The method according to claim 2, wherein the comparing includes applying heuristic logic to the comparison, the heuristic logic being based upon at least one of end-user input, machine learning techniques, metadata tags, previous schema negotiations, predefined custom connectors, heuristics, and combinations thereof.

4. The method according to claim 1, wherein the interrelating data points of the multiple data sources includes establishing at least one of a fuzzy or a concrete relationship between data points.

5. The method according to claim 1, wherein the interrelating data points of the multiple data sources includes associating related data points with metadata that describe an interrelationship between the data points.

6. The method according to claim 5, wherein the metadata may include any of a data attribute, a schema information for each data source, and a confidence level for interrelated sets of data points.

7. The method according to claim 1, further comprising receiving verification from an end user that an interrelationship between data points is correct.

8. The method according to claim 1, further comprising:

receiving a query for data against the negotiated schema from a client device;

fulfilling the query by;

obtaining data points from one or more of the multiple data sources by way of the negotiated schema; and

combining the data points together according to the interrelationships included in the negotiated schema to create a query response; and

providing the query response to the client device.

9. The method according to claim 8, automatically updating the negotiated schema when the different, unknown types of data structures of one or more of the data sources change.

10. The method according to claim 8, further comprising at least one of:

selecting one or more alternative data sources when one or more required data sources are unavailable; and

marking metadata in a returning result set appropriately if no alternative data source is available.

11. A system for generating a negotiated schema for multiple data sources, the system comprising:

a memory for storing executable instructions; and

a processor for executing the instructions, the instructions including;

an interrogation module that interrogates multiple data sources, each of the multiple data sources including different, unknown types of data structures that define how data is stored in each data source, the interrogation of at least one of the data sources including;

comparing the unknown data structure of the data source to a plurality of schema templates,

determining a fuzzy template match between the unknown data structure and each of the plurality of schema templates based on the comparison, and

selecting a schema representing the unknown data structure that is a fuzzy template match between the unknown data structure and a schema template of one or more of the plurality of schema templates; and

a schema generator that generates a negotiated schema by;

applying heuristic logic to determine correspondence between data points of the schemas of the multiple data sources to at least one of one another and one or more pre-existing schemas;

interrelating data points of the multiple data sources that correspond to one another;

interrelating the negotiated schema with selected schema for each of the multiple data sources based upon the interrelation of the data points; and

storing the negotiated schema in a storage media.

12. The system according to claim 11, wherein the interrogation module further:

compares the different, unknown types of data structures to a plurality of schema templates; and

selects one of the plurality of schema templates that at least partially matches the different, unknown types of data structures.

13. The system according to claim 12, wherein the interrogation module compares the different, unknown types of data structures to a plurality of schema templates by applying heuristic logic, the heuristic logic being based upon at least one of end-user input, machine learning techniques, metadata tags, previous schema negotiations, custom predefined connectors, heuristics, and combinations thereof.

14. The system according to claim 11, wherein the interrelating data points of the multiple data sources includes the schema generator determining at least one of a fuzzy and a concrete relationship between data points of two or more data sources that at least partially correspond to one another.

15. The system according to claim 14, wherein the schema generator is configured to provide metadata that describes an interrelationship between the data points.

16. The system according to claim 15, wherein the metadata may include any of data type, schema determination information for each data source, and a confidence level for interrelated sets of data points.

17. The system according to claim 11, wherein the schema generator receives verification from an end user that an interrelationship between data points is correct.

18. The system according to claim 11, wherein the interrogation module is further configured to:

receive a query for data against the negotiated schema from a client device;

fulfill the query by;

obtaining data points from one or more of the multiple data sources by way of the negotiated schema; and

combining the data points together according to the interrelationships included in the negotiated schema to create a query response; and

provide the query response to the client device.

19. The system according to claim 18, wherein the schema generator obtains metadata associated with data points and combines the metadata with the data points before the interrogation module provides the query response to the client device.

20. A non-transitory computer readable storage media having a program embodied thereon, the program being executable by a processor to perform a method for generating a negotiated schema for multiple data sources, the method comprising:

interrogating multiple data sources via a computing device communicatively couplable with each of the multiple data sources, each of the multiple data sources including one or more different, unknown types of data structures that define how data is stored in each data source, the interrogation of each of the data sources including;

comparing the unknown data structure of the data source to a plurality of schema templates,

identifying a plurality of schema templates that partially correspond to the unknown data structure of the data source, and

using heuristic logic to select a schema for representing the unknown data structure, the selected schema being a closest partial match between the unknown data structure and the plurality of identified schema templates;

generating a negotiated schema by;

establishing linkages between multiple data sources using an adaptive algorithm; and

interrelating data points of the multiple data sources that correspond to one another;

interrelating the negotiated schema with the selected schema for each of the multiple data sources based upon the interrelation of the data points; and

storing the negotiated schema in a storage media by way of the computing device.