Systems and methods for generating schemas that represent multiple data sources

• US 9,020,981 B2
• Filed: 09/30/2011
• Issued: 04/28/2015
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Saama Technologies LLC (Saama Technologies, Inc.)
• Original Assignee: Comprehend Systems, Inc. (Saama Technologies, Inc.)
• Inventors: Morrison, Rick, Gardner, Jud, Saffer, Matthew
• Primary Examiner(s): Dwivedi, Mahesh
• Assistant Examiner(s): GORTAYO, DANGELINO N
• Application Number:
  US13/251,149
• Publication Number:
  US 20130086104A1
• Time in Patent Office: 1,306 Days
• Field of Search: 707/769, 707/714, 707/758, 707/779, 707/793, 707/802, 707/803, 707/100, 707/104, 707/809
• US Class Current: 707/803
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/2468 Fuzzy queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 16/835 Query processing
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods generating schemas that represent multiple data sources are provided herein. According to some embodiments, methods may include determining a schema for each of the multiple data sources via a computing device communicatively couplable with each of the multiple data sources, each of the multiple data sources including one or more data structures that define how data is stored in the data source, generating a negotiated schema by comparing the schemas of the multiple data sources to one another and interrelating data points of the multiple data sources based upon the schemas, interrelating the negotiated schema with the schema for each of the multiple data sources based upon the interrelation of the data points, and storing the negotiated schema in a storage media by way of the computing device.

Claims

1. A method for generating a negotiated schema for multiple data sources, the method comprising:

interrogating multiple data sources via a computing device communicatively couplable with each of the multiple data sources, each of the multiple data sources including different, unknown types of data structures that define how data is stored in the data source, the interrogation of each data source including;

comparing the unknown data structure of the data source to a plurality of schema templates;

determining a match between the unknown data structure and each of the plurality of schema templates based on the comparison; and

determining a schema representing the unknown data structure, the determination including selecting a schema template based on the determined match between the unknown data structure and each of the plurality of schema templates;

generating a negotiated schema that represents the data structures of the multiple data sources by;

comparing the data structure of each of the determined schemas of the multiple data sources to one another; and

linking together interrelated data points that are in different schemas of the multiple data sources;

interrelating the negotiated schema with the schema for each of the multiple data sources based upon the interrelation of the data points such that the negotiated schema represents the data structures of the multiple data sources; and

storing the negotiated schema in a storage media by way of the computing device.

2. The method according to claim 1, wherein determining a match between the unknown data structure and each of the plurality of schema templates based on the comparison includes:

determining if the match is a correspondence between the unknown data structure and the plurality of schema templates;

determining if the match is a correspondence between the unknown data structure and an existing negotiated schema template; and

determining a matching strength for a fuzzy match using heuristics, if the match is a fuzzy match.

3. The method according to claim 2, wherein determining a schema representing the unknown data structure comprises selecting the most appropriate of:

a specified schema if a schema is specified in the data source;

a schema template based on correspondence, if the match is a correspondence between the unknown data structure and an existing negotiated schema template or one or more of the plurality of schema templates;

a schema template based on matching strength of the schema templates, if the match is a fuzzy match that at least partially matches the one or more of the plurality of schema templates; and

a created schema for the data source, if an appropriate correspondence between the data source and the plurality of schema templates is not found.

4. The method according to claim 1, wherein the interrelated data points of the multiple data sources include a first blood pressure for a first patient in a first data source having a first schema for the data structure of the first data source and a second blood pressure for a second patient in a second data source having a second schema for the data structure of the second data source.

5. The method according to claim 1, wherein generating a negotiated schema further comprises:

applying heuristic logic to identify one or more data points in each data source that may be directly linked to data points in other data sources based on the comparison; and

applying heuristic logic to identify one or more data points in each data source that may be indirectly linked to data points in other data sources based on the comparison.

6. The method according to claim 1, further comprising appending metadata to the negotiated schema, the metadata including information corresponding to linking of data points of schemas for generating negotiated schema, the metadata further including a confidence level associated with linking together interrelated data points of nodes in different schemas.

7. The method according to claim 1, further comprising receiving verification from an end user that an interrelationship between the data points is correct.

8. The method according to claim 1, further comprising:

receiving a query for data against the negotiated schema from a client device;

fulfilling the query by;

obtaining data points from one or more of the multiple data sources by way of the negotiated schema; and

combining the data points together according to the interrelationships included in the negotiated schema to create a query response; and

providing the query response to the client device.

9. The method according to claim 8, automatically updating the negotiated schema when one or more data structures of one or more of the data sources changes.

10. The method according to claim 8, further comprising at least one of:

selecting one or more alternative data sources when one or more required data sources are unavailable; and

marking metadata in a returning result set appropriately if no alternative data source is available.

11. A system for generating a negotiated schema for multiple data sources, the system comprising:

a memory for storing executable instructions; and

a processor for executing the instructions, the instructions including;

an interrogation module that determines a schema for each of the multiple data sources, each of the multiple data sources including different, unknown types of structures that define how data is stored in the data source, the determination by the interrogation module including;

comparing the unknown data structure of the data source to a plurality of schema templates;

determining a match between the unknown data structure and each of the plurality of schema templates based on the comparison; and

determining a schema representing the unknown data structure, the determination including selecting a schema template based on the determined match between the unknown data structure and each of the plurality of schema templates; and

a schema generator that generates a negotiated schema that represents the data structures of the multiple data sources by;

comparing the data structure of each of the determined schemas of the multiple data sources to at least one of one another and one or more pre-existing schemas;

interrelating data points of the multiple data sources that correspond to one another;

interrelating the negotiated schema with the schema for each of the multiple data sources based upon the interrelation of the data points; and

storing the negotiated schema in a storage media.

12. The system according to claim 11, wherein the interrogation module further:

compares one or more data structures to the plurality of schema templates; and

selects one of the plurality of schema templates that at least partially matches the one or more data structures.

13. The system according to claim 12, wherein the interrogation module compares the one or more data structures to the plurality of schema templates by applying heuristic logic, the heuristic logic being based upon at least one of end-user input, machine learning techniques, metadata tags, previous schema negotiations, custom predefined connectors, heuristics, and combinations thereof.

14. The system according to claim 11, wherein the interrelating data points of the multiple data sources includes the schema generator determining at least one of a fuzzy and a concrete relationship between data points of two or more data sources that at least partially correspond to one another.

15. The system according to claim 14, wherein the schema generator is configured to provide metadata that describes an interrelationship between the data points.

16. The system according to claim 15, wherein the metadata may include any of data type, schema determination information for each data source, and a confidence level for interrelated sets of data points.

17. The system according to claim 11, wherein the schema generator receives verification from an end user that an interrelationship between data points is correct.

18. The system according to claim 11, wherein the interrogation module is further configured to:

receive a query for data against the negotiated schema from a client device;

fulfill the query by;

obtaining data points from one or more of the multiple data sources by way of the negotiated schema; and

combining the data points together according to the interrelationships included in the negotiated schema to create a query response; and

provide the query response to the client device.

19. The system according to claim 18, wherein the schema generator obtains metadata associated with the data points and combines the metadata with the data points before the interrogation module provides the query response to the client device.

20. A non-transitory computer readable storage media having a program embodied thereon, the program being executable by a processor to perform a method for generating a negotiated schema for multiple data sources, the method comprising:

determining a schema for each of the multiple data sources via a computing device communicatively couplable with each of the multiple data sources, each of the multiple data sources including one or more data structures that define how data is stored in the data source, the determination of each schema including;

comparing the unknown data structure of the data source to a plurality of schema templates;

determining a match between the unknown data structure and each of the plurality of schema templates based on the comparison; and

determining a schema representing the unknown data structure, the determination including selecting a schema template based on the determined match between the unknown data structure and each of the plurality of schema templates;

generating a negotiated schema that represents the data structures of the multiple data sources by;

comparing the data structure of each of the schemas of the multiple data sources to one another; and

interrelating data points of the multiple data sources that correspond to one another;

interrelating the negotiated schema with the schema for each of the multiple data sources based upon the interrelation of the data points; and

storing the negotiated schema in a storage media by way of the computing device.