System for linking diverse data systems

• US 9,953,082 B2
• Filed: 06/30/2016
• Issued: 04/24/2018
• Est. Priority Date: 02/26/2015
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Gomadam, Karthik, Tung, Teresa Sheausan
• Primary Examiner(s): Somers, Marc S
• Application Number:
  US15/198,655
• Publication Number:
  US 20160314202A1
• Time in Patent Office: 663 Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence

Abstract

A system creates an abstraction layer surrounding a diverse data system including multiple different databases. Data is received from data sources and ingested into the various databases according to a core model. New instances of the core model are created and added to a larger linked data model (LDM) when new data sources are added to the system. The LDM captures the linkages between different linked data objects and links across different databases. Accordingly, applications are able to access or explore the linked data stored in different databases without prior knowledge of the linking relationships.

Claims

1. A method comprising:

receiving a dataset at a communication interface, the dataset including dataset context information comprising metadata for the dataset;

reviewing, by a processor, the metadata for the dataset to determine a type of data of the dataset;

selecting, by the processor, a core model comprising a schema of structured relationships correlating to the type of data of the dataset, the core model comprising;

a first type node corresponding to a first datatype;

a first database node corresponding to a first database;

a first relationship between the first type node and the first database node establishing that data of the first datatype is stored in the first database;

a second type node corresponding to a second datatype;

a second database node corresponding to a second database; and

a second relationship between the second type node and the second database node establishing that data of the second datatype is stored in the second database;

the first database and the second database identified as part of a data storage architecture extending across multiple databases including the first and second databases;

determining, by the processor, that a first portion of the dataset has the first datatype;

reviewing, by the processor, the first relationship between the first type node and the first database node to determine to store the first portion of the dataset in the first database;

transmitting, by the communication interface, the first portion of the dataset to the first database for storage;

determining, by the processor, that a second portion of the dataset has the second datatype;

reviewing, by the processor, the second relationship between the second type node and the second database node to determine to store the second portion of the dataset in the second database;

transmitting, by the communication interface, the second portion of the dataset to the second database for storage;

creating a linked representation separate from the core model that links the first portion of the dataset to the second portion of the dataset, the linked representation comprising;

a representation of the first portion of the dataset as an instance of the first type node of the core model;

a representation of the first database as an instance of the first database node of the core model;

a representation of the second portion of the dataset as an instance of the second type node of the core model; and

a representation of the second database as an instance of the second database node of the core model;

creating, by the processor, a domain knowledge graph from the linked representation, the domain knowledge graph including multiple linked nodes corresponding to core model instances; and

receiving, by content aware routing circuitry, data from a new data source; and

performing, by the content aware routing circuitry, an onboarding procedure on the data from the new data source, the onboarding procedure including;

identifying a type of data for the data from the new data source;

determining a correct database into which to store the data from the new data source responsive to the type of data for the data from the new data source; and

instantiating a new node in the domain knowledge graph corresponding to the new data source.

2. The method of claim 1 further comprising:

receiving a second dataset at the communication interface, the second dataset including second dataset context information comprising metadata for the second dataset;

determining, by the processor and responsive to the metadata, that the core model correlates to the second dataset;

determining by the processor that a first portion of the second dataset has the first datatype;

determining, by the processor, to store the first portion of the second dataset in the first database due to the first relationship between the first type node and the first database node in the core model;

transmitting, by the communication interface, the first portion of the second dataset to the first database for storage; and

altering the linked representation to include a link between the first portion of the second dataset to the first database and a representation of the first portion of the second dataset as an instance of the first type node of the core model.

3. The method of claim 1 further comprising:

prior to transmitting the first portion of the dataset to the first database for storage;

reviewing, by data consistency checking circuitry, the domain knowledge graph to determine attributes for storing the first portion of the dataset in the first database; and

determining, by data consistency checking circuitry, that the attributes are present for the dataset.

4. The method of claim 1 further comprising:

after performing the onboarding procedure;

receiving, by the content aware routing circuitry, data from the new data source;

querying, by the content aware routing circuitry, the domain knowledge graph to determine that the data from the new data source corresponds to the new node; and

identifying the correct database for storage of the data received from the new data source by determining a relationship within the new node associating the data from the new data source to the correct database.

5. The method of claim 1 further comprising:

receiving, by the processor, an update to the core model, the update comprising;

a third database node corresponding to a third database, and

a third relationship between the first type node and the third database node establishing that data of the first datatype is stored in the third database; and

updating, by the processor, the linked representation to include a representation of the third database as an instance of the third database node of the core model.

6. The method of claim 1 further comprising:

receiving, by the communication interface, a query for at least the first portion of the dataset from a querying entity;

referencing, by the processor, the linked representation to determine the first database as the database in which the first portion of the dataset is stored;

contacting by the processor via the communication interface the first database to retrieve the first portion of the dataset;

receiving by the communication interface the first portion of the dataset from the first database; and

transmitting by the communication interface the first portion of the dataset to the querying entity.

7. A system comprising:

a communication interface configured to receive a dataset including dataset context information comprising metadata for the dataset;

a processor coupled to the communication interface, the processor configured to;

review the metadata to determine a type of data of the dataset;

determine a core model comprising a schema of structured relationships correlating to the type of data of the dataset, the core model comprising;

a first type node corresponding to a first datatype;

a first database node corresponding to a first database;

a first relationship between the first type node and the first database node establishing that data of the first datatype is stored in the first database;

a second type node corresponding to a second datatype;

a second database node corresponding to a second database; and

a second relationship between the second type node and the second database node establishing that data of the second datatype is stored in the second database;

the first database and the second database identified as part of a data storage architecture extending across multiple databases including the first and second databases;

determine that a first portion of the dataset has the first datatype;

review the first relationship between the first type node and the first database node in the core model to determine to store the first portion of the dataset in the first database;

determine that a second portion of the dataset has the second datatype;

review the second relationship between the second type node and the second database node in the core model to determine to store the second portion of the dataset in the second database;

create a linked representation separate from the core model that links the first portion of the dataset to the second portion of the dataset, the linked representation comprising;

a representation of the first portion of the dataset as an instance of the first type node of the core model;

a representation of the first database as an instance of the first database node of the core model;

a representation of the second portion of the dataset as an instance of the second type node of the core model; and

a representation of the second database as an instance of the second database node of the core model; and

create a domain knowledge graph from the linked representation, the domain knowledge graph including multiple linked nodes corresponding to core model instances; and

the communication interface further configured to;

transmit the first portion of the dataset to the first database for storage; and

transmit the second portion of the dataset to the second database for storage; and

content aware routing circuitry configured to;

receive data from a new data source; and

perform an onboarding procedure on the data from the new data source, the onboarding procedure including;

identifying a type of data for the data from the new data source;

determining a correct database into which to store the data from the new data source responsive to the type of data for the data from the new data source; and

instantiating a new node in the domain knowledge graph corresponding to the new data source.

8. The system of claim 7 wherein the communication interface is further configured to receive a second dataset including second dataset context information comprising metadata for the second dataset;

and

wherein the processor is further configured to;

determine, responsive to the metadata, that the core model correlates to the second dataset;

determine that a first portion of the second dataset has the first datatype;

determine to store the first portion of the second dataset in the first database due to the first relationship between the first type node and the first database node in the core model;

alter the linked representation to include a link between the first portion of the second dataset to the first database and a representation of the first portion of the second dataset as an instance of the first type node of the core model; and

wherein the communication interface is further configured to transmit the first portion of the second dataset to the first database for storage.

9. The system of claim 7 further comprising:

data consistency checking circuitry configured to, prior to the communication interface transmitting the first portion of the dataset to the first database for storage;

review the domain knowledge graph to determine attributes for storing the first portion of the dataset in the first database; and

determine that the attributes are present for the dataset.

10. The system of claim 7 wherein the content aware routing circuitry is further configured to, after performing the onboarding procedure:

receive data from the new data source;

query the domain knowledge graph to determine that the data from the new data source corresponds to the new node; and

identify the correct database for storage of the data received from the new data source by determining a relationship within the new node associating the data from the new data source to the correct database.

11. The system of claim 7 wherein the processor is further configured to:

receive an update to the core model, the update comprising;

a third database node corresponding to a third database, and

a third relationship between the first type node and the third database node establishing that data of the first datatype is stored in the third database; and

update the linked representation to include a representation of the third database as an instance of the third database node of the core model.

12. The system of claim 7 wherein the communication interface is further configured to receive a query for at least the first portion of the dataset from a querying entity;

and

wherein the processor is further configured to;

reference the linked representation to determine the first database as the database in which the first portion of the dataset is stored;

contact via the communication interface the first database to retrieve the first portion of the dataset;

wherein the communication interface is also configured to;

receive the first portion of the dataset from the first database; and

transmit the first portion of the dataset to the querying entity.