Unified data model for integration between relational and non-relational databases

• US 10,061,787 B2
• Filed: 01/29/2016
• Issued: 08/28/2018
• Est. Priority Date: 01/29/2016
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: Wang, Zheng, Yang, Bowen, Sang, Di, Zhao, Xiaomeng, Gou, Shuai, Li, Jing, Wang, Xin, Jia, Tianyu, Gong, Dahan
• Primary Examiner(s): Lu, Kuen S
• Application Number:
  US15/011,009
• Publication Number:
  US 20170220606A1
• Time in Patent Office: 942 Days
• Field of Search: 707803
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Schema-less databases can make data modeling and data management difficult and can detrimentally affect integration with an RDBMS. Inferring a schema from a schema-less database can improve integration by indicating a structure or organization of data in the schema-less database. A schema analyzer can infer a schema by processing data of the schema-less database to identify statistically significant data fields. The schema analyzer then creates a schema that comprises the statistically significant data fields. A data modeler can use the resulting schema along with a schema for a RDBMS to generate a unified data model. A user may submit a query based on the unified data model to obtain results from both databases. The data modeler translates the query from the unified model to be compatible with each of the schemas so that data may be written to or retrieved from each of the schema-less database and the RDBMS.

Claims

1. A method comprising:

parsing a set of records retrieved from a first database to identify a set of one or more data fields in each of the set of records, wherein the first database does not have a defined schema;

determining statistically significant data fields of the set of data fields across the set of records, wherein statistically significant data fields are determined relative to a number of records in the set of records or determined relative to a number of indications for each of the set of data fields;

identifying a first plurality of database entities from the statistically significant data fields;

determining, from a defined schema of a second database, a second plurality of database entities; and

generating a unified data model for the first database and the second database based, at least in part, on the first and the second plurality of database entities.

2. The method of claim 1, wherein generating the unified data model further comprises:

determining a set of database entities based, at least in part, on merging and deduplicating the first plurality of database entities and the second plurality of database entities;

indicating relationships among the set of database entities in the unified data model based, at least in part, on identification of associated database entities in at least one of the first and the second databases; and

indicating properties of the set of database entities in the unified data model.

3. The method of claim 2, wherein indicating properties of the set of database entities in the unified data model comprises:

analyzing query histories of the first database and the second database to identify those of the first and second plurality of database entities queried more than a first threshold as frequently queried database entities;

analyzing the first database and the second database to identify those of the first and second plurality of database entities that have more than a threshold number of instances in at least one of the first and the second databases as large volume database entities; and

indicating the frequently queried database entities and the large volume database entities in the unified data model.

4. The method of claim 1, further comprising:

generating a first schema for the first database based, at least in part, on the statistically significant data fields;

retrieving a second schema from the second database;

receiving a query that complies with the unified data model;

translating the query into a first translated query that is compatible with the first schema;

executing the first translated query on the first database;

translating the query into a second translated query that is compatible with the second schema;

executing the second translated query on the second database; and

responding to the query with results from the first database and the second database.

5. The method of claim 1, wherein determining the statistically significant data fields of the set of data fields across the set of records comprises determining whether a percentage of the set of records in which a first data field of the set of data fields is indicated is greater than a threshold.

6. The method of claim 1, wherein determining the statistically significant data fields of the set of data fields across the set of records comprises determining whether a first data field of the set of data fields is indicated a threshold number of times in the set of records.

7. The method of claim 1, wherein determining the statistically significant data fields of the set of data fields across the set of records comprises comparing a number of indications of a first data field of the set of data fields to a number of indications of a second data field of the set of data fields.

8. The method of claim 1, wherein a database entity of the first database comprises information related to one or more data fields indicated in a document, a set of key value pairs, a column family, or a graph node of the first database and a database entity of the second database comprises information related to one or more data fields indicated in a database table.

9. The method of claim 1, wherein the set of records comprises a random sample of records from the first database.

10. One or more non-transitory machine-readable media having program code for a data modeler stored therein, the program code comprising instructions to:

infer a first schema corresponding to a first database based, at least in part, on identification of statistically significant data fields determined from entities of the first database, wherein the first database does not have a defined schema, wherein statistically significant data fields are determined relative to a number of records in the first database or determined relative to a number of indications for each data field in the first database;

retrieve, from a second database, a second schema, wherein the second database has a defined schema;

generate a unified data model based, at least in part, on the first schema and the second schema; and

migrate data from the second database to the first database using the unified data model.

11. The machine-readable storage media of claim 10, wherein the instructions to migrate data from the second database to the first database using the unified data model comprise instructions to:

read the data from the second database according to the second schema;

transform the data of the second database into data compatible with the unified data model;

transform the data compatible with the unified data model into data compatible with the first schema; and

write the data compatible with the first schema to the first database in accordance with the first schema.

12. An apparatus comprising:

a processor; and

a machine-readable medium having program code executable by the processor to cause the apparatus to,

parse a set of records retrieved from a first database to identify a set of one or more data fields in each of the set of records, wherein the first database does not have a defined schema;

determine statistically significant data fields of the set of data fields across the set of records, wherein statistically significant data fields are determined relative to a number of records in the set of records or determined relative to a number of indications for each of the set of data fields;

identify a first plurality of database entities from the statistically significant data fields;

determine, from a defined schema of a second database, a second plurality of database entities; and

generate a unified data model for the first database and the second database based, at least in part, on the first and the second plurality of database entities.

13. The apparatus of claim 12, wherein the program code executable by the processor to cause the apparatus to generate the unified data model further comprises program code executable by the processor to cause the apparatus to:

determine a set of database entities based, at least in part, on merging and deduplicating the first plurality of database entities and the second plurality of database entities;

indicate relationships among the set of database entities in the unified data model based, at least in part, on identification of associated database entities in at least one of the first and the second databases; and

indicate properties of the set of database entities in the unified data model.

14. The apparatus of claim 13, wherein the program code executable by the processor to cause the apparatus to indicate properties of the set of database entities in the unified data model comprises program code executable by the processor to cause the apparatus to:

analyze query histories of the first database and the second database to identify those of the first and second plurality of database entities queried more than a first threshold as frequently queried database entities;

analyze the first database and the second database to identify those of the first and second plurality of database entities that have more than a threshold number of instances in at least one of the first and the second databases as large volume database entities; and

indicate the frequently queried database entities and the large volume database entities in the unified data model.

15. The apparatus of claim 12, further comprising program code executable by the processor to cause the apparatus to:

generate a first schema for the first database based, at least in part, on the statistically significant data fields;

retrieve a second schema from the second database;

receive a query that complies with the unified data model;

translate the query into a first translated query that is compatible with the first schema;

execute the first translated query on the first database;

translate the query into a second translated query that is compatible with the second schema;

execute the second translated query on the second database; and

respond to the query with results from the first database and the second database.

16. The apparatus of claim 12, wherein the program code executable by the processor to cause the apparatus to determine the statistically significant data fields of the set of data fields across the set of records comprises program code executable by the processor to cause the apparatus to determine whether a percentage of the set of records in which a first data field of the set of data fields is indicated is greater than a threshold.

17. The apparatus of claim 12, wherein the program code executable by the processor to cause the apparatus to determine the statistically significant data fields of the set of data fields across the set of records comprises program code executable by the processor to cause the apparatus to determine whether a first data field of the set of data fields is indicated a threshold number of times in the set of records.

18. The apparatus of claim 12, wherein the program code executable by the processor to cause the apparatus to determine the statistically significant data fields of the set of data fields across the set of records comprises program code executable by the processor to cause the apparatus to compare a number of indications of a first data field of the set of data fields to a number of indications of a second data field of the set of data fields.

19. The apparatus of claim 12, wherein a database entity of the first database comprises information related to one or more data fields indicated in a document, a set of key value pairs, a column family, or a graph node of the first database and a database entity of the second database comprises information related to one or more data fields indicated in a database table.

20. The apparatus of claim 12, wherein the set of records comprises a random sample of records from the first database.