Validating data integrations using a secondary data store

• US 10,102,229 B2
• Filed: 11/09/2016
• Issued: 10/16/2018
• Est. Priority Date: 11/09/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Wilson, Timothy, Morgan, Kevin
• Primary Examiner(s): Jacob, Ajith
• Application Number:
  US15/347,637
• Publication Number:
  US 20180129684A1
• Time in Patent Office: 706 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/213 with details for schema evolution support
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation

Abstract

A data integration system provides a secondary validation data store that may be used as a staging area for testing and validating data integrations for a set of structured data. Ontology data that defines a schema for integrated data may be modified and tweaked. The secondary validation data store is more efficient than a production system for such schema modifications. Once the ontology data is satisfactory, the structured data and the ontology data may be loaded into a production system.

Claims

1. A method, comprising:

receiving a first set of data comprising a plurality of unstructured data records;

receiving a schema that describes characteristics of the plurality of unstructured data records and relationships between one or more data records of the plurality of unstructured data records;

generating, based on the first set of data and the schema, a first plurality of structured data records, wherein the first plurality of structured data records organizes the plurality of unstructured data records based on the schema;

causing to be displayed, at a client computing device, the first plurality of structured data records;

receiving schema modification instructions;

in response to receiving the schema modification instructions, modifying the schema based on the schema modification instructions;

generating, based on the modified schema and the first set of data, a second plurality of structured data records, wherein the second plurality of structured data records organizes the plurality of unstructured data records based on modified schema;

in response to receiving the schema modification instructions, causing to be displayed, at the client computing device, the second plurality of structured data records;

wherein the method is performed using one or more processors.

2. The method of claim 1, further comprising:

extracting a second set of data from one or more data sources; and

applying a data modification rule to the second set of data to generate, at least in part, the first set of data.

3. The method of claim 2, wherein the data modification rule comprises a rule for removing a field from the second set of data.

4. The method of claim 2, wherein the data modification rule comprises a rule for normalizing a value for a field from the second set of data.

5. The method of claim 2, wherein the data modification rule comprises a rule for joining the second set of data with a third set of data.

6. The method of claim 2, wherein the second set of data is a file of comma separated values.

7. The method of claim 1, wherein the first set of data is a set of data in a JavaScript Object Notation (JSON) format.

8. The method of claim 1, further comprising:

in response to receiving schema approval input, causing the first set of data and the schema to be transmitted to a second system for processing.

9. One or more non-transitory computer-readable media storing instructions, which when executed by one or more hardware processors cause:

receiving a first set of data comprising a plurality of unstructured data records;

receiving a schema that describes characteristics of the plurality of unstructured data records and relationships between one or more data records of the plurality of unstructured data records;

storing a first set of data using a first system, wherein the first system comprises a schema that describes objects and properties in the first set of data;

generating, based on the first set of data and the schema, a first plurality of structured data records, wherein the first plurality of structured data records organizes the plurality of unstructured data records based on the schema;

causing to be displayed, at a client computing device, the first plurality of structured data records;

receiving schema modification instructions;

in response to receiving the schema modification instructions, modifying the schema based on the schema modification instructions;

generating, based on the modified schema and the first set of data, a second plurality of structured data records, wherein the second plurality of structured data records organizes the plurality of unstructured data records based on modified schema;

in response to receiving the schema modification instructions, causing to be displayed, at the client computing device, the second plurality of structured data records.

10. The one or more non-transitory computer-readable media of claim 9, the instructions further comprising instructions, which when executed by one or more hardware processors cause:

extracting a second set of data from one or more data sources; and

applying a data modification rule to the second set of data to generate, at least in part, the first set of data.

11. The one or more non-transitory computer-readable media of claim 10, wherein the data modification rule comprises a rule for removing a field from the second set of data.

12. The one or more non-transitory computer-readable media of claim 10, wherein the data modification rule comprises a rule for normalizing a value for a field from the second set of data.

13. The one or more non-transitory computer-readable media of claim 10, wherein the data modification rule comprises a rule for joining the second set of data with a third set of data.

14. The one or more non-transitory computer-readable media of claim 10, wherein the second set of data is a file of comma separated values.

15. The one or more non-transitory computer-readable media of claim 9, wherein the first set of data is a set of data in a JavaScript Object Notation (JSON) format.

16. The one or more non-transitory computer-readable media of claim 9, the instructions further comprising instructions, which when executed by one or more hardware processors cause:

in response to receiving schema approval input, causing the first set of data and the schema to be transmitted to a second system for processing.

17. A method comprising:

receiving a first set of data, wherein the first set of data comprises a plurality of data fields;

analyzing a particular data field of the plurality of data fields to determine a particular property type of the data field;

using the particular property type, at least in part, to generate a database schema;

storing the first set of data in a secondary data store using the database schema;

wherein the method is performed using one or more processors.

18. The method of claim 17, further comprising:

indexing the secondary data store asynchronously.

19. The method of claim 17, wherein the first set of data is formatted in a JavaScript Object Notation (JSON) format.

20. The method of claim 17, wherein the secondary data store does not store logging data.