Systems and methods for facilitating data transformation

• US 9,946,777 B1
• Filed: 12/19/2016
• Issued: 04/17/2018
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Ossher, Joel, Xiao, David
• Primary Examiner(s): Hasan, Syed
• Application Number:
  US15/383,787
• Time in Patent Office: 484 Days
• Field of Search: None
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

Systems and methods are provided for facilitating the transformation of data from a tabular data set organized according to a data schema to an object based data set organized according to data ontology. The data schema and the data ontology may be used to generate a custom coding language suitable for facilitating the creation of a transform script. Error checking, type checking, autocomplete, and preview tools may be provided to permit the verification of the transform script as it is created.

Claims

1. A system for performing an integration of an origin data set into a target data set, the system comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to;

receive the origin data set, the origin data set being organized by an origin data set schema specifying a tabular format for data of the origin data set;

receive a target data set ontology, the target data set ontology defining data objects of the target data set;

generate, according to the origin data set schema and the target data set ontology, a domain-specific transform programming language specific to the origin data set schema and the target data set ontology;

receive transform instructions programmed in the domain-specific transform programming language;

generate a preview target data set from at least a portion of the origin data set using the transform instructions, the preview target data set generated in response to receiving the transform instructions; and

integrate the at least a portion of the origin data set into the target data set according to the received transform instructions in response to the preview target data set corresponding to the target data set ontology.

2. The system of claim 1, wherein the system is further caused to:

store data transformation information about the integration, the data transformation information including information about the transform instructions and a time and date of integration.

3. The system of claim 2, wherein to integrate the at least a portion of the origin data set into the target data set, the system is further caused to select the at least a portion of the origin data set according to the data transformation information and integrate the selected portion of the origin data set.

4. The system of claim 1, wherein the preview target data set is generated while the system receives the transform instructions.

5. The system of claim 1, wherein the system is further caused to provide a transform instructions template for receiving the transform instructions.

6. The system of claim 1, wherein the system caused to receive transform instructions is further caused to:

analyze the received transform instructions for errors as they are received; and

notify a user of the errors.

7. The system of claim 6, wherein to analyze the received transform instructions for errors the system is further caused to compare the received transform instructions to the target data set ontology and to the origin data set schema.

8. The system of claim 1, wherein the system caused to receive the transform instructions is further caused to provide autocomplete suggestions.

9. A computer implemented method of integrating an origin data set into a target data set, the method being performed on a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method, the method comprising:

receiving, by the computer system, the origin data set, the origin data set being organized by an origin data set schema specifying a tabular format for data of the origin data set;

receiving, by the computer system, a target data set ontology, the target data set ontology defining data objects of the target data set;

generating, by the computer system, according to the origin data set schema and the target data set ontology, a domain-specific transform programming language specific to the origin data set schema and the target data set ontology;

receiving, by the computer system, transform instructions programmed in the domain-specific transform programming language;

generating, by the computer system, a preview target data set from at least a portion of the origin data set using the transform instructions, the preview target data set generated in response to receiving the transform instructions; and

integrating, by the computer system, the at least a portion of the origin data set into the target data set according to the received transform instructions in response to the preview target data set corresponding to the target data set ontology.

10. The method of claim 9, further comprising:

storing, by the computer system, data transformation information about the integration, the data transformation information including information about the transform instructions and a time and date of integration.

11. The method of claim 10, wherein the integrating the at least a portion of the origin data set into the target data set includes selecting, by the computer system, the at least a portion of the origin data set according to the data transformation information and integrating the selected portion of the origin data set.

12. The method of claim 10, wherein receiving the transform instructions includes

analyzing, by the computer system, the received transform instructions for errors as they are received;

and

notifying, by the computer system, a user of the errors.

13. The method of claim 12, wherein analyzing the received transform instructions for errors includes comparing the received transform instructions to the target data set ontology and to the origin data set schema.

14. The method of claim 9, wherein the preview target data set is generated while the system receives transform instructions.

15. The method of claim 9, further comprising providing, by the computer system, a transform instructions template for receiving the transform instructions.

16. The method of claim 9, wherein receiving the transform instructions includes providing, by the computer system, autocomplete suggestions.

17. A system for performing an integration of an origin data set into a target data set, the system comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to;

receive a target data set ontology, the target data set ontology defining data objects of the target data set;

access an origin data set schema, the origin data set schema including an organizational structure of the origin data set;

generate, according to the origin data set schema and the target data set ontology, a domain-specific transform programming language specific to the origin data set schema and the target data set ontology;

receive transform instructions from a user input device programmed in the domain-specific transform programming language; and

transfer the transform instructions to a remote computer system for integrating the origin data set into the target data set according to the received transform instructions.

18. The system of claim 17, wherein the system is further caused to provide a preview target data set based on the received transform instructions.