Domain-specific language for dataset transformations

• US 9,965,534 B2
• Filed: 12/05/2016
• Issued: 05/08/2018
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Tolnay, David, Biswal, Punyashloka, Colombi, Andrew, Fu, Yupeng, Fuadi, Ashar, Kim, Mingyu, Nepywoda, Paul, Pundle, Akshay, Tamayo, Juan
• Primary Examiner(s): Ortiz Ditren, Belix M
• Application Number:
  US15/369,753
• Publication Number:
  US 20170083595A1
• Time in Patent Office: 519 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/9027 Trees

Abstract

Techniques related to a domain-specific language for dataset transformations are disclosed. A server computer may process a table definition composed in a domain-specific language. The table definition may include a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table. The sequence may include a customized transformation. A source dataset may be provided as input to an implementation of the customized transformation. An output dataset may be generated as a result of executing the implementation. An intermediate table may be generated based on performing at least one dataset transformation on a particular source table. A supplemental portion for the intermediate table may be generated based on performing the at least one dataset transformation on an appended portion of the particular source table. The target table may be generated based on combining the supplemental portion with the intermediate table.

Claims

1. A data processing method comprising:

processing a table definition composed in a domain-specific language, the table definition comprising a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table;

retrieving an intermediate table that was generated based on performing a first dataset transformation of the one or more dataset transformations on a previous version of a particular source table of the one or more source tables;

generating a supplemental portion for the intermediate table based on performing a second dataset transformation on an appended portion of an updated version of the particular source table;

generating the target table based on performing a third dataset transformation on the intermediate table and the supplemental portion for the intermediate table;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the first dataset transformation and the second dataset transformation are different dataset transformations.

3. The method of claim 1, wherein the second dataset transformation and the third dataset transformation are performed without being specified by an end user.

4. The method of claim 1, wherein generating the supplemental portion comprises performing a fourth dataset transformation on an output dataset of the first dataset transformation and an output dataset of the second dataset transformation.

5. The method of claim 4, wherein the fourth dataset transformation is reversible.

6. The method of claim 1, wherein the target table is generated based on persisting an output dataset of the third dataset transformation.

7. The method of claim 1, wherein the sequence of one or more dataset transformations is determined based on a graphical representation of the one or more dataset transformations.

8. The method of claim 7, wherein the graphical representation is a directed acyclic graph.

9. The method of claim 7, wherein the graphical representation comprises one or more leaf nodes and one or more non-leaf nodes, the one or more leaf nodes representing one or more tables, the one or more non-leaf nodes representing the one or more dataset transformations.

10. The method of claim 1, wherein one or more references to the one or more source tables are stored at a server computer, without copies of the one or more source tables being stored at the server computer.

11. A computer system comprising:

one or more processors;

one or more non-transitory data storage media coupled to the one or more processors and storing instructions which, when executed by the one or more processors, cause;

processing a table definition composed in a domain-specific language, the table definition comprising a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table;

retrieving an intermediate table that was generated based on performing a first dataset transformation of the one or more dataset transformations on a previous version of a particular source table of the one or more source tables;

generating a supplemental portion for the intermediate table based on performing a second dataset transformation on an appended portion of an updated version of the particular source table;

generating the target table based on performing a third dataset transformation on the intermediate table and the supplemental portion for the intermediate table.

12. The system of claim 11, wherein the first dataset transformation and the second dataset transformation are different dataset transformations.

13. The system of claim 11, wherein the one or more storage media further comprise sequences of instructions which when executed cause performing the second dataset transformation and the third dataset transformation without being specified by an end user.

14. The system of claim 11, wherein the one or more storage media further comprise sequences of instructions which when executed cause performing generating the supplemental portion by performing a fourth dataset transformation on an output dataset of the first dataset transformation and an output dataset of the second dataset transformation.

15. The system of claim 14, wherein the fourth dataset transformation is reversible.

16. The system of claim 11, wherein the one or more storage media further comprise sequences of instructions which when executed cause generating the target table based on persisting an output dataset of the third dataset transformation.

17. The system of claim 11, wherein the one or more storage media further comprise sequences of instructions which when executed cause determining the sequence of one or more dataset transformations based on a graphical representation of the one or more dataset transformations.

18. The system of claim 17, wherein the graphical representation is a directed acyclic graph.

19. The system of claim 17, wherein the graphical representation comprises one or more leaf nodes and one or more non-leaf nodes, the one or more leaf nodes representing one or more tables, the one or more non-leaf nodes representing the one or more dataset transformations.

20. The system of claim 11, wherein the one or more storage media further comprise sequences of instructions which when executed cause storing one or more references to the one or more source tables at a server computer, without corresponding copies of the one or more source tables being stored at the server computer.