History preserving data pipeline

• US 9,483,506 B2
• Filed: 10/09/2015
• Issued: 11/01/2016
• Est. Priority Date: 11/05/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Meacham, Jacob, Harris, Michael, Brodman, Gustav, Cuthriell, Lynn, Korus, Hannah, Toth, Brian, Hsiao, Jonathan, Elliot, Mark, Schimpf, Brian, Garland, Michael, Nguyen, Evelyn
• Primary Examiner(s): Hoang, Son T
• Application Number:
  US14/879,916
• Publication Number:
  US 20160125000A1
• Time in Patent Office: 389 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1865 Transactional file systems
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A history preserving data pipeline computer system and method. In one aspect, the history preserving data pipeline system provides immutable and versioned datasets. Because datasets are immutable and versioned, the system makes it possible to determine the data in a dataset at a point in time in the past, even if that data is no longer in the current version of the dataset.

Claims

1. A method comprising:

at one or more computing devices comprising one or more processors and one or more storage media storing one or more computer programs executed by the one or more processors to perform the method, performing operations comprising;

maintaining a build catalog comprising a plurality of build catalog entries;

wherein each build catalog entry, of the plurality of build catalog entries, comprises;

an identifier of a version of a derived dataset corresponding to the build catalog entry,

one or more dataset build dependencies of the version of the derived dataset corresponding to the build catalog entry, each of the one or more dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the build catalog entry is derived, and

a derivation program build dependency of the version of the derived dataset corresponding to the build catalog entry, the derivation program build dependency comprising an identifier of a version of a derivation program executed to generate the version of the derived dataset corresponding to the build catalog entry;

creating a new version of a particular derived dataset in context of a successful transaction; and

adding a new build catalog entry to the build catalog, the new build catalog entry comprising an identifier of the new version of the particular derived dataset, the identifier of the new version of the particular derived dataset being a transaction commit identifier assigned to the successful transaction.

2. The method of claim 1, wherein the operation of creating the new version of the particular derived dataset comprises an operation of executing a particular version of a particular derivation program;

and wherein the new build catalog entry comprises an identifier of the particular version of the particular derivation program.

3. The method of claim 1, wherein a particular build catalog entry, of the plurality of build catalog entries, comprises a plurality of dataset build dependencies of the version of the derived dataset corresponding to the particular build catalog entry, each of the plurality of dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the particular build catalog entry is derived.

4. The method of claim 1, wherein the build catalog comprises a particular build catalog entry comprising an identifier of a version of a derived dataset corresponding to the build catalog entry but does not include any of a dataset build dependency or a derivation program build dependency.

5. One or more non-transitory storage media storing one or more computer programs, the one or more computer programs comprising instructions for performing operations comprising:

maintaining a build catalog comprising a plurality of build catalog entries;

wherein each build catalog entry, of the plurality of build catalog entries, comprises;

an identifier of a version of a derived dataset corresponding to the build catalog entry,

one or more dataset build dependencies of the version of the derived dataset corresponding to the build catalog entry, each of the one or more dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the build catalog entry is derived, and

a derivation program build dependency of the version of the derived dataset corresponding to the build catalog entry, the derivation program build dependency comprising an identifier of a version of a derivation program executed to generate the version of the derived dataset corresponding to the build catalog entry;

creating a new version of a particular derived dataset in context of a successful transaction; and

adding a new build catalog entry to the build catalog, the new build catalog entry comprising an identifier of the new version of the particular derived dataset, the identifier of the new version of the particular derived dataset being a transaction commit identifier assigned to the successful transaction.

6. The one or more non-transitory storage media of claim 5, wherein the instructions for performing the operation of creating the new version of the particular derived dataset comprise instructions for performing an operation of executing a particular version of a particular derivation program;

and wherein the new build catalog entry comprises an identifier of the particular version of the particular derivation program.

7. The one or more non-transitory storage media of claim 5, wherein a particular build catalog entry, of the plurality of build catalog entries, comprises a plurality of dataset build dependencies of the version of the derived dataset corresponding to the particular build catalog entry, each of the plurality of dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the particular build catalog entry is derived.

8. The one or more non-transitory storage media of claim 5, wherein the build catalog comprises a particular build catalog entry comprising an identifier of a version of a derived dataset corresponding to the build catalog entry but does not include any of a dataset build dependency or a derivation program build dependency.

9. A system comprising:

one or more hardware processors;

one or more computer programs; and

one or more storage media storing the one or more computer programs for execution by the one or more hardware processors, the one or more computer programs comprising instructions for performing operations comprising;

maintaining a build catalog comprising a plurality of build catalog entries;

wherein each build catalog entry, of the plurality of build catalog entries, comprises;

an identifier of a version of a derived dataset corresponding to the build catalog entry,

one or more dataset build dependencies of the version of the derived dataset corresponding to the build catalog entry, each of the one or more dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the build catalog entry is derived, and

a derivation program build dependency of the version of the derived dataset corresponding to the build catalog entry, the derivation program build dependency comprising an identifier of a version of a derivation program executed to generate the version of the derived dataset corresponding to the build catalog entry;

creating a new version of a particular derived dataset in context of a successful transaction; and

adding a new build catalog entry to the build catalog, the new build catalog entry comprising an identifier of the new version of the particular derived dataset, the identifier of the new version of the particular derived dataset being a transaction commit identifier assigned to the successful transaction.

10. The system of claim 9, wherein the instructions for performing the operation of creating the new version of the particular derived dataset comprise instructions for performing an operation of executing a particular version of a particular derivation program;

and wherein the new build catalog entry comprises an identifier of the particular version of the particular derivation program.

11. The system of claim 9, wherein a particular build catalog entry, of the plurality of build catalog entries, comprises a plurality of dataset build dependencies of the version of the derived dataset corresponding to the particular build catalog entry, each of the plurality of dataset build dependencies comprising an identifier of a version of a child dataset from which the version of the derived dataset corresponding to the particular build catalog entry is derived.

12. The system of claim 9, wherein the build catalog comprises a particular build catalog entry comprising an identifier of a version of a derived dataset corresponding to the build catalog entry but does not include any of a dataset build dependency or a derivation program build dependency.