Automatically executing tasks and configuring access control lists in a data transformation system

• US 10,754,872 B2
• Filed: 12/18/2017
• Issued: 08/25/2020
• Est. Priority Date: 12/28/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Zarum, Asaf, Kruszewski, Robert
• Primary Examiner(s): Bromell, Alexandria Y
• Application Number:
  US15/845,879
• Publication Number:
  US 20180181632A1
• Time in Patent Office: 981 Days
• Field of Search: 707756, 707781, 707803
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: G06F 16/86 Mapping to a database

Abstract

A computer-implemented system or process is programmed or configured to use a configuration file to specify one or more tasks to apply to raw ingested data. A task may be a sequence of instructions programmed or configured to format raw ingested data into a dataset in a CSV format. Examples of tasks may include: a parser to parse Cobol data into a CSV, a parser to parse XML into a CSV, a parser to parse text using fixed-width fields to a CSV, a parser to parse files in a zip archive into a CSV, a regular expression search/replace function, or formatting logic to remove lines or blank lines from raw ingested data. In one embodiment, the configuration file may specify a schema definition for a task to use for generating a dataset. In one embodiment, the configuration file may also include one or more access control list (ACL) definitions for the generated dataset. In one embodiment, the building of datasets using the configuration file is automated, for example, on a nightly basis.

Claims

1. A method comprising:

retrieving a first configuration file, the first configuration file comprising;

a plurality of different data transformation tasks, each of the tasks denoted using a task identifier that identifies a particular task to apply to a set of input data and associated with task-specific criteria for execution of the particular task;

a schema definition for a dataset, wherein the schema definition defines a plurality of columns and a data type for each column of the plurality of columns;

retrieving a second configuration file, the second configuration file comprising an access control list that defines one or more access control permissions for the dataset;

receiving an input file that includes an input dataset;

in response to receiving the input file, based on reading the first configuration file and reading the second configuration file;

applying the plurality of different data transformation tasks to the input dataset based upon the first configuration file to generate an output file including an output dataset that is formatted differently from the input dataset, wherein the output dataset is formatted according to the task-specific criteria and aligns with the plurality of columns as defined by the schema definition of the first configuration file;

determining output access control permissions for the output dataset based on the access control list of the second configuration file;

wherein the method is performed using one or more processors.

2. The method of claim 1, wherein the output file is a file in a comma separated value (CSV) format.

3. The method of claim 1, wherein the input file is a text file, and further comprising using the first configuration file to apply the particular task to the input dataset, the using comprises:

applying a search-and-replace regular expression, specified in the first configuration file, to each line of the input dataset.

4. The method of claim 1, wherein the input file is a COBOL binary file, and further comprising using the first configuration file to apply the particular task to the input dataset, the using comprises:

using an expected byte size specified in the first configuration file to identify a location of a field in the input dataset; and

retrieving the field from the input dataset.

5. The method of claim 1, wherein the input file is an extensible markup language (XML) file, and further comprising using the first configuration file to apply the particular task to the input dataset, the using comprises:

mapping tagged fields of the XML file to the output dataset.

6. The method of claim 1, wherein the input dataset comprises a single text column with fixed-width fields, and further comprising using the first configuration file to apply the particular task to the input dataset, the using comprises:

using an array of fixed-width values specified in the first configuration file to map the fixed-width fields of the input dataset to the output dataset.

7. The method of claim 1, wherein the input file is a zip archive comprising text files, and further comprising using the first configuration file to apply the particular task to the input dataset, the using comprises:

mapping content of the text files to the output dataset.

8. The method of claim 7, wherein the zip archive is encrypted, and wherein the using further comprises:

decrypting the zip archive.

9. The method of claim 1, wherein the input file is a text file, and further comprising using the first configuration file to apply the particular task to the input dataset, the using comprises:

removing all blank lines from the input dataset.

10. The method of claim 1, wherein the input dataset is formatted as rows of text, and further comprising using the first configuration file to apply the particular task to the input dataset, the using comprises:

using a first setting specified in the first configuration file to identify a number of header rows to remove from the input dataset; and

using a second setting specified in the first configuration file to identify a number of footer rows to remove from the input dataset.

11. The method of claim 1, the input dataset being a single-column dataset, and further comprising using the first configuration file to transform the single-column dataset into a multi-column dataset that is delimited according to the schema definition.

12. One or more non-transitory computer-readable media storing instructions, which when executed by one or more processors cause:

retrieving a first configuration file, the first configuration file comprising;

a plurality of different data transformation tasks, each of the tasks denoted using a task identifier that identifies a particular task to apply to a set of input data and associated with task-specific criteria for execution of the particular task;

a schema definition for a dataset, wherein the schema definition defines a plurality of columns and a data type for each column of the plurality of columns;

retrieving a second configuration file, the second configuration file comprising an access control list that defines one or more access control permissions for the dataset;

receiving an input file that includes an input dataset;

in response to receiving the input dataset, file, based on reading the first configuration file and reading the second configuration file;

applying the plurality of different data transformation tasks to the input dataset based upon the first configuration file to generate an output file including an output dataset that is formatted differently from the input dataset, wherein the output dataset is formatted according to the task-specific criteria and aligns with the plurality of columns as defined by the schema definition of the first configuration file;

determining output access control permissions for the output dataset based on the access control list of the second configuration file.

13. The one or more non-transitory computer-readable media of claim 12, wherein the output file is a file in a comma separated value (CSV) format.

14. The one or more non-transitory computer-readable media of claim 12, wherein the input file is a text file, and wherein the instructions, which when executed by the one or more processors, further cause:

applying a search-and-replace regular expression, specified in the first configuration file, to each line of the input dataset.

15. The one or more non-transitory computer-readable media of claim 12, wherein the input file is a COBOL binary file, and wherein the instructions, which when executed by the one or more processors, further cause:

using an expected byte size specified in the first configuration file to identify a location of a field in the input dataset; and

retrieving the field from the input dataset.

16. The one or more non-transitory computer-readable media of claim 12, wherein the input file is an extensible markup language (XML) file, and wherein the instructions, which when executed by the one or more processors, further cause:

mapping tagged fields of the XML file to the output dataset.

17. The one or more non-transitory computer-readable media of claim 12, wherein the input dataset comprises a single text column with fixed-width fields, and wherein the instructions, which when executed by the one or more processors, further cause:

using an array of fixed-width values specified in the first configuration file to map the fixed-width fields of the input dataset to the output dataset.

18. The one or more non-transitory computer-readable media of claim 12, wherein the input file is a zip archive comprising text files and wherein the instructions, which when executed by the one or more processors, further cause:

mapping content of the text files to the output dataset.

19. The one or more non-transitory computer-readable media of claim 18, wherein the zip archive is encrypted, and wherein the instructions, which when executed by the one or more processors, further cause:

decrypting the zip archive.

20. The one or more non-transitory computer-readable media of claim 12, wherein the input file is a text, and wherein the instructions, which when executed by the one or more processors, further cause:

removing all blank lines from the input dataset.

21. The one or more non-transitory computer-readable media of claim 12, wherein the input dataset is formatted as rows of text, and wherein the instructions, which when executed by the one or more processors, further cause:

using a first setting specified in the first configuration file to identify a number of header rows to remove from the input dataset; and

using a second setting specified in the first configuration file to identify a number of footer rows to remove from the input dataset.