Techniques for data extraction

• US 10,133,782 B2
• Filed: 08/01/2016
• Issued: 11/20/2018
• Est. Priority Date: 08/01/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Pryce, Huw, Neale, James, Fink, Robert, Newman, Jared, Dennis, Graham, Nordling, Viktor, Jonkisz, Artur, Fox, Daniel, de Souza, Felix, Singh, Harkirat, Elliot, Mark
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US15/225,437
• Publication Number:
  US 20180032575A1
• Time in Patent Office: 841 Days
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2458 Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

Computer-implemented techniques for data extraction are described. The techniques include a method and system for retrieving an extraction job specification, wherein the extraction job specification comprises a source repository identifier that identifies a source repository comprising a plurality of data records; a data recipient identifier that identifies a data recipient; and a schedule that indicates a timing of when to retrieve the plurality of data records. The method and system further include retrieving the plurality of data records from the source repository based on the schedule, creating an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata, and sending the extraction transaction to the data recipient.

Claims

1. A method, comprising:

retrieving an extraction job specification from an extraction job specification repository implemented on a first computing device, wherein the extraction job specification is defined in one or more configuration files, and wherein the extraction job specification comprises;

a source repository identifier that identifies a source repository comprising a plurality of data records;

a data recipient identifier that identifies a data recipient;

a schedule that indicates a timing of when to retrieve the plurality of data records;

wherein the one or more configuration files are implemented in one or more of;

extensible markup language (XML), YAML Ain'"'"'t Markup Language (YAML), JavaScript Object Notation (JSON), and/or a markup language;

using the extraction job specification, retrieving, by a second computing device, the plurality of data records from the source repository based on the schedule;

using the extraction job specification, creating, by the second computing device, an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata;

sending, by the second computing device, the extraction transaction to the data recipient; and

wherein the method is performed using one or more processors.

2. The method of claim 1 wherein the extraction job specification further comprises an inline processor and the method further comprises, using the extraction job specification, applying the inline processor to the plurality of data records before creating the extraction transaction from the plurality of data records.

3. The method of claim 2 wherein the inline processor comprises instructions that specify one or more processes for filtering the plurality of data records.

4. The method of claim 3 wherein the instructions for filtering the plurality of data records comprise one or more regular expressions.

5. The method of claim 3 wherein the instructions for filtering the plurality of data records comprise a structured query language (SQL) expression.

6. The method of claim 2 wherein the inline processor comprises instructions that specify one or more processes for grouping a subset of the plurality of data records into a single transaction.

7. The method of claim 1 wherein the extraction job specification further comprises a completion strategy data processor and the method further comprises:

using the extraction job specification, applying the completion strategy data processor to the plurality of data records after sending the extraction transaction to the data recipient.

8. The method of claim 7 wherein the completion strategy data processor comprises instructions which when execute cause performing one or more of:

deleting the plurality of data records;

encrypting the plurality of data records; and

moving the plurality of data records to a storage location.

9. The method of claim 1 wherein the extraction job specification comprises a dynamic link library (DLL), Java Archive (JAR) file, or a device driver for accessing the data recipient.

10. A computer system, comprising:

one or more digital data storage media;

one or more processors that are communicatively coupled to the storage media;

one or more programs stored in the storage media and configured for execution by the one or more processors, the one or more programs comprising instructions which when executed using the one or more processors cause the one or more processors to perform;

retrieving an extraction job specification from an extraction job specification repository implemented on a first computing device, wherein the extraction job specification is defined in one or more configuration files, and wherein the extraction job specification comprises;

a source repository identifier that identifies a source repository comprising a plurality of data records;

a data recipient identifier that identifies a data recipient;

a schedule that indicates a timing of when to retrieve the plurality of data records;

wherein the one or more configuration files are implemented in one or more of;

extensible markup language (XML), YAML Ain′

t Markup Language (YAML), JavaScript Object Notation (JSON), and/or a markup language;

using the extraction job specification, retrieving, by a second computing device, the plurality of data records from the source repository based on the schedule;

using the extraction job specification, creating, by the second computing device, an extraction transaction from the plurality of data records, wherein the extraction transaction comprises a subset of the plurality of data records and metadata; and

sending, by the second computing device, the extraction transaction to the data recipient.

11. The system of claim 10 wherein the extraction job specification further comprises an inline processor and the instructions further comprise instructions which when executed cause, using the extraction job specification, applying the inline processor to the plurality of data records before creating the extraction transaction from the plurality of data records.

12. The system of claim 11 wherein the inline processor comprises additional instructions that specify one or more processes for filtering the plurality of data records.

13. The system of claim 12 wherein the additional instructions for filtering the plurality of data records comprise one or more regular expressions.

14. The system of claim 12 wherein the additional instructions for filtering the plurality of data records comprise a structured query language (SQL) expression.

15. The system of claim 11 wherein the inline processor comprises additional instructions for grouping a subset of the plurality of data records into a single transaction.

16. The system of claim 10 wherein the extraction job specification further comprises a completion strategy data processor and the instructions further comprise instructions which when executed cause, using the extraction job specification, applying the completion strategy data processor to the plurality of data records after sending the extraction transaction to the data recipient.

17. The system of claim 16 wherein the completion strategy data processor comprises additional instructions which when executed cause performing one or more of:

deleting the plurality of data records;

encrypting the plurality of data records; and

moving the plurality of data records to a storage location.

18. The system of claim 10 wherein the extraction job specification comprises a dynamic link library (DLL), Java Archive (JAR) file, or a device driver for accessing the data recipient.