Systems and methods for importing data from electronic data files

• US 9,946,776 B1
• Filed: 10/12/2016
• Issued: 04/17/2018
• Est. Priority Date: 09/04/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Yazicioglu, Stephen, Luck, Christopher, Giardina, Robert, Streufert, Justin, Slatcher, Timothy, O'Connor, Gregory, Marc-Aurele, Brandon, Zhu, Olivia, Schindel, Howard, Tung, Henry, Ray, Lucas, Leech, Christopher, Jeney, Eric, Negrus, Stefan, Lee, Jason, Mingione, Alessandro, Doyle, John McKinstry, Pitelka, Hunter, Lozano, Ethan, Ossher, Joel, Fedderly, Matthew
• Primary Examiner(s): Le, Hung
• Application Number:
  US15/292,075
• Time in Patent Office: 552 Days
• Field of Search: None
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: G06F 40/186 Templates

Abstract

Computer implemented systems and methods are disclosed for importing data from electronic data files. In accordance with some embodiments, source electronic data files are received at a data importation system and managed by the data importation system. The data importation system may load detector/transformer plugins and determined whether any of the load detector/transformer plugins apply to the received source electronic data files. The data importation system may also generate transformed data using a transformer corresponding to a detector that applies to the source electronic data files. The data importation system may import the transformed data into at least one of a data analysis system and a database.

Claims

1. A system comprising:

one or more memory devices that store a set of instructions; and

at least one processor that executes the set of instructions to perform operations for generating an updated transformed electronic data file using a detector/transformer plugin, the detector/transformer plugin including a detector and a transformer, the detector configured to detect whether an updated electronic source data file is compatible with the transformer, the transformer configured to transform the updated source electronic data file into the updated transformed electronic data file, the operations comprising;

receiving the source electronic data file;

receiving an update to the source electronic data file as an updated electronic source data file;

loading the detector/transformer plugin from a database based on a detection by the detector of the detector/transformer plugin that the updated electronic source data file is compatible with the transformer, the detection based on a file type of the updated electronic source data file;

performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the transformer of the detector/transformer plugin; and

providing the updated transformed electronic data file for importing into a data analysis system.

2. The system of claim 1, wherein an order in which detectors of a plurality of detector/transformer plugins in the database are determined to apply to the updated source electronic data file is determined based on priorities assigned to the plurality of detector/transformer plugins.

3. The system of claim 1, the operations further comprising storing transformed data in the updated transformed electronic data file.

4. The system of claim 1, the operations further comprising:

receiving a request to delete the source electronic data file; and

deleting the source electronic data file.

5. The system of claim 4, the operations further comprising providing a request to the data analysis system to delete transformed data corresponding to the source electronic data file.

6. A method comprising:

performing operations for generating an updated transformed electronic data file using a detector/transformer plugin, the detector/transformer plugin including a detector and a transformer, the detector configured to detect whether an updated electronic source data file is compatible with the transformer, the transformer configured to transform the updated source electronic data file into the updated transformed electronic data file, the operations comprising;

receiving the source electronic data file;

receiving an update to the source electronic data file as an updated electronic source data file;

loading the detector/transformer plugin from a database based on a detection by the detector of the detector/transformer plugin that the updated electronic source data file is compatible with the transformer, the detection based on a file type of the updated electronic source data file; and

performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the transformer of the detector/transformer plugin; and

providing the updated transformed electronic data file for importing into data analysis system, one or more modules incorporated into one or more memories of a computer system to configure one or more processors of the computer system to perform the generating of the updated transformed electronic data file using the detector/transformer plugin.

7. The method of claim 6, wherein an order in which detectors of a plurality of detector/transformer plugins in the database are determined to apply to the updated source electronic data file is determined based on priorities assigned to the plurality of detector/transformer plugins.

8. The method of claim 6, the operations further comprising storing transformed data in the updated transformed electronic data file.

9. The method of claim 6, the operations further comprising:

receiving a request to delete the source electronic data file; and

deleting the source electronic data file.

10. The method of claim 9, the operations further comprising providing a request to the data analysis system to delete transformed data corresponding to the source electronic data file.

11. The method of claim 6, further comprising providing the transformed updated electronic data file to the data analysis system.

12. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor of a machine, cause the machine to perform operations for generating an updated transformed electronic data file using a detector/transformer plugin, the detector/transformer plugin including a detector and a transformer, the detector configured to detect whether an updated electronic source data file is compatible with the transformer, the transformer configured to transform the updated source electronic data file into the updated transformed electronic data file, the operations comprising:

receiving a source electronic data file;

receiving an update to the source electronic data file as an updated electronic source data file;

loading the detector/transformer plugin from a database based on a detection by the detector of the detector/transformer plugin that the updated electronic source data file is compatible with the transformer, the detection based on a file type of the updated electronic source data file;

performing the transforming of the updated source electronic data file into the updated transformed electronic data file using the transformer of the detector/transformer plugin; and

providing the updated transformed electronic data file for importing into the data analysis system.