Systems and methods for importing data from electronic data files

• US 10,380,138 B1
• Filed: 10/12/2016
• Issued: 08/13/2019
• Est. Priority Date: 09/04/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Yazicioglu, Stephen, Luck, Christopher, Giardina, Robert, Streufert, Justin, Slatcher, Timothy, O'Connor, Gregory, Marc-Aurele, Brandon, Zhu, Olivia, Schindel, Howard, Tung, Henry, Ray, Lucas, Leech, Christopher, Jeney, Eric, Negrus, Stefan, Lee, Jason, Mingione, Alessandro, Doyle, John McKinstry, Pitelka, Hunter, Lozano, Ethan, Ossher, Joel, Fedderly, Matthew
• Primary Examiner(s): Le, Hung D
• Application Number:
  US15/292,078
• Time in Patent Office: 1,035 Days
• Field of Search: None
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: G06F 40/186 Templates

Abstract

Computer implemented systems and methods are disclosed for importing data from electronic data files. In accordance with some embodiments, a file format is assigned to a source electronic data files by a data importation system. The data importation system may further identify a file type identifier associated with the source electronic data file and map the source electronic data file to a transformation template. The data importation system may further store the file format, file type identifier, and an indication of the transformation template as a file type profile associated with the source electronic data file in a database.

Claims

1. A system comp rising:

one or more memory devices that store a set of instructions;

at least one processor that executes the set of instructions to perform operations for generating a transformed source electronic data file from a source electronic data file based on inputs received from an interactive mapping graphical user interface presented on a client device, the interactive mapping graphical user interface facilitating a mapping of a format of the source electronic data file to a canonical format included in a transformation template, the operations comprising;

detecting that a file type associated with the source electronic data file is an unknown file type;

providing for a selection of the transformation template via the interactive mapping graphical user interface for associating with the file type;

generating the interactive mapping graphical user interface for presentation on the client device, the interactive mapping graphical user interface including interactive user interface elements for combining data fields in the source electronic data file and mapping the combined data fields to a canonical data field included in the transformation template;

generating the transformed source electronic data file from the source electronic data file based on mappings received via the interactive mapping graphical user interface presented on the client device; and

storing the mappings as a file type profile in a database for subsequent use in importing updates of additional source electronic data files of the file type into a data analysis system without user intervention.

2. The system of claim 1, wherein the detecting of the file type is based on properties of the source electronic data file.

3. The system of claim 1, wherein the operations further include receiving an assignment of the transformation template to the source electronic data file via the interactive mapping graphical user interface.

4. The system of claim 1, wherein the detecting of the file type includes determining file type identifier associated with the source electronic data file, the file type identifier including at least one of a structure of the source electronic data file, a combination of file headers included in the source electronic data file, and an order of file headers included in the source electronic data file.

5. The system of claim 1, wherein the detecting of the file type includes determining a file type identifier associated with the source electronic data file, the file type identifier including at least one of a string of text included in the source electronic data file and a type of data included in the source electronic data file.

6. The system of claim 1, wherein the detecting of the file type identifier associated with the source electronic data file includes receiving an identification of the file type identifier associated with the source electronic data file as user input via the interactive mapping graphical user interface.

7. The system of claim 1, wherein the generating of the transformed source electronic data file includes conforming a format of the data fields included in the source electronic data file to a format of the canonical data fields included in the transformation template.

8. The system of claim 1, the operations further comprising:

receiving an up date as an up dated electronic source data file;

loading a detector/transformer plugin from a detector/transformer plugin database based on a file type of the up dated electronic source data file;

using the detector/transformer plugin to generate an updated transformed electronic data file based on data included in the update and replace the transformed electronic data file with the updated transformed electronic data file.

9. The system of claim 8, the operations further comprising selecting the detector/transformer plugin from a plurality of detector/transformer plugins based on a priority of the detector/transformer plugin being higher than priorities of other detector/transformer plugins of the plurality of detector/transformer plugins.

10. The system of claim 1, wherein the data fields in the source electronic data file represent bank wire transfer data and the combined data fields include a first field for a date and a second field for a time.

11. The system of claim 1, wherein the canonical field defines an arrangement of content in the canonical field and the interactive mapping graphical user interface includes elements for mapping an arrangement of content in the combined data field with the arrangement of content in the canonical field.

12. A method comprising:

incorporating one or more instructions into one or more memory devices, the one or more instructions configuring one or more computer processors to perform operations for generating a transformed source electronic data file from a source electronic data file based on inputs received from an interactive mapping graphical user interface presented on a client device, the interactive mapping graphical user interface facilitating a mapping a format of the source electronic data file to a canonical format included in a transformation template, the operations comprising;

detecting that a file type associated with the source electronic data file is an unknown file type;

providing for a selection of the transformation template via the interactive mapping graphical user interface for associating with the file type;

generating the interactive mapping graphical user interface for presentation on the client device, the interactive mapping graphical user interface including interactive user interface elements for combining data fields in the source electronic data file and mapping the combined data fields to a canonical data field included in the transformation template;

generating the transformed source electronic data file from the source electronic data file based on mappings received via the interactive mapping graphical user interface presented on the client device; and

storing the mappings as a file type profile in a database for use in automatically transforming additional source electronic data files of the file type without user intervention.

13. The method of claim 12, wherein the detecting of the file type is based on properties of the source electronic data file.

14. The method of claim 12, wherein the operations further include receiving an assignment of the transformation template to the source electronic data file via the interactive mapping graphical user interface.

15. The method of claim 12, wherein the detecting of the file type includes determining file type identifier associated with the source electronic data file, the file type identifier including at least one of a structure of the source electronic data file, a combination of file headers included in the source electronic data file, and an order of file headers included in the source electronic data file.

16. The method of claim 12, wherein the detecting of the file type includes determining a file type identifier associated with the source electronic data file, the file type identifier including at least one of a string of text included in the source electronic data file and a type of data included in the source electronic data file.

17. The method of claim 12, wherein the detecting of the file type identifier associated with the source electronic data file includes receiving an identification of the file type identifier associated with the source electronic data file as user input via the interactive mapping graphical user interface.

18. The method of claim 12, wherein the generating of the transformed source electronic data file includes conforming a format of the data fields included in the source electronic data file to a format of the canonical data fields included in the transformation template.

19. The system of claim 12, wherein the generating of the transformed source electronic data file includes conforming data included in the source electronic data file to canonical data included in the transformation template.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor of a machine, cause the machine to perform operations for generating a transformed source electronic data file from a source electronic data file based on inputs received from an interactive mapping graphical user interface presented on a client device, the interactive mapping graphical user interface facilitating a mapping a format of the source electronic data file to a canonical format included in a transformation template, the operations comprising:

detecting a file type associated with the source electronic data file;

providing for a selection of the transformation template via the interactive mapping graphical user interface for associating with the file type;

generating the interactive mapping graphical user interface for presentation on the client device, the interactive mapping graphical user interface including interactive user interface elements for combining data fields in the source electronic data file and mapping the combined data fields to a canonical data field included in the transformation template;

generating the transformed source electronic data file from the source electronic data file based on mappings received via the interactive mapping graphical user interface presented on the client device; and

storing the mappings as a file type profile in a database for use in automatically transforming additional source electronic data files of the file type without user intervention.