DATA LOADING SYSTEMS AND METHODS

• US 20110302226A1
• Filed: 02/22/2011
• Published: 12/08/2011
• Est. Priority Date: 06/04/2010
• Status: Active Grant

• Current Assignee: Yale University
• Original Assignee: Yale University
• Inventors: Abadi, Daniel, Abouzied, Azza
• Granted Patent: US 9,336,263 B2
• Time in Patent Office: Days
• US Class Current: 707/825
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

System, method, and computer program product for processing data are disclosed. The system is configured to perform transfer of data from a file system to a database system. Such transfer is accomplished through receiving a request for loading data into a database system, wherein the data includes a plurality of attributes, determining at least one attribute of the data for loading into the database system, and loading the at least one attribute of the data into the database system while continuing to process remaining attributes of the data.

Claims

1. A method for processing and transferring data from a file system to a database system, the method comprising the steps of:

receiving a request for loading data into a database system, wherein the data includes a plurality of attributes;

determining, based on the request, at least one attribute of the data for loading into the database system; and

loading the at least one attribute of the data into the database system while continuing to process remaining attributes of the data.

2. The method according to claim 1, wherein the receiving further comprises parsing the attributes of the data as a result of the received request.

3. The method according to claim 2, further comprising

incrementally loading, based on the request, each attribute of the data into the database system as soon as each parsed attributed is processed.

4. The method according to claim 3, further comprising

indexing each processed parsed attribute in the database system.

5. The method according to claim 4, wherein the indexing of the attributes is configured to be performed incrementally as soon as each processed parsed attributed is loaded into the database system.

6. The method according to claim 5, further comprising

accessing the received data loaded into the database system;

determining attributes of the data that have not been loaded into the database system;

parsing the attributes that have not been loaded into the database system;

processing the parsed unloaded attributes to determine at least one unloaded attribute for loading into the database system; and

loading the at least one processed parsed unloaded attribute of the data into the database system while continuing to process the remaining unloaded attributes of the data.

7. The method according to claim 6, further comprising

indexing each loaded processed parsed unloaded attribute in the database system.

8. The method according to claim 1, wherein the loading further includes loading the attributes on the column-store basis, whereby each different column of the received data is loaded independently into the database system.

9. The method according to claim 1, wherein the loading is selected from a group consisting of:

direct loading, wherein parsed attributes of the received data are loaded into the database system as soon as the attributes are processed, and

delayed loading, wherein parsed attributes of the received data are initially temporarily stored in a temporary storage and then loaded into the database system.

10. The method according to claim 1, wherein the loading further comprises

(a) dividing a column of loaded attributes into a plurality of portions;

(b) sorting the plurality of portions to obtain a first order of attributes within the column of loaded attributes;

(c) dividing the plurality of sorted portions of attributes into further plurality of portions;

(d) sorting the further plurality of portions to obtain a second order of attributes within the column of loaded attributes;

(e) repeating steps (c)-(d) to obtain a final order of attributes of the data.

11. A data processing system for transferring data having a plurality of attributes from a file system to a database system, the system comprising:

a processor configured to

receive a request for loading data into a database system, wherein the data includes a plurality of attributes;

determine, based on the request, at least one attribute of the data for loading into the database system; and

a data loader module configured to load the at least one attribute of the data into the database system while the processor continues to process remaining attributes of the data.

12. The system according to claim 11, further comprising

a parsing module configured to parse the attributes of the data as a result of the request.

13. The system according to claim 12, wherein each parsed attribute of the data is configured to be incrementally loaded, based on the request, into the database system as soon as each parsed attribute is processed.

14. The system according to claim 13, further comprising

an indexing module configured to index each processed parsed attribute in the database system.

15. The system according to claim 14, wherein the indexing of the attributes is configured to be performed incrementally as soon as each processed parsed attribute is loaded into the database system.

16. The system according to claim 15, wherein upon accessing the received data loaded into the database system, attributes of the data that have not been loaded into the database system are determined;

wherein

the parsing module is configured to parse the attributes that have not been loaded into the database system;

the processor is configured to process the parsed unloaded attributes to determine at least one unloaded attribute for loading into the database system; and

the data loader is configured to load the at least one processed parsed unloaded attribute of the data into the database system while continuing to process the remaining unloaded attributes of the data.

17. The system according to claim 16, wherein each loaded processed parsed unloaded attribute is configured to be indexed in the database system.

18. The system according to claim 11, wherein the loading further includes loading the attributes on the column-store basis, whereby each different column of the received data is loaded independently into the database system.

19. The system according to claim 11, wherein the data loader is configured to perform loading selected from a group consisting of:

direct loading, wherein parsed attributes of the received data are loaded into the database system as soon as the attributes are processed, and

delayed loading, wherein parsed attributes of the received data are initially temporarily stored in a temporary storage and then loaded into the database system.

20. The system according to claim 11, wherein the data loader is configured to

(a) divide a column of loaded attributes into a plurality of portions;

(b) sort the plurality of portions to obtain a first order of attributes within the column of loaded attributes;

(c) divide the plurality of sorted portions of attributes into further plurality of portions;

(d) sort the further plurality of portions to obtain a second order of attributes within the column of loaded attributes;

(e) repeat (c)-(d) to obtain a final order of attributes of the data.

21. A computer program product, tangibly embodied in a computer-readable medium, the computer program product being operable to cause a data processing system for transferring data from a file system to a database system, to perform operations comprising:

receiving a request for loading data into a database system, wherein he data includes a plurality of attributes;

determining at least one attribute of the data for loading into the database system; and

loading the at least one attribute of the data into the database system while continuing to process remaining attributes of the data.