Data extraction, transformation, and loading

• US 8,751,438 B2
• Filed: 04/13/2012
• Issued: 06/10/2014
• Est. Priority Date: 04/13/2012
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Kharod, Manish G., Sharma, Nityanand, Chatterjee, Sutap, John, Josy, Ratnam, Kopinath, Premkumar, Hrishikesh, Bhathivi, Ganesh R., Gudlavenkatasiva, Bhaskar R., Hussain, Naseemunnisa, Fariz, Salman
• Primary Examiner(s): LE, HUNG D
• Application Number:
  US13/446,604
• Publication Number:
  US 20130275360A1
• Time in Patent Office: 788 Days
• Field of Search: None
• US Class Current: 707/602
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A device receives instructions for extracting, transforming, and loading data; identifies a plurality of data processing activities based on the instructions; and determines a processing sequence for performing the plurality of data processing activities. The device further creates a plurality of processing objects corresponding to the plurality of data processing activities. Each one of the plurality of processing objects including a discrete data structure that is produced by performing a corresponding one of the plurality of data processing activities. The device performs the plurality of data processing activities in accordance with the processing sequence and by using the plurality of processing objects.

Claims

1. A method comprising:

receiving, by a computing device, instructions for extracting, transforming, and loading data;

mapping, by the computing device, the instructions to a plurality of classes that are in one or more types of object-oriented programming languages;

creating, by the computing device, a plurality of processing objects based on the plurality of classes and the instructions; and

performing, by the computing device, a plurality of data processing activities by using the plurality of processing objects.

2. The method of claim 1, where receiving the instructions includes:

receiving an Extensible Markup Language (XML) file that includes the instructions.

3. The method of claim 1, where performing the plurality of data processing activities includes:

extracting the data from one or more data sources,

sorting the data,

merging the data with other data to produce merged data,

deleting duplicate data in the merged data to produce processed data, and

providing the processed data to a destination device.

4. The method of claim 1, further comprising:

identifying processing dependencies between two or more of the plurality of data processing activities,

the processing dependencies indicating that a first activity of the two or more of the plurality of data processing activities is dependent on a second activity of the two or more of the plurality of data processing activities, and

determining a processing sequence based on the processing dependencies,

performing the plurality of data processing activities including;

performing the plurality of data processing activities in accordance with the processing sequence and by using the plurality of processing objects.

5. The method of claim 1, where creating the plurality of processing objects comprises:

annotating the plurality of classes to refer to the instructions that are mapped to the plurality of classes, and

creating the plurality of processing objects based on the plurality of classes and the instructions.

6. The method of claim 1, where performing the plurality of data processing activities includes:

receiving or retrieving data records that include portions of the data,

loading the data records into an in-memory data structure that is stored in a memory of the computing device,

sorting the data records in the memory of the computing device, and

outputting the sorted data records to another device.

7. The method of claim 6, where sorting the data records includes:

determining whether all of the data records, retrieved from a particular data source, have been loaded into the in-memory data structure, and

sorting the data records when all of the data records have been loaded into the in-memory data structure.

8. The method of claim 1, where performing the plurality of data processing activities includes:

adding data records, that include a portion of the data, to a batch,

providing the batch to a destination device,

determining whether an error has occurred while providing the batch to the destination device, and

providing the data records, from the batch, individually to the destination device when the error has occurred.

9. The method of claim 8, where adding the data records to the batch includes:

receiving a particular number from an operator of the computing device,

determining whether a quantity of the data records in the batch is less than the particular number, and

continuing to add more of the data records to the batch when the quantity of the data records in the batch is less than the particular number.

10. The method of claim 8, where providing the data records individually includes:

providing a particular data record, of the data records, to the destination device,

determining whether another error has occurred that is associated with the particular data record,

creating an error report that identifies the particular data record when the other error has occurred, and

transmitting the error report to an operator of the computing device.

11. The method of claim 8, where the error comprises one of:

a conflict between a first data record, of the data records, and a second data record that is stored by the destination device,

a failure associated with establishing a connection with the destination device, or

a failure associated with maintaining the connection with destination device.

12. A system comprising:

one or more processors to;

receive instructions for handling data from a plurality of data sources,

map each of the instructions to one of a plurality of classes that are in one or more types of object-oriented programming languages,

annotate the plurality of classes to refer to the instructions that are mapped to the plurality of classes,

create the plurality of processing objects based on the instructions and the plurality of annotated classes, and

perform, by using the plurality of processing objects, a plurality of data processing activities to process the data.

13. The system of claim 12, where, when performing the plurality of activities, the one or more processors are to:

identify the plurality of data processing activities based on the instructions,

identify processing dependencies between two or more of the plurality of data processing activities,

determine a processing sequence based on the processing dependencies, and

perform the plurality of data processing activities in accordance with the processing sequence.

14. The system of claim 12, where, when performing the plurality of data processing activities, the one or more processors are to:

receive the data from the plurality of data sources,

load the data into an in-memory data structure that is stored in a memory of the system,

sort the data in the memory, and

output the sorted data to another device.

15. The system of claim 12, where, when performing the plurality of data processing activities, the one or more processors are to:

create a batch based on data records that include a portion of the data,

determine that an error has occurred while providing the batch to a destination device, and

provide each of the data records individually to the destination device after determining that the error has occurred.

16. The system of claim 15, where, when providing each of the data records individually to the destination device, the one or more processors are to:

determine whether a processing error has occurred that is associated with a particular data record of the data records,

generate an error report that identifies the particular data record when the other error has occurred, and

provide the error report to a client device associated with the system.

17. A non-transitory computer-readable medium storing instructions, the instructions comprising:

one or more instructions that, when executed by one or more processors of a computer device, cause the one or more processors to;

receive instructions associated with extracting, transforming, and loading data;

identify a plurality of data processing activities based on the instructions;

determine a processing sequence for performing the plurality of data processing activities;

associate the instructions with a plurality of classes that are in one or more types of object-oriented programming languages;

create, based on the plurality of classes and the instructions, a plurality of processing objects corresponding to the plurality of data processing activities; and

perform the plurality of data processing activities in accordance with the processing sequence and by using the plurality of processing objects.

18. The non-transitory computer-readable medium of claim 17, where the instructions further comprise:

one or more instructions that, when executed by the one or more processors of the computer device, cause the one or more processors to;

receive data records that include portions of the data,

load the data records into an in-memory data structure that is stored in a memory of the computer device,

sort the data records in the memory of the computer device, and

output the sorted data records to another device.

19. The non-transitory computer-readable medium of claim 17, where the instructions further comprise:

one or more instructions that, when executed by the one or more processors of the computer device, cause the one or more processors to;

add data records to a batch,

the data records including a portion of the batch,

determine that a batch error has occurred while providing the batch to a destination device,

the batch error including one of;

a conflict between one of the data records and another data record stored by the destination device, or

a failure to establish or maintain a connection with the destination device, and

provide the data records, from the batch, individually to the destination device when the batch error has occurred.

20. The non-transitory computer-readable medium of claim 17, where the instructions further comprise:

one or more instructions that, when executed by the one or more processors of the computer device, cause the one or more processors to;

annotate, before creating the plurality of processing objects, the plurality of classes to refer to the instructions that are associated with the plurality of classes.