System and method for flexible data transfer

• US 8,918,524 B2
• Filed: 11/13/2012
• Issued: 12/23/2014
• Est. Priority Date: 07/30/2004
• Status: Expired due to Fees

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Clark, Nicholas James, Woods, William S.
• Primary Examiner(s): WILLIAMS, JENEE LAUREN
• Application Number:
  US13/675,824
• Publication Number:
  US 20130074079A1
• Time in Patent Office: 770 Days
• Field of Search: 709/203, 709217-219, 709/228, 709/230, 709/232, 709/246, 707600-602
• US Class Current: 709/228
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99954 Version management

Abstract

A method and system for flexibly transferring data from one or more data sources to one or more data destinations within an information network where each of the one or more data sources have data in a particular source format and each of the one or more data destinations have data in the same or another particular destination format using a parameter database that includes parameters to control the transfer of data, a scheduler that initiates the transfer of data, and a data loader in communications with the parameter database and scheduler that, upon initiation by the scheduler, extracts data from the one or more data sources, manipulates the extracted source data into one or more destination formats associated with the one or more data destinations, and inserts the data into one or more data destinations according to the parameters within the parameter database.

Claims

1. A data loading method, comprising:

assigning, by a data loader, different batch numbers to different groups of data loading operations;

launching concurrently a first group of threads corresponding to a first group of data loading operations having a first batch number;

awaiting completion of the first group of threads; and

responsive to detecting completion of the first group of threads, launching concurrently a second group of threads corresponding to a second group of data loading operations having a second batch number;

wherein launching the first group of threads includes launching a plurality of threads concurrently wherein each of the plurality of threads is associated with a corresponding one of the first group of data loading operations.

2. The method of claim 1, further comprising:

storing the different batch numbers assigned to the different groups of data loading operations in a parameter database; and

accessing the batch numbers stored in the parameter database to identify the first group of data loading operations.

3. The method of claim 1, wherein each group of data loading operations includes operations for:

extracting source information from a data source;

manipulating the source information into destination information having a destination format; and

inserting the destination information into a data destination.

4. The method of claim 3, wherein manipulating the source information includes invoking plug-ins to manipulate the source information.

5. The method of claim 4, wherein invoking the plug-ins includes determining a name and location of a plug-in from a parameter database.

6. The method of claim 3, wherein manipulating the source information includes converting source information content based on at least one of a computation, an algorithm, and a formula.

7. A system, comprising:

a processor;

a computer readable medium, accessible to the processor, including stored program instructions that, when executed by the processor, perform program operations including;

assigning, by a data loader, different batch numbers to different groups of data loading operations;

launching concurrently a first group of threads corresponding to a first group of data loading operations having a first batch number;

awaiting completion of the first group of threads; and

responsive to detecting completion of the first group of threads, launching concurrently a second group of threads corresponding to a second group of data loading operations having a second batch number;

wherein launching the first group of threads includes launching a plurality of threads concurrently wherein each of the plurality of threads is associated with a corresponding one of the first group of data loading operations.

8. The system of claim 7, wherein the program operations include:

storing the different batch numbers assigned to the different groups of data loading operations in a parameter database; and

accessing the batch numbers stored in the parameter database to identify the first group of data loading operations.

9. The system of claim 7, wherein each group of data loading operations includes operations for:

extracting source information from a data source;

manipulating the source information into destination information having a destination format; and

inserting the destination information into a data destination.

10. The system of claim 9, wherein manipulating the source information includes invoking plug-ins to manipulate the source information.

11. The system of claim 10, wherein invoking the plug-ins includes determining a name and location of a plug-in from a parameter database.

12. The system of claim 9, wherein manipulating the source information includes converting source information content based on at least one of a computation, an algorithm, and a formula.

13. A non-transitory computer readable medium including processor executable program instructions that, when executed by a processor, perform operations, comprising:

assigning, by a data loader, different batch numbers to different groups of data loading operations;

launching concurrently a first group of threads corresponding to a first group of data loading operations having a first batch number;

awaiting completion of the first group of threads; and

responsive to detecting completion of the first group of threads, launching concurrently a second group of threads corresponding to a second group of data loading operations having a second batch number;

wherein launching the first group of threads includes launching a plurality of threads concurrently wherein each of the plurality of threads is associated with a corresponding one of the first group of data loading operations.

14. The non-transitory computer readable medium of claim 13, wherein the program operations include:

storing the different batch numbers assigned to the different groups of data loading operations in a parameter database; and

accessing the batch numbers stored in the parameter database to identify the first group of data loading operations.

15. The non-transitory computer readable medium of claim 13, wherein each group of data loading operations includes operations for:

extracting source information from a data source;

manipulating the source information into destination information having a destination format; and

inserting the destination information into a data destination.

16. The non-transitory computer readable medium of claim 15, wherein manipulating the source information includes invoking plug-ins to manipulate the source information and wherein invoking the plug-ins includes determining a name and location of a plug-in from a parameter database.

17. The non-transitory computer readable medium of claim 15, wherein manipulating the source information includes converting source information content based on at least one of a computation, an algorithm, and a formula.