EFFICIENT FAILURE DETECTION FOR LONG RUNNING DATA TRANSFER JOBS

• US 20110271145A1
• Filed: 04/30/2010
• Published: 11/03/2011
• Est. Priority Date: 04/30/2010
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Silberstein, Adam E.
• Granted Patent: US 8,276,022 B2
• Time in Patent Office: Days
• US Class Current: 714/18
• CPC Class Codes: G06F 11/1443 Transmit or communication errors

Abstract

Disclosed are methods and apparatus for error handling within jobs that utilize a plurality of tasks for data transfer of individual data records to a storage destination. For each task, one or more failed records may be logged to a file for later insertion. If a high percentage of a task'"'"'s output (e.g., writes to another data storage system) is determined to be failing, the task short-circuits itself. Each task is also configured to perform checkpoint logging as the task completes work. If the entire job later short-circuits and is to be restarted, the restarted job only repeats a minimal amount of previously completed work for the tasks which have not already completed their data insertions. Together, these techniques can ensure that in the face of periodic failures, the job completes long-running job in a minimal time with minimal effects.

Claims

1. A computer implemented method of handling errors during a data transfer, comprising:

(a) for a first task that is configured to transfer a plurality of data records from a source to a destination storage system and when a specific record of such first task fails to be transferred to the destination storage system, causing the first task to retry transferring of the specific record to the destination storage system so that such retry is only performed a predefined number of times; and

(b) when the first task has been caused to retry transferring of a specific record of the first task more than the predefined number of times, storing the specific record in an error log for a later transfer attempt.

2. The method as recited in claim 1, wherein operations (a) and (b) are repeated for each of a plurality of substantially identical tasks that are configured by a job for transferring data record sets to the destination system, wherein each task is configured to transfer an individual, different data record set.

3. The method as recited in claim 2, wherein the job is a restart of such job, and the method further comprises:

for each task, determining whether the task is identified as completing; and

for each task, if it is determined that the task is identified as completing, stopping the task so as to inhibit data transfer for such task and performance of operations (a) and (b) for the task.

4. The method as recited in claim 3, further comprising:

when each task ends, using the failed records in the error log as input to one or more of the tasks after a predetermined time period so that the transfer of the failed records is retried.

5. The method as recited in claim 4, wherein using the failed records in the error log is accomplished by apportioning each of the failed records to a specific one of the tasks in which such failed record previously failed.

6. The method of claim 4, wherein the predetermined time period is an average recovery time for a temporary record failure.

7. The method as recited in claim 1, further comprising:

periodically determining whether the first task'"'"'s failure rate is above a predetermined threshold; and

if the first task is determined to be above the predetermined threshold, short-circuiting the first task by inhibiting further transfer by the first task and identifying the first task as being short-circuited.

8. An apparatus comprising at least a processor and a memory, wherein the processor and/or memory are configured to perform the following operations:

(a) for a first task that is configured to transfer a plurality of data records from a source to a destination storage system and when a specific record of such first task fails to be transferred to the destination storage system, causing the first task to retry transferring of the specific record to the destination storage system so that such retry is only performed a predefined number of times; and

(b) when the first task has been caused to retry transferring of a specific record of the first task more than the predefined number of times, storing the specific record in an error log for a later transfer attempt.

9. The apparatus as recited in claim 8, wherein operations (a) and (b) are repeated for each of a plurality of substantially identical tasks that are configured by a job for transferring data record sets to the destination system, wherein each task is configured to transfer an individual, different data record set.

10. The apparatus as recited in claim 9, wherein the job is a restart of such job, and the processor and/or memory are further configured:

for each task, determining whether the task is identified as completing; and

for each task, if it is determined that the task is identified as completing, stopping the task so as to inhibit data transfer for such task and performance of operations (a) and (b) for the task.

11. The apparatus as recited in claim 10, wherein the processor and/or memory are further configured:

when each task ends, using the failed records in the error log as input to one or more of the tasks after a predetermined time period so that the transfer of the failed records is retried.

12. The apparatus as recited in claim 11, wherein using the failed records in the error log is accomplished by apportioning each of the failed records to a specific one of the tasks in which such failed record previously failed.

13. The apparatus of claim 11, wherein the predetermined time period is an average recovery time for a temporary record failure.

14. The apparatus as recited in claim 8, wherein the processor and/or memory are further configured:

periodically determining whether the first task'"'"'s failure rate is above a predetermined threshold; and

if the first task is determined to be above the predetermined threshold, short-circuiting the first task by inhibiting further transfer by the first task and identifying the first task as being short-circuited.

15. A computer program product for partitioning a native table in a database, comprising at least one computer-readable medium having computer instructions stored therein which are operable to cause a computer device to perform the following operations:

(a) for a first task that is configured to transfer a plurality of data records from a source to a destination storage system and when a specific record of such first task fails to be transferred to the destination storage system, causing the first task to retry transferring of the specific record to the destination storage system so that such retry is only performed a predefined number of times; and

(b) when the first task has been caused to retry transferring of a specific record of the first task more than the predefined number of times, storing the specific record in an error log for a later transfer attempt.

16. The computer program product as recited in claim 15, wherein operations (a) and (b) are repeated for each of a plurality of substantially identical tasks that are configured by a job for transferring data record sets to the destination system, wherein each task is configured to transfer an individual, different data record set.

17. The computer program product as recited in claim 16, wherein the job is a restart of such job, and wherein the computer instructions are further operable to cause a computer device to perform the following operations:

for each task, determining whether the task is identified as completing and

for each task, if it is determined that the task is identified as completing, stopping the task so as to inhibit data transfer for such task and performance of operations (a) and (b) for the task.

18. The computer program product as recited in claim 17, wherein the computer instructions are further operable to cause a computer device to perform the following operations:

when each task ends, using the failed records in the error log as input to one or more of the tasks after a predetermined time period so that the transfer of the failed records is retried.

19. The computer program product as recited in claim 18, wherein using the failed records in the error log is accomplished by apportioning each of the failed records to a specific one of the tasks in which such failed record previously failed.

20. The computer program product of claim 18, wherein the predetermined time period is an average recovery time for a temporary record failure.

21. The computer program product as recited in claim 15, wherein the computer instructions are further operable to cause a computer device to perform the following operations:

periodically determining whether the first task'"'"'s failure rate is above a predetermined threshold; and

if the first task is determined to be above the predetermined threshold, short-circuiting the first task by inhibiting further transfer by the first task and identifying the first task as being short-circuited.