System and method for error handling and recovery

• US 6,687,846 B1
• Filed: 03/30/2000
• Issued: 02/03/2004
• Est. Priority Date: 03/30/2000
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Adrangi, Farid, Sloss, Reed J., Menon, Rama R., Gaddis, David W.
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): DUNCAN, MARC M
• Application Number:
  US09/539,271
• Time in Patent Office: 1,405 Days
• Field of Search: 714/4, 714/17, 714/704, 714/748, 714/774, 714/15, 714/1, 714/2, 714/29, 714/37, 714/43, 714/42, 714/48, 714/49
• US Class Current: 714/4.4
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage

Abstract

A system and method for error handling and recovery in a content distribution system is described in which errors corresponding to failed file operations (e.g., file transfer errors, file delete errors) are placed in an error queue. Error analysis logic reads the errors from the error queue and makes a determination as to whether the file operation errors are recoverable errors based on an error recovery policy. If the error analysis logic determines that recovery is possible, then one or more error recovery procedures are invoked. The procedures may be specific to the content delivery system (e.g., “Server X was down on 1/20 between 10:20 and 11:00 AM”), or may be more general (e.g., “attempt file transfers 5 times before quitting”). If it is determined that an error is not automatically recoverable, then the error is included in an error report.

Claims

1. An error recovery method comprising:

logging one or more file operation errors in an error queue in a content distribution network, said file operation errors including a file operation portion and an error code portion;

periodically reading said file operation errors from said error queue;

determining whether automatic error recovery is possible based on an error recovery policy;

performing an automated error recovery procedure if error recovery is possible;

wherein said error recovery policy includes information as to when specified portions of said network were inoperative, information as to when particular file servers were inoperative; and

information as to whether a file associated with said file operation error was not available on a specified source server.

2. The method as in claim 1 further comprising:

generating a report if error recovery is not possible.

3. The method as in claim 1 wherein said file operation errors comprise file transfer errors.

4. The method as in claim 1 wherein said file operation errors are file delete errors.

5. The method as in claim 1 wherein one of said error recovery procedures comprises:

reattempting file operations corresponding to said file operation errors if said file operations were previously attempted a number of times less than a predetermined threshold value.

6. The method as in claim 1 wherein one of said error recovery procedures comprises:

determining whether a group of said file operation errors have identical error causes over a finite period of time; and

reattempting file operations corresponding to said group of file operation errors.

7. An article of manufacture including a sequence of instructions which, when executed on a processor, cause the processor to:

log one or more file operation errors in an error queue in a content distribution network, said file operation errors including a file operation portion and an error code portion;

read said file operation errors from said error queue;

determine whether automatic error recovery is possible based on an error recovery policy; and

perform an automated error recovery procedure if error recovery is possible, wherein said error recovery policy includes information as to whether a file associated with said file operation error was not available on a specified source server.

8. The article of manufacture as in claim 7 including further instructions which cause said processor to:

generate a report if error recovery is not possible.

9. The article of manufacture as in claim 7 wherein said error recovery policy includes information as to when specified portions of said network were inoperative.

10. The article of manufacture as in claim 7 wherein said error recovery policy includes information as to when particular file servers were inoperative.

11. The article of manufacture as in claim 7 including further instructions defining an error recovery procedure which cause said processor to:

reattempting file operations corresponding to said file operation errors if said file operations were previously attempted a number of times less than a predetermined threshold value.

12. The article of manufacture as in claim 7 including further instructions defining an error recovery procedure which cause said processor to:

determining whether a group of said file operation errors have identical error causes over a finite period of time; and

reattempting file operations corresponding to said group of file operation errors.