Storage system with automatic redundant code component failure detection, notification, and repair

• US 7,711,989 B2
• Filed: 04/11/2006
• Issued: 05/04/2010
• Est. Priority Date: 04/01/2005
• Status: Expired due to Fees

• Current Assignee: Dot Hill Systems Corporation (Seagate Technology Holdings Plc)
• Original Assignee: Dot Hill Systems Corporation (Seagate Technology Holdings Plc)
• Inventors: Lintz, Dwight Oliver Jr., Wang, Yuanru Frank
• Primary Examiner(s): Baderman; Scott T
• Assistant Examiner(s): KO, CHAE M
• Application Number:
  US11/279,376
• Publication Number:
  US 20060236198A1
• Time in Patent Office: 1,484 Days
• Field of Search: 714/6, 714/758, 714/15, 714/36, 713/1, 713/2
• US Class Current: 714/36
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/1417 Boot up procedures

Abstract

A RAID system includes a non-volatile memory storing a first program and first and second copies of a second program, and a processor executing the first program. The first program detects the first copy of the second program is failed and repairs the failed first copy in the non-volatile memory using the second copy. The failures may be detected at boot time or during normal operation of the controller. In one embodiment, the failure is detected via a CRC check. In one embodiment, the controller repairs the failed copy by copying the good copy to the location of the failed copy. In one embodiment, the system includes multiple controllers, each having its own processor and non-volatile memory and program that detects and repairs failed program copies. The programs include a loader, an application, FPGA code, CPLD code, and a program for execution by a power supply microcontroller.

Claims

1. A RAID system, comprising:

a non-volatile memory, configured to store a first program and first and second versions of a second program and a third program, wherein the first and second versions of the second program are different;

a volatile memory;

a watch dog timer, for detecting a failure during a boot process of the RAID system, the watch dog timer having a predetermined maximum timeout period, wherein the boot process is normally longer than said predetermined maximum timeout period of said watch dog timer; and

a processor, coupled to said non-volatile memory and to said volatile memory and to said watch dog timer, configured to execute said first program, wherein said first program is configured to;

detect said first version of said second program is failed; and

repair said failed first version of said second program in said non-volatile memory using said second version of said second program;

wherein said second program comprises an application program for performing RAID control functions;

wherein said third program is configured to decompress said first program stored in said non-volatile memory to a decompressed form and to write said decompressed form to said volatile memory during the boot process, wherein said third program is further configured to disable the watch dog timer after writing said decompressed form of said first program to said volatile memory, wherein said third program is further configured to re-enable the watch dog timer prior to said processor executing said first program.

2. The RAID system as recited in claim 1, wherein said non-volatile memory comprises a FLASH memory.

3. The RAID system as recited in claim 1, further comprising:

a second non-volatile memory, coupled to said processor;

wherein said second version of said second program is stored in said second non-volatile memory rather than said first non-volatile memory, wherein said first program is configured to repair said failed first version of said second program in said first non-volatile memory using said second version of said second program in said second non-volatile memory.

4. The RAID system as recited in claim 1, wherein said first program is configured to detect said first version of said second program is failed by generating a first cyclic redundancy code (CRC) value of said first version of said second program and determining that said first CRC value mismatches a second CRC value of said first version of said second program.

5. The RAID system as recited in claim 4, wherein said second CRC value is previously generated and stored in said non-volatile memory when said first version of said second program is programmed into said non-volatile memory.

6. The RAID system as recited in claim 1, further comprising:

a non-volatile indication of whether said first version of said second program failed during initialization of the RAID controller, coupled to said processor;

wherein said first program is configured to detect said first version of said second program is failed by examining said non-volatile indication.

7. The RAID system as recited in claim 6, further comprising:

a watchdog timer, coupled to said non-volatile indication, configured to populate said non-volatile indication based on whether said first version of said second program failed to boot within a predetermined time during initialization of the RAID controller, wherein said watchdog timer resets the processor after populating the non-volatile indication.

8. The RAID system as recited in claim 1, wherein said second program comprises a loader program for loading an application program for execution by the processor.

9. The RAID system as recited in claim 8, further comprising:

a volatile memory, coupled to said processor, wherein said loader program is configured to load said application program from said non-volatile memory into said volatile memory for execution by the processor.

10. The RAID system as recited in claim 9, wherein said first program is further configured to detect said application program in said volatile memory is failed by performing a CRC check of said application program in said volatile memory.

11. The RAID system as recited in claim 10, wherein said first program is further configured to reboot said processor in response to detecting said application program in said volatile memory is failed.

12. The RAID system as recited in claim 11, wherein said first program is further configured to failover to a redundant controller of the RAID system prior to rebooting said processor.

13. The RAID system as recited in claim 1, wherein said second program comprises an application program for managing a RAID controller.

14. The RAID system as recited in claim 1, wherein said second program comprises an application program for monitoring and controlling an enclosure of the RAID system.

15. The RAID system as recited in claim 1, further comprising:

a field-programmable gate array (FPGA), coupled to said processor, wherein said second program comprises code for programming said FPGA.

16. The RAID system as recited in claim 1, further comprising:

a complex programmable logic device (CPLD), coupled to said processor, wherein said second program comprises code for programming said CPLD.

17. The RAID system as recited in claim 1, further comprising:

a power supply, coupled to provide power to said processor and said non-volatile memory, said power supply comprising a processor for executing said second program.

18. The RAID system as recited in claim 1, wherein said first version of said second program is at a first location in said non-volatile memory and said second version of said second program is at a second location in said non-volatile memory, wherein said first program is configured to copy said second version of said second program from said second location to said first location in said non-volatile memory to repair said failed first version of said second program in said non-volatile memory.

19. The RAID system as recited in claim 18, wherein said first program copies said second version of said second program from said second location to said first location in said non-volatile memory by reading said second version of said second program from said second location in said non-volatile memory and writing said second version of said second program read from said second location to said first location in said non-volatile memory.

20. The RAID system as recited in claim 18, further comprising:

a volatile memory, coupled to said processor;

wherein said first program copies said second version of said second program from said second location to said first location in said non-volatile memory via a temporary location in said volatile memory, wherein no modification is performed on said second version of said second program in said temporary location.

21. The RAID system as recited in claim 1, wherein said first program is further configured to notify a user that said first version of said second program is failed.

22. The RAID system as recited in claim 1, wherein said first program is further configured to update a count of a number of failures of said first version of said second program in response to detecting said first version of said second program is failed.

23. The RAID system as recited in claim 22, wherein said first program is further configured to notify a user that said first version of said second program is failed if said count exceeds a user-programmable threshold.

24. The RAID system as recited in claim 1, wherein said first program is further configured to log in said non-volatile memory an event specifying that said first version of said second program is failed.

25. A method for improving the data availability characteristics of a RAID system, comprising:

executing a first program on a processor of the RAID system;

detecting, by the first program, that a first version of a second program is failed, wherein said first version of said second program is stored in a non-volatile memory of the RAID system;

repairing, by the first program, said failed first version of said second program in said non-volatile memory using a second version of said second program stored in said non-volatile memory, wherein the first and second versions of the second program are different;

decompressing the first program to a decompressed form and writing the decompressed form to a volatile memory of the RAID system during a boot process of the RAID system; and

disabling a watch dog timer of the RAID system after writing the decompressed form of the first program to the volatile memory, wherein the watch dog timer is configured to detect a failure during the boot process, wherein the watch dog timer has a predetermined maximum timeout period, wherein the boot process is normally longer than the predetermined maximum timeout period of the watch dog timer; and

re-enabling the watch dog timer of the RAID system prior to executing the first program on the processor;

wherein said decompressing, said writing, and said disabling and then re-enabling are performed by a third program stored in the non-volatile memory;

wherein said second program comprises an application program for performing RAID control functions.

26. The method as recited in claim 25, wherein said non-volatile memory comprises a FLASH memory.

27. The method as recited in claim 25, wherein said second version of said second program is stored in a second non-volatile memory of the RAID system rather than said first non-volatile memory, wherein said repairing comprises repairing said failed first version of said second program in said first non-volatile memory using said second version of said second program in said second non-volatile memory.

28. The method as recited in claim 25, wherein said detecting said first version of said second program is failed comprises:

generating a first cyclic redundancy code (CRC) value of said first version of said second program; and

determining that said first CRC value mismatches a second CRC value of said first version of said second program.

29. The method as recited in claim 28, wherein said second CRC value is previously generated and stored in said non-volatile memory when said first version of said second program is programmed into said non-volatile memory.

30. The method as recited in claim 25, further comprising:

storing a non-volatile indication of whether said first version of said second program failed during initialization of the RAID controller;

wherein said detecting comprises examining said non-volatile indication and determining therefrom that said first version of said second program failed during initialization of the RAID controller.

31. The method as recited in claim 30, further comprising:

populating, by a watchdog timer, said non-volatile indication based on whether said first version of said second program failed to boot within a predetermined time during initialization of the RAID controller, wherein said watchdog timer resets the processor after populating the non-volatile indication.

32. The method as recited in claim 25, wherein said second program comprises a loader program for loading an application program for execution by the processor.

33. The method as recited in claim 32, further comprising:

loading, by said loader program, said application program from said non-volatile memory into a volatile memory of the RAID system for execution by the processor.

34. The method as recited in claim 33, further comprising:

detecting said application program in said volatile memory is failed by performing a CRC check of said application program in said volatile memory.

35. The method as recited in claim 34, further comprising:

rebooting said processor in response to said detecting said application program in said volatile memory is failed.

36. The method as recited in claim 35, further comprising:

failing over to a redundant controller of the RAID system prior to said rebooting said processor.

37. The method as recited in claim 25, wherein said second program comprises an application program for managing a RAID controller.

38. The method as recited in claim 25, wherein said second program comprises an application program for monitoring and controlling an enclosure of the RAID system.

39. The method as recited in claim 25, wherein said second program comprises code for programming a field-programmable gate array (FPGA).

40. The method as recited in claim 25, wherein said second program comprises code for programming a complex programmable logic device (CPLD).

41. The method as recited in claim 25, wherein said second program comprises code for execution by a power supply coupled to provide power to the RAID system.

42. The method as recited in claim 25, wherein said first version of said second program is at a first location in said non-volatile memory and said second version of said second program is at a second location in said non-volatile memory, wherein said repairing comprises:

copying said second version of said second program from said second location to said first location in said non-volatile memory.

43. The method as recited in claim 42, wherein said copying comprises:

reading said second version of said second program from said second location in said non-volatile memory; and

writing said second version of said second program read from said second location to said first location in said non-volatile memory.

44. The method as recited in claim 42, wherein said copying is performed via a temporary location in a volatile memory of the RAID system, wherein no modification is performed on said second version of said second program in said temporary location.

45. The method as recited in claim 25, further comprising:

notifying a user that said first version of said second program is failed, in response to said detecting.

46. The method as recited in claim 25, further comprising:

updating a count of a number of failures of said first version of said second program in response to said detecting said first version of said second program is failed.

47. The method as recited in claim 46, further comprising:

notifying a user that said first version of said second program is failed if said count exceeds a user-programmable threshold.

48. The method as recited in claim 25, further comprising:

logging in said non-volatile memory an event specifying that said first version of said second program is failed.

49. A RAID system, comprising:

a first controller, comprising;

a first non-volatile memory, configured to store a first program and first and second versions of a second program and a fifth program, wherein the first and second versions of the second program are different;

a first volatile memory;

a first watch dog timer, for detecting a failure during a boot process of the first controller, the first watch dog timer having a predetermined maximum timeout period, wherein the boot process of the first controller is normally longer than said predetermined maximum timeout period of said first watch dog timer; and

a first processor, coupled to said first non-volatile memory and to said first volatile memory and to said first watch dog timer, configured to execute said first program, wherein said first program is configured to;

detect said first version of said second program is failed; and

repair said failed first version of said second program in said first non-volatile memory using said second version of said second program;

wherein said fifth program is configured to decompress said first program stored in said first non-volatile memory to a decompressed form and to write said decompressed form to said first volatile memory during the boot process of the first controller, wherein said fifth program is further configured to disable the first watch dog timer after writing said decompressed form of said first program to said first volatile memory, wherein said fifth program is further configured to re-enable the first watch dog timer prior to said first processor executing said first program; and

a second controller, coupled to said first controller, comprising;

a second non-volatile memory, configured to store a third program and first and second versions of a fourth program and a sixth program, wherein the first and second versions of the fourth program are different;

a second volatile memory;

a second watch dog timer, for detecting a failure during a boot process of the second controller, the second watch dog timer having a predetermined maximum timeout period, wherein the boot process of the second controller is normally longer than said predetermined maximum timeout period of said second watch dog timer; and

a second processor, coupled to said second non-volatile memory and to said second volatile memory and to said second watch dog timer, configured to execute said third program, wherein said third program is configured to;

detect said first version of said fourth program is failed; and

repair said failed first version of said fourth program in said second non-volatile memory using said second version of said fourth program;

wherein said sixth program is configured to decompress said first program stored in said second non-volatile memory to a decompressed form and to write said decompressed form to said second volatile memory during the boot process of the second controller, wherein said sixth program is further configured to disable the second watch dog timer after writing said decompressed form of said first program to said second volatile memory, wherein said sixth program is further configured to re-enable the second watch dog timer prior to said second processor executing said third program;

wherein said second program comprises an application program for performing RAID control functions.

50. The RAID system as recited in claim 49, wherein said second program comprises program code for programming an FPGA.

51. The RAID system as recited in claim 49, wherein said second program comprises program code for programming a CPLD.

52. The RAID system as recited in claim 49, wherein said second program comprises a program for execution by a power supply microcontroller.

53. The RAID system as recited in claim 49, wherein said first controller comprises a RAID controller, wherein said second controller comprises a management controller for providing a management interface to a user for managing the RAID system.

54. The RAID system as recited in claim 49, wherein said first controller comprises a RAID controller, wherein said second controller comprises an enclosure controller for monitoring and controlling components of an enclosure of the RAID system.

55. The RAID system as recited in claim 49, further comprising:

a third controller, coupled to said first and second controllers, comprising;

a third non-volatile memory, configured to store a fifth program and first and second versions of a sixth program, wherein the first and second versions of the sixth program are different;

a third processor, coupled to said second non-volatile memory, configured to execute said fifth program, wherein said fifth program is configured to;

detect said first version of said sixth program is failed; and

repair said failed first version of said sixth program in said third non-volatile memory using said second version of said sixth program.