STORAGE SYSTEM WITH AUTOMATIC REDUNDANT CODE COMPONENT FAILURE DETECTION, NOTIFICATION, AND REPAIR

• US 20060236198A1
• Filed: 04/11/2006
• Published: 10/19/2006
• Est. Priority Date: 04/01/2005
• Status: Active Grant

• Current Assignee: Dot Hill Systems Corporation (Seagate Technology Holdings Plc)
• Original Assignee: Dot Hill Systems Corporation (Seagate Technology Holdings Plc)
• Inventors: Wang, Yuanru Frank, Lintz, Dwight Oliver Jr.
• Granted Patent: US 7,711,989 B2
• Time in Patent Office: Days
• US Class Current: 714/758
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/1417 Boot up procedures

Abstract

A RAID system includes a non-volatile memory storing a first program and first and second copies of a second program, and a processor executing the first program. The first program detects the first copy of the second program is failed and repairs the failed first copy in the non-volatile memory using the second copy. The failures may be detected at boot time or during normal operation of the controller. In one embodiment, the failure is detected via a CRC check. In one embodiment, the controller repairs the failed copy by copying the good copy to the location of the failed copy. In one embodiment, the system includes multiple controllers, each having its own processor and non-volatile memory and program that detects and repairs failed program copies. The programs include a loader, an application, FPGA code, CPLD code, and a program for execution by a power supply microcontroller.

Claims

1. A RAID system, comprising:

a non-volatile memory, configured to store a first program and first and second copies of a second program;

a processor, coupled to said non-volatile memory, configured to execute said first program, wherein said first program is configured to;

detect said first copy of said second program is failed; and

repair said failed first copy of said second program in said non-volatile memory using said second copy of said second program.

2. The RAID system as recited in claim 1, wherein said non-volatile memory comprises a FLASH memory.

3. The RAID system as recited in claim 1, further comprising:

a second non-volatile memory, coupled to said processor;

wherein said second copy of said second program is stored in said second non-volatile memory rather than said first non-volatile memory, wherein said first program is configured to repair said failed first copy of said second program in said first non-volatile memory using said second copy of said second program in said second non-volatile memory.

4. The RAID system as recited in claim 1, wherein said first program is configured to detect said first copy of said second program is failed by generating a first cyclic redundancy code (CRC) value of said first copy of said second program and determining that said first CRC value mismatches a second CRC value of said first copy of said second program.

5. The RAID system as recited in claim 4, wherein said second CRC value is previously generated and stored in said non-volatile memory when said first copy of said second program is programmed into said non-volatile memory.

6. The RAID system as recited in claim 1, further comprising:

a non-volatile indication of whether said first copy of said second program failed during initialization of the RAID controller, coupled to said processor;

wherein said first program is configured to detect said first copy of said second program is failed by examining said non-volatile indication.

7. The RAID system as recited in claim 6, further comprising:

a watchdog timer, coupled to said non-volatile indication, configured to populate said non-volatile indication based on whether said first copy of said second program failed to boot within a predetermined time during initialization of the RAID controller.

8. The RAID system as recited in claim 1, wherein said second program comprises a loader program for loading an application program for execution by the processor.

9. The RAID system as recited in claim 8, further comprising:

a volatile memory, coupled to said processor, wherein said loader program is configured to load said application program from said non-volatile memory into said volatile memory for execution by the processor.

10. The RAID system as recited in claim 9, wherein said first program is further configured to detect said application program in said volatile memory is failed by performing a CRC check of said application program in said volatile memory.

11. The RAID system as recited in claim 10, wherein said first program is further configured to reboot said processor in response to detecting said application program in said volatile memory is failed.

12. The RAID system as recited in claim 11, wherein said first program is further configured to failover to a redundant controller of the RAID system prior to rebooting said processor.

13. The RAID system as recited in claim 1, wherein said second program comprises an application program for performing RAID control functions.

14. The RAID system as recited in claim 1, wherein said second program comprises an application program for managing a RAID controller.

15. The RAID system as recited in claim 1, wherein said second program comprises an application program for monitoring and controlling an enclosure of the RAID system.

16. The RAID system as recited in claim 1, further comprising:

a field-programmable gate array (FPGA), coupled to said processor, wherein said second program comprises code for programming said FPGA.

17. The RAID system as recited in claim 1, further comprising:

a complex programmable logic device (CPLD), coupled to said processor, wherein said second program comprises code for programming said CPLD.

18. The RAID system as recited in claim 1, further comprising:

a power supply, coupled to provide power to said processor and said non-volatile memory, said power supply comprising a processor for executing said second program.

19. The RAID system as recited in claim 1, wherein said first copy of said second program is at a first location in said non-volatile memory and said second copy of said second program is at a second location in said non-volatile memory, wherein said first program is configured to copy said second copy of said second program from said second location to said first location in said non-volatile memory to repair said failed first copy of said second program in said non-volatile memory.

20. The RAID system as recited in claim 19, wherein said first program copies said second copy of said second program from said second location to said first location in said non-volatile memory by reading said second copy of said second program from said second location in said non-volatile memory and writing said second copy of said second program read from said second location to said first location in said non-volatile memory.

21. The RAID system as recited in claim 19, further comprising:

a volatile memory, coupled to said processor;

wherein said first program copies said second copy of said second program from said second location to said first location in said non-volatile memory via a temporary location in said volatile memory.

22. The RAID system as recited in claim 1, wherein said first program is further configured to notify a user that said first copy of said second program is failed.

23. The RAID system as recited in claim 1, wherein said first program is further configured to update a count of a number of failures of said first copy of said second program in response to detecting said first copy of said second program is failed.

24. The RAID system as recited in claim 23, wherein said first program is further configured to notify a user that said first copy of said second program is failed if said count exceeds a user-programmable threshold.

25. The RAID system as recited in claim 1, wherein said first program is further configured to log in said non-volatile memory an event specifying that said first copy of said second program is failed.

26. A method for improving the data availability characteristics of a RAID system, comprising:

executing a first program on a processor of the RAID system;

detecting, by the first program, that a first copy of a second program is failed, wherein said first copy of said second program is stored in a non-volatile memory of the RAID system; and

repairing, by the first program, said failed first copy of said second program in said non-volatile memory using a second copy of said second program stored in said non-volatile memory.

27. The method as recited in claim 26, wherein said non-volatile memory comprises a FLASH memory.

28. The method as recited in claim 26, wherein said second copy of said second program is stored in a second non-volatile memory of the RAID system rather than said first non-volatile memory, wherein said repairing comprises repairing said failed first copy of said second program in said first non-volatile memory using said second copy of said second program in said second non-volatile memory.

29. The method as recited in claim 26, wherein said detecting said first copy of said second program is failed comprises:

generating a first cyclic redundancy code (CRC) value of said first copy of said second program; and

determining that said first CRC value mismatches a second CRC value of said first copy of said second program.

30. The method as recited in claim 29, wherein said second CRC value is previously generated and stored in said non-volatile memory when said first copy of said second program is programmed into said non-volatile memory.

31. The method as recited in claim 26, further comprising:

storing a non-volatile indication of whether said first copy of said second program failed during initialization of the RAID controller;

wherein said detecting comprises examining said non-volatile indication and determining therefrom that said first copy of said second program failed during initialization of the RAID controller.

32. The method as recited in claim 31, further comprising:

populating, by a watchdog timer, said non-volatile indication based on whether said first copy of said second program failed to boot within a predetermined time during initialization of the RAID controller.

33. The method as recited in claim 26, wherein said second program comprises a loader program for loading an application program for execution by the processor.

34. The method as recited in claim 33, further comprising:

loading, by said loader program, said application program from said non-volatile memory into a volatile memory of the RAID system for execution by the processor.

35. The method as recited in claim 34, further comprising:

detecting said application program in said volatile memory is failed by performing a CRC check of said application program in said volatile memory.

36. The method as recited in claim 35, further comprising:

rebooting said processor in response to said detecting said application program in said volatile memory is failed.

37. The method as recited in claim 36, further comprising:

failing over to a redundant controller of the RAID system prior to said rebooting said processor.

38. The method as recited in claim 26, wherein said second program comprises an application program for performing RAID control functions.

39. The method as recited in claim 26, wherein said second program comprises an application program for managing a RAID controller.

40. The method as recited in claim 26, wherein said second program comprises an application program for monitoring and controlling an enclosure of the RAID system.

41. The method as recited in claim 26, wherein said second program comprises code for programming a field-programmable gate array (FPGA).

42. The method as recited in claim 26, wherein said second program comprises code for programming a complex programmable logic device (CPLD).

43. The method as recited in claim 26, wherein said second program comprises code for execution by a power supply coupled to provide power to the RAID system.

44. The method as recited in claim 26, wherein said first copy of said second program is at a first location in said non-volatile memory and said second copy of said second program is at a second location in said non-volatile memory, wherein said repairing comprises:

copying said second copy of said second program from said second location to said first location in said non-volatile memory.

45. The method as recited in claim 44, wherein said copying comprises:

reading said second copy of said second program from said second location in said non-volatile memory; and

writing said second copy of said second program read from said second location to said first location in said non-volatile memory.

46. The method as recited in claim 44, wherein said copying is performed via a temporary location in a volatile memory of the RAID system.

47. The method as recited in claim 26, further comprising:

notifying a user that said first copy of said second program is failed, in response to said detecting.

48. The method as recited in claim 26, further comprising:

updating a count of a number of failures of said first copy of said second program in response to said detecting said first copy of said second program is failed.

49. The method as recited in claim 48, further comprising:

notifying a user that said first copy of said second program is failed if said count exceeds a user-programmable threshold.

50. The method as recited in claim 26, further comprising:

logging in said non-volatile memory an event specifying that said first copy of said second program is failed.

51. A RAID system, comprising:

a first controller, comprising;

a first non-volatile memory, configured to store a first program and first and second copies of a second program; and

a first processor, coupled to said first non-volatile memory, configured to execute said first program, wherein said first program is configured to;

detect said first copy of said second program is failed; and

repair said failed first copy of said second program in said first non-volatile memory using said second copy of said second program; and

a second controller, coupled to said first controller, comprising;

a second non-volatile memory, configured to store a third program and first and second copies of a fourth program; and

a second processor, coupled to said second non-volatile memory, configured to execute said third program, wherein said third program is configured to;

detect said first copy of said fourth program is failed; and

repair said failed first copy of said fourth program in said second non-volatile memory using said second copy of said fourth program.

52. The RAID system as recited in claim 51, wherein said second program comprises program code for programming an FPGA.

53. The RAID system as recited in claim 51, wherein said second program comprises program code for programming a CPLD.

54. The RAID system as recited in claim 51, wherein said second program comprises a program for execution by a power supply microcontroller.

55. The RAID system as recited in claim 51, wherein said first controller comprises a RAID controller, wherein said second controller comprises a management controller for providing a management interface to a user for managing the RAID system.

56. The RAID system as recited in claim 51, wherein said first controller comprises a RAID controller, wherein said second controller comprises an enclosure controller for monitoring and controlling components of an enclosure of the RAID system.

57. The RAID system as recited in claim 51, further comprising:

a third controller, coupled to said first and second controllers, comprising;

a third non-volatile memory, configured to store a fifth program and first and second copies of a sixth program;

a third processor, coupled to said second non-volatile memory, configured to execute said fifth program, wherein said fifth program is configured to;

detect said first copy of said sixth program is failed; and

repair said failed first copy of said sixth program in said third non-volatile memory using said second copy of said sixth program.