Correcting multiple block data loss in a storage array using a combination of a single diagonal parity group and multiple row parity groups

• US 7,073,115 B2
• Filed: 12/28/2001
• Issued: 07/04/2006
• Est. Priority Date: 12/28/2001
• Status: Active Grant

• Current Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: English, Robert M., Kleiman, Steven R., Corbett, Peter F.
• Primary Examiner(s): Lamarre, Guy
• Assistant Examiner(s): KERVEROS, DEMETRIOS C
• Application Number:
  US10/035,603
• Publication Number:
  US 20030126522A1
• Time in Patent Office: 1,649 Days
• Field of Search: 714/763, 714/764, 714/770, 714/755, 714/756, 714/800, 714/804, 714/805, 714/6, 714/801, 714/802, 714/803, 714/758, 711/114
• US Class Current: 714/770
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems

Abstract

A technique efficiently corrects multiple storage device failures in a storage array using a combination of a single diagonal parity group and multiple row parity groups. The storage array includes a plurality of concatenated sub-arrays, wherein each sub-array includes a set of data storage devices and a parity storage device. Each row parity group is associated with a sub-array of the array. The array further includes a global parity storage device holding diagonal parity computed across the concatenation of the sub-arrays. Instead of requiring that each parity group contain both a row parity device and a diagonal parity device, the array is composed of a collection of row parity groups. Diagonal parity is calculated across the full array.

Claims

1. A system to correct multiple storage device failures in a storage array using a combination of multiple first parity groups and a single secondary parity group, the system comprising:

a storage array having a plurality of concatenated sub-arrays, each sub-array including a set of data storage devices and a first parity storage device, the array further including a global secondary storage device associated with the storage array and holding secondary parity values for the single secondary parity group, the secondary parity values computed across the concatenation of the sub-arrays.

2. The system of claim 1 wherein the first parity group is a row parity group, the first parity storage device is a row parity storage device and wherein each row parity group is associated with one of the sub-arrays of the storage array such that the array is composed of the multiple row parity groups.

3. The system of claim 2 wherein the secondary parity group is a diagonal parity group, the secondary storage device is a diagonal parity storage device and wherein the secondary parity values are diagonal parity values.

4. The system of claim 3 further comprising:

a storage operating system configured to implement double failure protection encoding of the concatenated sub-arrays, wherein row parity values for each sub-array are stored on each row parity storage device and diagonal parity values for the entire array are stored on the global diagonal parity storage device; and

a processing element configured to execute the storage operating system to thereby invoke storage access operations to and from the array in accordance with the double failure protection encoding.

5. The system of claim 4 wherein the double failure protection encoding is row-diagonal parity encoding.

6. The system of claim 4 wherein the double failure protection encoding is EVENODD parity encoding.

7. The system of claim 1 wherein each sub-array is organized as a concentrated parity disk array.

8. The system of claim 1 wherein each sub-array is organized as a distributed parity disk array.

9. The system of claim 1 wherein the storage devices are video tape, magnetic tape, optical, DVD, bubble memory, electronic random access memory or magnetic disk devices.

10. A method for correcting double failures in a storage array using a combination of a single diagonal parity group and multiple row parity groups, the method comprising the steps of:

organizing the storage array as a plurality of concatenated sub-arrays based on double failure protection encoding, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

computing the diagonal parity for the single diagonal parity group across the concatenated sub-arrays; and

correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device associated with the storage array.

11. A method for correcting double failures in a storage array using a combination of a single diagonal parity group and multiple row parity groups, the method comprising the steps of:

organizing the storage array as a plurality of concatenated sub-arrays based on double failure protection encoding, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

computing the diagonal parity for the single diagonal parity group across the concatenated sub-arrays;

correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device associated with the storage array;

encoding the double failure protection as row-diagonal parity encoding;

determining whether the storage device failure is to a single storage device in a sub-array;

if the storage device failure is to a single storage device in the sub-array, reconstructing the failed storage device using local row parity associated with the sub-array; and

if the storage device failure is not to a single storage device in the sub-array, reconstructing the failed global diagonal parity storage device using all data and row parity storage devices of all sub-arrays of the array.

12. The method of claim 11 wherein the step of correcting storage device failure further comprises the steps of:

if the storage device failure is not a single storage device failure, determining whether the storage device failure is a double failure within the sub-array;

if the storage device failure is not a double failure within the sub-array, determining whether one of the failures includes the diagonal parity storage device; and

if one of the failures does not include the diagonal parity storage device, reconstructing the failed storage device in each sub-array using local row parity.

13. The method of claim 12 wherein the step of correcting storage device failure further comprises the steps of:

if one of the failures includes the diagonal parity storage device, determining whether another of the failed storage devices includes a row parity storage device;

if the another of the failed storage devices includes the row parity storage device, reconstructing the row parity storage device from the data storage devices of the sub-array; and

reconstructing the diagonal parity storage device from all data and row parity storage devices of all sub-arrays of the array.

14. The method of claim 13 wherein the step of correcting storage device failure further comprises the steps of:

if the another of the failed storage devices does not include the row parity storage device, reconstructing the data storage device using local row parity associated with the sub-array; and

reconstructing the diagonal parity storage device from all data and row parity storage devices of all sub-arrays of the array.

15. The method of claim 14 wherein the step of correcting storage device failures further comprises the step of, if the storage device failure is a double failure within the sub-array, recovering two failed storage devices within the sub-array using a row-diagonal reconstruction process.

16. The method of claim 15 wherein the step of recovering comprises the steps of:

using the diagonal parity storage device to recover at least one data block from a first of the failed storage devices of the sub-array; and

once the data block is recovered, using row parity within the sub-array to recover a corresponding block in a second of the failed storage devices.

17. The method of claim 11 further comprising the step of organizing each sub-array as a concentrated parity disk array.

18. The method of claim 11 further comprising the step of organizing each sub-array as a distributed parity disk array.

19. A method for correcting double failures in a storage array using a combination of a single diagonal parity group and multiple row parity groups, the method comprising the steps of:

organizing the storage array as a plurality of concatenated sub-arrays based on double failure protection encoding, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

computing the diagonal parity for the single diagonal parity group across the concatenated sub-arrays;

correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device associated with the storage array;

encoding the double failure protection as EVENODD parity encoding;

determining whether the storage device failure is to a single storage device in a sub-array;

if the storage device failure is to a single storage device in the sub-array, reconstructing the failed storage device using local row parity associated with the sub-array; and

if the storage device failure is not to a single storage device in the sub-array, reconstructing the failed global diagonal parity storage device using all data storage devices of all sub-arrays of the array.

20. The method of claim 19 wherein the step of correcting storage device failure further comprises the steps of:

if the storage device failure is not a single storage device failure, determining whether the storage device failure is a double failure within the sub-array;

if the storage device failure is not a double failure within the sub-array, determining whether one of the failures includes the diagonal parity storage device; and

if one of the failures does not include the diagonal parity storage device, reconstructing the failed storage device in each sub-array using local row parity.

21. The method of claim 20 wherein the step of correcting storage device failure further comprises the steps of:

if one of the failures includes the diagonal parity storage device, determining whether another of the failed storage devices includes a row parity storage device;

if the another of the failed storage devices includes the row parity storage device, reconstructing the row parity storage device from the data storage devices of the sub-array; and

reconstructing the diagonal parity storage device from all of the data storage devices of the array.

22. The method of claim 21 wherein the step of correcting storage device failure further comprises the steps of:

if the another of the failed storage devices does not include the row parity storage device, reconstructing the data storage device using local row parity associated with the sub-array; and

reconstructing the diagonal parity storage device from all data storage devices of the array.

23. The method of claim 22 wherein the step of correcting storage device failures further comprises the step of, if the storage device failure is a double failure within the sub-array, recovering two failed storage devices within the sub-array using an EVENODD reconstruction process.

24. Apparatus for correcting double failures in a storage array using a combination of a single diagonal parity group and multiple row parity groups, the apparatus comprising:

means for organizing the storage array as a plurality of concatenated sub-arrays based on double failure protection encoding, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

means for computing the diagonal parity for the single diagonal parity group across the concatenated sub-arrays; and

means for correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device associated with the storage array.

25. A computer readable medium containing executable program instructions for correcting double failures in a storage array using a combination of a single diagonal parity group and multiple row parity groups, the executable program instructions comprising program instructions for:

organizing the storage array as a plurality of concatenated sub-arrays based on double failure protection encoding, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

computing the diagonal parity for the single diagonal parity group across the concatenated sub-arrays;

correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device associated with the storage array.

26. A system to correct multiple storage element failures in an array using a combination of multiple first failure recovery groups and a single secondary failure recovery group, the system comprising:

a storage array having a plurality of concatenated sub-arrays, each sub-array including a set of data storage elements and a first failure recovery storage element storing first values used to correct a single failure within the sub-array, the array further including a global failure recovery storage element associated with the storage array and holding secondary values for the single secondary failure recovery group, the secondary values computed across the concatenation of the sub-arrays.

27. The system of claim 26 wherein the storage elements are packets and wherein the failure recovery is parity.

28. A method for operating a storage array, comprising:

organizing the storage array as a plurality of concatenated sub-arrays based on double failure protection encoding, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

computing the diagonal parity for the single diagonal parity group across the concatenated sub-arrays;

correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device associated with the storage array;

determining whether the storage device failure is to a single storage device in a sub-array;

if the storage device failure is to a single storage device in the sub-array, reconstructing the failed storage device using local row parity associated with the sub-array; and

if the storage device failure is not to a single storage device in the sub-array, reconstructing the failed global diagonal parity storage device using all data storage devices of all sub-arrays of the array.

29. A storage array, comprising:

means for organizing the storage array as a plurality of concatenated sub-arrays based on double failure protection encoding, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

means for computing the diagonal parity for the single diagonal parity group across the concatenated sub-arrays;

means for correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device associated with the storage array;

means for determining whether the storage device failure is to a single storage device in a sub-array;

if the storage device failure is to a single storage device in the sub-array, means for reconstructing the failed storage device using local row parity associated with the sub-array; and

if the storage device failure is not to a single storage device in the sub-array, means for reconstructing the failed global diagonal parity storage device using all data storage devices of all sub-arrays of the array.

30. A method for correcting double failures in a storage array, comprising:

organizing the storage array as a plurality of concatenated sub-arrays, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

computing the diagonal parity across the concatenated sub-arrays; and

correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device.

31. The method of claim 30, further comprising:

storing all row parity data on a dedicated disk storage device.

32. The method of claim 30, further comprising:

storing all diagonal parity data on a dedicated disk storage device.

33. The method of claim 30, further comprising:

encoding the double failure protection as row-diagonal parity encoding.

34. The method of claim 30, further comprising:

encoding the double failure protection as EVENODD parity encoding.

35. The method of claim 30, further comprising:

determining whether the storage device failure is to a single storage device in one of the sub-arrays;

if the storage device failure is to a single storage device in one of the sub-arrays, reconstructing the failed storage device using local row parity associated with the sub-array; and

if the storage device failure is not to a single storage device in one of the sub-arrays, re-constructing the failed global diagonal parity storage device using all data and row parity storage devices of all sub-arrays of the array.

36. The method of claim 30, further comprising:

if the storage device failure is not a single storage device failure, determining whether the storage device failure is a double failure within one of the sub-arrays;

if the storage device failure is not a double failure within one of the sub-arrays, determining whether one of the failures includes the diagonal parity storage device; and

if one of the failures does not include the diagonal parity storage device, reconstructing the failed storage device in each sub-array using local row parity.

37. The method of claim 30, further comprising:

if one of the failures includes the diagonal parity storage device, determining whether another of the failed storage devices includes a row parity storage device;

if the another of the failed storage devices includes the row parity storage device, reconstructing the row parity storage device from the data storage devices of the sub-array; and

reconstructing the diagonal parity storage device from all data and row parity storage devices of all sub-arrays of the array.

38. The method of claim 30, further comprising:

if the another of the failed storage devices does not include the row parity storage device, reconstructing the data storage device using local row parity associated with the sub-array; and

reconstructing the diagonal parity storage device from all data and row parity storage devices of all sub-arrays of the array.

39. The method of claim 30, further comprising:

if the storage device failure is a double failure within one of the sub-arrays, recovering two failed storage devices within the sub-array using a row-diagonal reconstruction process.

40. The method of claim 30, further comprising:

using the diagonal parity storage device to recover at least one data block from a first of the failed storage devices of one of the sub-arrays; and

once the data block is recovered, using row parity within the same sub-array to recover a corresponding block in a second of the failed storage devices.

41. The method of claim 30, further comprising:

organizing each sub-array as a concentrated parity disk array.

42. The method of claim 30, further comprising:

organizing each sub-array as a distributed parity disk array.

43. A storage array, comprising:

means for organizing the storage array as a plurality of concatenated sub-arrays, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

means for computing the diagonal parity across the concatenated sub-arrays; and

means for correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device.

44. The storage array of claim 43, further comprising:

means for storing all row parity data on a dedicated disk storage device.

45. The storage array of claim 43, further comprising:

means for storing all diagonal parity data on a dedicated disk storage device.

46. The storage array of claim 43, further comprising:

means for encoding the double failure protection as row-diagonal parity encoding.

47. The storage array of claim 43, further comprising:

means for encoding the double failure protection as EVENODD parity encoding.

48. The storage array of claim 43, further comprising:

means for determining whether the storage device failure is to a single storage device in one of the sub-arrays;

if the storage device failure is to a single storage device in one of the sub-arrays, means for reconstructing the failed storage device using local row parity associated with the sub-array; and

if the storage device failure is not to a single storage device in one of the sub-arrays, means for re-constructing the failed global diagonal parity storage device using all data and row parity storage devices of all sub-arrays of the array.

49. The storage array of claim 43, further comprising:

if the storage device failure is not a single storage device failure, means for determining whether the storage device failure is a double failure within one of the sub-arrays;

if the storage device failure is not a double failure within one of the sub-arrays, means for determining whether one of the failures includes the diagonal parity storage device; and

if one of the failures does not include the diagonal parity storage device, means for reconstructing the failed storage device in each sub-array using local row parity.

50. The storage array of claim 43, further comprising:

if one of the failures includes the diagonal parity storage device, means for determining whether another of the failed storage devices includes a row parity storage device;

if the another of the failed storage devices includes the row parity storage device, means for reconstructing the row parity storage device from the data storage devices of the sub-array; and

means for reconstructing the diagonal parity storage device from all data and row parity storage devices of all sub-arrays of the array.

51. The storage array of claim 43, further comprising:

if the another of the failed storage devices does not include the row parity storage device, means for reconstructing the data storage device using local row parity associated with the sub-array; and

means for reconstructing the diagonal parity storage device from all data and row parity storage devices of all sub-arrays of the array.

52. The storage array of claim 43, further comprising:

if the another of the failed storage devices does not include the row parity storage device, means for reconstructing the data storage device using local row parity associated with the sub-array; and

means for reconstructing the diagonal parity storage device from all data and row parity storage devices of all sub-arrays of the array.

53. The storage array of claim 43, further comprising:

means for using the diagonal parity storage device to recover at least one data block from a first of the failed storage devices of one of the sub-arrays; and

once the data block is recovered, means for using row parity within the same sub-array to recover a corresponding block in a second of the failed storage devices.

54. The storage array of claim 43, further comprising:

means for organizing each sub-array as a concentrated parity disk array.

55. The storage array of claim 43, further comprising:

means for organizing each sub-array as a distributed parity disk array.

56. A computer readable media, comprising:

said computer readable media containing instructions for execution on a processor for the practice of a method for correcting double failures in a storage array, having the steps,

organizing the storage array as a plurality of concatenated sub-arrays, each sub-array including a set of data storage devices and a row parity storage device, the storage array further including a global diagonal parity storage device for holding diagonal parity;

computing the diagonal parity across the concatenated sub-arrays; and

correcting storage device failure within the array using the row parity storage device associated with each sub-array and the global diagonal parity storage device.

57. A method of correcting failures in a storage array comprising:

organizing the storage array into a plurality of sub-arrays, each sub-array including a plurality of data storage devices and at least one row parity storage device for storing parity information for the data storage devices;

computing global diagonal parity information across the plurality of sub-arrays, the global diagonal parity information computed from both the data storage devices and the row parity storage devices in the plurality of sub-arrays.

storing the global diagonal parity information in a global diagonal parity storage device;

detecting a storage device failure;

if the storage device failure is a single failed data storage device in one of the sub-arrays, reconstructing the single failed data storage device using row parity from the row parity storage device of that one of the sub-arrays;

if the storage device failure is two failed storage devices within one of the sub-arrays, reconstructing the two failed storage data devices using a row-diagonal reconstruction process.

58. The method of claim 57 further comprising:

if the storage device failure is a failed row parity storage device in one of the sub-arrays, reconstructing the failed row parity storage device using the data storage devices in that one of the sub-arrays.

59. The method of claim 57 further comprising:

if the storage device failure is a failed global diagonal parity storage device, reconstructing the failed global parity storage device using all the data storage devices and the row parity storage devices of all the plurality of sub-arrays.

60. The method of claim 57 further comprising:

if the storage device failure is a failed row parity storage device in a one of the sub arrays and a failed global diagonal parity storage device, first reconstructing the failed row parity storage device using the data storage devices in that one of the sub-arrays, and then reconstructing the failed global parity storage device using all the data storage devices and all the row parity storage devices in all the plurality of sub-arrays.

61. The method of claim 57 further comprising:

if the storage device failure is the failure of two data storage devices that reside in different sub-arrays, reconstructing each failed data storage device using row parity information from the row parity storage device of the sub-array in which the failed data storage device resides.

62. An apparatus for correcting failures in a storage array comprising:

means for organizing the storage array into a plurality of sub-arrays, each sub-array including a plurality of data storage devices and at least one row parity storage device for storing parity information for the data storage devices;

means for computing global diagonal parity information across the plurality of sub-arrays, the global diagonal parity information computed from both the data storage devices and the row parity storage devices in the plurality of sub-arrays.

means storing the global diagonal parity information in a global diagonal parity storage device;

means for detecting a storage device failure;

if the storage device failure is a single failed data storage device in one of the sub-arrays, means for reconstructing the single failed data storage device using row parity from the row parity storage device of that one of the sub-arrays;

if the storage device failure is two failed storage devices within one of the sub-arrays, means for reconstructing the two failed storage data devices using a row-diagonal reconstruction process.

63. The apparatus of claim 62 further comprising:

if the storage device failure is a failed row parity storage device in one of the sub-arrays, means reconstructing the failed row parity storage device using the data storage devices in that one of the sub-arrays.

64. The apparatus of claim 62 further comprising:

if the storage device failure is a failed global diagonal parity storage device, means for reconstructing the failed global parity storage device using all the data storage devices and the row parity storage devices of all the plurality of sub-arrays.

65. The apparatus of claim 62 further comprising:

if the storage device failure is a failed row parity storage device in a one of the sub arrays and a failed global diagonal parity storage device, means for reconstructing the failed row parity storage device using the data storage devices in that one of the sub-arrays, and means for reconstructing the failed global parity storage device using all the data storage devices and all the row parity storage devices in all the plurality of sub-arrays.

66. The apparatus of claim 62 further comprising:

if the storage device failure is the failure of two data storage devices that reside in different sub-arrays, means for reconstructing each failed data storage device using row parity information from the row parity storage device of the sub-array in which the failed data storage device resides.

67. A computer readable medium containing executable program instructions for correcting failures in a storage array, the executable program instructions comprising program instructions for:

computing global diagonal parity information across the plurality of sub-arrays, the global diagonal parity information computed from both the data storage devices and the row parity storage devices in the plurality of sub-arrays.

storing the global diagonal parity information in a global diagonal parity storage device;

detecting a storage device failure;

if the storage device failure is a single failed data storage device in one of the sub-arrays, reconstructing the single failed data storage device using row parity from the row parity storage device of that one of the sub-arrays;

if the storage device failure is two failed storage devices within one of the sub-arrays, reconstructing the two failed storage data devices using a row-diagonal reconstruction process.