Distributed erasure coded virtual file system

• US 9,448,887 B1
• Filed: 08/22/2015
• Issued: 09/20/2016
• Est. Priority Date: 08/22/2015
• Status: Active Grant

• Current Assignee: Weka.IO Ltd.
• Original Assignee: Weka.IO Ltd.
• Inventors: Ben Dayan, Maor, Palmon, Omri, Zvibel, Liran
• Primary Examiner(s): Iqbal, Nadeem
• Application Number:
  US14/833,053
• Time in Patent Office: 395 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1415 at system level
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0664 at device level, e.g. emulation of a storage device or system
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices comprises one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed across the plurality of storage devices such that each of the plurality of failure resilient address spaces spans a plurality of the storage devices. Each one of the plurality of failure resilient address spaces is organized into a plurality of stripes. Each one or more stripes of the plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains. Each of the plurality of stripes may comprise a plurality of storage blocks. Each block of a particular one of the plurality of stripes may reside on a different one of the plurality of storage devices.

Claims

1. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes; and

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains, wherein said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces.

2. The system of claim 1, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

3. The system of claim 2, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

4. The system of claim 1, wherein said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

5. The system of claim 4, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

6. The system of claim 1, wherein said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

7. The system of claim 1, wherein said rank is based on read and write overhead for committing to each of said plurality of stripes.

8. The system of claim 1, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

9. The system of claim 8, wherein each of said computing devices owns multiple of said failure resilient address spaces.

10. The system of claim 1, wherein said plurality of storage devices are organized into a plurality of failure domains.

11. The system of claim 10, wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

12. The system of claim 10, wherein each one of said plurality of stripes spans said plurality of failure domains.

13. The system of claim 10, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

14. The system of claim 13, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

15. The system of claim 10, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

16. The system of claim 10, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

17. The system of claim 1, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

18. The system of claim 1, wherein:

said plurality of stripes are organized into a plurality of groups;

each of said plurality of groups comprises one or more of said plurality of stripes; and

for each of said groups, said plurality of computing devices are operable to rank said one or more of said plurality of stripes.

19. The system of claim 1, wherein said plurality of computing devices are operable to:

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups.

20. The system of claim 19, wherein said criterion is based on how many blocks are available for new data to be written to.

21. The system of claim 1, wherein said plurality of computing devices are operable to buffer multiple unrelated portions of data from multiple client application write operations and commit said multiple unrelated portions of data to a single one of said plurality of stripes.

22. The system of claim 1, wherein:

one of said plurality of FEC correction domains comprises a plurality of data storage blocks, a first protection storage block, and a second protection storage block;

digits stored in said first protection storage block are calculated based on contents of a subset of said plurality of data storage blocks; and

digits stored in said second protection storage block are calculated based on contents of at least a subset of said plurality of data storage blocks, and said first protection storage block.

23. The system of claim 1, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

replicate said plurality of data portions and said first protection portion, said replication resulting in a first set comprising said plurality of data portions, one or more replicas of said plurality of data portions, said first protection portion, and one or more replicas of said first protection portion;

calculate a second protection portion based on all of said data portions and all of said first protection portions of said first set; and

commit said first set and said second protection portion to one of said plurality of FEC protection domains.

24. The system of claim 1, wherein:

of said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either;

one of a plurality of data portions, or protection digits calculated based on said plurality of data portions; and

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks.

25. The system of claim 1, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

calculate a second protection portion based on a first subset of said first protection portion and said plurality of data portions;

commit said first subset of said first protection portion and said plurality of data portions and said first protection portion to a first one of said plurality of FEC protection domains;

calculate a second protection portion based on a second subset of said first protection portion and said plurality of data portions; and

commit said second subset of said first protection portion and said plurality of data portions and said second protection portion to a second one of said plurality of FEC protection domains.

191. The system of claim 2, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

192. The system of claim 191, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

193. The system of claim 192, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

26. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains;

said plurality of stripes are organized into a plurality of groups;

each of said plurality of groups comprises one or more of said plurality of stripes; and

for each of said groups, said plurality of computing devices are operable to rank said one or more of said plurality of stripes.

27. The system of claim 26, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

28. The system of claim 27, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

29. The system of claim 26, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

30. The system of claim 26, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

31. The system of claim 26, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

32. The system of claim 26, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

33. The system of claim 26, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

34. The system of claim 33, wherein each of said computing devices owns multiple of said failure resilient address spaces.

35. The system of claim 26, wherein said plurality of storage devices are organized into a plurality of failure domains.

36. The system of claim 35, wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

37. The system of claim 35, wherein each one of said plurality of stripes spans said plurality of failure domains.

38. The system of claim 35, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

39. The system of claim 38, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

40. The system of claim 35, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

41. The system of claim 35, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

42. The system of claim 26, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

43. The system of claim 26, wherein said plurality of computing devices are operable to:

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups.

44. The system of claim 43, wherein said criterion is based on how many blocks are available for new data to be written to.

45. The system of claim 26, wherein said plurality of computing devices are operable to buffer multiple unrelated portions of data from multiple client application write operations, and commit said multiple unrelated portions of data to a single one of said plurality of stripes.

46. The system of claim 26, wherein:

one of said plurality of FEC correction domains comprises a plurality of data storage blocks, a first protection storage block, and a second protection storage block;

digits stored in said first protection storage block are calculated based on contents of a subset of said plurality of data storage blocks; and

digits stored in said second protection storage block are calculated based on contents of at least a subset of said plurality of data storage blocks, and said first protection storage block.

47. The system of claim 26, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

replicate said plurality of data portions and said first protection portion, said replication resulting in a first set comprising said plurality of data portions, one or more replicas of said plurality of data portions, said first protection portion, and one or more replicas of said first protection portion;

calculate a second protection portion based on all of said data portions and all of said first protection portions of said first set; and

commit said first set and said second protection portion to one of said plurality of FEC protection domains.

48. The system of claim 26, wherein:

of said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either;

one of a plurality of data portions, or protection digits calculated based on said plurality of data portions; and

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks.

49. The system of claim 26, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

calculate a second protection portion based on a first subset of said first protection portion and said plurality of data portions;

commit said first subset of said first protection portion and said plurality of data portions and said first protection portion to a first one of said plurality of FEC protection domains;

calculate a second protection portion based on a second subset of said first protection portion and said plurality of data portions; and

commit said second subset of said first protection portion and said plurality of data portions and said second protection portion to a second one of said plurality of FEC protection domains.

50. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains; and

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups.

51. The system of claim 50, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

52. The system of claim 51, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

53. The system of claim 50, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

54. The system of claim 53, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

55. The system of claim 50, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

56. The system of claim 50, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

57. The system of claim 50, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

58. The system of claim 57, wherein each of said computing devices owns multiple of said failure resilient address spaces.

59. The system of claim 50, wherein said plurality of storage devices are organized into a plurality of failure domains.

60. The system of claim 59, wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

61. The system of claim 59, wherein each one of said plurality of stripes spans said plurality of failure domains.

62. The system of claim 59, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

63. The system of claim 62, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

64. The system of claim 59, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

65. The system of claim 59, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

66. The system of claim 50, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

67. The system of claim 50, wherein said criterion is based on how many blocks are available for new data to be written to.

68. The system of claim 50, wherein said plurality of computing devices are operable to buffer multiple unrelated portions of data from multiple client application write operations, and commit said multiple unrelated portions of data to a single one of said plurality of stripes.

69. The system of claim 50, wherein:

one of said plurality of FEC correction domains comprises a plurality of data storage blocks, a first protection storage block, and a second protection storage block;

digits stored in said first protection storage block are calculated based on contents of a subset of said plurality of data storage blocks; and

digits stored in said second protection storage block are calculated based on contents of at least a subset of said plurality of data storage blocks, and said first protection storage block.

70. The system of claim 50, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

replicate said plurality of data portions and said first protection portion, said replication resulting in a first set comprising said plurality of data portions, one or more replicas of said plurality of data portions, said first protection portion, and one or more replicas of said first protection portion;

calculate a second protection portion based on all of said data portions and all of said first protection portions of said first set; and

commit said first set and said second protection portion to one of said plurality of FEC protection domains.

71. The system of claim 50, wherein:

of said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either;

one of a plurality of data portions, or protection digits calculated based on said plurality of data portions; and

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks.

72. The system of claim 50, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

calculate a second protection portion based on a first subset of said first protection portion and said plurality of data portions;

commit said first subset of said first protection portion and said plurality of data portions and said first protection portion to a first one of said plurality of FEC protection domains;

calculate a second protection portion based on a second subset of said first protection portion and said plurality of data portions; and

commit said second subset of said first protection portion and said plurality of data portions and said second protection portion to a second one of said plurality of FEC protection domains.

73. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains; and

said plurality of computing devices are operable to buffer multiple unrelated portions of data from multiple client application write operations, and commit said multiple unrelated portions of data to a single one of said plurality of stripes.

74. The system of claim 73, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

75. The system of claim 74, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

76. The system of claim 73, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

77. The system of claim 76, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

78. The system of claim 73, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

79. The system of claim 73, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

80. The system of claim 73, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

81. The system of claim 80, wherein each of said computing devices owns multiple of said failure resilient address spaces.

82. The system of claim 73, wherein said plurality of storage devices are organized into a plurality of failure domains.

83. The system of claim 82, wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

84. The system of claim 82, wherein each one of said plurality of stripes spans said plurality of failure domains.

85. The system of claim 82, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

86. The system of claim 85, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

87. The system of claim 82, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

88. The system of claim 82, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

89. The system of claim 73, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

90. The system of claim 73, wherein:

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

91. The system of claim 73, wherein:

one of said plurality of FEC correction domains comprises a plurality of data storage blocks, a first protection storage block, and a second protection storage block;

digits stored in said first protection storage block are calculated based on contents of a subset of said plurality of data storage blocks; and

digits stored in said second protection storage block are calculated based on contents of at least a subset of said plurality of data storage blocks, and said first protection storage block.

92. The system of claim 73, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

replicate said plurality of data portions and said first protection portion, said replication resulting in a first set comprising said plurality of data portions, one or more replicas of said plurality of data portions, said first protection portion, and one or more replicas of said first protection portion;

calculate a second protection portion based on all of said data portions and all of said first protection portions of said first set; and

commit said first set and said second protection portion to one of said plurality of FEC protection domains.

93. The system of claim 73, wherein:

of said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either;

one of a plurality of data portions, or protection digits calculated based on said plurality of data portions; and

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks.

94. The system of claim 73, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

calculate a second protection portion based on a first subset of said first protection portion and said plurality of data portions;

commit said first subset of said first protection portion and said plurality of data portions and said first protection portion to a first one of said plurality of FEC protection domains;

calculate a second protection portion based on a second subset of said first protection portion and said plurality of data portions; and

commit said second subset of said first protection portion and said plurality of data portions and said second protection portion to a second one of said plurality of FEC protection domains.

95. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes; and

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains;

one of said plurality of FEC correction domains comprises a plurality of data storage blocks, a first protection storage block, and a second protection storage block;

digits stored in said first protection storage block are calculated based on contents of a subset of said plurality of data storage blocks; and

digits stored in said second protection storage block are calculated based on contents of at least a subset of said plurality of data storage blocks, and said first protection storage block.

96. The system of claim 95, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

97. The system of claim 96, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

98. The system of claim 95, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

99. The system of claim 98, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

100. The system of claim 95, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

101. The system of claim 95, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

102. The system of claim 95, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

103. The system of claim 102, wherein each of said computing devices owns multiple of said failure resilient address spaces.

104. The system of claim 95, wherein said plurality of storage devices are organized into a plurality of failure domains.

105. The system of claim 104, wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

106. The system of claim 104, wherein each one of said plurality of stripes spans said plurality of failure domains.

107. The system of claim 104, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

108. The system of claim 107, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

109. The system of claim 104, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

110. The system of claim 104, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

111. The system of claim 95, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

112. The system of claim 95, wherein:

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

113. The system of claim 95, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

replicate said plurality of data portions and said first protection portion, said replication resulting in a first set comprising said plurality of data portions, one or more replicas of said plurality of data portions, said first protection portion, and one or more replicas of said first protection portion;

calculate a second protection portion based on all of said data portions and all of said first protection portions of said first set; and

commit said first set and said second protection portion to one of said plurality of FEC protection domains.

114. The system of claim 95, wherein:

of said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either;

one of a plurality of data portions, or protection digits calculated based on said plurality of data portions; and

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks.

115. The system of claim 95, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

calculate a second protection portion based on a first subset of said first protection portion and said plurality of data portions;

commit said first subset of said first protection portion and said plurality of data portions and said first protection portion to a first one of said plurality of FEC protection domains;

calculate a second protection portion based on a second subset of said first protection portion and said plurality of data portions; and

commit said second subset of said first protection portion and said plurality of data portions and said second protection portion to a second one of said plurality of FEC protection domains.

116. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains; and

said plurality of computing devices are operable to;

calculate a first protection portion based on a plurality of data portions;

replicate said plurality of data portions and said first protection portion, said replication resulting in a first set comprising said plurality of data portions, one or more replicas of said plurality of data portions, said first protection portion, and one or more replicas of said first protection portion;

calculate a second protection portion based on all of said data portions and all of said first protection portions of said first set; and

commit said first set and said second protection portion to one of said plurality of FEC protection domains.

117. The system of claim 116, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

118. The system of claim 117, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

119. The system of claim 116, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

120. The system of claim 119, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

121. The system of claim 116, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

122. The system of claim 116, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

123. The system of claim 116, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

124. The system of claim 123, wherein each of said computing devices owns multiple of said failure resilient address spaces.

125. The system of claim 116, wherein said plurality of storage devices are organized into a plurality of failure domains.

126. The system of claim 125, wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

127. The system of claim 125, wherein each one of said plurality of stripes spans said plurality of failure domains.

128. The system of claim 125, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

129. The system of claim 128, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

130. The system of claim 125, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

131. The system of claim 125, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

132. The system of claim 116, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

133. The system of claim 116, wherein:

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

134. The system of claim 116, wherein:

of said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either;

one of a plurality of data portions, or protection digits calculated based on said plurality of data portions; and

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks.

135. The system of claim 116, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

calculate a second protection portion based on a first subset of said first protection portion and said plurality of data portions;

commit said first subset of said first protection portion and said plurality of data portions and said first protection portion to a first one of said plurality of FEC protection domains;

calculate a second protection portion based on a second subset of said first protection portion and said plurality of data portions; and

commit said second subset of said first protection portion and said plurality of data portions and said second protection portion to a second one of said plurality of FEC protection domains.

136. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains;

of said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either;

one of a plurality of data portions, or protection digits calculated based on said plurality of data portions; and

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks.

137. The system of claim 136, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

138. The system of claim 137, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

139. The system of claim 136, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

140. The system of claim 139, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

141. The system of claim 136, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

142. The system of claim 136, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

143. The system of claim 136, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

144. The system of claim 143, wherein each of said computing devices owns multiple of said failure resilient address spaces.

145. The system of claim 136, wherein said plurality of storage devices are organized into a plurality of failure domains.

146. The system of claim 145, wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

147. The system of claim 145, wherein each one of said plurality of stripes spans said plurality of failure domains.

148. The system of claim 145, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

149. The system of claim 148, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

150. The system of claim 145, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

151. The system of claim 145, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

152. The system of claim 136, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

153. The system of claim 136, wherein:

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

154. The system of claim 136, wherein said plurality of computing devices are operable to:

calculate a first protection portion based on a plurality of data portions;

calculate a second protection portion based on a first subset of said first protection portion and said plurality of data portions;

commit said first subset of said first protection portion and said plurality of data portions and said first protection portion to a first one of said plurality of FEC protection domains;

calculate a second protection portion based on a second subset of said first protection portion and said plurality of data portions; and

commit said second subset of said first protection portion and said plurality of data portions and said second protection portion to a second one of said plurality of FEC protection domains.

158. The system of claim 136, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

159. The system of claim 158, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

155. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes; and

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains; and

said plurality of computing devices are operable to;

calculate a first protection portion based on a plurality of data portions;

calculate a second protection portion based on a first subset of said first protection portion and said plurality of data portions;

commit said first subset of said first protection portion and said plurality of data portions and said first protection portion to a first one of said plurality of FEC protection domains;

calculate a second protection portion based on a second subset of said first protection portion and said plurality of data portions; and

commit said second subset of said first protection portion and said plurality of data portions and said second protection portion to a second one of said plurality of FEC protection domains.

156. The system of claim 155, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

157. The system of claim 156, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

160. The system of claim 155, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

161. The system of claim 155, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

162. The system of claim 155, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

163. The system of claim 162, wherein each of said computing devices owns multiple of said failure resilient address spaces.

164. The system of claim 155, wherein said plurality of storage devices are organized into a plurality of failure domains.

165. The system of claim 164, wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

166. The system of claim 164, wherein each one of said plurality of stripes spans said plurality of failure domains.

167. The system of claim 164, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

168. The system of claim 167, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

169. The system of claim 164, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

170. The system of claim 164, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

171. The system of claim 155, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

172. The system of claim 155, wherein:

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

173. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains;

said plurality of storage devices are organized into a plurality of failure domains; and

each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

174. The system of claim 173, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

175. The system of claim 174, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

176. The system of claim 173, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

177. The system of claim 176, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

178. The system of claim 173, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

179. The system of claim 173, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

180. The system of claim 173, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

181. The system of claim 180, wherein each of said computing devices owns multiple of said failure resilient address spaces.

182. The system of claim 173, wherein each one of said plurality of stripes spans said plurality of failure domains.

183. The system of claim 173, wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

184. The system of claim 183, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

185. The system of claim 173, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

186. The system of claim 173, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

187. The system of claim 173, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

188. The system of claim 173, wherein:

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

189. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains;

said plurality of storage devices are organized into a plurality of failure domains; and

in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

190. The system of claim 189, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

194. The system of claim 189, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

195. The system of claim 189, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

196. The system of claim 189, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

197. The system of claim 196, wherein each of said computing devices owns multiple of said failure resilient address spaces.

198. The system of claim 189, wherein each one of said plurality of stripes spans said plurality of failure domains.

199. The system of claim 189, wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

200. The system of claim 189, wherein said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

201. The system of claim 189, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

202. The system of claim 189, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

203. The system of claim 189, wherein:

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

217. The system of claim 189, wherein:

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

204. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains;

said plurality of storage devices are organized into a plurality of failure domains; and

said plurality of computing devices are operable to, in the event of a failure of one or more of said failure domains, determine a rate at which any particular lost block is reconstructed based on how many other blocks of a same one of said plurality of stripes have also been lost.

205. The system of claim 204, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

206. The system of claim 205, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

207. The system of claim 204, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

208. The system of claim 207, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

209. The system of claim 204, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

210. The system of claim 204, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

211. The system of claim 204, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

212. The system of claim 211, wherein each of said computing devices owns multiple of said failure resilient address spaces.

213. The system of claim 204, wherein each one of said plurality of stripes spans said plurality of failure domains.

214. The system of claim 204, wherein:

in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes; and

said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

215. The system of claim 204, wherein one or more of said plurality of failure domains comprises a plurality of said storage devices.

216. The system of claim 204, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.

218. A system comprising:

a plurality of computing devices communicatively coupled to each other via a network, wherein;

each of said plurality of computing devices comprises one or more of a plurality of storage devices;

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans a plurality of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains; and

said plurality of computing devices are operable to;

perform successive commit operations to a selected one of said plurality of groups until said one or more of said plurality of stripes of said group no longer meets a determined criterion; and

upon said selected one of said plurality of groups no longer meeting said determined criterion, select a different one of said plurality of groups; and

said criterion is based on how many blocks are available for new data to be written to.

219. The system of claim 218, wherein:

each of said plurality of stripes comprises a plurality of storage blocks; and

each block of a particular one of said plurality of stripes resides on a different one of said plurality of storage devices.

220. The system of claim 219, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

a first portion of said plurality of storage blocks are for storage of data digits;

a second portion of said plurality of storage blocks are for storage of protection digits calculated based, at least in part, on said data digits;

a first quantity corresponds to how many blocks are in said plurality of storage blocks;

a second quantity corresponds to how many failure domains are in said plurality of failure domains; and

said first quantity is less than said second quantity.

221. The system of claim 218, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many protected and/or unprotected storage blocks are in each of said plurality of stripes.

222. The system of claim 221, wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

223. The system of claim 218, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

224. The system of claim 218, wherein:

said plurality of computing devices are operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces; and

said rank is based on read and write overhead for committing to each of said plurality of stripes.

225. The system of claim 218, wherein:

each of said failure resilient address spaces is owned by only one of said plurality of computing devices at any given time; and

each one of said plurality of failure resilient address spaces may be read and written only by its owner.

226. The system of claim 225, wherein each of said computing devices owns multiple of said failure resilient address spaces.

227. The system of claim 218, wherein:

said plurality of storage devices are organized into a plurality of failure domains;

in an instance when multiple of said plurality of stripes have one or more failed blocks, said plurality of computing devices are operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes; and

said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said plurality of computing devices perform said reconstruction of said plurality of stripes which have only one failed block.

228. The system of claim 218, wherein:

said plurality of storage devices are organized into a plurality of failure domains one or more of said plurality of failure domains comprises a plurality of said storage devices.

229. The system of claim 218, wherein each of said plurality of FEC protection domains spans multiple stripes of said plurality of stripes.