Erasure Coded Storage Aggregation in Data Centers

• US 20110029840A1
• Filed: 07/31/2009
• Published: 02/03/2011
• Est. Priority Date: 07/31/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Calder, Bradley, Puri, Siddhartha, Mehr, John D., Pope, Matt, McKelvie, Samuel J., Ozzie, Raymond E.
• Granted Patent: US 8,458,287 B2
• Time in Patent Office: Days
• US Class Current: 714/763
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/35 Unequal or adaptive error protection, e.g. by providing a different level of protection according to significance of source information or by adapting the coding according to the change of transmission channel characteristics

Abstract

Embodiments of erasure coded storage aggregation are disclosed. The erasure coded storage aggregation includes storing a data file as erasure coded fragments in a plurality of nodes of one or more data centers. The erasure coded storage aggregation further includes monitoring an access frequency of the data file. Based on the comparison between the access frequency and a predetermined threshold, the data file is either reconstructed from the erasure coded fragments and stored in a storage node or retained as erasure coded fragments in the plurality of nodes of the one or more data centers.

Claims

1. One or more computer readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

storing a data file as erasure coded fragments in a plurality of nodes of at least one data center;

monitoring an access frequency of the data file that is stored as the erasure coded fragments;

reconstructing the data file from the erasure coded fragments and storing the data file in a storage node when the access frequency of the data file meets or exceeds a predetermined frequency threshold; and

retaining the data file as erasure coded fragments in the plurality of nodes when the access frequency of the data file is below the predetermined frequency threshold.

2. The one or more computer readable media of claim 1, wherein the storing further includes performing hierarchical erasure coding on the data file to obtain erasure coded fragments of the data file.

3. The one or more computer readable media of claim 1, wherein the reconstructing the data file further includes:

retrieving at least some of the erasure coded fragment from the plurality of nodes; and

reconstructing the data file from the retrieved erasure coded fragments.

4. The one or more computer readable media of claim 3, wherein the storing includes receiving the data file from a client device, and further storing an instruction that, when executed, cause the one or more processors to perform an act comprising providing the reconstructed data file to the client.

5. The one or more computer readable media of claim 3, further storing an instruction that, when executed, causes the one or more processors to perform an act comprising storing metadata in multiple locations of the plurality of nodes, and wherein the retrieving includes retrieving the erasure coded fragments using the stored metadata.

6. The one or more computer readable media of claim 1, further storing an instruction that when executed, causes the one or more processors to perform an act comprising selecting the plurality of nodes from one or more computation nodes or one or more storage nodes.

7. The one or more computer readable media of claim 1, further storing an instruction that, when executed, causes the one or more processors to perform an act comprising selecting the plurality of nodes from available nodes on the at least one data center, each of the selected nodes being selected based at least in part on an estimated availability, an estimated health status or a proximity to another selected node.

8. The one or more computer readable media of claim 1, wherein the plurality of nodes comprises a first plurality of nodes, and further storing an instruction that, when executed, cause the one or more processors to perform acts comprising:

monitoring an integrity of each of the erasure coded fragments that are stored in the first plurality of nodes;

reconstructing the data file when the monitoring determines that a predetermined quantity of the erasure coded fragments with a valid integrity is less than a predetermined minimal threshold;

performing erasure coding on the reconstructed data file to obtain erasure coded fragments of the data file; and

storing the erasure coded fragments of the data file in a second plurality of nodes of the at least one data center, wherein at least one node of the second plurality of nodes differs from the first plurality of nodes.

9. One or more computer readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

receiving a data file for storage from a client device or a data center storage node;

selecting a first storage option or a second storage option, the first storage option corresponding to storing the data file as an intact file, and the second storage option corresponding to storing the data file as a plurality of erasure coded fragments;

storing the data file as an intact file in a storage node of at least one data center when the first storage option is selected; and

storing the data file as a plurality of erasure coded fragments in a plurality of nodes of the at least one data center when the second storage option is selected.

10. The one or more computer readable media of claim 9, wherein the selecting includes selecting the first option or the second option based at least in part on a past access frequency of the data file, a predicted future access frequency of the data file or a file type of the data file.

11. The one or more computer readable media of claim 9, further storing instructions that, when executed, cause the one or more processors to perform acts comprising:

when the second storage option is selected;

retrieving at least some of the erasure coded fragment from the plurality of nodes; and

reconstructing the data file from the retrieved erasure coded fragments.

12. The one or more computer readable media of claim 11, further storing an instruction that, when executed, causes the one or more processors to perform an act comprising providing the reconstructed data file to the client device or the data center storage node when the second option is selected.

13. The one or more computer readable media of claim 9, further storing an instruction that, when executed, causes the one or more processors to perform an act comprising storing metadata in multiple locations of the plurality of nodes, and wherein the retrieving includes retrieving the erasure coded fragments using the stored metadata when the second option is selected.

14. The one or more computer readable media of claim 9, wherein the storing further includes selecting the plurality of nodes from one or more computation nodes or one or more storage nodes.

15. The one or more computer readable media of claim 14, wherein the selecting includes selecting the plurality of nodes from available nodes on the at least one data center, each of the selected nodes being selected based at least in part on an estimated availability, an estimated health status or a proximity to another selected node.

16. The one or more computer readable media of claim 9, wherein the plurality of nodes comprises a first plurality of nodes, and further storing an instruction that, when executed, causes the one or more processors to perform acts comprising:

when the second option is selected;

monitoring an integrity of each of the erasure coded fragments that are stored in the first plurality of nodes;

reconstructing the data file when the monitoring determines that a predetermined quantity of the erasure coded fragments have an integrity that is less than a predetermined minimal threshold;

performing erasure coding on the data file to obtain erasure coded fragments of the data file; and

storing the erasure coded fragments of the data file in a second plurality of nodes of the at least one data center, wherein at least one node of the second plurality of nodes differs from the first plurality of nodes.

17. A system, comprising:

one or more processors;

a memory that includes a plurality of computer-executable components, the plurality of computer-executable components comprising;

a front end to switch a data file between a first storage option or a second storage option, the first storage option corresponding to storing the data file as an intact file, and the second storage option corresponding to storing the data file as a plurality of erasure coded fragments;

an allocation coordinator to select a plurality of nodes from one or more computation nodes or one or more storage nodes;

an erasure coder to perform erasure coding on the data file to obtain erasure coded fragments of the data file; and

an aggregation agent to store the erasure coded fragments in the plurality of nodes.

18. The system of claim 17, wherein the allocation coordinator is to further command the aggregation agent to retrieve at least some of the erasure coded fragments from the plurality of nodes, activate the erasure coder to reconstruct the data file from the retrieved erasure coded fragments and provide the reconstructed data file to a client device or a data storage node via the front end.

19. The system of claim 17, wherein the aggregation agent is stored in one of a driver layer, a root operating system (OS), or a virtual machine (VM) scheduler of a data center.

20. The system of claim 17, wherein the front end is further to:

(i) switch the data file to the first storage option when an access frequency of the data file exceeds a predetermined frequency threshold, and (ii) switch the data file to the second storage option when the access frequency of the data file falls below the predetermined frequency threshold.