Erasure coded storage aggregation in data centers

• US 8,918,478 B2
• Filed: 06/03/2013
• Issued: 12/23/2014
• Est. Priority Date: 07/31/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ozzie, Raymond E., Mehr, John D., Puri, Siddhartha, McKelvie, Samuel J., Calder, Bradley, Pope, Matt
• Primary Examiner(s): Lazaro, David
• Assistant Examiner(s): SHAW, ROBERT A
• Application Number:
  US13/908,132
• Publication Number:
  US 20130275390A1
• Time in Patent Office: 568 Days
• Field of Search: 709/217
• US Class Current: 709/217
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/35 Unequal or adaptive error protection, e.g. by providing a different level of protection according to significance of source information or by adapting the coding according to the change of transmission channel characteristics

Abstract

Embodiments of erasure coded storage aggregation are disclosed. The erasure coded storage aggregation includes storing a data file as erasure coded fragments in a plurality of nodes of one or more data centers. The erasure coded storage aggregation further includes monitoring an access frequency of the data file. Based on the comparison between the access frequency and a predetermined threshold, the data file is either reconstructed from the erasure coded fragments and stored in a storage node or retained as erasure coded fragments in the plurality of nodes of the one or more data centers.

Claims

1. One or more device memories storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

receiving a data file for storage;

selecting a first storage option or a second storage option based in part on at least one of a past access frequency of the data file or a predicted future access frequency of the data file, the first storage option corresponding to storing the data file as an intact file, and the second storage option corresponding to storing the data file as a plurality of erasure coded fragments;

storing the data file as the intact file in a storage node of at least one data center if the first storage option is selected; and

storing the data file as the plurality of erasure coded fragments in a plurality of nodes of the at least one data center if the second storage option is selected.

2. The one or more device memories of claim 1, wherein the acts further comprise:

after the second storage option is selected;

retrieving at least some of the plurality of erasure coded fragments from the plurality of nodes; and

reconstructing the data file from the at least some of the plurality of erasure coded fragments retrieved.

3. The one or more device memories of claim 2, wherein the acts further comprise providing the data file to a client device or a data center storage node after the data file is reconstructed.

4. The one or more device memories of claim 2, wherein the acts further comprise storing metadata in multiple locations across the plurality of nodes, and wherein the retrieving includes retrieving the at least some of the plurality erasure coded fragments based at least in part on the metadata.

5. The one or more device memories of claim 1, wherein the acts further comprise selecting the plurality of nodes from one or more computation nodes or one or more storage nodes.

6. The one or more device memories of claim 5, wherein each of the plurality of nodes are selected based at least in part on an estimated availability, an estimated health status or a proximity to another selected node.

7. The one or more device memories of claim 1, wherein the plurality of nodes comprises a first plurality of nodes and the plurality of erasure coded fragments comprises a first plurality of erasure coded fragments, and wherein the acts further comprise:

after the second storage option is selected;

monitoring an integrity of each of the first plurality of erasure coded fragments that are stored in the first plurality of nodes;

reconstructing the data file when the monitoring determines that a predetermined quantity of the first plurality of erasure coded fragments have an integrity that is less than a predetermined minimal threshold;

performing erasure coding on the data file to obtain a second plurality of erasure coded fragments of the data file; and

storing the second plurality of erasure coded fragments of the data file in a second plurality of nodes of the at least one data center, wherein at least one node of the second plurality of nodes differs from the first plurality of nodes.

8. A system, comprising:

one or more processors;

one or more memories;

a front end stored on the one or more memories and operable on the one or more processors to select between a first storage option or a second storage option, the first storage option corresponding to storing a data file as an intact file, and the second storage option corresponding to storing the data file as a plurality of erasure coded fragments, wherein the front end is configured to select the first storage option if an access frequency of the data file meets or exceeds a predetermined frequency threshold and the front end is configured to select the second storage option if the access frequency of the data file falls below the predetermined frequency threshold;

an allocation coordinator stored on the one or more memories and operable on the one or more processors to select a plurality of nodes to store the data file as the plurality of erasure coded fragments;

an erasure coder stored on the one or more memories and operable on the one or more processors to perform erasure coding on the data file to obtain the plurality of erasure coded fragments of the data file; and

an aggregation agent stored on the one or more memories and operable on the one or more processors to store the plurality of erasure coded fragments in the plurality of nodes.

9. The system of claim 8, wherein the allocation coordinator further commands the aggregation agent to retrieve at least some of the plurality of erasure coded fragments from the plurality of nodes, to activate the erasure coder to reconstruct the data file from the at least some of the plurality of erasure coded fragments retrieved and to provide the data file to a client device or a data storage node via the front end.

10. The system of claim 8, wherein the aggregation agent is stored in one of a driver layer, a root operating system (OS), or a virtual machine (VM) scheduler of a data center.

11. A method comprising:

receiving a data file for storage;

selecting, by one or more processors configured with executable instructions and based in part on at least one of a past access frequency of the data file or a predicted future access frequency of the data file, a first storage option or a second storage option, the first storage option corresponding to storing the data file as an intact file and the second storage option corresponding to storing the data file as a plurality of erasure coded fragments;

storing the data file as the intact file in a storage node of at least one data center if the first storage option is selected; and

storing the data file as the plurality of erasure coded fragments in a plurality of nodes of the at least one data center if the second storage option is selected.

12. The method of claim 11, further comprising:

after the second storage option is selected;

retrieving at least some of the plurality of erasure coded fragments from the plurality of nodes; and

reconstructing the data file from the at least some of the plurality of erasure coded fragments retrieved.

13. The method of claim 12, further comprising providing the data file to a client device or a data center storage node after the data file is reconstructed.

14. The method of claim 12, further comprising storing metadata in multiple locations across the plurality of nodes, and wherein the retrieving includes retrieving the at least some of the plurality erasure coded fragments based at least in part on the metadata.

15. The method of claim 11, further comprising selecting the plurality of nodes from one or more computation nodes or one or more storage nodes.

16. The method of claim 15, wherein each of the plurality of nodes are selected based at least in part on an estimated availability, an estimated health status or a proximity to another selected node.

17. The method of claim 11, wherein the plurality of nodes comprises a first plurality of nodes and the plurality of erasure coded fragments comprises a first plurality of erasure coded fragments, the method further comprising:

after the second storage option is selected;

monitoring an integrity of each of the first plurality of erasure coded fragments that are stored in the first plurality of nodes;

reconstructing the data file when the monitoring determines that a predetermined quantity of the first plurality of erasure coded fragments have an integrity that is less than a predetermined minimal threshold;

performing erasure coding on the data file to obtain a second plurality of erasure coded fragments of the data file; and

storing the second plurality of erasure coded fragments of the data file in a second plurality of nodes of the at least one data center, wherein at least one node of the second plurality of nodes differs from the first plurality of nodes.