Dynamic object replication within a distributed storage system

• US 7,778,972 B1
• Filed: 03/08/2006
• Issued: 08/17/2010
• Est. Priority Date: 12/29/2005
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Atlas, Alan B., Sorenson, James Christopher III, Wagner, Eric M., Cormie, John David
• Primary Examiner(s): Truong; Cam Y T
• Application Number:
  US11/370,664
• Time in Patent Office: 1,623 Days
• Field of Search: 707/104.1
• US Class Current: 707/626
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0667 at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Dynamic object replication in distributed storage systems. A system may implement a web services interface, storage nodes, storage node selection logic and a coordinator. The web services interface may be configured to receive, according to a web services protocol, client requests to store data objects including a given client request to store a given one of the data objects. The storage nodes may be configured to store replicas of the data objects. The storage node selection logic may be configured to dynamically determine a write plan for storing replicas of the given data object to corresponding storage nodes, where the write plan is dynamically determined according to current state information of the storage nodes. The coordinator may be configured to receive the client requests from the web services interface, and for the given client request, to store the replicas of the given data object to the corresponding storage nodes according to the write plan.

Claims

1. A system, comprising:

a plurality of computing nodes, each computing node comprising at least one processor and memory, wherein the plurality of computing nodes is configured to implement;

a web services interface configured to present a data storage web service to clients, wherein said data storage web service includes one or more web services endpoints, wherein each web services endpoint of the one or more web services endpoints is addressable by ones of said clients to receive, according to a web services protocol, client requests to store data objects supplied by said clients within said data storage web service, wherein said client requests include a given client request to store a given one of said data objects;

a plurality of storage nodes configured to store replicas of said data objects;

storage node selection logic configured to dynamically determine a write plan for storing one or more replicas of said given data object to one or more corresponding ones of said plurality of storage nodes, wherein a plurality of distinct storage policies defined within said data storage web service are applicable to said given data object;

wherein said write plan is dynamically determined according to current state information of said plurality of storage nodes, wherein to dynamically determine said write plan, said storage node selection logic is further configured to select said one or more corresponding ones of said plurality of storage nodes such that a number of said plurality of storage policies satisfied by storing said one or more replicas of said given data object according to said write plan is maximized, and such that on at least some occasions of dynamically determining said write plan, at least two of said storage policies are satisfied by said write plan;

wherein a number of storage policies that are satisfiable by said write plan differs on different occasions of dynamically determining said write plan, dependent upon said current state information of said plurality of storage nodes; and

a coordinator configured to receive said client requests to store data objects from said web services interface, wherein in response to said given client request, said coordinator is configured to store said one or more replicas of said given data object to said one or more corresponding storage nodes according to said write plan.

2. The system as recited in claim 1, wherein said storage node selection logic is further configured to dynamically determine said write plan in response to said coordinator receiving said given client request.

3. The system as recited in claim 1, wherein said current state information includes operational status information corresponding respectively to said plurality of storage nodes and indicative of whether a corresponding storage node is in a normal operational state, and wherein to dynamically determine said write plan, said storage node selection logic is further configured to select said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said operational status information corresponding to said one or more storage nodes.

4. The system as recited in claim 1, wherein said current state information includes load status information corresponding respectively to said plurality of storage nodes and indicative of a level of load under which a corresponding storage node is operating, and wherein to dynamically determine said write plan, said storage node selection logic is further configured to select said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said load status information corresponding to said one or more storage nodes.

5. The system as recited in claim 4, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of processor utilization corresponding to said given storage node.

6. The system as recited in claim 4, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of memory utilization corresponding to said given storage node.

7. The system as recited in claim 4, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of storage device input/output bandwidth utilization corresponding to said given storage node.

8. The system as recited in claim 4, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of network interface bandwidth utilization corresponding to said given storage node.

9. The system as recited in claim 1, wherein said current state information includes network cost information corresponding respectively to said plurality of storage nodes and indicative of a cost of communication with a corresponding storage node, and wherein to dynamically determine said write plan, said storage node selection logic is further configured to select said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said network cost information corresponding to said one or more storage nodes.

10. The system as recited in claim 9, wherein said network cost information varies over time.

11. The system as recited in claim 9, wherein for a given one of said plurality of storage nodes, said network cost information includes an indication of network communication latency corresponding to said given storage node.

12. The system as recited in claim 9, wherein for a given one of said plurality of storage nodes, said network cost information includes an indication of a network communication economic cost corresponding to said given storage node.

13. The system as recited in claim 1, wherein to dynamically determine said write plan, said storage node selection logic is further configured to minimize a cost associated with storing said one or more replicas of said given data object according to said write plan.

14. The system as recited in claim 13, wherein said cost varies according to time, and wherein to minimize said cost, said storage node selection logic is further configured to identify a time period during which a particular one of said one or more replicas is to be stored to a particular one of said plurality of storage nodes, dependent upon a cost associated with storing said particular replica to said particular storage node during said time period.

15. The system as recited in claim 1, wherein a given one of said storage policies specifies a minimum number of replicas of said given data object to be stored to corresponding ones of said storage nodes.

16. The system as recited in claim 1, wherein said plurality of storage nodes is distributed among a plurality of areas, and wherein a given one of said storage policies specifies a minimum number of said plurality of areas among which said one or more replicas of said given data object are to be distributed.

17. The system as recited in claim 1, wherein said plurality of computing nodes is further configured to implement a replicator configured to:

subsequent to said one or more replicas of said given data object being stored to said corresponding one or more storage nodes, determine whether any of said one or more replicas is inaccessible;

in response to determining that any of said one or more replicas is inaccessible, request a new write plan from said storage node selection logic for storing one or more additional replicas of said given data object, wherein said storage node selection logic is configured to dynamically determine said new write plan according to current state information of said plurality of storage nodes; and

subsequent to receiving said new write plan, store said one or more additional replicas to one or more corresponding ones of said storage nodes according to said new write plan.

18. The system as recited in claim 1, wherein said plurality of computing nodes is further configured to implement a replicator configured to:

subsequent to said one or more replicas of said given data object being stored to said corresponding one or more storage nodes, determine whether a number of storage policies satisfied by said stored one or more replicas of said given data object is insufficient;

in response to determining that said number of satisfied storage policies is insufficient, request a new write plan from said storage node selection logic for storing one or more additional replicas of said given data object, wherein said storage node selection logic is configured to dynamically determine said new write plan according to current state information of said plurality of storage nodes, and wherein dynamically determining said write plan includes maximizing a number of storage policies satisfied by storing said one or more additional replicas of said given data object according to said new write plan; and

subsequent to receiving said new write plan, store said one or more additional replicas to one or more corresponding ones of said storage nodes according to said new write plan.

19. The system as recited in claim 18, wherein to determine whether a number of storage policies satisfied by said stored one or more replicas of said given data object is insufficient, said replicator is further configured to determine whether said number of satisfied storage policies is less than a threshold value.

20. The system as recited in claim 1:

wherein the storage node selection logic is further configured to dynamically determine a particular write plan for storing one or more replicas of a particular data object to one or more corresponding ones of said plurality of storage nodes in response to a client request to store said particular data object, wherein said plurality of storage policies are applicable to said particular data object, and wherein each of said storage policies has a corresponding weight;

wherein said particular write plan for storing said particular data object is dynamically determined according to current state information of said plurality of storage nodes, wherein to dynamically determine said particular write plan, said storage node selection logic is further configured to select said one or more corresponding ones of said plurality of storage nodes such that a total of the corresponding weights of said plurality of storage policies satisfied by storing said one or more replicas of said given data object according to said particular write plan is maximized; and

wherein a number of storage policies that are satisfiable by said particular write plan differs on different occasions of dynamically determining said particular write plan, dependent upon said current state information of said plurality of storage nodes.

21. The system as recited in claim 1, wherein one or more of said storage policies are mandatory storage policies, and wherein in response to determining that no write plan exists that satisfies a given mandatory storage policy for said given data object, said storage node selection logic is further configured to indicate that said given client request to store said given data object has failed.

22. The system as recited in claim 1, wherein said plurality of storage policies comprises one or more of the following:

a locality policy configured to give preference to storage nodes in a same area as a node determining said write plan;

a load balancing policy configured to equalize write traffic among said storage nodes;

or

a space balancing policy configured to equalize storage resource capacity utilization among said storage nodes.

23. The system as recited in claim 1, wherein said plurality of storage policies comprises a lowest-cost-chain policy configured to minimize a total cost of a sequence of storage node write operations included in said write plan.

24. The system as recited in claim 1:

wherein to receive said client requests according to said web services protocol, each web services endpoint is configured to implement a corresponding web services application programming interface (API) defining data storage web service operations that are available to ones of said clients via web services calls, and is addressable to receive said web services calls, formatted according to said web services API to specify one or more of said data storage web service operations, from ones of said clients according to an Internet-based application layer data transport protocol;

wherein in response to receiving web services calls formatted according to said web services API that are indicative of client requests to store data objects, each said web services endpoint is further configured to store data objects supplied by said clients within said data storage web service; and

wherein each said web services endpoint is further configured to receive web services calls formatted according to said web services API that are indicative of client requests for access to ones of said data objects, wherein a given one of said client requests for access to a given one of said data objects includes a key value corresponding to said given data object.

25. A method, comprising:

presenting a data storage web service to clients, wherein said data storage web service includes one or more web services endpoints, wherein each web services endpoint of the one or more web services endpoints is addressable by ones of said clients according to a web services protocol to store data objects supplied by said clients within said data storage web service;

receiving, via said one or more web services endpoints, client requests to store ones of said data objects including a given client request to store a given one of said data objects;

dynamically determining a write plan for storing one or more replicas of said given data object to one or more corresponding ones of a plurality of storage nodes, wherein a plurality of distinct storage policies defined within said data storage web service are applicable to said given data object;

wherein said write plan is dynamically determined according to current state information of said plurality of storage nodes, and wherein dynamically determining said write plan comprises selecting said one or more corresponding ones of said plurality of storage nodes such that a number of said plurality of storage policies satisfied by storing said one or more replicas of said given data object according to said write plan is maximized, and such that on at least some occasions of dynamically determining said write plan, at least two of said storage policies are satisfied by said write plan;

wherein a number of storage policies that are satisfiable by said write plan differs on different occasions of dynamically determining said write plan, dependent upon said current state information of said plurality of storage nodes; and

in response to said given client request, storing said one or more replicas of said given data object to said one or more corresponding storage nodes according to said write plan.

26. The method as recited in claim 25, wherein dynamically determining said write plan occurs in response to receiving said given client request.

27. The method as recited in claim 25, wherein said current state information includes operational status information corresponding respectively to said plurality of storage nodes and indicative of whether a corresponding storage node is in a normal operational state, and wherein dynamically determining said write plan includes selecting said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said operational status information corresponding to said one or more storage nodes.

28. The method as recited in claim 25, wherein said current state information includes load status information corresponding respectively to said plurality of storage nodes and indicative of a level of load under which a corresponding storage node is operating, and wherein dynamically determining said write plan includes selecting said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said load status information corresponding to said one or more storage nodes.

29. The method as recited in claim 28, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of processor utilization corresponding to said given storage node.

30. The method as recited in claim 28, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of memory utilization corresponding to said given storage node.

31. The method as recited in claim 28, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of storage device input/output bandwidth utilization corresponding to said given storage node.

32. The method as recited in claim 28, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of network interface bandwidth utilization corresponding to said given storage node.

33. The method as recited in claim 25, wherein said current state information includes network cost information corresponding respectively to said plurality of storage nodes and indicative of a cost of communication with a corresponding storage node, and wherein dynamically determining said write plan includes selecting said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said network cost information corresponding to said one or more storage nodes.

34. The method as recited in claim 33, wherein said network cost information varies over time.

35. The method as recited in claim 33, wherein for a given one of said plurality of storage nodes, said network cost information includes an indication of network communication latency corresponding to said given storage node.

36. The method as recited in claim 33, wherein for a given one of said plurality of storage nodes, said network cost information includes an indication of a network communication economic cost corresponding to said given storage node.

37. The method as recited in claim 25, wherein dynamically determining said write plan includes minimizing a cost associated with storing said one or more replicas of said given data object according to said write plan.

38. The method as recited in claim 37, wherein said cost varies according to time, and wherein minimizing said cost includes identifying a time period during which a particular one of said one or more replicas is to be stored to a particular one of said plurality of storage nodes, wherein identifying said time period is dependent upon a cost associated with storing said particular replica to said particular storage node during said time period.

39. The method as recited in claim 25, wherein a given one of said storage policies specifies a minimum number of replicas of said given data object to be stored to corresponding ones of said storage nodes.

40. The method as recited in claim 25, wherein said plurality of storage nodes is distributed among a plurality of areas, and wherein a given one of said storage policies specifies a minimum number of said plurality of areas among which said one or more replicas of said given data object are to be distributed.

41. The method as recited in claim 25, further comprising:

subsequent to said one or more replicas of said given data object being stored to said corresponding one or more storage nodes, determining whether any of said one or more replicas is inaccessible;

in response to determining that any of said one or more replicas is inaccessible, dynamically determining a new write plan for storing one or more additional replicas of said given data object according to current state information of said plurality of storage nodes; and

storing said one or more additional replicas to one or more corresponding ones of said storage nodes according to said new write plan.

42. The method as recited in claim 25, further comprising:

subsequent to said one or more replicas of said given data object being stored to said corresponding one or more storage nodes, determine whether a number of storage policies satisfied by said stored one or more replicas of said given data object is insufficient;

in response to determining that said number of satisfied storage policies is insufficient, dynamically determining a new write plan for storing one or more additional replicas of said given data object according to current state information of said plurality of storage nodes, wherein dynamically determining said write plan includes maximizing a number of storage policies satisfied by storing said one or more additional replicas of said given data object according to said new write plan; and

storing said one or more additional replicas to one or more corresponding ones of said storage nodes according to said new write plan.

43. The method as recited in claim 42, wherein determining whether a number of storage policies satisfied by said stored one or more replicas of said given data object is insufficient comprises determining whether said number of satisfied storage policies is less than a threshold value.

44. The method as recited in claim 25, further comprising:

dynamically determining a particular write plan for storing one or more replicas of a particular data object to one or more corresponding ones of said plurality of storage nodes in response to a client request to store said particular data object, wherein said plurality of storage policies are applicable to said particular data object, and wherein each of said storage policies has a corresponding weight;

wherein said particular write plan for storing said particular data object is dynamically determined according to current state information of said plurality of storage nodes, wherein to dynamically determine said write plan, said storage node selection logic is further configured to select said one or more corresponding ones of said plurality of storage nodes such that a total of the corresponding weights of said plurality of storage policies satisfied by storing said one or more replicas of said given data object according to said particular write plan is maximized; and

wherein a number of storage policies that are satisfiable by said particular write plan differs on different occasions of dynamically determining said particular write plan, dependent upon said current state information of said plurality of storage nodes.

45. The method as recited in claim 25, wherein one or more of said storage policies are mandatory storage policies, and wherein the method further comprises:

in response to determining that no write plan exists that satisfies a given mandatory storage policy for said given data object, indicating that said given client request to store said given data object has failed.

46. The method as recited in claim 25, wherein said plurality of storage policies comprises one or more of the following:

a locality policy configured to give preference to storage nodes in a same area as a node determining said write plan;

a load balancing policy configured to equalize write traffic among said storage nodes;

or

a space balancing policy configured to equalize storage resource capacity utilization among said storage nodes.

47. The method as recited in claim 25, wherein said plurality of storage policies comprises a lowest-cost-chain policy configured to minimize a total cost of a sequence of storage node write operations included in said write plan.

48. A computer-accessible storage medium storing instructions, wherein the instructions are computer-executable to:

present a data storage web service to clients, wherein said data storage web service includes one or more web services endpoints, wherein each web services endpoint of the one or more web services endpoints is addressable by ones of said clients according to a web services protocol to store data objects supplied by said clients within said data storage web service;

receive, via said one or more web services endpoints, client requests to store ones of said data objects including a given client request to store a given one of said data objects;

dynamically determine a write plan for storing one or more replicas of a given data object to one or more corresponding ones of a plurality of storage nodes, wherein a plurality of distinct storage policies defined within said data storage web service are applicable to said given data object;

wherein said write plan is dynamically determined according to current state information of said plurality of storage nodes, and wherein to dynamically determine said write plan, the instructions are further computer-executable to select said one or more corresponding ones of said plurality of storage nodes such that a number of said plurality of storage policies satisfied by storing said one or more replicas of said given data object according to said write plan is maximized, and such that on at least some occasions of dynamically determining said write plan, at least two of said storage policies are satisfied by said write plan;

wherein a number of storage policies that are satisfiable by said write plan differs on different occasions of dynamically determining said write plan, dependent upon said current state information of said plurality of storage nodes; and

subsequent to a given client request to store said given data object being received according to a web services protocol, instruct that said one or more replicas of said given data object be stored to said one or more corresponding storage nodes according to said write plan.

49. The computer-accessible storage medium as recited in claim 48, wherein dynamically determining said write plan occurs in response to receiving said given client request.

50. The computer-accessible storage medium as recited in claim 48, wherein said current state information includes operational status information corresponding respectively to said plurality of storage nodes and indicative of whether a corresponding storage node is in a normal operational state, and wherein dynamically determining said write plan includes selecting said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said operational status information corresponding to said one or more storage nodes.

51. The computer-accessible storage medium as recited in claim 48, wherein said current state information includes load status information corresponding respectively to said plurality of storage nodes and indicative of a level of load under which a corresponding storage node is operating, and wherein dynamically determining said write plan includes selecting said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said load status information corresponding to said one or more storage nodes.

52. The computer-accessible storage medium as recited in claim 51, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of processor utilization corresponding to said given storage node.

53. The computer-accessible storage medium as recited in claim 51, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of memory utilization corresponding to said given storage node.

54. The computer-accessible storage medium as recited in claim 51, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of storage device input/output bandwidth utilization corresponding to said given storage node.

55. The computer-accessible storage medium as recited in claim 51, wherein for a given one of said plurality of storage nodes, said load status information is indicative of a level of network interface bandwidth utilization corresponding to said given storage node.

56. The computer-accessible storage medium as recited in claim 48, wherein said current state information includes network cost information corresponding respectively to said plurality of storage nodes and indicative of a cost of communication with a corresponding storage node, and wherein dynamically determining said write plan includes selecting said one or more storage nodes for storing a corresponding replica of said given data object dependent upon said network cost information corresponding to said one or more storage nodes.

57. The computer-accessible storage medium as recited in claim 56, wherein said network cost information varies over time.

58. The computer-accessible storage medium as recited in claim 56, wherein for a given one of said plurality of storage nodes, said network cost information includes an indication of network communication latency corresponding to said given storage node.

59. The computer-accessible storage medium as recited in claim 56, wherein for a given one of said plurality of storage nodes, said network cost information includes an indication of a network communication economic cost corresponding to said given storage node.

60. The computer-accessible storage medium as recited in claim 48, wherein dynamically determining said write plan includes minimizing a cost associated with storing said one or more replicas of said given data object according to said write plan.

61. The computer-accessible storage medium as recited in claim 60, wherein said cost varies according to time, and wherein minimizing said cost includes identifying a time period during which a particular one of said one or more replicas is to be stored to a particular one of said plurality of storage nodes, wherein identifying said time period is dependent upon a cost associated with storing said particular replica to said particular storage node during said time period.

62. The computer-accessible storage medium as recited in claim 48, wherein a given one of said storage policies specifies a minimum number of replicas of said given data object to be stored to corresponding ones of said storage nodes.

63. The computer-accessible storage medium as recited in claim 48, wherein said plurality of storage nodes is distributed among a plurality of areas, and wherein a given one of said storage policies specifies a minimum number of said plurality of areas among which said one or more replicas of said given data object are to be distributed.

64. The computer-accessible storage medium as recited in claim 48, wherein the instructions are further computer-executable to:

subsequent to said one or more replicas of said given data object being stored to said corresponding one or more storage nodes, detect a request to generate a new write plan, wherein said request is generated in response to determining that any of said one or more replicas is inaccessible;

in response to said request, dynamically determine a new write plan for storing one or more additional replicas of said given data object according to current state information of said plurality of storage nodes; and

instruct that said one or more additional replicas be stored to one or more corresponding ones of said storage nodes according to said new write plan.

65. The computer-accessible storage medium as recited in claim 48, wherein the instructions are further computer-executable to:

subsequent to said one or more replicas of said given data object being stored to said corresponding one or more storage nodes, detect a request to generate a new write plan, wherein said request is generated in response to determining that a number of storage policies satisfied by said stored one or more replicas of said given data object is insufficient;

in response to said request, dynamically determine a new write plan for storing one or more additional replicas of said given data object according to current state information of said plurality of storage nodes, wherein dynamically determining said write plan includes maximizing a number of storage policies satisfied by storing said one or more additional replicas of said given data object according to said new write plan; and

instruct that said one or more additional replicas be stored to one or more corresponding ones of said storage nodes according to said new write plan.

66. The computer-accessible storage medium as recited in claim 65, wherein determine that a number of storage policies satisfied by said stored one or more replicas of said given data object is insufficient comprises determining that said number of satisfied storage policies is less than a threshold value.

67. The computer-accessible storage medium as recited in claim 48, wherein the instructions are further computer-executable to:

dynamically determine a particular write plan for storing one or more replicas of a particular data object to one or more corresponding ones of said plurality of storage nodes in response to a client request to store said particular data object, wherein said plurality of storage policies are applicable to said particular data object, and wherein each of said storage policies has a corresponding weight;

wherein said particular write plan for storing said particular data object is dynamically determined according to current state information of said plurality of storage nodes, wherein to dynamically determine said write plan, said storage node selection logic is further configured to select said one or more corresponding ones of said plurality of storage nodes such that a total of the corresponding weights of said plurality of storage policies satisfied by storing said one or more replicas of said given data object according to said particular write plan is maximized; and

wherein a number of storage policies that are satisfiable by said particular write plan differs on different occasions of dynamically determining said particular write plan, dependent upon said current state information of said plurality of storage nodes.

68. The computer-accessible storage medium as recited in claim 48, wherein one or more of said storage policies are mandatory storage policies, and wherein the instructions are further computer-executable to:

in response to determining that no write plan exists that satisfies a given mandatory storage policy for said given data object, indicate that said given client request to store said given data object has failed.

69. The computer-accessible storage medium as recited in claim 48, wherein said plurality of storage policies comprises one or more of the following:

a locality policy configured to give preference to storage nodes in a same area as a node determining said write plan;

a load balancing policy configured to equalize write traffic among said storage nodes;

or

a space balancing policy configured to equalize storage resource capacity utilization among said storage nodes.

70. The computer-accessible storage medium as recited in claim 48, wherein said plurality of storage policies comprises a lowest-cost-chain policy configured to minimize a total cost of a sequence of storage node write operations included in said write plan.