Hierarchic storage policy for distributed object storage systems

• US 10,379,951 B2
• Filed: 09/12/2018
• Issued: 08/13/2019
• Est. Priority Date: 12/29/2016
• Status: Active Grant

• Current Assignee: JP Morgan Chase Bank, N.A. (JP Morgan Chase & Co)
• Original Assignee: Western Digital Technologies Incorporated (Western Digital Corporation)
• Inventors: De Keyser, Koen, De Schrijver, Frederik Jacqueline Luc, Blyweert, Stijn
• Primary Examiner(s): Abraham, Esaw T
• Application Number:
  US16/129,364
• Publication Number:
  US 20190012235A1
• Time in Patent Office: 335 Days
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Techniques for distributing data in a distributed data storage system using a hierarchy rule that is generated based on a spreading policy and a set of tolerable failures specified by a user in absence of system deployment information are disclosed. The system includes a controller node which receives a request including a spreading policy and a protection level for spreading a first data object. The controller node determines a hierarchy rule corresponding to the spreading policy based on the protection level. The controller node distributes the first data object in the system using the hierarchy rule and the spreading policy. The controller node receives a reconfiguration of system deployment. The controller node distributes a second data object in the system based on providing protection of the protection level to the second data object without affecting protection of the same protection level applied to the first data object.

Claims

1. A computer-implemented method comprising:

receiving, by one or more processors of a controller node, a request for spreading data of a data object, the request including;

a spreading policy defining a number of data storage devices storing a portion of the data object that are allowed to fail; and

a protection level defining a set of tolerable failure scenarios;

determining, by the one or more processors, a hierarchy rule corresponding to the spreading policy based on the protection level; and

distributing, by the one or more processors, the data of the data object in a distributed object storage system using the hierarchy rule and the spreading policy, such that the protection level applies to the data object and a previously-stored data object to ensure that data of the data object and data of the previously-stored data object is recovered responsive to a failure scenario of the set of tolerable failure scenarios occurring.

2. The computer-implemented method of claim 1, wherein determining the hierarchy rule corresponding to the spreading policy comprises:

determining whether the spreading policy can be translated to a value associated with a hierarchy level based on the set of tolerable failure scenarios, the value indicating a maximum number of data storage devices on the hierarchy level for spreading the data of the data object to;

determining whether the value associated with the hierarchy level has been identified; and

in response to determining that the value associated with the hierarchy level has been identified, generating the hierarchy rule corresponding to the spreading policy based on the identified value.

3. The computer-implemented method of claim 2, further comprising:

identifying a hierarchical deployment configuration of the distributed object storage system; and

determining whether the hierarchy rule is compatible with the hierarchical deployment configuration of the distributed object storage system, wherein distributing the data of the data object in the distributed object storage system is based on determining that the hierarchy rule is compatible with the hierarchical deployment configuration of the distributed object storage system.

4. The computer-implemented method of claim 3, wherein:

determining whether the hierarchy rule is compatible with the hierarchical deployment configuration of the distributed object storage system comprises determining whether each entity of the hierarchy level meets the maximum number of data storage devices associated with the hierarchy level without receiving information of data storage devices used at other entities of the hierarchy level; and

an entity includes at least one of a storage node, a rack, and a data center.

5. The computer-implemented method of claim 3, further comprising:

in response to determining that the hierarchy rule is incompatible with the hierarchical deployment configuration of the distributed object storage system;

notifying a user; and

instructing the user to modify the spreading policy or the protection level.

6. The computer-implemented method of claim 2, wherein the hierarchy level comprises:

a level where the data storage devices are grouped according to respective storage nodes;

a level where the storage nodes are grouped to respective racks; and

a level where the racks are grouped according to respective data centers.

7. The computer-implemented method of claim 1, wherein the spreading policy comprises:

a desired spreading width; and

a maximum concurrent failure tolerance.

8. A distributed object storage system comprising:

a controller node having a memory and one or more processors configured to execute instructions stored on the memory, wherein the controller node is configured to perform operations comprising;

receiving a request for spreading data of a data object, the request including;

a spreading policy defining a number of data storage devices storing a portion of the data object that are allowed to fail; and

a protection level defining a set of tolerable failure scenarios;

determining a hierarchy rule corresponding to the spreading policy based on the protection level; and

distributing the data of the data object in the distributed object storage system using the hierarchy rule and the spreading policy, such that the protection level applies to the data object and a previously-stored data object to ensure that data of the data object and data of the previously-stored data object is recovered responsive to a failure scenario of the set of tolerable failure scenarios occurring.

9. The distributed object storage system of claim 8, wherein to determine the hierarchy rule corresponding to the spreading policy, the operations further comprise:

determining whether the spreading policy can be translated to a number associated with a hierarchy level based on the set of tolerable failure scenarios, the number indicating a threshold number of data storage devices on the hierarchy level onto which the data object is to be spread;

determining whether the number associated with each hierarchy level of a hierarchical configuration has been identified; and

in response to determining that the number associated with each hierarchy level of the hierarchical configuration has been identified, generating the hierarchy rule corresponding to the spreading policy based on the identified number.

10. The distributed object storage system of claim 9, wherein the operations further comprise:

identifying a hierarchical deployment configuration of the distributed object storage system; and

determining whether the hierarchy rule is compatible with the hierarchical deployment configuration of the distributed object storage system, wherein distributing the data of the data object in the distributed object storage system is based on determining that the hierarchy rule is compatible with the hierarchical deployment configuration of the distributed object storage system.

11. The distributed object storage system of claim 10, wherein to determine whether the hierarchy rule is compatible with the hierarchical deployment configuration of the distributed object storage system, the controller node is further configured to perform the operations comprising determining whether an entity of the hierarchy level meets the threshold number of data storage devices associated with the hierarchy level without receiving information of data storage devices used at other entities of the hierarchy level.

12. The distributed object storage system of claim 10, wherein the operations further comprise:

in response to determining that the hierarchy rule is incompatible with the hierarchical deployment configuration of the distributed object storage system;

notifying a user; and

instructing the user to modify at least one of the spreading policy and the protection level.

13. The distributed object storage system of claim 9, wherein the hierarchy level comprises:

a level where the data storage devices are grouped according to respective storage nodes;

a level where the storage nodes are grouped to respective racks; and

a level where the racks are grouped according to respective data centers.

14. The distributed object storage system of claim 8, wherein the spreading policy comprises:

a desired spreading width; and

a maximum concurrent failure tolerance.

15. A distributed object storage system comprising:

means for receiving a request for spreading data of a data object, the request including;

a spreading policy defining a number of data storage devices storing a portion of the data object that are allowed to fail; and

a protection level defining a set of tolerable failure scenarios;

means for determining a hierarchy rule corresponding to the spreading policy based on the protection level; and

means for distributing the data of the data object in a distributed object storage system using the hierarchy rule and the spreading policy, such that the protection level applies to the data object and a previously-stored data object to ensure that data of the data object and data of the previously-stored data object is recovered responsive to a failure scenario of the set of tolerable failure scenarios occurring.

16. The distributed object storage system of claim 15, wherein the means for determining the hierarchy rule corresponding to the spreading policy comprises:

means for determining whether the spreading policy can be translated to a value associated with a hierarchy level based on the set of tolerable failure scenarios, the value indicating a threshold number of data storage devices on the hierarchy level over which to spread the data object; and

means for generating the hierarchy rule corresponding to the spreading policy based on the value associated with the hierarchy level.

17. The distributed object storage system of claim 16, further comprising:

means for identifying a hierarchical deployment configuration of the distributed object storage system; and

means for determining whether the hierarchy rule is compatible with the hierarchical deployment configuration of the distributed object storage system, wherein distributing the data of the data object in the distributed object storage system is based on determining that the hierarchy rule is compatible with the hierarchical deployment configuration of the distributed object storage system.

18. The distributed object storage system of claim 17, further comprising:

means for notifying a user in response to determining that the hierarchy rule is incompatible with the hierarchical deployment configuration of the distributed object storage system; and

means for instructing the user to modify the spreading policy and/or the protection level.

19. The distributed object storage system of claim 16, wherein the hierarchy level comprises:

a level where the data storage devices are grouped according to respective storage nodes;

a level where the storage nodes are grouped to respective racks; and

a level where the racks are grouped according to respective data centers.

20. The distributed object storage system of claim 15, wherein the spreading policy comprises:

a desired spreading width; and

a maximum concurrent failure tolerance.