Load balancing and service differentiation within a dispersed storage network

• US 9,971,538 B2
• Filed: 11/21/2016
• Issued: 05/15/2018
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Badanahatti, Venkata Giri, Hegde, Harsha
• Primary Examiner(s): Tu, Christine T.
• Application Number:
  US15/356,929
• Publication Number:
  US 20170153853A1
• Time in Patent Office: 540 Days
• Field of Search: 714763, 714764, 714767
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/616 Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561

Abstract

Methods for use in a dispersed storage network (DSN) to determine appropriate resources for storing data. In various examples, a network control unit obtains status information relating to the DSN. The status information, which may be updated periodically, may include information relating to storage unit utilization, network loading and performance levels, data size indicators, access profiles specifying differentiated service levels, etc. Based on the status information, the network control unit maps each of a plurality of data classes to a plurality of sets of storage units, and generates network control information based on the mapping. The network control information, which may include routing instructions and storage set utilization information, is issued to a network routing unit for use in facilitating storage (e.g., via IP address translation) of differing classes of data in the sets of storage units in accordance with the mapping.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the DSN having a network routing unit and storage resources including a plurality of sets of storage units, the method comprises:

obtaining status information relating to storage units of the plurality of sets of storage units;

mapping, based at least in part on the status information, each of a plurality of data classes to the plurality of sets of storage units, including mapping at least a first data class to a first set of storage units and a second data class to a second set of storage units, the plurality of data classes relating to characteristics of data objects to be stored in a dispersed manner in the plurality of sets of storage units;

generating network control information based on the mapping, the network control information including at least one of routing instructions or storage set utilization information for use in facilitating storage of data of the first data class and the second data class in the plurality of sets of storage units; and

issuing the network control information to the network routing unit.

2. The method of claim 1, wherein obtaining status information relating to the storage units includes at least one of:

interpreting a network query, receiving a portion of the status information from a computing device of the DSN, receiving a portion of the status information from at least one of the storage units of the plurality of sets of storage units, receiving the status information from the network routing unit, or interpreting a test result.

3. The method of claim 1, wherein the status information includes information relating to at least one of:

a utilization level of the storage units, a DSN network loading level, a DSN network performance level, a storage unit access performance level, a data storage reliability level, a data size indicator, a slice size indicator, an accessing entity identifier (ID), or an access profile.

4. The method of claim 3, wherein the status information is updated on a periodic basis.

5. The method of claim 1, wherein the plurality of data classes include data classes relating to at least one of:

data types, data object size indicators, slice size indicators, data owner identifiers, data availability requirements, desired data storage reliability levels, source names, data names, or system registry information.

6. The method of claim 1, wherein the first set of storage units of the plurality of sets of storage units is affiliated with a first family of Internet Protocol (IP) addresses and the second set of storage units of the plurality of sets of storage units is affiliated with a second family of IP addresses, and wherein the network control information references the first family of IP addresses and the second family of IP addresses.

7. The method of claim 1 further comprises:

receiving a data object for storage;

dispersed storage error encoding the data object to produce a plurality of sets of encoded data slices, wherein the data object is segmented into a plurality of data segments and wherein each data segment is dispersed storage error encoded into a respective set of encoded data slices; and

facilitating storage of the plurality of sets of encoded data slices in at least one of the first set of storage units and the second set of storage units in accordance with the network control information.

8. The method of claim 7, wherein facilitating storage of the plurality of sets of encoded data slices includes Internet Protocol (IP) address translation involving IP addresses affiliated with at least one of the first set of storage units or the second set of storage units.

9. A computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processing modules of a dispersed storage network (DSN) that include a processor and a memory, causes the one or more processing modules to;

obtain status information relating to at least one of a plurality of sets of storage units of the DSN or data to be stored in the plurality of sets of storage units;

map, based at least in part on the status information, each of a plurality of data classes to the plurality of sets of storage units, including mapping at least a first data class to a first set of storage units and a second data class to a second set of storage units, the plurality of data classes relating to characteristics of data objects to be stored in the plurality of sets of storage units;

generate network control information based on the mapping, the network control information including routing instructions for use in facilitating storage of data of the first data class and the second data class in the plurality of sets of storage units; and

issue the network control information to a network routing unit of the DSN.

10. The computer readable storage medium of claim 9, wherein obtaining status information relating to at least one of a plurality of sets of storage units includes at least one of:

interpreting a network query, receiving a portion of the status information from a computing device of the DSN, receiving a portion of the status information from at least one of storage unit of the plurality of sets of storage units, receiving the status information from a network routing unit, or interpreting a test result.

11. The computer readable storage medium of claim 9, wherein the status information includes information relating to at least one of:

a utilization level of storage units of the first set of storage units and the second set of storage units, a DSN network loading level, a DSN network performance level, a storage unit access performance level, a data storage reliability level, a data size indicator, a slice size indicator, an accessing entity identifier (ID), or an access profile.

12. The computer readable storage medium of claim 9 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more processing modules to obtain updated status information on a periodic basis.

13. The computer readable storage medium of claim 9, wherein the plurality of data classes include data classes relating to at least one of:

data types, data object size indicators, slice size indicators, data owner identifiers, data availability requirements, desired data storage reliability levels, source names, data names, or system registry information.

14. The computer readable storage medium of claim 9, wherein the first set of storage units of the plurality of sets of storage units is affiliated with a first family of Internet Protocol (IP) addresses and the second set of storage units of the plurality of sets of storage units is affiliated with a second family of IP addresses, the first family of IP addresses and the second family of IP addresses referenced by the network control information.

15. A computing device of a group of computing devices of a dispersed storage network (DSN), the DSN having a network routing unit and storage resources including a plurality of sets of storage units, the computing device comprises:

a network interface;

a local memory; and

a processing module operably coupled to the network interface and the local memory, wherein the processing module operates to;

obtain, via the network interface, status information relating to storage units of the plurality of sets of storage units;

map, based at least in part on the status information, each of a plurality of data classes to the plurality of sets of storage units, including mapping at least a first data class to a first set of storage units and a second data class to a second set of storage units, the plurality of data classes relating to characteristics of data objects to be stored in the plurality of sets of storage units;

generate network control information based on the mapping, the network control information including at least one of routing instructions or storage set utilization information for use in facilitating storage of data of the first data class and the second data class in the plurality of sets of storage units; and

issue, via the network interface, the network control information to the network routing unit.

16. The computing device of claim 15, wherein obtaining status information relating to the storage units includes at least one of:

interpreting a network query, receiving a portion of the status information from a computing device of the DSN, receiving a portion of the status information from at least one of the storage units of the plurality of sets of storage units, receiving the status information from the network routing unit, or interpreting a test result.

17. The computing device of claim 15, wherein the status information includes information relating to at least one of:

a utilization level of the storage units, a DSN network loading level, a DSN network performance level, a storage unit access performance level, a data storage reliability level, a data size indicator, a slice size indicator, an accessing entity identifier (ID), or an access profile.

18. The computing device of claim 15, wherein the processing module further operates to:

obtain, via the network interface, updated status information on a periodic basis.

19. The computing device of claim 15, wherein the plurality of data classes include data classes relating to at least one of:

data types, data object size indicators, slice size indicators, data owner identifiers, data availability requirements, desired data storage reliability levels, source names, data names, or system registry information.

20. The computing device of claim 15, wherein the first set of storage units of the plurality of sets of storage units is affiliated with a first family of Internet Protocol (IP) addresses and the second set of storage units of the plurality of sets of storage units is affiliated with a second family of IP addresses, the first family of IP addresses and the second family of IP addresses referenced by the network control information.