Generating additional slices based on data access frequency

• US 10,171,111 B2
• Filed: 08/26/2016
• Issued: 01/01/2019
• Est. Priority Date: 09/24/2015
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Peake, Andrew G.
• Primary Examiner(s): Torres, Joseph D
• Application Number:
  US15/249,130
• Publication Number:
  US 20170091032A1
• Time in Patent Office: 858 Days
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/154 Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 63/061 for key exchange, e.g. in peer-to-peer networks cryptographic mechanisms or cryptographic arrangements for key agreement H04L9/0838
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70

Abstract

A method for execution by a computing device of a dispersed storage network. The method begins by determining whether frequency of access to a set of encoded data slices exceeds a frequently accessed threshold. The method continues, when the frequency of access exceeds the frequently accessed threshold, by determining an access amount indicative of a degree that the frequency of access exceeds the frequently accessed threshold. The method continues by generating a number of additional encoded data slices and storing the number of additional encoded data slices in a number of additional storage units, wherein the set of storage units and the number of additional storage units produce an expanded set of storage units. The method continues by sending a plurality of data access requests to subsets of the expanded set of storage units in a distributed manner to improve processing efficiency of the plurality of data access requests.

Claims

1. A method for execution by a computing device of a dispersed storage network (DSN), the method comprises:

determining whether a frequency of access via the DSN from one or more other computing devices to a set of encoded data slices that is stored in a set of storage units of the DSN exceeds a frequently accessed threshold, wherein a data segment of a data object is dispersed storage error encoded in accordance with first dispersed error encoding parameters including a first encoding matrix to produce the set of encoded data slices that is stored in the set of storage units of the DSN, wherein the set of encoded data slices includes a pillar width number and a decode threshold number, wherein the pillar width number corresponds to number of encoded data slices in the set of encoded data slices, and wherein the decode threshold number corresponds to a number of encoded data slices of the set of encoded data slices to retrieve a corresponding data segment of the data object;

when the frequency of access via the DSN from the one or more other computing devices to the set of encoded data slices that is stored in the set of storage units of the DSN exceeds the frequently accessed threshold, determining an access amount indicative of a degree in which the frequency of access exceeds the frequently accessed threshold;

generating a number of additional encoded data slices for the set of encoded data slices based on the access amount in accordance with second dispersed error encoding parameters including a second encoding matrix that includes at least one of more rows or more columns than the first encoding matrix;

storing the number of additional encoded data slices in a number of additional storage units within the DSN, wherein the set of storage units and the number of additional storage units produce an expanded set of storage units within the DSN that includes more storage units than the set of storage units; and

sending, via the DSN from at least one of the computing device or the one or more other computing devices, a plurality of data access requests for the set of encoded data slices to different respective subsets of the expanded set of storage units in a distributed manner to load balance the plurality of data access requests for the set of encoded data slices among the expanded set of storage units within the DSN, wherein, over time, each storage unit of the expanded set of storage units within the DSN receives approximately an equal number of the plurality of data access requests and less than all of the plurality of data access requests.

2. The method of claim 1 further comprises:

when the frequency of access has exceeded the frequently accessed threshold and when the frequency of access is decreasing, determining a rate of the decreasing;

based on the rate of decreasing, determining whether one or more of the additional encoded data slices are to be deleted; and

when the one or more of the additional encoded data slices are to be deleted, deleting the one or more of the additional encoded data slices.

3. The method of claim 2 further comprises:

when the additional encoded data slices have been deleted, determining whether the frequency of access is below a second frequently accessed threshold, wherein the second frequently accessed threshold is less than the frequently accessed threshold;

when the frequency of access is below the second frequently accessed threshold, determining whether to delete one or more encoded data slices of the set of encoded data slices; and

when the one or more encoded data slices of the set of encoded data slices is to be deleted, deleting the one or more encoded data slices of the set of encoded data slices.

4. The method of claim 1 further comprises:

determining whether the frequency of access is below a second frequently accessed threshold, wherein the second frequently accessed threshold is less than the frequently accessed threshold;

when the frequency of access is below the second frequently accessed threshold, determining whether to delete one or more encoded data slices of the set of encoded data slices; and

when the one or more encoded data slices of the set of encoded data slices is to be deleted, deleting the one or more encoded data slices of the set of encoded data slices.

5. The method of claim 1, wherein the sending the plurality of data access requests for the set of encoded data slices to the subsets of the expanded set of storage units further comprises:

sending a first data access request of the plurality of data access requests to a first subset of the expanded set of storage units; and

sending, substantially concurrently with the first data access request, a second data access request of the plurality of data access requests to a second subset of the expanded set of storage units when an expanded pillar width number is equal to or greater than twice a read threshold number, wherein the expanded pillar width number corresponds to the expanded set of storage units.

6. The method of claim 1, wherein the generating the number of additional encoded data slices further comprises:

determining a cost associated with the expanded set of storage units;

when the cost is below a first cost threshold, generating a first number of additional encoded data slices;

when the cost is equal to or above the first cost threshold, generating a second number of additional encoded data slices, wherein the second number is less than the first number, and wherein the cost associated with the expanded set of storage units includes on at least one of;

historical performance;

bandwidth;

or

available storage.

7. The method of claim 1, wherein the determining the access amount comprises one or more of:

determining a volume of the plurality of data access requests;

determining a rate of increase of the plurality of data access requests; and

determining that the frequency of access has exceeded a second frequently accessed threshold, which is greater than the frequently accessed threshold.

8. A computing device comprises:

an interface configured to interface and communicate with a dispersed storage network (DSN);

memory; and

a processing module operably coupled to the memory and the interface, wherein the processing module is operable to;

determine whether a frequency of access via the DSN from one or more other computing devices to a set of encoded data slices that is stored in a set of storage units of the DSN exceeds a frequently accessed threshold, wherein a data segment of a data object is dispersed storage error encoded in accordance with first dispersed error encoding parameters including a first encoding matrix to produce the set of encoded data slices that is stored in the set of storage units of the DSN, wherein the set of encoded data slices includes a pillar width number and a decode threshold number, wherein the pillar width number corresponds to number of encoded data slices in the set of encoded data slices, and wherein the decode threshold number corresponds to a number of encoded data slices of the set of encoded data slices to retrieve a corresponding data segment of the data object;

when the frequency of access via the DSN from the one or more other computing devices to the set of encoded data slices that is stored in the set of storage units of the DSN exceeds the frequently accessed threshold, determine an access amount indicative of a degree in which the frequency of access exceeds the frequently accessed threshold;

generate a number of additional encoded data slices for the set of encoded data slices based on the access amount in accordance with second dispersed error encoding parameters including a second encoding matrix that includes at least one of more rows or more columns than the first encoding matrix;

store the number of additional encoded data slices in a number of additional storage units within the DSN, wherein the set of storage units and the number of additional storage units produce an expanded set of storage units within the DSN;

DSN that includes more storage units than the set of storage units; and

send, via the interface and via the DSN from at least one of the computing device or the one or more other computing devices, a plurality of data access requests for the set of encoded data slices to different respective subsets of the expanded set of storage units in a distributed manner to load balance the plurality of data access requests for the set of encoded data slices among the expanded set of storage units within the DSN, wherein, over time, each storage unit of the expanded set of storage units within the DSN receives approximately an equal number of the plurality of data access requests and less than all of the plurality of data access requests.

9. The computing device of claim 8, wherein the processing module is further operable to:

when the frequency of access has exceeded the frequently accessed threshold and when the frequency of access is decreasing, determine a rate of the decreasing;

based on the rate of decreasing, determine whether one or more of the additional encoded data slices are to be deleted; and

when the one or more of the additional encoded data slices are to be deleted, delete one or more of the additional encoded data slices.

10. The computing device of claim 9, wherein the processing module is further operable to:

when the additional encoded data slices have been deleted, determine whether the frequency of access is below a second frequently accessed threshold, wherein the second frequently accessed threshold is less than the frequently accessed threshold;

when the frequency of access is below the second frequently accessed threshold, determine whether to delete one or more encoded data slices of the set of encoded data slices; and

when the one or more encoded data slices of the set of encoded data slices is to be deleted, delete the one or more encoded data slices of the set of encoded data slices.

11. The computing device of claim 8, wherein the processing module is further operable to:

determine whether the frequency of access is below a second frequently accessed threshold, wherein the second frequently accessed threshold is less than the frequently accessed threshold;

when the frequency of access is below the second frequently accessed threshold, determine whether to delete one or more encoded data slices of the set of encoded data slices; and

when the one or more encoded data slices of the set of encoded data slices is to be deleted, delete the one or more encoded data slices of the set of encoded data slices.

12. The computing device of claim 8, wherein the processing module is further operable to send, via the interface, the plurality of data access requests for the set of encoded data slices to the subsets of the expanded set of storage units by:

sending, via the interface, a first data access request of the plurality of data access requests to a first subset of the expanded set of storage units; and

sending, via the interface, substantially concurrently with the first data access request, a second data access request of the plurality of data access requests to a second subset of the expanded set of storage units when an expanded pillar width number is equal to or greater than twice a read threshold number, wherein the expanded pillar width number corresponds to the expanded set of storage units.

13. The computing device of claim 8, wherein the processing module is further operable to generate the number of additional encoded data slices by:

determining a cost associated with the expanded set of storage units;

when the cost is below a first cost threshold, generating a first number of additional encoded data slices;

when the cost is equal to or above the first cost threshold, generating a second number of additional encoded data slices, wherein the second number is less than the first number, and wherein the cost associated with the expanded set of storage units includes on at least one of;

historical performance;

bandwidth;

or

available storage.

14. The computing device of claim 8, wherein the processing module is further operable to determine the access amount by:

determining a volume of the plurality of data access requests;

determining a rate of increase of the plurality of data access requests; and

determining that the frequency of access has exceeded a second frequently accessed threshold, which is greater than the frequently accessed threshold.