Reducing data stored when using multiple information dispersal algorithms

• US 10,241,694 B2
• Filed: 11/16/2016
• Issued: 03/26/2019
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Wozniak, Ethan S.
• Primary Examiner(s): McMahon, Daniel F.
• Application Number:
  US15/352,710
• Publication Number:
  US 20170192700A1
• Time in Patent Office: 860 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 12/0684 with feedback, e.g. presence or absence of unit detected by addressing, overflow detection
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/0888 using selective caching, e.g. bypass
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/2246 Trees, e.g. B+trees
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/263 Network storage, e.g. SAN or NAS
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0616 in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0652 : Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/0653 : Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 : Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 : Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0665 : at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 : using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 43/0852 : Delays
• CPC Class Codes: H04L 43/0876 : Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/0888 : Throughput
• CPC Class Codes: H04L 43/16 : Threshold monitoring
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Systems and methods for storing data in a dispersed storage network using at least two information dispersal algorithms (IDA'"'"' s) having different widths and thresholds are disclosed. In multiple IDA configurations, at least two IDA'"'"'s with different widths and thresholds are paired and used to store the data multiple times, where some IDA'"'"'s provide “wider” IDA configurations that are more reliable and other IDA'"'"'s provide “narrower” configurations with a lower threshold and lower reliability. Data can be written in the less reliable IDA configurations as a performance optimization to reduce the input/output operations necessary for reading the data. As a further optimization, the processing unit can determine to write only a subset of the IDA configurations. Similarly, dispersed storage units themselves, when reaching the capacity limits for their memory devices, can begin to delete slices they hold for some of the IDA configurations, to free up space.

Claims

1. A method of storing data in a dispersed storage network, the dispersed storage network including a first set of dispersed storage units and a second set of dispersed storage units, the method comprising:

encoding data utilizing first dispersal parameters to produce a first plurality of sets of encoded data slices;

facilitating storage of respective sets of encoded data slices of the first plurality of sets of encoded data slices in respective dispersed storage units of the first set of dispersed storage units;

encoding the data utilizing second dispersal parameters to produce a second plurality of sets of encoded data slices; and

facilitating storage of respective sets of encoded data slices of the second plurality of sets of encoded data slices in respective dispersed storage units of the second set of dispersed storage units.

2. The method of claim 1, further comprising selecting the first dispersal parameters based on one or more of a predetermination, a reliability requirement and a request.

3. The method of claim 1, wherein the step of facilitating storage of respective sets of encoded data slices of the first plurality of sets of encoded data slices further comprises identifying the first set of dispersed storage units, generating at least one set of first write slice requests where the at least one set of first write slice requests include the first plurality of sets of encoded data slices, and sending the at least one set of first write slice requests to the first set of dispersed storage units.

4. The method of claim 1, wherein the step of facilitating storage of respective sets of encoded data slices of the second plurality of sets of encoded data slices further comprises identifying the second set of dispersed storage units, generating at least one set of second write slice requests where the at least one set of second write slice requests include the second plurality of sets of encoded data slices, and sending the at least one set of second write slice requests to the second set of dispersed storage units.

5. The method of claim 1, further comprising determining whether to encode the data utilizing second dispersal parameters based on one or more of encoding processing capacity availability, storage capacity availability, network capacity availability and expected retrieval frequency of the data.

6. The method of claim 5, further comprising determining whether to delete the second plurality of sets of encoded data slices.

7. The method of claim 6, further comprising at least one of:

indicating to delete the second plurality of sets of encoded data slices when data retrieval frequency is lower than a first threshold;

indicating to delete the second plurality of sets of encoded data slices when a storage availability level is less than a second threshold;

or indicating to delete the second plurality of sets of encoded data slices when a network capacity availability level is greater than a third threshold.

8. The method of claim 6, further comprising issuing delete requests to the second set of dispersed storage units.

9. The method of claim 1, further comprising determining the second dispersal parameters based on one or more of a predetermination, a data retrieval performance requirement and a request.

10. A dispersed storage processing unit for use in a dispersed storage network, the dispersed storage network including a first set of dispersed storage units and a second set of dispersed storage units, the dispersed storage processing unit comprising:

a communications interface;

a memory; and

a computer processor;

where the memory includes instructions for causing the computer processor to;

encode data utilizing first dispersal parameters to produce a first plurality of sets of encoded data slices;

facilitate storage of respective sets of encoded data slices of the first plurality of sets of encoded data slices in respective dispersed storage units of the first set of dispersed storage units;

encode the data utilizing second dispersal parameters to produce a second plurality of sets of encoded data slices; and

facilitate storage of respective sets of encoded data slices of the second plurality of sets of encoded data slices in respective dispersed storage units of the second set of dispersed storage units.

11. The dispersed storage processing unit of claim 10, wherein the memory further comprises instructions for causing the computer processor to select the first dispersal parameters based on one or more of a predetermination, a reliability requirement and a request.

12. The dispersed storage processing unit of claim 10, wherein the memory further comprises instructions for causing the computer processor to identify the first set of dispersed storage units, generate at least one set of first write slice requests where the at least one set of first write slice requests include the first plurality of sets of encoded data slices, and send the at least one set of first write slice requests to the first set of dispersed storage units.

13. The dispersed storage processing unit of claim 10, wherein the memory further comprises instructions for causing the computer processor to identify the second set of dispersed storage units, generate at least one set of second write slice requests where the at least one set of second write slice requests include the second plurality of sets of encoded data slices, and send the at least one set of second write slice requests to the second set of dispersed storage units.

14. The dispersed storage processing unit of claim 10, wherein the memory further comprises instructions for causing the computer processor to determine whether to encode the data utilizing second dispersal parameters based on one or more of encoding processing capacity availability, storage capacity availability, network capacity availability and expected retrieval frequency of the data.

15. The dispersed storage processing unit of claim 14, wherein the memory further comprises instructions for causing the computer processor to determine whether to delete the second plurality of sets of encoded data slices.

16. The dispersed storage processing unit of claim 15, wherein the memory further comprises instructions for causing the computer processor to at least one of:

indicate to delete the second plurality of sets of encoded data slices when data retrieval frequency is lower than a first threshold;

indicate to delete the second plurality of sets of encoded data slices when a storage availability level is less than a second threshold;

or indicate to delete the second plurality of sets of encoded data slices when a network capacity availability level is greater than a third threshold.

17. The dispersed storage processing unit of claim 15, wherein the memory further comprises instructions for causing the computer processor to issue delete requests to the second set of dispersed storage units.

18. The dispersed storage processing unit of claim 10, wherein the memory further comprises instructions for causing the computer processor to determine the second dispersal parameters based on one or more of a predetermination, a data retrieval performance requirement and a request.

19. A dispersed storage network comprising:

a first set of dispersed storage units;

a second set of dispersed storage units; and

a disperse storage processing unit including;

a communications interface;

a memory; and

a computer processor;

where the memory includes instructions for causing the computer processor to;

encode data utilizing first dispersal parameters to produce a first plurality of sets of encoded data slices;

facilitate storage of respective sets of encoded data slices of the first plurality of sets of encoded data slices in respective dispersed storage units of the first set of dispersed storage units;

encode the data utilizing second dispersal parameters to produce a second plurality of sets of encoded data slices; and

facilitate storage of respective sets of encoded data slices of the second plurality of sets of encoded data slices in respective dispersed storage units of the second set of dispersed storage units.

20. The dispersed storage network of claim 19, wherein the memory further comprises instructions for causing the computer processor to select the first dispersal parameters based on one or more of a predetermination, a reliability requirement and a request.