Adjusting a dispersal parameter of dispersedly stored data

• US 8,843,804 B2
• Filed: 03/06/2012
• Issued: 09/23/2014
• Est. Priority Date: 04/01/2011
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Resch, Jason K., Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): Merant, Guerrier
• Assistant Examiner(s): GANDHI, DIPAKKUMAR B
• Application Number:
  US13/413,452
• Publication Number:
  US 20120254691A1
• Time in Patent Office: 931 Days
• US Class Current: 714/763
• CPC Class Codes: G06F 11/1044 with specific ECC/EDC distribution
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1441 Resetting or repowering
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0646 Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/09 Error detection only, e.g. using cyclic redundancy check [CRC] codes or single parity bit
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/23 using convolutional codes, e.g. unit memory codes
• CPC Class Codes: H03M 13/2707 Simple row-column interleaver, i.e. pure block interleaving
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a processing module storing data files utilizing a dispersed storage error coding function that includes a pillar width parameter and a decode threshold parameter. The method continues with the processing module determining whether to adjust the pillar width parameter based one or more memory performance characteristics. When the pillar width parameter is to be decreased, the method continues with the processing module identifying one or more pillars within a memory to delete to produce one or more identified pillars, identifying encoded data slices of one or more of the data files stored in the one or more identified pillars to produce identified encoded data slices, and deleting the identified encoded data slices.

Claims

1. A method comprises:

storing data files utilizing a dispersed storage error coding function, wherein a data file of the data files is encoded using the dispersed storage error coding function to produce a plurality of sets of encoded data slices, wherein the plurality of sets of encoded data slices is stored in memory, and wherein the dispersed storage error coding function includes a pillar width parameter and a decode threshold parameter, where the pillar width parameter is at least 1.8 times the decode threshold parameter;

determining whether to adjust the pillar width parameter based one or more memory performance characteristics; and

when the pillar width parameter is to be decreased;

identifying one or more pillars within the memory to delete to produce one or more identified pillars;

identifying encoded data slices of one or more of the data files stored in the one or more identified pillars to produce identified encoded data slices; and

deleting the identified encoded data slices.

2. The method of claim 1, wherein determining to decrease the pillar width parameter comprises:

determining a memory utilization indicator associated with the memory;

determining a memory reliability indicator associated with the memory; and

when the memory utilization indicator is unfavorable and the memory reliability indicator is favorable, indicating a decrease of the pillar width parameter.

3. The method of claim 1, wherein the identifying one or more pillars within the memory to delete comprises:

determining an amount of memory space to reclaim based on at least one of;

a memory utilization indicator and a memory reliability indicator;

identifying one or more of the data files based on data file criteria; and

determining a number of pillars to be deleted based on the amount of memory space to reclaim and the identified one or more of the data files.

4. The method of claim 1, wherein the deleting the identified encoded data slices comprises:

reclaiming memory space of the deleted encoded data slices to produce reclaimed memory space; and

updating pillar mapping of the memory in accordance with the decreasing of the pillar width parameter and the reclaimed memory space.

5. The method of claim 1, wherein the identifying encoded data slices comprises:

identifying the one or more of the data files; and

for each of the one or more identified data files;

determining which of the encoded data slices of a respective plurality of sets of encoded data slices are stored in the one or more identified pillars to produce data file specific encoded data slices, wherein the identified encoded data slices includes the data file specific encoded data slices for each of the one or more identified data files.

6. The method of claim 1, wherein the memory comprises one or more of:

a local memory; and

a dispersed storage network (DSN) memory.

7. The method of claim 1 further comprises:

encoding a subsequent data file utilizing the decreasing of the pillar width parameter, the decode threshold parameter, and the dispersed storage error coding function to produce a subsequent plurality of sets of encoded data slices; and

storing the subsequent plurality of sets of encoded data slices in the memory.

8. A method comprises:

storing data files utilizing a dispersed storage error coding function, wherein a data file of the data files is encoded using the dispersed storage error coding function to produce a plurality of sets of encoded data slices, wherein the plurality of sets of encoded data slices is stored in memory, and wherein the dispersed storage error coding function includes a pillar width parameter and a decode threshold parameter, where the pillar width parameter is greater than the decode threshold parameter;

determining whether to adjust the pillar width parameter based one or more memory performance characteristics; and

when the pillar width parameter is to be increased;

determining a number of additional pillars to produce an increased pillar width parameter;

identifying one or more of the data files based on data file criteria; and

for each of the one or more data files;

encoding a data file of the one or more data files utilizing the increased pillar width parameter, the decode threshold parameter, and the dispersed storage error coding function to produce a plurality of subsets of encoded data slices relating to the number of additional pillars; and

storing the plurality of subsets of encoded data slices in the memory corresponding to the additional pillars.

9. The method of claim 8 further comprises:

the determining to increase the pillar width parameter including;

determining a memory utilization indicator associated with the memory;

determining a memory reliability indicator associated with the memory; and

when the memory utilization indicator is favorable and the memory reliability indicator is unfavorable, indicating an increase of the pillar width parameter; and

the determining the number of additional pillars including;

determining a level of unfavorability of the memory reliability indicator; and

determining the number of additional pillars based on the level of unfavorability.

10. The method of claim 8, wherein the storing the plurality of subsets of encoded data slices comprises:

updating pillar mapping of the memory in accordance with the increasing of the pillar width parameter.

11. The method of claim 8, wherein the memory comprises one or more of:

a local memory; and

a dispersed storage network (DSN) memory.

12. The method of claim 8 further comprises:

encoding a subsequent data file utilizing the increased pillar width parameter, the decode threshold parameter, and the dispersed storage error coding function to produce a subsequent plurality of sets of encoded data slices; and

storing the subsequent plurality of sets of encoded data slices in the memory.

13. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

store data files utilizing a dispersed storage error coding function, wherein a data file of the data files is encoded using the dispersed storage error coding function to produce a plurality of sets of encoded data slices, wherein the plurality of sets of encoded data slices is stored in memory, and wherein the dispersed storage error coding function includes a pillar width parameter and a decode threshold parameter, where the pillar width parameter is at least 1.8 times the decode threshold parameter;

a second module, when operable within the computing device, causes the computing device to;

determine whether to adjust the pillar width parameter based one or more memory performance characteristics; and

when the pillar width parameter is to be decreased, a third module, when operable within the computing device, causes the computing device to;

identify one or more pillars within the memory to delete to produce one or more identified pillars;

identify encoded data slices of one or more of the data files stored in the one or more identified pillars to produce identified encoded data slices; and

delete the identified encoded data slices.

14. The DS module of claim 13, wherein the second module functions to determine to decrease the pillar width parameter by:

determining a memory utilization indicator associated with the memory;

determining a memory reliability indicator associated with the memory; and

when the memory utilization indicator is unfavorable and the memory reliability indicator is favorable, indicating a decrease of the pillar width parameter.

15. The DS module of claim 13, wherein the third module functions to identify the one or more pillars within the memory to delete by:

determining an amount of memory space to reclaim based on at least one of;

a memory utilization indicator and a memory reliability indicator;

identifying one or more of the data files based on data file criteria; and

determining a number of pillars to be deleted based on the amount of memory space to reclaim and the identified one or more of the data files.

16. The DS module of claim 13, wherein the third module functions to delete the identified encoded data slices by:

reclaiming memory space of the deleted encoded data slices; and

updating pillar mapping of the memory in accordance with the decreasing of the pillar width parameter and the reclaimed memory space.

17. The DS module of claim 13, wherein the third module functions to identify the encoded data slices by:

identifying the one or more of the data files; and

for each of the one or more identified data files;

determining which of the encoded data slices of a respective plurality of sets of encoded data slices are stored in the one or more identified pillars to produce data file specific encoded data slices, wherein the identified encoded data slices includes the data file specific encoded data slices for each of the one or more identified data files.

18. The DS module of claim 13, wherein the memory comprises one or more of:

a local memory; and

a dispersed storage network (DSN) memory.

19. The DS module of claim 13 further comprises:

the first module, when operable within the computing device, further causes the computing device to;

encode a subsequent data file utilizing the decreased pillar width parameter, the decode threshold parameter, and the dispersed storage error coding function to produce a subsequent plurality of sets of encoded data slices; and

store the subsequent plurality of sets of encoded data slices in the memory.

20. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

store data files utilizing a dispersed storage error coding function, wherein a data file of the data files is encoded using the dispersed storage error coding function to produce a plurality of sets of encoded data slices, wherein the plurality of sets of encoded data slices is stored in memory, and wherein the dispersed storage error coding function includes a pillar width parameter and a decode threshold parameter, where the pillar width parameter is greater than the decode threshold parameter;

a second module, when operable within the computing device, causes the computing device to;

determine whether to adjust the pillar width parameter based one or more memory performance characteristics; and

when the pillar width parameter is to be increased, a third module, when operable within the computing device, causes the computing device to;

determine a number of additional pillars to produce an increased pillar width parameter;

identify one or more of the data files based on data file criteria; and

for each of the one or more data files;

encode a data file of the one or more data files utilizing the increased pillar width parameter, the decode threshold parameter, and the dispersed storage error coding function to produce a plurality of subsets of encoded data slices relating to the number of additional pillars; and

store the plurality of subsets of encoded data slices in the memory corresponding to the additional pillars.

21. The DS module of claim 20 further comprises:

the second module, when operable within the computing device, further causes the computing device to determine to increase the pillar width parameter by one or more of;

determining a memory utilization indicator associated with the memory;

determining a memory reliability indicator associated with the memory; and

when the memory utilization indicator is favorable and the memory reliability indicator is unfavorable, indicating an increase of the pillar width parameter; and

the third module, when operable within the computing device, further causes the computing device to determine the number of additional pillars by one or more of;

determining a level of unfavorability of the memory reliability indicator; and

determining the number of additional pillars based on the level of unfavorability.

22. The DS module of claim 20, wherein the third module further functions to store the plurality of subsets of encoded data slices by:

updating pillar mapping of the memory in accordance with the increasing of the pillar width parameter.

23. The DS module of claim 20, wherein the memory comprises one or more of:

a local memory; and

a dispersed storage network (DSN) memory.

24. The DS module of claim 20 further comprises:

the first module, when operable within the computing device, further causes the computing device to;

encode a subsequent data file utilizing the increased pillar width parameter, the decode threshold parameter, and the dispersed storage error coding function to produce a subsequent plurality of sets of encoded data slices; and

store the subsequent plurality of sets of encoded data slices in the memory.