Compacting dispersed storage space

• US 10,216,647 B2
• Filed: 10/11/2011
• Issued: 02/26/2019
• Est. Priority Date: 02/27/2010
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Volvovski, Ilya, Baptist, Andrew, Dhuse, Greg, Resch, Jason K.
• Primary Examiner(s): Shanmugasundaram, Kannan
• Application Number:
  US13/270,528
• Publication Number:
  US 20120110038A1
• Time in Patent Office: 2,695 Days
• Field of Search: 707822
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 2212/7205 Cleaning, compaction, garbage collection, erase control
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: H04L 41/06 Management of faults, events, alarms or notifications
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a processing module receiving an encoded data slice for storage in memory that is organized as a plurality of log files and identifying a log file based on information regarding the encoded data slice to produce an identified log file, wherein the identified log file is storing at least one other encoded data slice. The method continues with the processing module comparing storage parameters of the identified log file with desired storage parameters associated with the encoded data slice. The method continues with the processing module attempting to identify a second log file based on an alternate log file storage protocol when the storage parameters of the identified log file compare unfavorably with the desired storage parameters and when the second log file is identified, storing the encoded data slice in the second log file.

Claims

1. A method for execution by a storage unit, the method comprises:

receiving an encoded data slice for storage in physical memory of the storage unit, wherein the physical memory includes a plurality of storage locations, wherein the physical memory is virtually divided into a plurality of log files, and wherein each log file of the plurality of log files is associated with a unique set of storage locations of the plurality of storage locations, wherein a data object is partitioned into a plurality of data segments, wherein a data segment of the plurality of data segments is error encoded and sliced in accordance with distributed data storage parameters to produce a plurality of encoded data slices for distributed storage in a plurality of storage units that includes the storage unit, and wherein the plurality of encoded data slices includes the encoded data slice that is received for storage in physical memory of the storage unit;

determining a storage location of the plurality of storage locations for storing the encoded data slice by;

identifying a log file of the plurality of log files based on information regarding the encoded data slice corresponding to information regarding the log file to produce an identified log file, wherein the information regarding the encoded data slice includes at least one of;

a data identifier (ID) of a file associated with the encoded data slice, a user ID associated with the encoded data slice, and an indication of the log file contained in a message accompanying the encoded data slice, and wherein the identified log file is storing at least one other encoded data slice;

comparing storage parameters of the identified log file with desired storage parameters associated with the encoded data slice; and

when the storage parameters of the identified log file include at least one of;

the log file is identified as a most recently compacted log file;

the log file is identified in a slice location table lookup; and

the log file is identified based on a slice name associated with the encoded data slice,

identifying a storage location within the unique set of storage locations associated with the identified log file.

2. The method of claim 1 further comprises:

the comparing of the storage parameters of the identified log file with the desired storage parameters being unfavorable when the identified log file includes a number of storage gaps that compares unfavorably to a gap threshold.

3. The method of claim 1 further comprises:

the comparing of the storage parameters of the identified log file with the desired storage parameters being unfavorable when a storage balance between the identified log file and another log file compares unfavorably to a storage balance threshold.

4. The method of claim 1 further comprises:

the comparing of the storage parameters of the identified log file with the desired storage parameters being unfavorable when a storage capacity of the identified log file compares unfavorably to a storage threshold.

5. The method of claim 1 further comprises:

when the storage parameters of the identified log file compare unfavorably with the desired storage parameters;

attempting to identify a second log file of the plurality of log files based on an alternate log file storage protocol; and

when the second log file is not identified;

creating another log file; and

storing the encoded data slice in the other log file.

6. The method of claim 1 further comprises:

when the storage parameters of the identified log file compare unfavorably with the desired storage parameters;

attempting to identify a second log file of the plurality of log files based on an alternate log file storage protocol; and

when the second log file is identified, storing the encoded data slice in a storage location of the unique set of storage locations affiliated with the second log file.

7. A computer comprises:

an interface;

a memory; and

a processing module operable to;

receive, via the interface, an encoded data slice for storage physical memory of the storage unit, wherein the physical memory includes a plurality of storage locations, wherein the physical memory is virtually divided into a plurality of log files, and wherein each log file of the plurality of log files is associated with a unique set of storage locations of the plurality of storage locations, wherein a data object is partitioned into a plurality of data segments, wherein a data segment of the plurality of data segments is error encoded and sliced in accordance with distributed data storage parameters to produce a plurality of encoded data slices for distributed storage in a plurality of storage units that includes the storage unit, and wherein the plurality of encoded data slices includes the encoded data slice that is received for storage in physical memory of the storage unit;

determine a storage location of the plurality of storage locations for storing the encoded data slice by;

identifying a log file of the plurality of log files based on information regarding the encoded data slice corresponding to information regarding the log file to produce an identified log file, wherein the information regarding the encoded data slice includes at least one of;

a data identifier (ID) of a file associated with the encoded data slice, a user ID associated with the encoded data slice, and an indication of the log file contained in a message accompanying the encoded data slice, and wherein the identified log file is storing at least one other encoded data slice;

comparing storage parameters of the identified log file with desired storage parameters associated with the encoded data slice; and

when the storage parameters of the identified log file include at least one of;

the log file is identified as a most recently compacted log file;

the log file is identified in a slice location table lookup; and

the log file is identified based on a slice name associated with the encoded data slice

identifying a storage location within the unique set of storage locations associated with the identified log file.

8. The computer of claim 7, wherein the processing module further functions to:

compare the storage parameters of the identified log file with the desired storage parameters as unfavorable when the identified log file includes a number of storage gaps that compares unfavorably to a gap threshold.

9. The computer of claim 7, wherein the processing module further functions to:

compare the storage parameters of the identified log file with the desired storage parameters as unfavorable when a storage balance between the identified log file and another log file compares unfavorably to a storage balance threshold.

10. The computer of claim 7, wherein the processing module further functions to:

compare the storage parameters of the identified log file with the desired storage parameters as unfavorable when a storage capacity of the identified log file compares unfavorably to a storage threshold.

11. The computer of claim 7, wherein the processing module further functions to:

when the storage parameters of the identified log file compare unfavorably with the desired storage parameters;

attempting to identify a second log file of the plurality of log files based on an alternate log file storage protocol; and

when the second log file is not identified;

create another log file; and

store the encoded data slice in the other log file.

12. The computer of claim 7, wherein the processing module further functions to:

when the storage parameters of the identified log file compare unfavorably with the desired storage parameters;

attempt to identify a second log file of the plurality of log files based on an alternate log file storage protocol; and

when the second log file is identified, store the encoded data slice in a storage location of the unique set of storage locations affiliated with the second log file.