DISPERSED STORAGE PROCESSING UNIT AND METHODS WITH DATA AGGREGATION FOR USE IN A DISPERSED STORAGE SYSTEM

• US 20110055473A1
• Filed: 03/30/2010
• Published: 03/03/2011
• Est. Priority Date: 09/30/2005
• Status: Active Grant

• Current Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Mark, Zachary J.
• Granted Patent: US 8,938,591 B2
• Time in Patent Office: Days
• US Class Current: 711/114
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0622 in relation to access
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A new data block to be stored in the dispersed storage system is received. When it is determined that a previous data segment contains sufficient space for the new data block, the previous data segment is retrieved from a plurality of dispersed storage units. A revised data segment is generated by aggregating the new data block with at least one existing data block of the previous data segment. A plurality of slices are generated for the revised data segment. The plurality of slices are stored in the plurality of dispersed storage units.

Claims

1. A method for use in a dispersed storage system, the method comprising:

receiving a new data block to be stored in the dispersed storage system;

determining when a previous data segment contains sufficient space for the new data block;

when the previous data segment contains sufficient space for the new data block;

retrieving the previous data segment from a plurality of dispersed storage units;

generating a revised data segment by aggregating the new data block with at least one existing data block of the previous data segment;

generating a plurality of slices for the revised data segment; and

storing the plurality of slices in the plurality of dispersed storage units.

2. The method of claim 1 wherein retrieving the previous data segment from the plurality of dispersed storage units, includes:

retrieving a plurality of previous slices corresponding to the previous data segment; and

deslicing and decoding the plurality of previous slices to regenerate the previous data segment.

3. The method of claim 1 further comprising encoding the revised data segment.

4. The method of claim 1 further comprising:

generating metadata associated with the revised data segment.

5. The method of claim 1 wherein generating the metadata includes aggregating the metadata with metadata associated with the previous data segment.

6. The method of claim 1 wherein generating the revised data segment includes:

padding the revised data segment with padding data to fill the revised data segment when the aggregation of the new data block with the at least one existing data block of the previous data segment does not fill the revised data segment.

7. The method of claim 1 wherein generating the revised data segment includes:

removing padding data from the previous data segment.

8. The method of claim 1 wherein generating the revised data segment includes:

storing an indication of available space when the aggregation of the new data block with the at least one existing data block of the previous data segment does not fill the revised data segment.

9. The method of claim 1 wherein generating the revised data segment includes:

storing a segment filled indication when the aggregation of the new data block with the at least one existing data block of the previous data segment fills the revised data segment.

10. The method of claim 1 further comprising, when the previous data segment contains sufficient space for the new data block:

generating a new data segment;

storing an indication of available space associated with the new data segment;

generating a plurality of slices for the new data segment; and

storing the plurality of slices in the plurality of dispersed storage units.

11. A dispersed storage processing unit for use in a dispersed storage system, the dispersed storage processing unit comprising:

an access module that receives a new data block to be stored in the dispersed storage system, that determines when a previous data segment contains sufficient space for the new data block, when the previous data segment contains sufficient space for the new data block, the access module;

retrieves the previous data segment from a plurality of dispersed storage units; and

generates a revised data segment by aggregating the new data block with at least one existing data block of the previous data segment;

a grid module, coupled to the access module, that generates a plurality of slices for the revised data segment; and

a storage module, coupled to the grid module, that stores the plurality of slices in the plurality of dispersed storage units.

12. The dispersed storage processing unit of claim 11 wherein the access module commands the storage module and the grid module to retrieve the previous data segment from the plurality of dispersed storage units by:

retrieving a plurality of previous slices corresponding to the previous data segment; and

deslicing and decoding the plurality of previous slices to regenerate the previous data segment.

13. The dispersed storage processing unit of claim 11 wherein the access module further:

generates metadata associated with the revised data segment.

14. The dispersed storage processing unit of claim 13 wherein the access module generates the metadata by aggregating metadata for the new data block with metadata associated with the previous data segment.

15. The dispersed storage processing unit of claim 11 wherein the access module generates the revised data segment by further:

padding the revised data segment with padding data to fill the revised data segment when the aggregation of the new data block with the at least one existing data block of the previous data segment does not fill the revised data segment.

16. The dispersed storage processing unit of claim 11 wherein the access module generates the revised data segment by further:

removing padding data from the previous data segment.

17. The dispersed storage processing unit of claim 11 wherein the access module generates the revised data segment by further:

storing an indication of available space when the aggregation of the new data block with the at least one existing data block of the previous data segment does not fill the revised data segment.

18. The dispersed storage processing unit of claim 11 wherein the access module generates the revised data segment by further:

storing a segment filled indication when the aggregation of the new data block with the at least one existing data block of the previous data segment fills the revised data segment.

19. The dispersed storage processing unit of claim 11 when the previous data segment contains sufficient space for the new data block, the access module:

generates a new data segment; and

stores an indication of available space associated with the new data segment;

wherein the grid module generates a plurality of slices for the new data segment; and

wherein the storage module stores the plurality of slices in the plurality of dispersed storage units.

20. A method for use in a dispersed storage system, the method comprising:

receiving a plurality of data blocks to be stored in the dispersed storage system, each of the plurality of data blocks including data block metadata and data;

mapping each of the plurality of data blocks to a corresponding one of a plurality of aggregated data blocks of fixed length;

packing the data from each of the a plurality of data blocks into a data section of the corresponding one of the plurality of aggregated data blocks, wherein data from at least one of the plurality of data blocks has a length that is less than a length of the data section of the corresponding one of the plurality of aggregated data blocks;

generating aggregated metadata for each of the plurality of aggregated data blocks, based in part on the data block metadata from a corresponding one of the plurality of data blocks; and

storing the plurality of aggregated data blocks in a plurality of dispersed storage units.

21. A method for use in a dispersed storage system, the method comprising:

receiving a new data block to be stored in the dispersed storage system;

generating a revised data segment by aggregating the new data block into an existing data segment stored in a plurality of dispersed storage units.