Dispersed storage processing unit and methods with data aggregation for use in a dispersed storage system

• US 8,938,591 B2
• Filed: 03/30/2010
• Issued: 01/20/2015
• Est. Priority Date: 09/30/2005
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Mark, Zachary J., Gladwin, S. Christopher
• Primary Examiner(s): Bradley, Matthew
• Assistant Examiner(s): TSUI, DANIEL D
• Application Number:
  US12/749,592
• Publication Number:
  US 20110055473A1
• Time in Patent Office: 1,757 Days
• Field of Search: None
• US Class Current: 711/154
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0622 in relation to access
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A new data block to be stored in the dispersed storage system is received. When it is determined that a previous data segment contains sufficient space for the new data block, the previous data segment is retrieved from a plurality of dispersed storage units. A revised data segment is generated by aggregating the new data block with at least one existing data block of the previous data segment. A plurality of slices are generated for the revised data segment. The plurality of slices are stored in the plurality of dispersed storage units.

Claims

1. A method for use in a dispersed storage system, the method comprising:

receiving a new data block to be stored in the dispersed storage system;

determining when a previous data segment contains sufficient space for the new data block;

when the previous data segment contains sufficient space for the new data block;

retrieving the previous data segment from a plurality of dispersed storage units at a corresponding plurality of different geographical locations;

retrieving original metadata corresponding to the previous data segment;

generating a revised data segment by aggregating the new data block with at least one existing data block of the previous data segment;

generating aggregated metadata for the revised data segment, the aggregated metadata including at least one portion of the original metadata and at least one new metadata portion;

generating a plurality of slices for the revised data segment; and

storing the plurality of slices in the plurality of dispersed storage units at the corresponding plurality of different geographical locations.

2. The method of claim 1 wherein retrieving the previous data segment from the plurality of dispersed storage units, includes:

retrieving a plurality of previous slices corresponding to the previous data segment; and

deslicing and decoding the plurality of previous slices to regenerate the previous data segment.

3. The method of claim 1 further comprising encoding the revised data segment.

4. The method of claim 1 wherein the at least one new metadata portion includes a new block number of the new data block.

5. The method of claim 4 wherein the at least one portion of the original metadata includes at least one existing block number of the at least one existing data block of the previous data segment.

6. The method of claim 1 wherein generating the revised data segment includes:

padding the revised data segment with padding data to fill the revised data segment when the aggregation of the new data block with the at least one existing data block of the previous data segment does not fill the revised data segment.

7. The method of claim 1 wherein generating the revised data segment includes:

removing padding data from the previous data segment.

8. The method of claim 1 wherein generating the revised data segment includes:

storing an indication of available space when the aggregation of the new data block with the at least one existing data block of the previous data segment does not fill the revised data segment.

9. The method of claim 1 wherein generating the revised data segment includes:

storing a segment filled indication when the aggregation of the new data block with the at least one existing data block of the previous data segment fills the revised data segment.

10. The method of claim 1 further comprising, when the previous data segment does not contain sufficient space for the new data block:

generating a new data segment;

storing an indication of available space associated with the new data segment;

generating a plurality of slices for the new data segment; and

storing the plurality of slices in the plurality of dispersed storage units.

11. A dispersed storage processing unit for use in a dispersed storage system, the dispersed storage processing unit comprising:

a device that includes;

an access module that receives a new data block to be stored in the dispersed storage system, that determines when a previous data segment contains sufficient space for the new data block, when the previous data segment contains sufficient space for the new data block, the access module;

retrieves the previous data segment from a plurality of dispersed storage units at a corresponding plurality of different geographical locations;

retrieves original metadata corresponding to the previous data segment;

generates a revised data segment by aggregating the new data block with at least one existing data block of the previous data segment; and

generates aggregated metadata for the revised data segment, the aggregated metadata including at least one portion of the original metadata and at least one new metadata portion;

a grid module, coupled to the access module, that generates a plurality of slices for the revised data segment; and

a storage module, coupled to the grid module, that stores the plurality of slices in the plurality of dispersed storage units at the corresponding plurality of geographical different locations.

12. The dispersed storage processing unit of claim 11 wherein the access module commands the storage module and the grid module to retrieve the previous data segment from the plurality of dispersed storage units by:

retrieving a plurality of previous slices corresponding to the previous data segment; and

deslicing and decoding the plurality of previous slices to regenerate the previous data segment.

13. The dispersed storage processing unit of claim 11 wherein the at least one new metadata portion includes a new block number of the new data block.

14. The dispersed storage processing unit of claim 13 wherein the at least one portion of the original metadata includes at least one existing block number of the at least one existing data block of the previous data segment.

15. The dispersed storage processing unit of claim 11 wherein the access module generates the revised data segment by further:

padding the revised data segment with padding data to fill the revised data segment when the aggregation of the new data block with the at least one existing data block of the previous data segment does not fill the revised data segment.

16. The dispersed storage processing unit of claim 11 wherein the access module generates the revised data segment by further:

removing padding data from the previous data segment.

17. The dispersed storage processing unit of claim 11 wherein the access module generates the revised data segment by further:

storing an indication of available space when the aggregation of the new data block with the at least one existing data block of the previous data segment does not fill the revised data segment.

18. The dispersed storage processing unit of claim 11 wherein the access module generates the revised data segment by further:

storing a segment filled indication when the aggregation of the new data block with the at least one existing data block of the previous data segment fills the revised data segment.

19. The dispersed storage processing unit of claim 11 when the previous data segment does not contain sufficient space for the new data block, the access module:

generates a new data segment; and

stores an indication of available space associated with the new data segment;

wherein the grid module generates a plurality of slices for the new data segment; and

wherein the storage module stores the plurality of slices in the plurality of dispersed storage units.

20. A method for use in a dispersed storage system, the method comprising:

receiving a plurality of data blocks to be stored in the dispersed storage system, each of the plurality of data blocks including data block metadata and data;

mapping each of the plurality of data blocks to a corresponding one of a plurality of aggregated data blocks of fixed length;

packing the data from each of the a plurality of data blocks into a data section of the corresponding one of the plurality of aggregated data blocks, wherein data from at least one of the plurality of data blocks has a length that is less than a length of the data section of the corresponding one of the plurality of aggregated data blocks;

generating aggregated metadata for each of the plurality of aggregated data blocks, based in part on the data block metadata from a corresponding one of the plurality of data blocks; and

storing the plurality of aggregated data blocks in a plurality of dispersed storage units at a corresponding plurality of different geographical locations.

21. A method for use in a dispersed storage system, the method comprising:

receiving a new data block to be stored in the dispersed storage system;

generating a revised data segment by aggregating the new data block into an existing data segment stored in a plurality of dispersed storage units at a corresponding plurality of different geographical locations;

retrieving original metadata corresponding to the previous data segment; and

generating aggregated metadata for the revised data segment, the aggregated metadata including the original metadata and at least one new metadata portion;

wherein the at least one new metadata portion includes a new block number of the new data block; and

wherein the original metadata includes at least one existing block number of the at least one existing data block of the previous data segment.