Dispersed storage unit and methods with metadata separation for use in a dispersed storage system

• US 9,235,350 B2
• Filed: 03/30/2010
• Issued: 01/12/2016
• Est. Priority Date: 08/27/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Mark, Zachary J.
• Primary Examiner(s): Ehichioya, Fred I
• Assistant Examiner(s): MOBIN, HASANUL
• Application Number:
  US12/749,583
• Publication Number:
  US 20110055178A1
• Time in Patent Office: 2,114 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0622 in relation to access
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A data slice to be stored in a dispersed storage unit of the dispersed storage system is received. A plurality of data blocks are generated from the data slice and metadata are generated associated with each of the plurality of data blocks. A data file is generated based on the plurality of data blocks. A metadata file is generated based on the metadata associated with the plurality of data blocks. The metadata file and the data file are stored in at least one memory of the dispersed storage unit.

Claims

1. A method for use in a dispersed storage system, the method comprising:

receiving a data slice of a data object to be stored in a dispersed storage unit of the dispersed storage system, the data slice having an associated revision number;

generating a plurality of data blocks from the data slice and metadata associated with each of the plurality of data blocks, wherein the metadata includes revision data indicating the revision number associated with the data slice;

generating a data file based on the plurality of data blocks;

generating a metadata file based on the metadata associated with the plurality of data blocks; and

storing the metadata file and the data file in at least one memory of the dispersed storage unit.

2. The method of claim 1 wherein the metadata file is stored separately from the data file.

3. The method of claim 1 wherein the metadata file indicates the plurality of data blocks.

4. The method of claim 1 wherein the metadata associated with the plurality of data blocks includes a block number for each data block;

wherein the metadata file includes a range of block numbers corresponding to the block numbers for the plurality of data blocks.

5. The method of claim 1 wherein the metadata associated with the plurality of data blocks includes a checksum for each data block.

6. The method of claim 1 wherein the metadata associated with the plurality of data blocks includes data length for each data block.

7. The method of claim 1 wherein the plurality of data blocks are of equal length and wherein generating the plurality of data blocks includes:

padding one of the plurality of data blocks with padding data to fill the one of the plurality of data blocks.

8. The method of claim 1 wherein at least two of the plurality of data blocks are of unequal length.

9. The method of claim 1 wherein the metadata file includes a compilation of the metadata associated with the plurality of data blocks.

10. The method of claim 1 wherein the metadata file includes at least one data element associated with the data object.

11. A dispersed storage unit for use in a dispersed storage system, the dispersed storage unit comprising:

a storage unit control module that includes;

a data block generator that receives a data slice of a data object, the data slice having an associated revision number, and that generates a plurality of data blocks from the data slice and metadata associated with each of the plurality of data blocks, wherein the metadata includes revision data indicating the revision number associated with the data slice;

a data file generator that generates a data file based on the plurality of data blocks; and

a meta data file generator that generates a metadata file based on the metadata associated with the plurality of data blocks; and

at least one memory that stores the metadata file and the data file.

12. The dispersed storage unit of claim 11 wherein the at least one memory stores the metadata file separately from the data file.

13. The dispersed storage unit of claim 11 wherein the metadata file generator that generates the metadata file to indicate each of the plurality of data blocks.

14. The dispersed storage unit of claim 11 wherein the metadata file generator generates the metadata associated with the plurality of data blocks to include a block number for each data block.

15. The dispersed storage unit of claim 11 wherein the metadata file generator generates the metadata associated with the plurality of data blocks to include a checksum for each data block.

16. The dispersed storage unit of claim 11 wherein the metadata file generator generates the metadata associated with the plurality of data blocks to include a data length for each data block.

17. The dispersed storage unit of claim 11 wherein the plurality of data blocks are of equal length and wherein the data block generator pads one of the plurality of data blocks with padding data to fill the one of the plurality of data blocks.

18. The dispersed storage unit of claim 11 wherein at least two of the plurality of data blocks are of unequal length.

19. The dispersed storage unit of claim 11 wherein the metadata file includes a compilation of the metadata associated with the plurality of data blocks.

20. The dispersed storage unit of claim 11 wherein the metadata file includes at least one data element associated with the data object.