Metadata access in a dispersed storage network

• US 10,289,688 B2
• Filed: 06/07/2011
• Issued: 05/14/2019
• Est. Priority Date: 06/22/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K., Dhuse, Greg, Baptist, Andrew, Leggette, Wesley
• Primary Examiner(s): Harper, Eliyah S.
• Application Number:
  US13/154,725
• Publication Number:
  US 20110314072A1
• Time in Patent Office: 2,898 Days
• Field of Search: 707812, 707807, 707999102, 707999101
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/141 for bus or memory accesses
• CPC Class Codes: G06F 11/167 Error detection by comparing the memory output
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 2209/043 of tables, e.g. lookup, substitution or mapping
• CPC Class Codes: H04L 2209/30 Compression, e.g. Merkle-Damgard construction
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 2209/56 Financial cryptography, e.g. electronic payment or e-cash
• CPC Class Codes: H04L 2209/80 Wireless network architectures or network communication protocols for wireless network security H04W12/00
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/06 for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/3242 involving keyed hash functions, e.g. message authentication codes [MACs], CBC-MAC or HMAC
• CPC Class Codes: H04L 9/3247 : involving digital signatures
• CPC Class Codes: H04L 9/3263 : involving certificates, e.g. public key certificate [PKC] or attribute certificate [AC]; Public key infrastructure [PKI] arrangements network architectures or network communication protocols for supporting authentication of entities using certificates in a packet data network H04L63/0823
• CPC Class Codes: H04L 9/3271 : using challenge-response
• CPC Class Codes: H04W 12/041 : Key generation or derivation
• CPC Class Codes: H04W 12/0431 : Key distribution or pre-distribution; Key agreement
• CPC Class Codes: H04W 12/10 : Integrity
• CPC Class Codes: H04W 12/35 : Protecting application or service provisioning, e.g. securing SIM application provisioning

Abstract

A method begins by a processing module dispersed storage error encoding a data segment to produce a set of encoded data slices and dispersed storage error encoding metadata associated with the data segment to produce a set of encoded metadata slices. The method continues with the processing module creating a set of data slice names for the set of encoded data slices and creating a set of metadata slice names based on the set of data slice names. The method continues with the processing module sending the set of encoded data slices and the set of data slice names to a dispersed storage network (DSN) memory for storage therein. The method continues with the processing module sending the set of encoded metadata slices and the set of metadata slice names to the DSN memory for storage therein.

Claims

1. A method comprises:

dispersed storage error encoding a data segment to produce a set of encoded data slices;

dispersed storage error encoding metadata associated with the data segment to produce a set of encoded metadata slices, wherein the metadata includes at least one of an object identifier (ID), object size, object type, object format, directory information, a file name, a file path, a source name, a dispersed storage network (DSN) address, a snapshot identifier, a segmentation allocation table (SAT) source name, object hash, access permissions, or a timestamp;

creating a set of data slice names for the set of encoded data slices;

creating a set of metadata slice names based on the set of data slice names;

sending the set of encoded data slices and the set of data slice names to a dispersed storage network (DSN) memory, wherein the DSN memory stores an encoded data slice of the set of encoded data slices based on a corresponding one of the set of data slice names using a first level of memory allocation; and

sending the set of encoded metadata slices and the set of metadata slice names to the DSN memory, wherein the DSN memory stores an encoded metadata slice of the set of encoded metadata slices based on a corresponding one of the set of metadata slice names using a second level of memory allocation, and wherein the second level of memory allocation is smaller than the first level of memory allocation.

2. The method of claim 1 further comprises:

sending the encoded data slice and the corresponding one of the set of data slice names to a first dispersed storage (DS) unit of the DSN memory, wherein memory space of the first DS unit is partitioned in accordance with the first level of memory allocation; and

sending the encoded metadata slice and the corresponding one of the set of metadata slice names to a second DS unit of the DSN memory, wherein memory space of the second DS unit is partitioned in accordance with the second level of memory allocation.

3. The method of claim 1 further comprises:

sending the encoded data slice and the corresponding one of the set of data slice names to a dispersed storage (DS) unit of the DSN memory, wherein a first portion of memory space of the DS unit is partitioned in accordance with the first level of memory allocation; and

sending the encoded metadata slice and the corresponding one of the set of metadata slice names to the DS unit, wherein a second portion of the memory space of the DS unit is partitioned in accordance with the second level of memory allocation.

4. The method of claim 1, wherein the creating the set of data slice names is based on at least one of:

a data identifier (ID) associated with the data segment;

a vault ID lookup;

a directory lookup;

a source name;

a vault source name;

the set of encoded data slices;

a hash of the data segment;

or

an object number associated with the data ID.

5. The method of claim 1, wherein the creating the set of metadata slice names comprises at least one of:

toggling a data/metadata flag of a data slice name of the set of data slice names to produce a corresponding metadata slice name of the set of metadata slice names;

performing an exclusive OR (XOR) logical function on the data slice name with a naming mask to produce the corresponding metadata slice name;

adding a constant value to the data slice name to produce the corresponding metadata slice name;

or

subtracting the constant value from the data slice name to produce the corresponding metadata slice name.

6. A method comprises:

determining a set of data slice names corresponding to a data segment previously stored in a dispersed storage network (DSN) memory as a set of encoded data slices;

determining a set of metadata slice names based on the set of data slice names, wherein the metadata slice names correspond to metadata previously stored in the DSN memory as a set of encoded metadata slices, wherein the metadata includes at least one of an object identifier (ID), object size, object type, object format, directory information, a file name, a file path, a source name, a dispersed storage network (DSN) address, a snapshot identifier, a segmentation allocation table (SAT) source name, object hash, access permissions, or a timestamp;

retrieving at least a decode threshold number of encoded data slices of the set of encoded data slices from the DSN memory to produce received encoded data slices utilizing the set of data slice names, wherein the DSN memory retrieves an encoded data slice of the set of encoded data slices based on a corresponding one of the set of data slice names using a first level of memory allocation;

retrieving at least a decode threshold number of encoded metadata slices of the set of encoded metadata slices from the DSN memory to produce received encoded metadata slices utilizing the set of metadata slice names, wherein the DSN memory retrieves an encoded metadata slice of the set of encoded metadata slices based on a corresponding one of the set of metadata slice names using a second level of memory allocation, and wherein the second level of memory allocation is smaller than the first level of memory allocation;

dispersed storage error decoding the received encoded data slices to reproduce the data segment; and

dispersed storage error decoding the received encoded metadata slices to reproduce the metadata.

7. The method of claim 6 further comprises:

transforming the data segment utilizing a metadata function to produce transformed data in accordance with the metadata; and

sending the transformed data to a requesting entity.

8. The method of claim 6, wherein the dispersed storage error decoding the received encoded data slices comprises:

dispersed storage error decoding the received encoded data slices based on the metadata to reproduce the data segment.

9. A computer comprises:

an interface;

a non-transitory memory that stores operational instructions; and

at least one processing device that executes the operational instructions, which cause the at least one processing device to;

dispersed storage error encode a data segment to produce a set of encoded data slices;

dispersed storage error encode metadata associated with the data segment to produce a set of encoded metadata slices, wherein the metadata includes at least one of an object identifier (ID), object size, object type, object format, directory information, a file name, a file path, a source name, a dispersed storage network (DSN) address, a snapshot identifier, a segmentation allocation table (SAT) source name, object hash, access permissions, or a timestamp;

create a set of data slice names for the set of encoded data slices;

create a set of metadata slice names based on the set of data slice names;

send, via the interface, the set of encoded data slices and the set of data slice names to a dispersed storage network (DSN) memory, wherein the DSN memory stores an encoded data slice of the set of encoded data slices based on a corresponding one of the set of data slice names using a first level of memory allocation; and

send, via the interface, the set of encoded metadata slices and the set of metadata slice names to the DSN memory, wherein the DSN memory stores an encoded metadata slice of the set of encoded metadata slices based on a corresponding one of the set of metadata slice names using a second level of memory allocation, and wherein the second level of memory allocation is smaller than the first level of memory allocation.

10. The computer of claim 9, wherein executing the operational instructions further cause the at least one processing device to:

send, via the interface, the encoded data slice and the corresponding one of the set of data slice names to a first dispersed storage (DS) unit of the DSN memory, wherein memory space of the first DS unit is partitioned in accordance with the first level of memory allocation; and

send, via the interface, the encoded metadata slice and the corresponding one of the set of metadata slice names to a second DS unit of the DSN memory, wherein memory space of the second DS unit is partitioned in accordance with the second level of memory allocation.

11. The computer of claim 9, wherein executing the operational instructions further cause the at least one processing device to:

send, via the interface, the encoded data slice and the corresponding one of the set of data slice names to a dispersed storage (DS) unit of the DSN memory, wherein a first portion of memory space of the DS unit is partitioned in accordance with the first level of memory allocation; and

send, via the interface, the encoded metadata slice and the corresponding one of the set of metadata slice names to the DS unit, wherein a second portion of the memory space of the DS unit is partitioned in accordance with the second level of memory allocation.

12. The computer of claim 9, wherein the at least one processing device functions to create the set of data slice names based on at least one of:

a data identifier (ID) associated with the data segment;

a vault ID lookup;

a directory lookup;

a source name;

a vault source name;

the set of encoded data slices;

a hash of the data segment;

or

an object number associated with the data ID.

13. The computer of claim 9, wherein the at least one processing device functions to create the set of metadata slice names by at least one of:

toggling a data/metadata flag of a data slice name of the set of data slice names to produce a corresponding metadata slice name of the set of metadata slice names;

performing an exclusive OR (XOR) logical function on the data slice name with a naming mask to produce the corresponding metadata slice name;

adding a constant value to the data slice name to produce the corresponding metadata slice name;

or

subtracting the constant value from the data slice name to produce the corresponding metadata slice name.

14. A computer comprises:

an interface;

a non-transitory memory that stores operational instructions; and

a at least one processing device that executes the operational instructions, which cause the at least one processing device to;

determine a set of data slice names corresponding to a data segment previously stored in a dispersed storage network (DSN) memory as a set of encoded data slices;

determine a set of metadata slice names based on the set of data slice names, wherein the metadata slice names correspond to metadata previously stored in the DSN memory as a set of encoded metadata slices, wherein the metadata includes at least one of an object identifier (ID), object size, object type, object format, directory information, a file name, a file path, a source name, a dispersed storage network (DSN) address, a snapshot identifier, a segmentation allocation table (SAT) source name, object hash, access permissions, or a timestamp;

retrieve, via the interface, at least a decode threshold number of encoded data slices of the set of encoded data slices from the DSN memory to produce received encoded data slices utilizing the set of data slice names, wherein the DSN memory retrieves an encoded data slice of the set of encoded data slices based on a corresponding one of the set of data slice names using a first level of memory allocation;

retrieve, via the interface, at least a decode threshold number of encoded metadata slices of the set of encoded metadata slices from the DSN memory to produce received encoded metadata slices utilizing the set of metadata slice names, wherein the DSN memory retrieves an encoded metadata slice of the set of encoded metadata slices based on a corresponding one of the set of metadata slice names using a second level of memory allocation, and wherein the second level of memory allocation is smaller than the first level of memory allocation;

dispersed storage error decode the received encoded data slices to reproduce the data segment; and

dispersed storage error decode the received encoded metadata slices to reproduce the metadata.

15. The computer of claim 14, wherein executing the operational instructions further cause the at least one processing device to:

transform the data segment utilizing a metadata function to produce transformed data in accordance with the metadata; and

send, via the interface, the transformed data to a requesting entity.

16. The computer of claim 14, wherein executing the operational instructions further cause the at least one processing device to dispersed storage error decode the received encoded data slices by:

dispersed storage error decoding the received encoded data slices based on the metadata to reproduce the data segment.