Storing data and associated metadata in a dispersed storage network

• US 10,248,506 B2
• Filed: 07/26/2016
• Issued: 04/02/2019
• Est. Priority Date: 07/31/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Horan, Scott M., Leggette, Wesley B.
• Primary Examiner(s): Nguyen, Thien
• Application Number:
  US15/219,741
• Publication Number:
  US 20170031761A1
• Time in Patent Office: 980 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/108 Parity data distribution in semiconductor storages, e.g. in SSD
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/645 using a third party
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0622 in relation to access
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/065 : Replication mechanisms
• CPC Class Codes: G06F 3/0653 : Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0668 : adopting a particular infrastructure
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 : Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H03M 13/2906 : using block codes H03M13/2957 takes precedence
• CPC Class Codes: H03M 13/3761 : using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 63/0428 : wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/061 : for key exchange, e.g. in peer-to-peer networks cryptographic mechanisms or cryptographic arrangements for key agreement H04L9/0838
• CPC Class Codes: H04L 63/0853 : using an additional device, e.g. smartcard, SIM or a different communication terminal cryptographic mechanisms or cryptographic arrangements for entity authentication involving additional secure or trusted devices H04L9/3234
• CPC Class Codes: H04L 63/108 : when the policy decisions are valid for a limited amount of time
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/63 : Routing a service request depending on the request content or context
• CPC Class Codes: H04L 9/0861 : Generation of secret information including derivation or calculation of cryptographic keys or passwords

Abstract

A method begins by a processing module generating metadata for a data object. The method continues by a first disperse storage error encoding the metadata to produce a set of metadata slices. The method continues by partitioning the data to produce a plurality of data segments. The method continues by a second disperse storage error encoding the plurality of data segments to produce a plurality of sets of encoded data slices. The method continues by applying a distributed agreement protocol function to a data object identifier for the data object to produce ranked scoring information with regards to a plurality of storage sets. The method continues by selecting a storage set of the plurality of storage sets based on the ranked scoring information. The method continues by facilitating storage of the set of metadata slices and the plurality of sets of encoded data slices in the selected storage set.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

generating metadata for a data object;

first disperse storage error encoding the metadata to produce a set of metadata slices, wherein the first disperse storage error encoding includes utilizing first dispersal parameters to store metadata, the first dispersal parameters including a decode threshold of 1;

partitioning the data object to produce a plurality of data segments;

first disperse storage error encoding at least a first data segment of the plurality of data segments to produce a set of data slices, wherein the first disperse storage error encoding includes utilizing the first dispersal parameters of the metadata, the first dispersal parameters including a decode threshold of 1;

second disperse storage error encoding additional data segments of the plurality of data segments to produce a plurality of sets of encoded data slices, wherein the additional data segments do not include the at least a first data segment and wherein the second disperse storage error encoding includes utilizing second dispersal parameters, the second dispersal parameters different from the first dispersal parameters and including a decode threshold greater than 1; and

facilitating storage of the set of metadata slices and the plurality of sets of encoded data slices in one or more storage units of the DSN; and

wherein subsequent accessing of the metadata slices reproduces the metadata and the first data segment of the data object, making the first data segment of the data object available for use by the one or more computing devices while continuing to retrieve the additional data segments from the DSN.

2. The method of claim 1, wherein the first disperse storage error encoding includes utilizing the first dispersal parameters to store a copy of the metadata in each storage unit.

3. The method of claim 2, wherein the first dispersal parameters to store metadata in each storage unit further include an information dispersal algorithm threshold of width 1.

4. The method of claim 1, wherein the partitioning is in accordance with a data segmentation approach.

5. The method of claim 4, wherein the data segmentation approach includes any of:

fixed size, ramping up size, or ramping down size.

6. The method of claim 1 further comprising, for each storage unit of the selected storage set, issuing a write slice request to the storage unit, where the write slice request includes an associated metadata slice of the set of metadata slices and a plurality of associated encoded data slices of each set of encoded data slices.

7. The method of claim 6, wherein the plurality of associated encoded data slices of each set of encoded data slices includes a common pillar.

8. A computing device of a group of computing devices of a dispersed storage network (DSN), the computing device comprises:

an interface;

a local memory; and

a processing module operably coupled to the interface and the local memory, wherein the processing module functions to;

generate metadata for a data object;

first disperse storage error encode the metadata to produce a set of metadata slices, wherein the first disperse storage error encode includes utilizing first dispersal parameters to store metadata, the first dispersal parameters including a decode threshold of 1;

partition the data object to produce a plurality of data segments;

first disperse storage error encode at least a first data segment of the plurality of data segments to produce a set of data slices, wherein the first disperse storage error encoding includes utilizing the first dispersal parameters of the metadata, the first dispersal parameters including a decode threshold of 1;

second disperse storage error encode additional data segments of the plurality of data segments to produce a plurality of sets of encoded data slices, wherein the additional data segments do not include the at least a first data segment and wherein the second disperse storage error encoding includes utilizing second dispersal parameters, the second dispersal parameters different from the first dispersal parameters and including a decode threshold greater than 1; and

facilitate storage of the set of metadata slices and the plurality of sets of encoded data slices in one or more storage units of the DSN; and

wherein subsequent accessing of the metadata slices reproduces the metadata and the first data segment of the data object, making the first data segment of the data object available for use by the computing device while continuing to retrieve the additional data segments from the DSN.

9. The computing device of claim 8, wherein the first dispersal parameters to store metadata in each storage unit further include an information dispersal algorithm threshold of width 1.

10. The computing device of claim 8, wherein the partitioning is in accordance with a data segmentation approach.

11. The computing device of claim 10, wherein the data segmentation approach includes any of:

fixed size, ramping up size, or ramping down size.

12. The computing device of claim 8 further comprising, for each storage unit of the selected storage set, issuing a write slice request to the storage unit, where the write slice request includes an associated metadata slice of the set of metadata slices and a plurality of associated encoded data slices of each set of encoded data slices.