Slice metadata for optimized dispersed storage network memory storage strategies

• US 10,409,661 B2
• Filed: 09/29/2017
• Issued: 09/10/2019
• Est. Priority Date: 09/29/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Leggette, Wesley B.
• Primary Examiner(s): McMahon, Daniel F.
• Application Number:
  US15/720,140
• Publication Number:
  US 20190102241A1
• Time in Patent Office: 711 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing unit of a dispersed storage network (DSN) generating a hint regarding data stored or to be stored. When the data is to be stored, the DS processing module divides the data into data segments and dispersed storage error encodes a data segment of the data segments to produce a set of encoded data slices. The method continues by the DS processing unit generating a set of hints based on the hint and affiliating the set of hints with the set of encoded data slices to produce a set of affiliated encoded data slices. The method continues by the DS processing unit sending the set of affiliated encoded data slices to a set of storage units of the DSN such that a storage unit of the set of storage units stores an encoded data slice in accordance with a corresponding hint.

Claims

1. A method comprises:

generating, by a dispersed storage (DS) processing unit of a dispersed storage network (DSN), a hint regarding data stored or to be stored, wherein the hint includes one or more storage directives;

when the data is to be stored;

dividing, by the DS processing unit, the data into data segments;

dispersed storage error encoding, by the DS processing unit, a data segment of the data segments to produce a set of encoded data slices;

generating, by the DS processing unit, a set of hints based on the hint;

affiliating, by the DS processing unit, the set of hints with the set of encoded data slices to produce a set of affiliated encoded data slices; and

sending, by the DS processing unit, the set of affiliated encoded data slices to a set of storage units of the DSN, wherein a storage unit of the set of storage units stores an encoded data slice of the set of encoded data slices in accordance with a corresponding hint of the set of hints; and

when the data is stored;

determining whether to send the hint as a unique message or embedded in a DSN message of a plurality of DSN messages;

when sending the hint as the unique message;

creating a set of unique messages that includes slice names and the hint appended to the slice names; and

when sending the hint as the embedded message;

appending the hint to a set of messages regarding the DSN message.

2. The method of claim 1, wherein a storage directive of the one or more storage directives comprises one of:

a type of data, a location constraint, a modification constraint, an access frequency, an expected access latency, a desired access latency, a security requirement, and a compression requirement.

3. The method of claim 1, wherein the generating the hint comprises one or more of:

interpreting the data to determine one or more data characteristics of the data;

interpreting ancillary data of the data to determine one or more storage requests; and

interpreting an update to the ancillary data to determine updated storage requests.

4. The method of claim 1, wherein the affiliating the set of hints comprises:

appending the corresponding hint to the encoded data slice to produce an affiliated encoded data slice of the set of affiliated encoded data slices.

5. The method of claim 1, wherein the affiliating the set of hints comprises:

appending the corresponding hint to a slice name of the encoded data slice to produce an affiliated encoded data slice of the set of affiliated encoded data slices.

6. The method of claim 1, wherein the generating the hint further comprises:

generating a first hint that includes a first one or more desired storage directives; and

generating a second hint that includes a second one or more desired storage directives, wherein one or more of the first one or more desired storage directives is a different desired storage directive than one or more of the second one or more desired storage directives.

7. A dispersed storage (DS) processing unit of a dispersed storage network (DSN) comprises:

memory;

an interface; and

a processing module, wherein the memory and the interface are operably coupled to the processing module, and wherein the processing module is operable to;

generate a hint regarding data stored or to be stored, wherein the hint includes one or more storage directives;

when the data is to be stored;

divide the data into data segments;

dispersed storage error encode a data segment of the data segments to produce a set of encoded data slices;

generate a set of hints based on the hint;

affiliate the set of hints with the set of encoded data slices to produce a set of affiliated encoded data slices; and

send, via the interface, the set of affiliated encoded data slices to a set of storage units of the DSN, wherein a storage unit of the set of storage units stores an encoded data slice of the set of encoded data slices in accordance with a corresponding hint of the set of hints and

when the data is stored;

determine whether to send the hint as a unique message or embedded in a DSN message of a plurality of DSN messages;

when sending the hint as the unique message;

create a set of unique messages that includes slice names and the hint appended to the slice names; and

when sending the hint as the embedded message;

append the hint to a set of messages regarding the DSN message.

8. The DS processing unit of claim 7, wherein the processing module is operable to generate a storage directive of the one or more storage directives to include one of:

a type of data, a location constraint, a modification constraint, an access frequency, an expected access latency, a desired access latency, a security requirement, and a compression requirement.

9. The DS processing unit of claim 7, wherein the processing module is operable to generate the hint by one or more of:

interpreting the data to determine one or more data characteristics of the data;

interpreting ancillary data of the data to determine one or more storage requests; and

interpreting an update to the ancillary data to determine updated storage requests.

10. The DS processing unit of claim 7, wherein the processing module is operable to affiliate the set of hints by:

appending the corresponding hint to the encoded data slice to produce an affiliated encoded data slice of the set of affiliated encoded data slices.

11. The DS processing unit of claim 7, wherein the processing module is operable to affiliate the set of hints by:

appending the corresponding hint to a slice name of the encoded data slice to produce an affiliated encoded data slice of the set of affiliated encoded data slices.

12. The DS processing unit of claim 7, wherein the processing module is operable to generate the hint by:

generating a first hint that includes a first one or more desired storage directives; and

generating a second hint that includes a second one or more desired storage directives, wherein one or more of the first one or more desired storage directives is a different desired storage directive than one or more of the second one or more desired storage directives.

13. A computer readable memory device of a dispersed storage network (DSN) comprises:

a first memory element for storing operational instructions that, when executed by a dispersed storage (DS) processing unit of the DSN, causes the DS processing unit to;

generate a hint regarding data stored or to be stored, wherein the hint includes one or more storage directives;

when the data is to be stored;

divide the data into data segments;

dispersed storage error encode a data segment of the data segments to produce a set of encoded data slices;

generate a set of hints based on the hint; and

affiliate the set of hints with the set of encoded data slices to produce a set of affiliated encoded data slices; and

send the set of affiliated encoded data slices to a set of storage units of the DSN, wherein a storage unit of the set of storage units stores an encoded data slice of the set of encoded data slices in accordance with a corresponding hint of the set of hints; and

when the data is stored;

determine whether to send the hint as a unique message or embedded in a DSN message of a plurality of DSN messages; and

when sending the hint as the unique message;

create a set of unique messages that includes slice names and hints appended to the slice names; and

when sending the hint as the embedded message;

append the hint to a set of messages regarding the DSN message.

14. The computer readable memory device of claim 13, wherein the first memory element stores further operational instructions that, when executed by the DS processing unit, causes the DS processing unit to:

interpret the data to determine one or more data characteristics of the data;

interpret ancillary data of the data to determine one or more storage requests; and

interpret an update to the ancillary data to determine updated storage requests.

15. The computer readable memory device of claim 13, wherein the first memory element stores further operational instructions, that when executed by the DS processing unit, causes the DS processing unit to:

append the corresponding hint to the encoded data slice to produce an affiliated encoded data slice of the set of affiliated encoded data slices.

16. The computer readable memory device of claim 13, wherein the first memory element stores further operational instructions, that when executed by the DS processing unit, causes the DS processing unit to:

append the corresponding hint to a slice name of the encoded data slice to produce an affiliated encoded data slice of the set of affiliated encoded data slices.

17. The computer readable memory device of claim 13, wherein the first memory element stores further operational instructions that, when executed by the DS processing unit, causes the DS processing unit to:

generate a first hint that includes a first one or more desired storage directives; and

generate a second hint that includes a second one or more desired storage directives, wherein one or more of the first one or more desired storage directives is a different desired storage directive than one or more of the second one or more desired storage directives.