Combining deduplication with locality for efficient and fast storage

• US 10,146,622 B2
• Filed: 12/08/2017
• Issued: 12/04/2018
• Est. Priority Date: 09/08/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Algie, Teague S., Viraraghavan, Praveen
• Primary Examiner(s): Abraham, Esaw T
• Application Number:
  US15/836,805
• Publication Number:
  US 20180101432A1
• Time in Patent Office: 361 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by receiving a storage request to store one or more portions of a data object. The method continues by determining whether the one or more portions are stored within a dispersed storage network memory. When stored, the method continues by determining a retrieval demand for each portion. When the retrieval demand for a first portion is below a first threshold, the method continues by indicating that an existing copy of the first portion will be used for retrieval requests of the first portion. When the retrieval demand is above the first threshold, the method continues by copying the existing copy to create a first portion copy and indicating that the existing copy will be used for retrieval requests from a first group of user computing devices and indicating that the first portion copy will be used for retrieval requests from a second group of user computing devices.

Claims

1. A method comprises:

receiving, by a computing device of a dispersed storage network (DSN), a storage request to store one or more portions of a data object, wherein the data object includes a plurality of portions;

determining, by the computing device, whether the one or more portions are already stored within a DSN memory;

when the one or more portions are stored;

determining, by the computing device, a retrieval demand for each of the one or more portions;

when the retrieval demand for a first portion of the one or more portions is below a first threshold, indicating, by the computing device, that an existing copy of the first portion will be used for retrieval requests of the first portion within the DSN;

when the retrieval demand for the first portion of the one or more portions is above the first threshold;

copying, by the computing device, the existing copy of the first portion to create a first portion copy;

indicating, by the computing device, that the existing copy of the first portion will be used for retrieval requests from a first group of user computing devices of the DSN; and

indicating, by the computing device, that the first portion copy will be used for retrieval requests from a second group of user computing devices of the DSN.

2. The method of claim 1 further comprises:

dividing the data object into a plurality of data segments, wherein a portion of the plurality of portions includes one or more data segments, and wherein the plurality of data segments is dispersed storage error encoded into pluralities of sets of encoded data slices.

3. The method of claim 1, wherein the determining the retrieval demand comprises one or more of:

an anticipated retrieval demand level;

a historical retrieval demand level based on related content;

indicator from a content provider; and

indicator from a system operator.

4. The method of claim 1 further comprises:

when the retrieval demand for the first portion of the one or more portions is above a second threshold, wherein the second threshold is greater than the first threshold;

copying, by the computing device, the existing copy or the first portion copy to create a second first portion copy; and

indicating, by the computing device, that the second first portion copy will be used for retrieval requests from a third group of user computing devices of the DSN.

5. The method of claim 1 further comprises:

receiving from a first user computing device, a retrieval request for a particular portion of the data object;

determining whether the particular portion has been replicated; and

when the particular portion has been replicated;

determining a set of storage units to retrieve the particular portion from based on the first user computing device affiliation with a group of user computing devices.

6. The method of claim 5 further comprises:

when the particular portion has not been replicated;

identifying a set of storage units within the DSN that is storing the particular portion.

7. The method of claim 5 further comprises:

retrieving one or more sets of encoded data slices from the set of storage units;

decoding the one or more sets of encoded data slices to produce the particular portion; and

sending the particular portion to the first user computing device.

8. A computing device of a dispersed storage network (DSN) comprises:

memory;

an interface; and

a processing module operably coupled to the memory and the interface, wherein the processing module is operable to;

receive, via the interface, a storage request to store one or more portions of a data object, wherein the data object includes a plurality of portions;

determine whether the one or more portions are already stored within a DSN memory;

when the one or more portions are stored;

determine a retrieval demand for each of the one or more portions;

when the retrieval demand for a first portion of the one or more portions is below a first threshold;

indicate that an existing copy of the first portion will be used for retrieval requests of the first portion within the DSN;

when the retrieval demand for the first portion of the one or more portions is above the first threshold;

copy the existing copy of the first portion to create a first portion copy;

indicate that the existing copy of the first portion will be used for retrieval requests from a first group of user computing devices of the DSN; and

indicate that the first portion copy will be used for retrieval requests from a second group of user computing devices of the DSN.

9. The computing device of claim 8, wherein the processing module is further operable to:

divide the data object into a plurality of data segments, wherein a portion of the plurality of portions includes one or more data segments, and wherein the plurality of data segments is dispersed storage error encoded into pluralities of sets of encoded data slices.

10. The computing device of claim 8, wherein the processing module determines the retrieval demand by one or more of:

determining an anticipated retrieval demand level;

determining a historical retrieval demand level based on related content;

receiving, via the interface, an indicator from a content provider; and

receiving, via the interface, an indicator from a system operator.

11. The computing device of claim 8, wherein the processing module is further operable to:

when the retrieval demand for the first portion of the one or more portions is above a second threshold, wherein the second threshold is greater than the first threshold;

copy the existing copy or the first portion copy to create a second first portion copy; and

indicate that the second first portion copy will be used for retrieval requests from a third group of user computing devices of the DSN.

12. The computing device of claim 8, wherein the processing module is further operable to:

receive, via the interface, from a first user computing device, a retrieval request for a particular portion of the data object;

determine whether the particular portion has been replicated; and

when the particular portion has been replicated;

determine a set of storage units to retrieve the particular portion from based on the first user computing device affiliation with a group of user computing devices.

13. The computing device of claim 12, wherein the processing module is further operable to:

when the particular portion has not been replicated;

identify a set of storage units within the DSN that is storing the particular portion.

14. The computing device of claim 12, wherein the processing module is further operable to:

retrieve one or more sets of encoded data slices from the set of storage units;

decode the one or more sets of encoded data slices to produce the particular portion; and

send, via the interface, the particular portion to the first user computing device.