Priority based rebuilding

• US 10,061,650 B2
• Filed: 01/24/2017
• Issued: 08/28/2018
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Gladwin, S. Christopher, Resch, Jason K., Shirley, Thomas F.
• Primary Examiner(s): Thammavong, Prasith
• Application Number:
  US15/414,452
• Publication Number:
  US 20170132081A1
• Time in Patent Office: 581 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/108 Parity data distribution in semiconductor storages, e.g. in SSD
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 2211/1035 Keeping track, i.e. keeping track of data and parity changes
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/06 using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 65/00 Network arrangements, protocols or services for supporting real-time applications in data packet communication real-time or near real-time messaging, e.g. instant messaging [IM] H04L51/04; selective video distribution H04N21/00
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 69/00 Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass networks security protocols H04L9/40; wireless communication networks H04W
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18
• CPC Class Codes: H04N 21/2181 : comprising remotely distributed storage units, e.g. when movies are replicated over a plurality of video servers distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: H04N 21/23109 : by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23116 : involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/2393 : involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/8586 : by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02

Abstract

A method includes determining, by a computing device of a dispersed storage network (DSN), a data priority level for a data object. The method further includes determining, by the computing device, slice priority levels based on the data priority level. The method further includes encoding, by the computing device, the data object into sets of encoded data slices. While writing the sets of encoded data slices to storage units, the method further includes detecting, by a rebuilding module of the DSN, a slice error for an encoded data slice. When the data priority level is below a data priority threshold and the slice has the first slice priority level, the method further includes facilitating rebuilding of the slice. When the data priority level is below the data priority threshold and the slice has the second slice priority level, the method further includes foregoing rebuilding of the slice.

Claims

1. A method comprises:

determining, by a computing device of a dispersed storage network (DSN), a data priority level for a data object to be stored in the DSN in accordance with dispersed storage error encoding parameters;

determining, by the computing device, slice priority levels based on the data priority level;

encoding, by the computing device, the data object in accordance with the dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein, based on the slice priority levels, a first subset of encoded data slices of a set of encoded data slices of the plurality of sets of encoded data slices has a first slice priority level and wherein, based on the slice priority levels, a second subset of encoded data slices of the set of encoded data slices has a second slice priority level;

while writing the plurality of sets of encoded data slices to storage units of the DSN;

detecting, by a rebuilding module of the DSN, a slice error for an encoded data slice of the set of encoded data slices;

determining, by the rebuilding module, whether the encoded data slice has the first or the second slice priority level;

when the data priority level is below a data priority threshold and the encoded data slice has the first slice priority level, facilitating, by the rebuilding module, rebuilding of the encoded data slice; and

when the data priority level is below the data priority threshold and the encoded data slice has the second slice priority level, foregoing, by the rebuilding module, rebuilding of the encoded data slice.

2. The method of claim 1, wherein determining the data priority level comprises:

determining a data retention timeframe for storing the data object;

establishing the data priority level to be above the data priority threshold when the data retention timeframe is equal to or greater than an average retention timeframe; and

establishing the data priority level to be below the data priority threshold when the data retention timeframe is less than the average retention timeframe.

3. The method of claim 1, wherein the determining slice priority levels comprises:

when the data priority level is below the data priority threshold;

establishing the first subset of encoded data slices to include a read threshold number of encoded data slices; and

establishing the second subset of encoded data slices to include a number of encoded data slices equal to a total number less the read threshold number.

4. The method of claim 1, wherein the determining slice priority levels comprises:

when the data priority level is above the data priority threshold;

establishing the first subset of encoded data slices to include a write threshold number of encoded data slices; and

establishing the second subset of encoded data slices to include a number of encoded data slices equal to a total number less the write threshold number.

5. The method of claim 1 further comprises:

when the data priority level is equal to or above the data priority threshold, facilitating, by the rebuilding module, rebuilding of the encoded data slice.

6. The method of claim 1 further comprises:

determining, by the computing device, the slice priority levels such that a third subset of encoded data slices of the set of encoded data slices has a third slice priority level, wherein the first subset of encoded data slices includes a read threshold number of encoded data slices, the second subset of encoded data slices includes a number of encoded data slices equal to a write number less the read threshold number, and the third subset of encoded data slices includes a number of encoded data slices equal to a total number less the write threshold number.

7. The method of claim 6 further comprises:

when the data priority level is between the data priority level and a second data priority threshold;

facilitating, by the rebuilding module, rebuilding of the encoded data slice when the encoded data slice has the first or the second slice priority level; and

foregoing, by the rebuilding module, rebuilding of the encoded data slice when the encoded data slice has the third slice priority level.

8. A non-transitory computer readable memory comprises:

a first memory section that stores operational instructions that, when executed by a computing device of a dispersed storage network (DSN), causes the computing device to;

determine a data priority level for a data object to be stored in the DSN in accordance with dispersed storage error encoding parameters;

determine slice priority levels based on the data priority level;

encode the data object in accordance with the dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein a first subset of encoded data slices of a set of encoded data slices of the plurality of sets of encoded data slices has a first slice priority level and wherein a second subset of encoded data slices of the set of encoded data slices has a second slice priority level; and

a second memory section that stores operational instructions that, when executed by a rebuilding module of the DSN, causes the rebuilding module to;

while writing the plurality of sets of encoded data slices to storage units of the DSN;

detect a slice error for an encoded data slice of the set of encoded data slices;

determine whether the encoded data slice has the first or the second slice priority level;

when the data priority level is below a data priority threshold and the encoded data slice has the first slice priority level, facilitate rebuilding of the encoded data slice; and

when the data priority level is below the data priority threshold and the encoded data slice has the second slice priority level, forego rebuilding of the encoded data slice.

9. The non-transitory computer readable memory of claim 8, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to determine the data priority level by:

determining a data retention timeframe for storing the data object;

establishing the data priority level to be above the data priority threshold when the data retention timeframe is equal to or greater than an average retention timeframe; and

establishing the data priority level to be below the data priority threshold when the data retention timeframe is less than the average retention timeframe.

10. The non-transitory computer readable memory of claim 8, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to determine slice priority levels by:

when the data priority level is below the data priority threshold;

establishing the first subset of encoded data slices to include a read threshold number of encoded data slices; and

establishing the second subset of encoded data slices to include a number of encoded data slices equal to a total number less the read threshold number.

11. The non-transitory computer readable memory of claim 8, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to determine slice priority levels by:

when the data priority level is above the data priority threshold;

establishing the first subset of encoded data slices to include a write threshold number of encoded data slices; and

establishing the second subset of encoded data slices to include a number of encoded data slices equal to a total number less the write threshold number.

12. The non-transitory computer readable memory of claim 8, wherein the second memory section further stores operational instructions that, when executed by the rebuilding module, causes the rebuilding module to:

when the data priority level is equal to or above the data priority threshold, facilitate rebuilding of the encoded data slice.

13. The non-transitory computer readable memory of claim 8, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to:

determine the slice priority levels such that a third subset of encoded data slices of the set of encoded data slices has a third slice priority level, wherein the first subset of encoded data slices includes a read threshold number of encoded data slices, the second subset of encoded data slices includes a number of encoded data slices equal to a write number less the read threshold number, and the third subset of encoded data slices includes a number of encoded data slices equal to a total number less the write threshold number.

14. The non-transitory computer readable memory of claim 13, wherein the second memory section further stores operational instructions that, when executed by the rebuilding module, causes the rebuilding module to:

when the data priority level is between the data priority level and a second data priority threshold;

facilitate rebuilding of the encoded data slice when the encoded data slice has the first or the second slice priority level; and

forego rebuilding of the encoded data slice when the encoded data slice has the third slice priority level.