Prioritized deleting of slices stored in a dispersed storage network

• US 8,984,371 B2
• Filed: 06/16/2014
• Issued: 03/17/2015
• Est. Priority Date: 06/06/2011
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Gladwin, S. Christopher, Shirley, Thomas Franklin Jr., Resch, Jason K.
• Primary Examiner(s): Trimmings, John P
• Application Number:
  US14/305,880
• Publication Number:
  US 20140298138A1
• Time in Patent Office: 274 Days
• Field of Search: 714/763
• US Class Current: 714/763
• CPC Class Codes: G06F 11/108 Parity data distribution in semiconductor storages, e.g. in SSD
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 2211/1035 Keeping track, i.e. keeping track of data and parity changes
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 9/06 using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
• CPC Class Codes: H03M 13/1105 Decoding
• CPC Class Codes: H04L 65/00 Network arrangements, protocols or services for supporting real-time applications in data packet communication real-time or near real-time messaging, e.g. instant messaging [IM] H04L51/04; selective video distribution H04N21/00
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04L 69/00 Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass networks security protocols H04L9/40; wireless communication networks H04W
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/2393 involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02

Abstract

A method begins, as data objects are ingested, by determining, for each of some of the data objects, a priority indicator to produce a listing of priority indicators. The method continues for a data object by determining encoding parameters based on a corresponding priority indicator. The method continues by encoding the data object in accordance with the encoding parameters to produce a plurality of sets of encoded data slices and storing them. The method continues by identifying a first data object for analysis based on a corresponding priority indicator and an analysis priority. The method continues by decoding a plurality of sets of encoded data slices to recover the first data object and analyzing it in accordance with analysis criteria to determine its relevancy. The method continues by issuing a command to delete the plurality of sets of encoded data slices when the relevancy is below a threshold.

Claims

1. A method for execution by one or more computing devices of a dispersed storage network (DSN), the method comprises:

as a plurality of data objects is ingested, determining, for each data object of at least some of the plurality of data objects, a priority indicator to produce a listing of priority indicators corresponding to the at least some of the plurality of data objects;

for a data object of the at least some of the plurality of data objects;

determining dispersed storage error encoding parameters based on a corresponding priority indicator of the listing of priority indicators;

dispersed storage error encoding the data object in accordance with the dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices; and

facilitating storage of the plurality of sets of encoded data slices in memory of the DSN;

identifying a first data object of the at least some of the plurality of data objects for analysis based on a corresponding priority indicator of the first data object and an analysis priority;

dispersed storage error decoding a first plurality of sets of encoded data slices to recover the first data object;

analyzing the recovered first data object in accordance with analysis criteria to determine relevancy of the first data object; and

when the relevancy of the recovered first data object is below a threshold, issuing a command to delete the first plurality of sets of encoded data slices.

2. The method of claim 1 further comprises:

when the relevancy of the first data object is above the threshold, updating the corresponding priority indicator of the first data object.

3. The method of claim 1 further comprises:

ingesting the plurality of data objects during an ingest time period;

establishing an analysis time period during which time at least higher priority data objects of the at least some of the plurality of data objects will be analyzed; and

determining, for the data object of the at least some of the plurality of data objects, the priority indicator based on the analysis time period, the quantity of data objects of the at least some of the plurality of data objects, and previously determined priority indicators of other data objects of the at least some of the plurality of data objects.

4. The method of claim 1, wherein the determining the priority indicator for the data object of the at least some of the plurality of data objects comprises:

as the data object of the at least some of the plurality of data objects is being ingested, identifying one or more data characteristics of the data object, wherein the one or more data characteristics are from a list of data characteristics, wherein the list of data characteristics includes a data source factor, a data content factor, and a data location origination factor.

5. The method of claim 1, wherein the determining dispersed storage error encoding parameters comprises:

establishing a likelihood of analysis for the data object based on an ingest time period, an analysis time period, the priority indicator of the data object, and priority indicators of other data objects of the at least some of the plurality of data objects;

when the likelihood of analysis is high, establishing the dispersed storage error encoding parameters having a first pillar width number and a first decode threshold number; and

when the likelihood of analysis is low, establishing the dispersed storage error encoding parameters having a second pillar width number and a second decode threshold number, wherein the second pillar width number is greater than the first pillar width number and wherein the second decode threshold number is greater than the first decode threshold number.

6. The method of claim 5 further comprises:

as an analysis time period passes and when the likelihood of analysis is low, incrementally deleting an encoded data slice from one or more sets of encoded data slices of the plurality of sets of encoded data slices until the analysis time period expires, the data object is analyzed, or the second decode threshold number of encoded data slices of the one or more sets of encoded data slices of the plurality of sets of encoded data slices is reached; and

when the analysis time period expires and the data object has not been analyzed, deleting remaining encoded data slices of the plurality of sets of encoded data slices.

7. The method of claim 1, wherein the analysis priority comprises one or more factors from a list of factors, wherein the list of factors includes:

likelihood of the data object including a search term;

likelihood of the data object including a character string;

identity of a source of the data object;

identity of a destination of the data object;

likelihood of the data object including a data pattern;

identity of a location from which the data object was originated;

identity of a location to which the data object was sent; and

timing of when the data object was originated, transmitted, received, or stored.

8. A computer readable memory comprises:

a first storage memory section that stores operational instructions that, when read by a first computing device, causes the first computing device to;

as a plurality of data objects is ingested, determine, for each data object of at least some of the plurality of data objects, a priority indicator to produce a listing of priority indicators corresponding to the at least some of the plurality of data objects; and

for a data object of the at least some of the plurality of data objects;

determine dispersed storage error encoding parameters based on a corresponding priority indicator of the listing of priority indicators;

dispersed storage error encode the data object in accordance with the dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices; and

facilitate storage of the plurality of sets of encoded data slices in memory of the DSN; and

a second storage memory section that stores operational instructions that, when read by the first computing device or a second computing device, causes the first or the second computing device to;

identify a first data object of the at least some of the plurality of data objects for analysis based on a corresponding priority indicator of the first data object and an analysis priority;

dispersed storage error decode a first plurality of sets of encoded data slices to recover the first data object;

analyze the recovered first data object in accordance with analysis criteria to determine relevancy of the first data object; and

when the relevancy of the recovered first data object is below a threshold, issue a command to delete the first plurality of sets of encoded data slices.

9. The computer readable memory of claim 8 comprises:

the second storage memory section further stores operational instructions that, when read by the first computing device or the second computing device, causes the first or the second computing device to;

when the relevancy of the first data object is above the threshold, update the corresponding priority indicator of the first data object.

10. The computer readable memory of claim 8 comprises:

ingesting the plurality of data objects during an ingest time period;

the first storage memory section further stores operational instructions that, when read by the first computing device, causes the first computing device to;

establish an analysis time period during which time at least higher priority data objects of the at least some of the plurality of data objects will be analyzed; and

determine, for the data object of the at least some of the plurality of data objects, the priority indicator based on the analysis time period, the quantity of data objects of the at least some of the plurality of data objects, and previously determined priority indicators of other data objects of the at least some of the plurality of data objects.

11. The computer readable memory of claim 8, wherein the first storage memory section further stores operational instructions that, when read by the first computing device, causes the first computing device to determine the priority indicator for the data object of the at least some of the plurality of data objects by:

as the data object of the at least some of the plurality of data objects is being ingested, identifying one or more data characteristics of the data object, wherein the one or more data characteristics are from a list of data characteristics, wherein the list of data characteristics includes a data source factor, a data content factor, and a data location origination factor.

12. The computer readable memory of claim 8, wherein the first storage memory section further stores operational instructions that, when read by the first computing device, causes the first computing device to determine the dispersed storage error encoding parameters by:

establishing a likelihood of analysis for the data object based on an ingest time period, an analysis time period, the priority indicator of the data object, and priority indicators of other data objects of the at least some of the plurality of data objects;

when the likelihood of analysis is high, establishing the dispersed storage error encoding parameters having a first pillar width number and a first decode threshold number; and

when the likelihood of analysis is low, establishing the dispersed storage error encoding parameters having a second pillar width number and a second decode threshold number, wherein the second pillar width number is greater than the first pillar width number and wherein the second decode threshold number is greater than the first decode threshold number.

13. The computer readable memory of claim 12 further comprises:

the second storage memory section further stores operational instructions that, when read by the first computing device or the second computing device, causes the first or the second computing device to;

as an analysis time period passes and when the likelihood of analysis is low, incrementally facilitating deleting an encoded data slice from one or more sets of encoded data slices of the plurality of sets of encoded data slices until the analysis time period expires, the data object is analyzed, or the second decode threshold number of encoded data slices of the one or more sets of encoded data slices of the plurality of sets of encoded data slices is reached; and

when the analysis time period expires and the data object has not been analyzed, facilitating deleting remaining encoded data slices of the plurality of sets of encoded data slices.

14. The computer readable memory of claim 8, wherein the analysis priority comprises one or more factors from a list of factors, wherein the list of factors includes:

likelihood of the data object including a search term;

likelihood of the data object including a character string;

identity of a source of the data object;

identity of a destination of the data object;

likelihood of the data object including a data pattern;

identity of a location from which the data object was originated;

identity of a location to which the data object was sent; and

timing of when the data object was originated, transmitted, received, or stored.