Storing low retention priority data in a dispersed storage network

• US 9,749,414 B2
• Filed: 06/30/2014
• Issued: 08/29/2017
• Est. Priority Date: 08/29/2013
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K.
• Primary Examiner(s): Sison, June
• Assistant Examiner(s): Nguyen, Steven
• Application Number:
  US14/320,518
• Publication Number:
  US 20150067090A1
• Time in Patent Office: 1,156 Days
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a processing module of a dispersed storage network (DSN) sending a set of low retention priority write requests to storage units of the DSN, where each low retention priority write request includes a low retention priority query. For each storage unit of the storage units that receives a low retention priority write request of the set of low retention priority write requests, the method continues with the processing module determining a low retention priority response regarding availability for storing low retention priority data based on current storage of low priority data objects and available memory for storing the low retention priority data. The method continues with the processing module sending the low retention priority response. When a threshold number of favorable low retention priority responses have been received, the method continues at the processing module facilitating storage of a low retention priority data object.

Claims

1. A method for execution by one or more processors of one or more computing devices of a dispersed storage network (DSN), the method comprises:

sending a set of low retention priority write requests to storage units of the DSN based on a write facilitating request to store a low retention priority data object in the DSN, wherein each low retention priority write request of the set of low retention priority write requests includes a low retention priority query, wherein the low retention priority data object corresponds to scratch pad data associated with intermediate results generated when a task is performed on other data, and wherein the low retention priority data object requires only temporary storage in the DSN, wherein the low retention priority data object includes a given low retention priority level of a plurality of low retention priority level of low retention priority data that specify that first information within the DSN associated with a first retention priority level shall not be deleted from storage within the DSN before second information within the DSN associated with a second retention priority level when the first retention priority level indicates a greater requirement for retention than the second retention priority level;

for each storage unit of the storage units that receives a low retention priority write request of the set of low retention priority write requests;

determining a low retention priority response regarding availability for storing low retention priority data based on current storage of low priority data objects and available memory for storing the low retention priority data; and

sending the low retention priority response; and

when a threshold number of favorable low retention priority responses have been received, facilitating storage of the low retention priority data object.

2. The method of claim 1 further comprises:

including a set of encoded data slices within the set of low retention priority write requests when sending the set of low retention priority write requests, wherein a data segment of the low retention priority data object is dispersed storage error encoded to produce the set of encoded data slices; and

for each storage unit of the storage units that receives a low retention priority write request of the set of low retention priority write requests;

determining whether there is available memory for storing the low retention priority data;

when there is not available memory for storing the low retention priority data, comparing retention priority of an encoded data slice of another low retention priority data object of the current storage of low priority data objects with a retention priority of an encoded data slice of the set of encoded data slices; and

when the retention priority of the encoded data slice of the set of encoded data slices compares favorably to the retention priority of the encoded data slice of the other low retention priority data object;

overwriting the encoded data slice of the other low retention priority data object with the encoded data slice of the set of encoded data slices; and

sending a write response.

3. The method of claim 2, wherein the facilitating the storage of the low retention priority data object comprises:

sending a set of write commit requests to each storage unit sending the write response.

4. The method of claim 2, wherein the determining whether there is available memory for storing the low retention priority data comprises:

determining total available memory within the storage unit;

determining a level of storage of higher priority data objects; and

determining whether there is available memory for storing the low retention priority data based on the total available memory and the level of storage the higher priority data objects.

5. The method of claim 1 further comprises:

the facilitating storage of the low retention priority data object by sending, for storage, a set of encoded data slices to the storage units providing the favorable low retention priority response, wherein a data segment of the low retention priority data object is dispersed storage error encoded to produce the set of encoded data slices; and

for each storage unit receiving an encoded data slice of the set of encoded data slices;

determining whether there is low retention priority memory currently available;

when the low retention priority memory is not currently available, comparing retention priority of an encoded data slice of another low retention priority data object of the current storage of low priority data objects with a retention priority of an encoded data slice of the set of encoded data slices; and

when the retention priority of the encoded data slice of the set of encoded data slices compares favorably to the retention priority of the encoded data slice of the other low retention priority data object;

overwriting the encoded data slice of the other low retention priority data object with the encoded data slice of the set of encoded data slices; and

sending a write response.

6. The method of claim 1 further comprises:

generating the low retention priority query to include queries regarding different levels of low retention priority data;

for the given low retention priority level of the low retention priority data object to be stored, determining whether the threshold number of favorable low retention priority responses have been received; and

when the threshold number of favorable low retention priority responses have been received for the given low retention priority level, facilitating the storage of the low retention priority data object.

7. A method for execution by a storage unit of a dispersed storage network (DSN), the method comprises:

receiving a low retention priority write request of a set of low retention priority write requests to storage units of the DSN based on a write facilitating request to store a low retention priority data object in the DSN, wherein the low retention priority write request includes a low retention priority query, wherein the low retention priority data object corresponds to scratch pad data associated with intermediate results generated when a task is performed on other data, and wherein the low retention priority data object requires only temporary storage in the DSN, wherein the low retention priority data object includes a given low retention priority level of a plurality of low retention priority level of low retention priority data that specify that first information within the DSN associated with a first retention priority level shall not be deleted from storage within the DSN before second information within the DSN associated with a second retention priority level when the first retention priority level indicates a greater requirement for retention than the second retention priority level;

determining a low retention priority response regarding availability for storing low retention priority data based on current storage of low priority data objects and available memory for storing the low retention priority data;

sending the low retention priority response; and

receiving, from a computing device, another write facilitating request regarding storage of a portion of the low retention priority data object when the computing device received a threshold number of favorable low retention priority responses.

8. The method of claim 7 further comprises:

receiving an encoded data slice of a set of encoded data slices within the low retention priority write request, wherein a data segment of the low retention priority data object is dispersed storage error encoded to produce the set of encoded data slices;

determining whether there is available memory for storing the low retention priority data;

when there is not available memory for storing the low retention priority data, comparing retention priority of an encoded data slice of another low retention priority data object of the current storage of low priority data objects with a retention priority of an encoded data slice of the set of encoded data slices; and

when the retention priority of the encoded data slice of the set of encoded data slices compares favorably to the retention priority of the encoded data slice of the other low retention priority data object;

overwriting the encoded data slice of the other low retention priority data object with the encoded data slice of the set of encoded data slices; and

sending a write response.

9. The method of claim 8, wherein the determining whether there is available memory for storing the low retention priority data comprises:

determining total available memory within the storage unit;

determining a level of storage of higher priority data objects; and

determining whether there is available memory for storing the low retention priority data based on the total available memory and the level of storage the higher priority data objects.

10. The method of claim 7 further comprises:

interpreting the low retention priority query to identify queries regarding different levels of low retention priority data;

for a given low retention priority level of the low retention priority data object to be stored, determining whether the threshold number of favorable low retention priority responses have been received; and

when the threshold number of favorable low retention priority responses have been received for the given low retention priority level, facilitating storage of the low retention priority data object.

11. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processors of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

send a set of low retention priority write requests to storage units of the DSN based on a write facilitating request to store a low retention priority data object in the DSN, wherein each low retention priority write request of the set of low retention priority write requests includes a low retention priority query, wherein the low retention priority data object corresponds to scratch pad data associated with intermediate results generated when a task is performed on other data, and wherein the low retention priority data object requires only temporary storage in the DSN, wherein the low retention priority data object includes a given low retention priority level of a plurality of low retention priority level of low retention priority data that specify that first information within the DSN associated with a first retention priority level shall not be deleted from storage within the DSN before second information within the DSN associated with a second retention priority level when the first retention priority level indicates a greater requirement for retention than the second retention priority level;

for each storage unit of the storage units that receives a low retention priority write request of the set of low retention priority write requests;

determine a low retention priority response regarding availability for storing low retention priority data based on current storage of low priority data objects and available memory for storing the low retention priority data; and

send the low retention priority response; and

when a threshold number of favorable low retention priority responses have been received, facilitate storage of the low retention priority data object.

12. The non-transitory computer readable storage medium of claim 11 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processors causes the one or more computing devices of the DSN to;

include a set of encoded data slices within the set of low retention priority write requests when sending the set of low retention priority write requests, wherein a data segment of the low retention priority data object is dispersed storage error encoded to produce the set of encoded data slices; and

for each storage unit of the storage units that receives a low retention priority write request of the set of low retention priority write requests;

determine whether there is available memory for storing the low retention priority data;

when there is not available memory for storing the low retention priority data, compare retention priority of an encoded data slice of another low retention priority data object of the current storage of low priority data objects with a retention priority of an encoded data slice of the set of encoded data slices; and

when the retention priority of the encoded data slice of the set of encoded data slices compares favorably to the retention priority of the encoded data slice of the other low retention priority data object;

overwrite the encoded data slice of the other low retention priority data object with the encoded data slice of the set of encoded data slices; and

send a write response.

13. The non-transitory computer readable storage medium of claim 12, wherein the one or more processors functions to execute the operational instructions stored by the at least one memory section to cause the one or more computing devices of the DSN to facilitate the storage of the low retention priority data object by:

sending a set of write commit requests to each storage unit sending the write response.

14. The non-transitory computer readable storage medium of claim 12, wherein the one or more processors functions to execute the operational instructions stored by the at least one memory section to cause the one or more computing devices of the DSN to determine whether there is available memory for storing the low retention priority data object by:

determining total available memory within the storage unit;

determining a level of storage of higher priority data objects; and

determining whether there is available memory for storing the low retention priority data based on the total available memory and the level of storage the higher priority data objects.

15. The non-transitory computer readable storage medium of claim 11 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processors causes the one or more computing devices of the DSN to;

facilitate the storage of the low retention priority data object by sending, for storage, a set of encoded data slices to the storage units providing the favorable low retention priority response, wherein a data segment of the low retention priority data object is dispersed storage error encoded to produce the set of encoded data slices; and

for each storage unit receiving an encoded data slice of the set of encoded data slices;

determine whether there is low retention priority memory currently available;

when the low retention priority memory is not currently available, compare retention priority of an encoded data slice of another low retention priority data object of the current storage of low priority data objects with a retention priority of an encoded data slice of the set of encoded data slices; and

when the retention priority of the encoded data slice of the set of encoded data slices compares favorably to the retention priority of the encoded data slice of the other low retention priority data object;

overwrite the encoded data slice of the other low retention priority data object with the encoded data slice of the set of encoded data slices; and

send a write response.

16. The non-transitory computer readable storage medium of claim 11 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processors causes the one or more computing devices of the DSN to;

generate the low retention priority query to include queries regarding different levels of low retention priority data;

for a given low retention priority level of the low retention priority data object to be stored, determine whether the threshold number of favorable low retention priority responses have been received; and

when the threshold number of favorable low retention priority responses have been received for the given low retention priority level, facilitate the storage of the low retention priority data object.

17. A storage device of a dispersed storage network (DSN), the storage device comprises:

an interface; and

memory that stores operational instructions; and

a processor operably coupled to the interface and to the memory, wherein the processor, when operable within the storage device based on the operational instructions, is configured to;

receive, via the interface, a low retention priority write request of a set of low retention priority write requests to storage units of the DSN based on a write facilitating request to store a low retention priority data object in the DSN, wherein the low retention priority write request includes a low retention priority query, wherein the low retention priority data object corresponds to scratch pad data associated with intermediate results generated when a task is performed on other data, and wherein the low retention priority data object requires only temporary storage in the DSN, wherein the low retention priority data object includes a given low retention priority level of a plurality of low retention priority level of low retention priority data that specify that first information within the DSN associated with a first retention priority level shall not be deleted from storage within the DSN before second information within the DSN associated with a second retention priority level when the first retention priority level indicates a greater requirement for retention than the second retention priority level;

determine a low retention priority response regarding availability for storing low retention priority data based on current storage of low priority data objects and available memory for storing the low retention priority data;

send, via the interface, the low retention priority response; and

receive, from a computing device via the interface, another write facilitating request regarding storage of a portion of the low retention priority data object when the computing device received a threshold number of favorable low retention priority responses.

18. The storage device of claim 17 further comprises:

the processor is further operable to;

receive, via the interface, an encoded data slice of a set of encoded data slices within the low retention priority write request, wherein a data segment of the low retention priority data object is dispersed storage error encoded to produce the set of encoded data slices;

determine whether there is available memory for storing the low retention priority data;

when there is not available memory for storing the low retention priority data, compare retention priority of an encoded data slice of another low retention priority data object of the current storage of low priority data objects with a retention priority of an encoded data slice of the set of encoded data slices; and

when the retention priority of the encoded data slice of the set of encoded data slices compares favorably to the retention priority of the encoded data slice of the other low retention priority data object;

overwrite the encoded data slice of the other low retention priority data object with the encoded data slice of the set of encoded data slices; and

send, via the interface, a write response.

19. The storage device of claim 18, wherein the storage device functions to determine whether there is available memory for storing the low retention priority data by:

determining total available memory within the storage device;

determining a level of storage of higher priority data objects; and

determining whether there is available memory for storing the low retention priority data based on the total available memory and the level of storage the higher priority data objects.

20. The storage device of claim 17 further comprises:

the processor is further operable to;

interpret the low retention priority query to identify queries regarding different levels of low retention priority data;

for a given low retention priority level of the low retention priority data object to be stored, determine whether the threshold number of favorable low retention priority responses have been received; and

when the threshold number of favorable low retention priority responses have been received for the given low retention priority level, facilitate storage of the low retention priority data object.