Dispersed storage network with data segment backup and methods for use therewith

• US 10,176,054 B2
• Filed: 07/01/2015
• Issued: 01/08/2019
• Est. Priority Date: 09/30/2005
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K., Leggette, Wesley
• Primary Examiner(s): Lwin, Maung T
• Application Number:
  US14/789,968
• Publication Number:
  US 20150301905A1
• Time in Patent Office: 1,287 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 63/10 for controlling access to devices or network resources
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins with a processing module providing a data segment. The method continues with the processing module retrieving a plurality of first slices, corresponding to a previous revision of the data segment, from the distributed storage network. The method continues with the processing module recreating the previous revision of the data segment from the plurality of first slices corresponding to the previous revision of the data segment. The method continues with the processing module determining if the previous revision of the data segment compares unfavorably to the data segment. The method continues with the processing module storing the data segment in the DSN when determined that the previous version of the data segment compares unfavorably to the data segment.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

receiving a backup command regarding a modified version of a data object, wherein a previous version of the data object is stored in the DSN as a plurality of sets of encoded data slices based on operational parameters, wherein, in accordance with the operational parameters, the previous version of the data object is partitioned into a first plurality of data segments that is encoded to produce the plurality of sets of encoded data slices;

partitioning the modified version of the data object into a second plurality of data segments in accordance with the operational parameters;

identifying a modified data segment of the second plurality of data segments based on a comparison between information derived from the modified data segment and information derived from a corresponding data segment of the first plurality of data segments;

for the modified data segment;

determining whether to encode the modified data segment using the operational parameters or different operational parameters to produce determined operational parameters;

encoding the modified data segment in accordance with the determined operational parameters to produce a revised set of encoded data slices;

updating a revision number associated with the modified data segment;

in response to determining to encode the modified data segment using different operational parameters, associating an indication of the different operational parameters with the revised set of encoded data slices; and

storing the revised set of encoded data slices in storage units of the DSN.

2. The method of claim 1, wherein the operational parameters comprise two or more of:

an error coding algorithm, a pillar width number, a read threshold, a write threshold, an encryption algorithm, a slicing parameter, a compression algorithm, an integrity check method, caching settings, or parallelism settings.

3. The method of claim 1, wherein the determining whether to encode the modified data segment using the operational parameters or the different operational parameters comprises:

determining a size of the modified data segment is different than a size of a corresponding data segment by a size difference;

determining whether the size difference between the modified data segment and the corresponding data segment is at or above a threshold; and

in response to determining that the size difference between the modified data segment and the corresponding data segment is at or above a threshold, determining to use the different operational parameters.

4. The method of claim 1, wherein the determining whether to encode the modified data segment using the operational parameters or the different operational parameters comprises:

determining a desired storage reliability for the modified version of the data object; and

determining the different operational parameters based on the desired storage reliability.

5. The method of claim 1, wherein the identifying a modified data segment comprises:

retrieving at least a threshold number of encoded data slices of a set of encoded data slices for the corresponding data segment;

decoding the at least the threshold number of encoded data slices to reconstruct the corresponding data segment;

comparing the corresponding data segment with the modified data segment; and

in response to determining that the modified data segment does not substantially match the corresponding data segment, identifying the modified data segment.

6. The method of claim 1, further comprises:

updating a user vault with a new revision number and the determined operational parameters for the modified data segment.

7. A dispersed storage (DS) processing unit for use in a dispersed storage network (DSN) comprises:

memory;

an interface; and

a processing module operably coupled to the memory and to the interface, wherein the processing module is operable to;

receive a backup command regarding a modified version of a data object, wherein a previous version of the data object is stored in the DSN as a plurality of sets of encoded data slices based on operational parameters, wherein, in accordance with the operational parameters, the previous version of the data object is partitioned into a first plurality of data segments that is encoded to produce the plurality of sets of encoded data slices;

partition the modified version of the data object into a second plurality of data segments in accordance with the operational parameters;

identify a modified data segment of the second plurality of data segments based on a comparison between information derived from the modified data segment and information derived from a corresponding data segment of the first plurality of data segments;

for the modified data segment;

determine whether to encode the modified data segment using the operational parameters or different operational parameters to produce determined operational parameters;

encode the modified data segment in accordance with the determined operational parameters to produce a revised set of encoded data slices;

update a revision number associated with the modified data segment;

in response to determining to encode the modified data segment using different operational parameters, associate an indication of the different operational parameters with the revised set of encoded data slices; and

output, via the interface, the revised set of encoded data slices to storage units of the DSN for storage therein.

8. The DS processing unit of claim 7, wherein the operational parameters comprise two or more of:

an error coding algorithm, a pillar width number, a read threshold, a write threshold, an encryption algorithm, a slicing parameter, a compression algorithm, an integrity check method, caching settings, or parallelism settings.

9. The DS processing unit of claim 7, wherein the processing module is further operable to determine whether to encode the modified data segment using the operational parameters or the different operational parameters by:

determining a size of the modified data segment is different than a size of a corresponding data segment by a size difference;

determining whether the size difference between the modified data segment and the corresponding data segment is at or above a threshold; and

in response to determining that the size difference between the modified data segment and the corresponding data segment is at or above a threshold, determining to use the different operational parameters.

10. The DS processing unit of claim 7, wherein the processing module is further operable to determine whether to encode the modified data segment using the operational parameters or the different operational parameters by:

determining a desired storage reliability for the modified version of the data object; and

determining the different operational parameters based on the desired storage reliability.

11. The DS processing unit of claim 7, wherein the processing module is further operable to identify the modified data segment by:

retrieving at least a threshold number of encoded data slices of a set of encoded data slices for the corresponding data segment;

decoding the at least the threshold number of encoded data slices to reconstruct the corresponding data segment;

comparing the corresponding data segment with the modified data segment; and

in response to determining that the modified data segment does not substantially match the corresponding data segment, identifying the modified data segment.

12. The DS processing unit of claim 7, wherein the processing module is further operable to:

updating a user vault with a new revision number and the determined operational parameters for the modified data segment.

13. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

receive a backup command regarding a modified version of a data object, wherein a previous version of the data object is stored in the DSN as a plurality of sets of encoded data slices based on operational parameters, wherein, in accordance with the operational parameters, the previous version of the data object is partitioned into a first plurality of data segments that is encoded to produce the plurality of sets of encoded data slices;

partition the modified version of the data object into a second plurality of data segments in accordance with the operational parameters;

identify a modified data segment of the second plurality of data segments based on a comparison between information derived from the modified data segment and information derived from a corresponding data segment of the first plurality of data segments;

for the modified data segment;

determine whether to encode the modified data segment using the operational parameters or different operational parameters to produce determined operational parameters;

encode the modified data segment in accordance with the determined operational parameters to produce a revised set of encoded data slices;

update a revision number associated with the modified data segment;

in response to determining to encode the modified data segment using different operational parameters, associate an indication of the different operational parameters with the revised set of encoded data slices; and

output the revised set of encoded data slices to storage units of the DSN for storage therein.

14. The non-transitory computer readable storage medium of claim 13, wherein the operational parameters comprise two or more of:

an error coding algorithm, a pillar width number, a read threshold, a write threshold, an encryption algorithm, a slicing parameter, a compression algorithm, an integrity check method, caching settings, or parallelism settings.

15. The non-transitory computer readable storage medium of claim 13, wherein the at least one memory section that stores operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices to determine whether to encode the modified data segment using the operational parameters or the different operational parameters by:

determining a size of the modified data segment is different than a size of a corresponding data segment by a size difference;

determining whether the size difference between the modified data segment and the corresponding data segment is at or above a threshold; and

in response to determining that the size difference between the modified data segment and the corresponding data segment is at or above a threshold, determining to use the different operational parameters.

16. The non-transitory computer readable storage medium of claim 13, wherein the at least one memory section that stores operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices to determine whether to encode the modified data segment using the operational parameters or the different operational parameters by:

determining a desired storage reliability for the modified version of the data object; and

determining the different operational parameters based on the desired storage reliability.

17. The non-transitory computer readable storage medium of claim 13, wherein the at least one memory section that stores operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices to identify the modified data segment by:

retrieving at least a threshold number of encoded data slices of a set of encoded data slices for the corresponding data segment;

decoding the at least the threshold number of encoded data slices to reconstruct the corresponding data segment;

comparing the corresponding data segment with the modified data segment; and

in response to determining that the modified data segment does not substantially match the corresponding data segment, identifying the modified data segment.

18. The non-transitory computer readable storage medium of claim 13, wherein the at least one memory section that stores operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices to:

updating a user vault with a new revision number and the determined operational parameters for the modified data segment.