Updating de-duplication tracking data for a dispersed storage network

• US 9,661,074 B2
• Filed: 06/30/2014
• Issued: 05/23/2017
• Est. Priority Date: 08/29/2013
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Peake, Andrew George
• Primary Examiner(s): Ly, Anh
• Application Number:
  US14/320,547
• Publication Number:
  US 20150066875A1
• Time in Patent Office: 1,058 Days
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module of a dispersed storage network (DSN) determining whether a change has occurred to a data object of a set of data objects. When a change has occurred, the method continues with the DS processing module accessing de-duplication tracking data for the set of data objects. When the change is deletion of an identified data object of the set of data objects, the method continues with the DS processing module determining whether the identified data object is the only data object in the set of data objects. When the identified data object is not the only data object in the set of data objects, the method continues with the DS processing module updating the linking information to delete linking the identified data object to addressing information.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

determining whether a change has occurred to a data object of a set of data objects, wherein each data object of the set of data objects has substantially identical data to other data objects of the set of data objects and wherein only one plurality of sets of encoded data slices corresponding to one data object of the set of data objects is stored in memory of the DSN; and

when a change has occurred;

accessing deduplication tracking data for the set of data objects, wherein the deduplication tracking data includes a representation of the substantially identical data, addressing information for the only one plurality of sets of encoded data slices, and linking information that links the set of data objects to the addressing information, wherein the linking information includes the number of copies of data objects having substantially identical data;

when the change is deletion of an identified data object of the set of data objects;

determining whether the identified data object is the only data object in the set of data objects; and

when the identified data object is not the only data object in the set of data objects, updating the linking information to delete linking the identified data object to the addressing information.

2. The method of claim 1 further comprises:

when the identified data object is the only data object in the set of data objects;

issuing a command to delete the only one plurality of sets of encoded data slices; and

deleting the deduplication tracking data.

3. The method of claim 1 further comprises:

when the change is addition of an identified data object of the set of data objects;

generating a linking object for linking DSN addressing of the identified data object to the addressing information; and

updating the linking information to include a representation of the linking object.

4. The method of claim 3 further comprises:

updating the linking information by incrementing a count of data objects in the set of data objects;

or

updating the linking information to include the linking object.

5. The method of claim 1, wherein the representation of the substantially identical data comprises:

a hashed representation of the data object.

6. The method of claim 1 further comprises:

identifying a plurality of linking objects stored in the memory of the DSN;

determining a count of linking objects of the plurality of linking objects that are associated with the set of data objects;

comparing the count of linking objects to a count of the linking objects of the linking information; and

when the count of linking objects compares unfavorably to the count of the linking objects of the linking information, updating the linking information based on a discrepancy between the count of linking objects and the count of the linking objects of the linking information.

7. A dispersed storage (DS) processing module of a dispersed storage network (DSN), the DS module comprises:

a first module, when operable within a computing device, causes the computing device to;

determine whether a change has occurred to a data object of a set of data objects, wherein each data object of the set of data objects has substantially identical data to other data objects of the set of data objects and wherein only one plurality of sets of encoded data slices corresponding to one data object of the set of data objects is stored in memory of the DSN; and

a second module, when operable within the computing device, causes the computing device to;

when a change has occurred;

access deduplication tracking data for the set of data objects, wherein the deduplication tracking data includes a representation of the substantially identical data, addressing information for the only one plurality of sets of encoded data slices, and linking information that links the set of data objects to the addressing information, wherein the linking information includes the number of copies of data objects having substantially identical data;

when the change is deletion of an identified data object of the set of data objects;

determine whether the identified data object is the only data object in the set of data objects; and

when the identified data object is not the only data object in the set of data objects, update the linking information to delete linking the identified data object to the addressing information.

8. The DS processing module of claim 7 further comprises:

the second module, when operable within the computing device, further causes the computing device to;

when the identified data object is the only data object in the set of data objects;

issue a command to delete the only one plurality of sets of encoded data slices; and

delete the deduplication tracking data.

9. The DS processing module of claim 7 further comprises:

the second module, when operable within the computing device, further causes the computing device to;

when the change is addition of an identified data object of the set of data objects;

generate a linking object for linking DSN addressing of the identified data object to the addressing information; and

update the linking information to include a representation of the linking object.

10. The DS processing module of claim 9 further comprises:

the second module, when operable within the computing device, further causes the computing device to;

update the linking information by incrementing a count of data objects in the set of data objects;

or

update the linking information to include the linking object.

11. The DS processing module of claim 7, wherein the representation of the substantially identical data comprises:

a hashed representation of the data object.

12. The DS processing module of claim 7 further comprises:

a third module, when operable within the computing device, causes the computing device to;

identify a plurality of linking objects stored in the memory of the DSN;

determine a count of linking objects of the plurality of linking objects that are associated with the set of data objects;

compare the count of linking objects to a count of the linking objects of the linking information; and

when the count of linking objects compares unfavorably to the count of the linking objects of the linking information, update the linking information based on a discrepancy between the count of linking objects and the count of the linking objects of the linking information.

13. A computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

determine whether a change has occurred to a data object of a set of data objects, wherein each data object of the set of data objects has substantially identical data to other data objects of the set of data objects and wherein only one plurality of sets of encoded data slices corresponding to one data object of the set of data objects is stored in memory of the DSN; and

when a change has occurred;

access deduplication tracking data for the set of data objects, wherein the deduplication tracking data includes a representation of the substantially identical data, addressing information for the only one plurality of sets of encoded data slices, and linking information that links the set of data objects to the addressing information, wherein the linking information includes the number of copies of data objects having substantially identical data;

when the change is deletion of an identified data object of the set of data objects;

determine whether the identified data object is the only data object in the set of data objects; and

when the identified data object is not the only data object in the set of data objects, update the linking information to delete linking the identified data object to the addressing information.

14. The computer readable storage medium of claim 13 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

when the identified data object is the only data object in the set of data objects;

issue a command to delete the only one plurality of sets of encoded data slices; and

delete the deduplication tracking data.

15. The computer readable storage medium of claim 13 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

when the change is addition of an identified data object of the set of data objects;

generate a linking object for linking DSN addressing of the identified data object to the addressing information; and

update the linking information to include a representation of the linking object.

16. The computer readable storage medium of claim 15 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

update the linking information by incrementing a count of data objects in the set of data objects;

or

update the linking information to include the linking object.

17. The computer readable storage medium of claim 13, wherein the representation of the substantially identical data comprises:

a hashed representation of the data object.

18. The computer readable storage medium of claim 13 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

identify a plurality of linking objects stored in the memory of the DSN;

determine a count of linking objects of the plurality of linking objects that are associated with the set of data objects;

compare the count of linking objects to a count of the linking objects of the linking information; and

when the count of linking objects compares unfavorably to the count of the linking objects of the linking information, update the linking information based on a discrepancy between the count of linking objects and the count of the linking objects of the linking information.