Efficient storage of data in a dispersed storage network

• US 10,075,523 B2
• Filed: 02/04/2014
• Issued: 09/11/2018
• Est. Priority Date: 04/01/2013
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Volvovski, Ilya, Gladwin, S. Christopher, Grube, Gary W., Markison, Timothy W., Resch, Jason K., Shirley, Jr., Thomas Franklin, Dhuse, Greg, Motwani, Manish, Baptist, Andrew, Leggette, Wesley
• Primary Examiner(s): Meky, Moustafa M
• Assistant Examiner(s): Kassa, Elizabeth
• Application Number:
  US14/172,140
• Publication Number:
  US 20140297776A1
• Time in Patent Office: 1,680 Days
• Field of Search: 709212
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module receiving data for storage and generating a dispersed storage network (DSN) source name for the data. The method continues with the DS processing module determining whether substantially identical data to the data has been previously stored in memory of the DSN. When the substantially identical data has been previously stored in the memory of the DSN, the method continues with the DS processing module generating an object linking file that links the data to the substantially identical data, dispersed storage error encoding the object linking file to produce a set of encoded link file slices, and outputting the set of encoded link file slices for storage in the memory of the DSN.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

receiving a data object “

B”

for storage;

generating a DSN source name for the data object “

B”

;

when the data object “

B”

is substantially identical to a previously stored data object “

A”

, generating an object linking file that links a DSN address of the previously stored data object “

A”

to the data object “

B”

, wherein the previously stored data object “

A”

has an associated DSN source name that differs from the DSN source name for the data object “

B”

;

dispersed storage error encoding the object linking file to produce a set of encoded link file slices; and

outputting the set of encoded link file slices for storage in memory of the DSN.

2. The method of claim 1, wherein the generating the DSN source name for the data object “

B”

comprises;

obtaining a vault identifier;

generating a random number; and

combining the vault identifier and the random number to produce the DSN source name.

3. The method of claim 1 further comprises:

the data object “

B”

is substantially identical to the previously stored data object “

A”

when a data tag of data object “

B”

substantially matches a data tag of data object “

A”

.

4. The method of claim 1, further comprises:

the data object “

B”

is substantially identical to the previously stored data object “

A”

when a data comparison function performed on the data object “

B” and

the previously stored data object “

A”

indicates that the data object “

B”

is substantially identical the previously stored data object “

A”

.

5. The method of claim 1, wherein the generating the object linking file further comprises:

labeling the object linking file with the DSN source name for the data object “

B”

.

6. The method of claim 5, wherein the generating the object linking file further comprises:

generating a data tag;

accessing, based on the data tag, a data tag list to identify an entry associated with the previously stored data object “

A”

; and

retrieving the DSN address of the previously stored data object “

A”

from the identified entry.

7. The method of claim 5, wherein the generating the object linking file further comprises:

generating a data tag;

searching, based on the data tag, a dispersed hierarchical index to identify an entry associated with the previously stored data object “

A”

; and

retrieving the DSN address of the previously stored data object “

A”

from the identified entry.

8. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

receive a data object “

B”

for storage; and

generate a dispersed storage network (DSN) source name for the data object “

B”

;

a second module, when operable within the computing device, causes the computing device to;

when the data object “

B”

is substantially identical to a previously stored data object “

A”

generate an object linking file that links a DSN address of the previously stored data object “

A”

to the data object “

B”

;

dispersed storage error encode the object linking file to produce a set of encoded link file slices; and

output the set of encoded link file slices for storage in memory of the DSN.

9. The DS module of claim 8, wherein the first module functions to cause the computing device to generate the DSN source name for the data object “

B”

by;

obtaining a vault identifier;

generating a random number; and

combining the vault identifier and the random number to produce the DSN source name.

10. The DS module of claim 8 further comprises:

the data object “

B”

is substantially identical to the previously stored data object “

A”

when a data tag of data object “

B”

substantially matches a data tag of data object “

A”

.

11. The DS module of claim 8 further comprises:

the data object “

B”

is substantially identical to the previously stored data object “

A”

when a data comparison function performed on the data object “

B” and

the previously stored data object “

A”

indicates that the data object “

B”

is substantially identical the previously stored data object “

A”

.

12. The DS module of claim 8, wherein the second module functions to cause the computing device to generate the object linking file by:

labeling the object linking file with the DSN source name for the data object “

B”

.

13. The DS module of claim 12, wherein the second module further functions to cause the computing device to generate the object linking file by:

generating a data tag;

accessing, based on the data tag, a data tag list to identify an entry associated with the previously stored data object “

A”

; and

retrieving the DSN address of the previously stored data object “

A”

from the identified entry.

14. The DS module of claim 12, wherein the second module further functions to cause the computing device to generate the object linking file by:

generating a data tag;

searching, based on the data tag, a dispersed hierarchical index to identify an entry associated with the previously stored data object “

A”

; and

retrieving the DSN address of the previously stored data object “

A”

from the identified entry.