DISPERSED STORAGE OF SOFTWARE

• US 20110161680A1
• Filed: 10/12/2010
• Published: 06/30/2011
• Est. Priority Date: 12/29/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: GRUBE, GARY W., MARKISON, TIMOTHY W.
• Granted Patent: US 8,762,343 B2
• Time in Patent Office: Days
• US Class Current: 713/193
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 2209/603 Digital right managament [DRM]
• CPC Class Codes: H04L 9/3236 using cryptographic hash functions

Abstract

A data de-duplication method begins by a processing module receiving a plurality of data storage requests from a plurality of requesting devices wherein a data storage request includes the data and a requester identifier (ID). The method continues with the processing module obtaining a data identifier (ID) for the data. For each of the plurality of data storage requests, the method continues with the processing module producing a requester storage record, dispersed storage error encoding the requester storage record to produce a set of encoded requester storage record slices, and sending the set of encoded requester storage record slices to a dispersed storage network (DSN) memory for storage therein. The method continues with the processing module dispersed storage error encoding at least a portion of the data to produce a set of encoded data slices and sending the set of encoded data slices to the DSN memory for storage therein.

Claims

1. A data de-duplication method, wherein the method comprises:

receiving a plurality of data storage requests from a plurality of requesting devices regarding storage of data, wherein a data storage request of the plurality of data storage requests includes the data and a requester identifier (ID) of a corresponding one of the plurality of requesting devices;

obtaining a data identifier (ID) for the data;

for each of the plurality of data storage requests;

combining the data ID and the requester ID to produce a requester storage record;

dispersed storage error encoding the requester storage record to produce a set of encoded requester storage record slices;

sending the set of encoded requester storage record slices to a dispersed storage network (DSN) memory for storage therein;

dispersed storage error encoding at least a portion of the data from one of the plurality of data storage requests to produce a set of encoded data slices; and

sending the set of encoded data slices to the DSN memory for storage therein.

2. The method of claim 1, wherein the obtaining the data ID comprises at least one of:

generating a calculated hash of the data;

retrieving the data ID from profile table based on the calculated hash; and

receiving the data ID from the requester.

3. The method of claim 1 further comprises:

determining, from data storage request to data storage request of the plurality of data storage requests, whether the data is substantially the same by at least one of;

determining whether a first data ID associated with the data of a first data storage request of the plurality of data storage requests substantially matches a second data ID associated with the data of a second data storage request of the plurality of data storage requests;

determining whether a first hash of the data of the first data storage request substantially matches a second hash of the data of the second data storage request; and

comparing the data of the first data storage request with the data of the second data storage request.

4. The method of claim 1, wherein the combining the data ID and the requester ID comprises at least one of:

selecting at least one of the data ID and the requester ID as the requester storage record;

encrypting at least one of the data ID and the requester ID using a random encryption key to produce the requester storage record;

encrypting the random encryption key using a public key to produce an encrypted random encryption key;

dispersed storage error encoding the encrypted random encryption key to produce a set of encoded key slices; and

sending the set of encoded key slices to the DSN memory for storage therein.

5. The method of claim 1, wherein the dispersed storage error encoding at least the portion of the data comprises at least one of:

encoding at least one of the portion of the data and the data ID using an error coding dispersal storage function to produce the set of encoded data slices;

encrypting at least one of the portion of the data and the data ID using a random encryption key to produce encrypted data and encoding the encrypted data using the error coding dispersal storage function to produce the set of encoded data slices; and

encrypting the random encryption key using a public key to produce an encrypted random encryption key and dispersed storage error encoding the encrypted random encryption key to produce a set of encoded key slices.

6. The method of claim 1 further comprises:

selecting the one of the plurality of data storage requests based on at least one of;

an ordering of receipt of the plurality of data storage requests;

the requester ID identifying a particular de-duplication authorization level;

a type of the data;

a size of the data; and

number of the plurality of data storage requests.

7. A method for retrieving de-duplicated stored data, wherein the method comprises:

receiving a plurality of data retrieval requests from a plurality of requesting devices to retrieve the de-duplicated stored data, wherein a data retrieval request of the plurality of data retrieval requests includes a requester storage record identifier (ID);

for each of the plurality of data retrieval requests;

retrieving a set of encoded requester storage record slices from a dispersed storage network (DSN) memory based on the requester storage record ID;

dispersed storage error decoding the set of encoded requester storage record slices to reproduce a requester storage record; and

extracting a data identifier (ID) and a requester identifier (ID) from the requester storage record;

retrieving a set of encoded data slices from the DSN memory based on the data ID of one of the plurality of data retrieval requests;

dispersed storage error decoding the set of encoded data slices to reproduce at least a portion of the de-duplicated stored data; and

sending the at least a portion of the de-duplicated stored data to each of the plurality of requesting devices.

8. The method of claim 7 further comprises:

the data retrieval request further includes a requester device provided requester ID; and

verifying the requester ID extracted from the requester storage record with the requester device provided requester ID.

9. The method of claim 7, wherein the dispersed storage error decoding the set of encoded requester storage record slices comprises:

retrieving a set of encoded key slices from the DSN memory;

dispersed storage error decoding the set of encoded key slices to produce an encrypted random encryption key;

decrypting the encrypted random encryption key using a private key to produce a random encryption key;

dispersed storage error decoding the set of encoded requester storage record to produce an encrypted requester storage record; and

decrypting the encrypted requester storage record using the random encryption key to produce the requester storage record.

10. The method of claim 7, wherein the dispersed storage error decoding the set of encoded data slices comprises at least one of:

retrieving a set of encoded key slices from the DSN memory;

dispersed storage error decoding the set of encoded key slices to produce an encrypted random encryption key;

decrypting the encrypted random encryption key using a private key to produce a random encryption key;

dispersed storage error decoding the plurality of encoded data slices to produce a plurality of encrypted data segments; and

decrypting each of the plurality of encrypted data segments using the random encryption key to produce the at least a portion of the de-duplicated stored data.

11. A computer comprises:

an interface; and

a processing module operable to;

receive, via the interface, a plurality of data storage requests from a plurality of requesting devices regarding storage of data, wherein a data storage request of the plurality of data storage requests includes the data and a requester identifier (ID) of a corresponding one of the plurality of requesting devices;

obtain a data identifier (ID) for the data;

for each of the plurality of data storage requests;

combine the data ID and the requester ID to produce a requester storage record;

dispersed storage error encode the requester storage record to produce a set of encoded requester storage record slices;

send, via the interface, the set of encoded requester storage record slices to a dispersed storage network (DSN) memory for storage therein;

dispersed storage error encode at least a portion of the data from one of the plurality of data storage requests to produce a set of encoded data slices; and

send, via interface, the set of encoded data slices to the DSN memory for storage therein.

12. The computer of claim 11, wherein the processing module further functions to obtain the data ID by least one of:

generating a calculated hash of the data;

retrieving the data ID from profile table based on the calculated hash; and

receiving, via the interface, the data ID from the requester.

13. The computer of claim 11, wherein the processing module further functions to:

determine, from data storage request to data storage request of the plurality of data storage requests, whether the data is substantially the same by at least one of;

determining whether a first data ID associated with the data of a first data storage request of the plurality of data storage requests substantially matches a second data ID associated with the data of a second data storage request of the plurality of data storage requests;

determining whether a first hash of the data of the first data storage request substantially matches a second hash of the data of the second data storage request; and

comparing the data of the first data storage request with the data of the second data storage request.

14. The computer of claim 11, wherein the processing module further functions to combine the data ID and the requester ID by at least one of:

selecting at least one of the data ID and the requester ID as the requester storage record;

encrypting at least one of the data ID and the requester ID using a random encryption key to produce the requester storage record;

encrypting the random encryption key using a public key to produce an encrypted random encryption key;

dispersed storage error encoding the encrypted random encryption key to produce a set of encoded key slices; and

sending, via the interface, the set of encoded key slices to the DSN memory for storage therein.

15. The computer of claim 11, wherein the processing module further functions to dispersed storage error encode at least the portion of the data by at least one of:

encoding at least one of the portion of the data and the data ID using an error coding dispersal storage function to produce the set of encoded data slices;

encrypting at least one of the portion of the data and the data ID using a random encryption key to produce encrypted data and encoding the encrypted data using the error coding dispersal storage function to produce the set of encoded data slices; and

encrypting the random encryption key using a public key to produce an encrypted random encryption key and dispersed storage error encoding the encrypted random encryption key to produce a set of encoded key slices.

16. The computer of claim 11, wherein the processing module further functions to:

select the one of the plurality of data storage requests based on at least one of;

an ordering of receipt of the plurality of data storage requests;

the requester ID identifying a particular de-duplication authorization level;

a type of the data;

a size of the data; and

number of the plurality of data storage requests.

17. A computer comprises:

an interface; and

a processing module operable to;

receive, via the interface, a plurality of data retrieval requests from a plurality of requesting devices to retrieve de-duplicated stored data, wherein a data retrieval request of the plurality of data retrieval requests includes a requester storage record identifier (ID);

for each of the plurality of data retrieval requests;

retrieve, via the interface, a set of encoded requester storage record slices from a dispersed storage network (DSN) memory based on the requester storage record ID;

dispersed storage error decode the set of encoded requester storage record slices to reproduce a requester storage record; and

extract a data identifier (ID) and a requester identifier (ID) from the requester storage record;

retrieve, via the interface, a set of encoded data slices from the DSN memory based on the data ID of one of the plurality of data retrieval requests;

dispersed storage error decode the set of encoded data slices to reproduce at least a portion of the de-duplicated stored data; and

send, via the interface, the at least a portion of the de-duplicated stored data to each of the plurality of requesting devices.

18. The computer of claim 17, wherein the processing module further functions to:

the data retrieval request further includes a requester device provided requester ID; and

verify the requester ID extracted from the requester storage record with the requester device provided requester ID.

19. The computer of claim 17, wherein the processing module functions to dispersed storage error decode the set of encoded requester storage record slices by:

retrieving, via the interface, a set of encoded key slices from the DSN memory;

dispersed storage error decoding the set of encoded key slices to produce an encrypted random encryption key;

decrypting the encrypted random encryption key using a private key to produce a random encryption key;

dispersed storage error decoding the set of encoded requester storage record to produce an encrypted requester storage record; and

decrypting the encrypted requester storage record using the random encryption key to produce the requester storage record.

20. The computer of claim 17, wherein the processing module functions to dispersed storage error decode the set of encoded data slices by at least one of:

retrieving, via the interface, a set of encoded key slices from the DSN memory;

dispersed storage error decoding the set of encoded key slices to produce an encrypted random encryption key;

decrypting the encrypted random encryption key using a private key to produce a random encryption key;

dispersed storage error decoding the plurality of encoded data slices to produce a plurality of encrypted data segments; and

decrypting each of the plurality of encrypted data segments using the random encryption key to produce the at least a portion of the de-duplicated stored data.