Dispersed storage of software

• US 8,762,343 B2
• Filed: 10/12/2010
• Issued: 06/24/2014
• Est. Priority Date: 12/29/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): PHAM, KHANH B
• Application Number:
  US12/902,684
• Publication Number:
  US 20110161680A1
• Time in Patent Office: 1,351 Days
• Field of Search: 707/652, 707/827
• US Class Current: 707/652
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 2209/603 Digital right managament [DRM]
• CPC Class Codes: H04L 9/3236 using cryptographic hash functions

Abstract

A data de-duplication method begins by a processing module receiving a plurality of data storage requests from a plurality of requesting devices wherein a data storage request includes the data and a requester identifier (ID). The method continues with the processing module obtaining a data identifier (ID) for the data. For each of the plurality of data storage requests, the method continues with the processing module producing a requester storage record, dispersed storage error encoding the requester storage record to produce a set of encoded requester storage record slices, and sending the set of encoded requester storage record slices to a dispersed storage network (DSN) memory for storage therein. The method continues with the processing module dispersed storage error encoding at least a portion of the data to produce a set of encoded data slices and sending the set of encoded data slices to the DSN memory for storage therein.

Claims

1. A data de-duplication method, wherein the method comprises:

receiving a plurality of data storage requests from a plurality of requesting devices regarding storage of substantially identical data, wherein a data storage request of the plurality of data storage requests includes the substantially identical data and a requester identifier (ID) of a corresponding one of the plurality of requesting devices;

obtaining a data identifier (ID) for the substantially identical data;

generating a requester storage record to include the data ID and the requester IDs from each of the plurality of data storage requests;

dispersed storage error encoding the requester storage record to produce a set of encoded requester storage record slices;

sending the set of encoded requester storage record slices to a dispersed storage network (DSN) memory for storage therein;

for the plurality of data storage requests, dispersed storage error encoding at least a portion of the substantially identical data from one of the plurality of data storage requests to produce a set of encoded data slices; and

sending the set of encoded data slices to the DSN memory for storage therein in accordance with the data ID.

2. The method of claim 1, wherein the obtaining the data ID comprises at least one of:

generating a calculated hash of the substantially identical data;

retrieving the data ID from profile table based on the calculated hash; and

receiving the data ID from the requester.

3. The method of claim 1 further comprises:

determining, from data storage request to data storage request of the plurality of data storage requests, whether the substantially identical data is substantially the same by at least one of;

determining whether a first data ID associated with the substantially identical data of a first data storage request of the plurality of data storage requests substantially matches a second data ID associated with the substantially identical data of a second data storage request of the plurality of data storage requests;

determining whether a first hash of the substantially identical data of the first data storage request substantially matches a second hash of the substantially identical data of the second data storage request; and

comparing the substantially identical data of the first data storage request with the substantially identical data of the second data storage request.

4. The method of claim 1, wherein the combining the data ID and the requester ID comprises at least one of:

selecting at least one of the data ID and the requester ID as the requester storage record;

encrypting at least one of the data ID and the requester ID using a random encryption key to produce the requester storage record;

encrypting the random encryption key using a public key to produce an encrypted random encryption key;

dispersed storage error encoding the encrypted random encryption key to produce a set of encoded key slices; and

sending the set of encoded key slices to the DSN memory for storage therein.

5. The method of claim 1, wherein the dispersed storage error encoding at least the portion of the substantially identical data comprises at least one of:

encoding at least one of the portion of the substantially identical data and the data ID using an error coding dispersal storage function to produce the set of encoded data slices;

encrypting at least one of the portion of the substantially identical data and the data ID using a random encryption key to produce encrypted data and encoding the encrypted data using the error coding dispersal storage function to produce the set of encoded data slices; and

encrypting the random encryption key using a public key to produce an encrypted random encryption key and dispersed storage error encoding the encrypted random encryption key to produce a set of encoded key slices.

6. The method of claim 1 further comprises:

selecting the one of the plurality of data storage requests based on at least one of;

an ordering of receipt of the plurality of data storage requests;

the requester ID identifying a particular de-duplication authorization level;

a type of the data;

a size of the data; and

number of the plurality of data storage requests.

7. A method for retrieving de-duplicated stored data, wherein the method comprises:

receiving a plurality of data retrieval requests from a plurality of requesting devices to retrieve substantially identical data, wherein a data retrieval request of the plurality of data retrieval requests includes a requester storage record identifier (ID);

for at least one of the plurality of data retrieval requests;

retrieving a set of encoded requester storage record slices from a dispersed storage network (DSN) memory based on the requester storage record ID;

dispersed storage error decoding the set of encoded requester storage record slices to reproduce a requester storage record; and

extracting a data identifier (ID) and a requester identifier (ID) from the requester storage record, wherein the data ID identifies the substantially identical data;

retrieving a set of encoded data slices from the DSN memory based on the data ID of one of the plurality of data retrieval requests;

dispersed storage error decoding the set of encoded data slices to reproduce at least a portion of the de-duplicated stored data; and

sending the at least a portion of the de-duplicated stored data to each of the plurality of requesting devices in accordance with the plurality of extracted requester IDs.

8. The method of claim 7 further comprises:

the data retrieval request further includes a requester device provided requester ID; and

verifying the requester ID extracted from the requester storage record with the requester device provided requester ID.

9. The method of claim 7, wherein the dispersed storage error decoding the set of encoded requester storage record slices comprises:

retrieving a set of encoded key slices from the DSN memory;

dispersed storage error decoding the set of encoded key slices to produce an encrypted random encryption key;

decrypting the encrypted random encryption key using a private key to produce a random encryption key;

dispersed storage error decoding the set of encoded requester storage record to produce an encrypted requester storage record; and

decrypting the encrypted requester storage record using the random encryption key to produce the requester storage record.

10. The method of claim 7, wherein the dispersed storage error decoding the set of encoded data slices comprises at least one of:

retrieving a set of encoded key slices from the DSN memory;

dispersed storage error decoding the set of encoded key slices to produce an encrypted random encryption key;

decrypting the encrypted random encryption key using a private key to produce a random encryption key;

dispersed storage error decoding the plurality of encoded data slices to produce a plurality of encrypted data segments; and

decrypting each of the plurality of encrypted data segments using the random encryption key to produce the at least a portion of the de-duplicated stored data.

11. A computer comprises:

an interface;

memory; and

one or more processing devices operably coupled to the interface and the memory, wherein the one or more processing devices is operable to;

receive, via the interface, a plurality of data storage requests from a plurality of requesting devices regarding storage of substantially identical data, wherein a data storage request of the plurality of data storage requests includes the substantially identical data and a requester identifier (ID) of a corresponding one of the plurality of requesting devices;

obtain a data identifier (ID) for the substantially identical data;

generate a requester storage record to include the data ID and the requester IDs from each of the plurality of data storage requests;

dispersed storage error encode the requester storage record to produce a set of encoded requester storage record slices;

send, via the interface, the set of encoded requester storage record slices to a dispersed storage network (DSN) memory for storage therein;

for the plurality of data storage requests, dispersed storage error encode at least a portion of the substantially identical data from one of the plurality of data storage requests to produce a set of encoded data slices; and

send, via interface, the set of encoded data slices to the DSN memory for storage therein in accordance with the data ID.

12. The computer of claim 11, wherein the one or more processing devices further functions to obtain the data ID by least one of:

generating a calculated hash of the substantially identical data;

retrieving the data ID from profile table based on the calculated hash; and

receiving, via the interface, the data ID from the requester.

13. The computer of claim 11, wherein the one or more processing devices further functions to:

determine, from data storage request to data storage request of the plurality of data storage requests, whether the substantially identical data is substantially the same by at least one of;

determining whether a first data ID associated with the substantially identical data of a first data storage request of the plurality of data storage requests substantially matches a second data ID associated with the substantially identical data of a second data storage request of the plurality of data storage requests;

determining whether a first hash of the substantially identical data of the first data storage request substantially matches a second hash of the substantially identical data of the second data storage request; and

comparing the substantially identical data of the first data storage request with the substantially identical data of the second data storage request.

14. The computer of claim 11, wherein the one or more processing devices further functions to combine the data ID and the requester ID by at least one of:

selecting at least one of the data ID and the requester ID as the requester storage record;

encrypting at least one of the data ID and the requester ID using a random encryption key to produce the requester storage record;

encrypting the random encryption key using a public key to produce an encrypted random encryption key;

dispersed storage error encoding the encrypted random encryption key to produce a set of encoded key slices; and

sending, via the interface, the set of encoded key slices to the DSN memory for storage therein.

15. The computer of claim 11, wherein the one or more processing devices further functions to dispersed storage error encode at least the portion of the substantially identical data by at least one of:

encoding at least one of the portion of the substantially identical data and the data ID using an error coding dispersal storage function to produce the set of encoded data slices;

encrypting at least one of the portion of the substantially identical data and the data ID using a random encryption key to produce encrypted data and encoding the encrypted data using the error coding dispersal storage function to produce the set of encoded data slices; and

encrypting the random encryption key using a public key to produce an encrypted random encryption key and dispersed storage error encoding the encrypted random encryption key to produce a set of encoded key slices.

16. The computer of claim 11, wherein the one or more processing devices further functions to:

select the one of the plurality of data storage requests based on at least one of;

an ordering of receipt of the plurality of data storage requests;

the requester ID identifying a particular de-duplication authorization level;

a type of the data;

a size of the data; and

number of the plurality of data storage requests.

17. A computer comprises:

an interface;

memory; and

one or more processing devices operably coupled to the interface and the memory, wherein the one or more processing devices is operable to;

receive, via the interface, a plurality of data retrieval requests from a plurality of requesting devices to retrieve substantially identical data, wherein a data retrieval request of the plurality of data retrieval requests includes a requester storage record identifier (ID);

for at least one of the plurality of data retrieval requests;

retrieve, via the interface, a set of encoded requester storage record slices from a dispersed storage network (DSN) memory based on the requester storage record ID;

dispersed storage error decode the set of encoded requester storage record slices to reproduce a requester storage record; and

extract a data identifier (ID) and a requester identifier (ID) from the requester storage record, wherein the data ID identifies the substantially identical data;

retrieve, via the interface, a set of encoded data slices from the DSN memory based on the data ID of one of the plurality of data retrieval requests;

dispersed storage error decode the set of encoded data slices to reproduce at least a portion of the de-duplicated stored data; and

send, via the interface, the at least a portion of the de-duplicated stored data to each of the plurality of requesting devices in accordance with the plurality of extracted requester IDs.

18. The computer of claim 17, wherein the one or more processing devices further functions to:

the data retrieval request further includes a requester device provided requester ID; and

verify the requester ID extracted from the requester storage record with the requester device provided requester ID.

19. The computer of claim 17, wherein the one or more processing devices functions to dispersed storage error decode the set of encoded requester storage record slices by:

retrieving, via the interface, a set of encoded key slices from the DSN memory;

dispersed storage error decoding the set of encoded key slices to produce an encrypted random encryption key;

decrypting the encrypted random encryption key using a private key to produce a random encryption key;

dispersed storage error decoding the set of encoded requester storage record to produce an encrypted requester storage record; and

decrypting the encrypted requester storage record using the random encryption key to produce the requester storage record.

20. The computer of claim 17, wherein the one or more processing devices functions to dispersed storage error decode the set of encoded data slices by at least one of:

retrieving, via the interface, a set of encoded key slices from the DSN memory;

dispersed storage error decoding the set of encoded key slices to produce an encrypted random encryption key;

decrypting the encrypted random encryption key using a private key to produce a random encryption key;

dispersed storage error decoding the plurality of encoded data slices to produce a plurality of encrypted data segments; and

decrypting each of the plurality of encrypted data segments using the random encryption key to produce the at least a portion of the de-duplicated stored data.