Accessing large amounts of data in a dispersed storage network

• US 9,274,864 B2
• Filed: 08/17/2012
• Issued: 03/01/2016
• Est. Priority Date: 10/04/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew, Volvovski, Ilya, Dhuse, Greg, Leggette, Wesley, Resch, Jason K.
• Primary Examiner(s): Decady, Albert
• Assistant Examiner(s): Vallecillo, Kyle
• Application Number:
  US13/588,391
• Publication Number:
  US 20130086448A1
• Time in Patent Office: 1,292 Days
• Field of Search: 714/763, 714/E11.034, 709/238, 709/226
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 21/33 using certificates
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 2209/04 Masking or blinding
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 63/0823 using certificates cryptographic mechanisms or cryptographic arrangements for entity authentication involving certificates H04L9/3263
• CPC Class Codes: H04L 9/085 Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/0863 involving passwords or one-time passwords network architectures or network communication protocols for using one-time keys in a packet data network H04L63/067
• CPC Class Codes: H04L 9/0869 involving random numbers or seeds
• CPC Class Codes: H04L 9/321 involving a third party or a trusted authority
• CPC Class Codes: H04L 9/3263 involving certificates, e.g. public key certificate [PKC] or attribute certificate [AC]; Public key infrastructure [PKI] arrangements network architectures or network communication protocols for supporting authentication of entities using certificates in a packet data network H04L63/0823

Abstract

A method begins by a dispersed storage (DS) processing module obtaining a plurality of data objects for storage in a dispersed storage network (DSN) and determining one or more common data object aspects of a data object of the plurality of data objects. The method continues with the DS processing module disperse storage error encoding at least a portion of the data object to produce a set of encoded data slices and generating a set of DSN addresses for the set of encoded data slices, wherein each of the set of DSN addresses includes a field referencing the one or more common data object aspects. The method continues with the DS processing module outputting the set of encoded data slices for storage in the DSN based on the set of DSN addresses.

Claims

1. A method for storing large amounts of data, the method comprises:

obtaining, by a processing module of a computing device, a plurality of data objects for storage in a dispersed storage network (DSN);

determining, by the processing module, that two data objects of the plurality of data objects have one or more common data object aspects wherein each of the two data objects includes a plurality of data segments;

disperse storage error encoding, by the processing module, the plurality of data segments of a first data object of the two data objects to produce a first plurality of sets of encoded data slices, wherein a data segment of the plurality of data segments is dispersed storage error encoded into a set of encoded data slices of the plurality of sets of encoded data slices and wherein a decode threshold number of encoded data slices of the set of encoded data slices in needed to recover the data segment;

generating, by the processing module, a first plurality of sets of DSN addresses for the first plurality of sets of encoded data slices, wherein DSN addresses of the first plurality of sets of DSN addresses includes a field referencing the one or more common data object aspects;

disperse storage error encoding, by the processing module, the plurality of data segments of a second data object of the two data objects to produce a second plurality of sets of encoded data slices;

generating, by the processing module, a second plurality of sets of DSN addresses for the second plurality of sets of encoded data slices, wherein DSN addresses of the second plurality of sets of DSN addresses includes the field referencing the one or more common data object aspects; and

outputting the first and second plurality of sets of encoded data slices for storage in the DSN based on the first and second plurality of sets of DSN addresses.

2. The method of claim 1 further comprises:

creating a record for the first data object in a common data object aspect database, wherein the record includes a data object identifier, information regarding the one or more common data object aspects, and a portions number indicating a number of portions constituting the first data object, wherein the common data object aspect database includes a plurality of records for at least some of the plurality of data objects.

3. The method of claim 1 further comprises:

the one or more common data object aspects is one of a plurality of common data object aspects that includes two or more of;

a time window aspect, a destination identifier (ID) of the data object, a source ID of the data object, one or more internet protocol (IP) addresses associated with the data object, a geographic location regarding the data object, data content information of the data object, data size of the data object, a data object type, a capturing unit ID that obtained the data object, and a vault ID.

4. The method of claim 1, wherein generating a DSN address of the first plurality of sets of DSN addresses comprises:

generating a slice index field;

generating a data object aspect field;

generating a data object ID; and

generating a segment field regarding the at least a portion of the first data object.

5. The method of claim 1 further comprises:

decoding a decode threshold number of the first and second plurality of sets of encoded data slices for each data segment of the first and second data objects to produce a reconstructed data object;

analyzing the reconstructed data object to identify additional common data object aspects of the reconstructed data object with a third data object of the plurality of data objects;

disperse storage error encoding at least a portion of the reconstructed data object to produce a new set of encoded data slices;

generating a new set of DSN addresses for the new set of encoded data slices, wherein each of the new set of DSN addresses includes a field referencing the one or more common data object aspects and the additional common data object aspects; and

outputting the new set of encoded data slices for storage in the DSN based on the new set of DSN addresses.

6. The method of claim 1 further comprises:

dividing the first data object into a plurality of portions;

disperse storage error encoding the plurality of portions to produce another plurality of sets of encoded data slices;

generating another plurality of sets of DSN addresses for the other plurality of sets of encoded data slices, wherein DSN addresses of the other plurality of sets of DSN addresses includes a field referencing the one or more common data object aspects; and

outputting the other plurality of sets of encoded data slices for storage in the DSN based on the other plurality of sets of DSN addresses.

7. A method for retrieving data objects having a common aspect, the method comprises:

selecting, by a processing module of a computing device, one or more common data object aspects from a plurality of common data object aspects to produce selected common data object aspects;

accessing, by the processing module, a common data object aspect database based on the selected common data object aspects to identify a set of records, wherein a record of the common data object aspect database includes a data object identifier of a data object, information regarding one or more common data object aspects of the data object, and a portions number indicating a number of portions constituting the data object;

generating, by the processing module, a first plurality of sets of dispersed storage network (DSN) addresses based on a first data object identifier, the information regarding the one or more common data object aspects, and the portions number of a first record of the set of records;

retrieving, by the processing module, at least a decode threshold number of encoded data slices of a first plurality of sets of encoded data slices from the DSN based on the first plurality of sets of DSN addresses;

decoding, by the processing module, the at least a decode threshold number of encoded data slices of the first plurality of sets of encoded data slices in accordance with a dispersed storage error encoding function to reproduce a plurality of portions of a first data object;

generating, by the processing module, a second plurality of sets of dispersed storage network (DSN) addresses based on a second data object identifier, the information regarding the one or more common data object aspects, and the portions number of a second record of the set of records;

retrieving, by the processing module, at least a decode threshold number of encoded data slices of a second plurality of sets of encoded data slices from the DSN based on the second plurality of sets of DSN addresses; and

decoding, by the processing module, the at least a decode threshold number of encoded data slices of the second plurality of sets of encoded data slices in accordance with a dispersed storage error encoding function to reproduce a plurality of portions of a second data object.

8. The method of claim 7, wherein the plurality of common data object aspects comprises two or more of:

a time window aspect, a destination identifier (ID) of the data object, a source ID of the data object, one or more internet protocol (IP) addresses associated with the data object, a geographic location regarding the data object, data content information of the data object, data size of the data object, a data object type, a capturing unit ID that obtained the data object, and a vault ID.

9. The method of claim 7, wherein generating a DSN address of the first and second plurality of sets of DSN addresses comprises:

generating a slice index field based on the dispersed storage error encoding function;

generating a data object aspect field based on the selected common data object aspects;

generating a data object ID based on the data object identifier; and

generating a segment field based on the portions number.

10. A capturing unit comprises:

a first module, when operable within a computing device, causes the computing device to obtain a plurality of data objects for storage in a dispersed storage network (DSN);

a second module, when operable within the computing device, causes the computing device to determine that two data objects of the plurality of data objects have one or more common data object aspects wherein each of the two data objects includes a plurality of data segments;

a third module, when operable within the computing device, causes the computing device to;

disperse storage error encode the plurality of data segments of a first data object of the two data objects to produce a first plurality of sets of encoded data slices, wherein a data segment of the plurality of data segments is dispersed storage error encoded into a set of encoded data slices of the first plurality of sets of encoded data slices and wherein a decode threshold number of encoded data slices of the set of encoded data slices in needed to recover the data segment; and

disperse storage error encode the plurality of data segments of a second data object of the two data objects to produce a second plurality of sets of encoded data slices;

a fourth module, when operable within the computing device, causes the computing device to;

generate a first plurality of sets of DSN addresses for the first plurality of sets of encoded data slices, wherein DSN addresses of the first plurality of sets of DSN addresses includes a field referencing the one or more common data object aspects; and

generate a second plurality of sets of DSN addresses for the second plurality of sets of encoded data slices, wherein DSN addresses of the second plurality of sets of DSN addresses includes the field referencing the one or more common data object aspects; and

a fifth module, when operable within the computing device, causes the computing device to output the first and second plurality of sets of encoded data slices for storage in the DSN based on the first and second plurality of sets of DSN addresses.

11. The capturing unit of claim 10 further comprises:

a sixth module, when operable within the computing device, causes the computing device to create a record for the first data object in a common data object aspect database, wherein the record includes a data object identifier, information regarding the one or more common data object aspects, and a portions number indicating a number of portions constituting the first data object, wherein the common data object aspect database includes a plurality of records for at least some of the plurality of data objects.

12. The capturing unit of claim 10 further comprises:

the one or more common data object aspects is one of a plurality of common data object aspects that includes two or more of;

a time window aspect, a destination identifier (ID) of the data object, a source ID of the data object, one or more internet protocol (IP) addresses associated with the data object, a geographic location regarding the data object, data content information of the data object, data size of the data object, a data object type, a capturing unit ID that obtained the data object, and a vault ID.

13. The capturing unit of claim 10, wherein the fourth module, when operable, generates a DSN address of the first plurality of sets of DSN addresses by:

generating a slice index field;

generating a data object aspect field;

generating a data object ID; and

generating a segment field regarding the at least a portion of the first data object.

14. The capturing unit of claim 10 further comprises:

a sixth module, when operable within the computing device, causes the computing device to decode a decode threshold number of the first and second plurality of sets of encoded data slices for each portion of the first and second data objects to produce a reconstructed data object;

the second module is further operable to analyze the reconstructed data object to identify additional common data object aspects of the reconstructed data object with a third data object of the plurality of data objects;

the third module is further operable to disperse storage error encode at least a portion of the reconstructed data object to produce a new set of encoded data slices;

the fourth module is further operable to generate a new set of DSN addresses for the new set of encoded data slices, wherein each of the new set of DSN addresses includes a field referencing the one or more common data object aspects and the additional common data object aspects; and

the fifth module is further operable to output the new set of encoded data slices for storage in the DSN based on the new set of DSN addresses.

15. The capturing unit of claim 10 further comprises:

the third module is further operable to;

divide the first data object into a plurality of portions; and

disperse storage error encode the plurality of portions to produce another plurality of sets of encoded data slices;

the fourth module is further operable to generate another plurality of sets of DSN addresses for the other plurality of sets of encoded data slices, wherein DSN addresses of the other plurality of sets of DSN addresses includes a field referencing the one or more common data object aspects; and

the fifth module is further operable to output the other plurality of sets of encoded data slices for storage in the DSN based on the other plurality of sets of DSN addresses.

16. A retrieving unit comprises:

a first module, when operable within a computing device, causes the computing device to select one or more common data object aspects from a plurality of common data object aspects to produce selected common data object aspects;

a second module, when operable within the computing device, causes the computing device to access a common data object aspect database based on the selected common data object aspects to identify a set of records, wherein a record of the common data object aspect database includes a data object identifier of a data object, information regarding one or more common data object aspects of the data object, and a portions number indicating a number of portions constituting the data object;

a third module, when operable within the computing device, causes the computing device to;

generate a first plurality of sets of dispersed storage network (DSN) addresses based on a first data object identifier, the information regarding the one or more common data object aspects, and the portions number of a first record of the set of records; and

generate a second plurality of sets of dispersed storage network (DSN) addresses based on a second data object identifier, the information regarding the one or more common data object aspects, and the portions number of a second record of the set of records;

a fourth module, when operable within the computing device, causes the computing device to;

retrieve at least a decode threshold number of encoded data slices of a first plurality of sets of encoded data slices from the DSN based on the first plurality of sets of DSN addresses; and

retrieve at least a decode threshold number of encoded data slices of a second plurality of sets of encoded data slices from the DSN based on the second plurality of sets of DSN addresses;

a fifth module, when operable within the computing device, causes the computing device to;

decode the at least a decode threshold number of encoded data slices of the first plurality of sets of encoded data slices in accordance with a dispersed storage error encoding function to reproduce a plurality of portions of a first data object; and

decode the at least a decode threshold number of encoded data slices of the second plurality of sets of encoded data slices in accordance with a dispersed storage error encoding function to reproduce a plurality of portions of a second data object.

17. The retrieving unit of claim 16, wherein the plurality of common data object aspects comprises two or more of:

a time window aspect, a destination identifier (ID) of the data object, a source ID of the data object, one or more internet protocol (IP) addresses associated with the data object, a geographic location regarding the data object, data content information of the data object, data size of the data object, a data object type, a capturing unit ID that obtained the data object, and a vault ID.

18. The retrieving unit of claim 16, wherein the third module is further operable to generate a DSN address of the first and second plurality of sets of DSN addresses by:

generating a slice index field based on the dispersed storage error encoding function;

generating a data object aspect field based on the selected common data object aspects;

generating a data object ID based on the data object identifier; and

generating a segment field based on the portions number.