Data modification in a dispersed storage network

• US 10,095,578 B2
• Filed: 08/06/2014
• Issued: 10/09/2018
• Est. Priority Date: 06/22/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K.
• Primary Examiner(s): HARPER, ELIYAH STONE
• Application Number:
  US14/452,774
• Publication Number:
  US 20140351624A1
• Time in Patent Office: 1,525 Days
• Field of Search: 707736, 707790
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/141 for bus or memory accesses
• CPC Class Codes: G06F 11/167 Error detection by comparing the memory output
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 2209/043 of tables, e.g. lookup, substitution or mapping
• CPC Class Codes: H04L 2209/30 Compression, e.g. Merkle-Damgard construction
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 2209/56 Financial cryptography, e.g. electronic payment or e-cash
• CPC Class Codes: H04L 2209/80 Wireless network architectures or network communication protocols for wireless network security H04W12/00
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/06 for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/3242 involving keyed hash functions, e.g. message authentication codes [MACs], CBC-MAC or HMAC
• CPC Class Codes: H04L 9/3247 : involving digital signatures
• CPC Class Codes: H04L 9/3263 : involving certificates, e.g. public key certificate [PKC] or attribute certificate [AC]; Public key infrastructure [PKI] arrangements network architectures or network communication protocols for supporting authentication of entities using certificates in a packet data network H04L63/0823
• CPC Class Codes: H04L 9/3271 : using challenge-response
• CPC Class Codes: H04W 12/041 : Key generation or derivation
• CPC Class Codes: H04W 12/0431 : Key distribution or pre-distribution; Key agreement
• CPC Class Codes: H04W 12/10 : Integrity
• CPC Class Codes: H04W 12/35 : Protecting application or service provisioning, e.g. securing SIM application provisioning

Abstract

A method for modifying data stored in a dispersed storage network (DSN). In various embodiments, a data object is received for storage in DSN memory. A dispersed storage processing unit determines a number of data segments for the data object and divides the data object into a plurality of data blocks. The data blocks are allocated to the data segments in a column-row orientation (for example, columns may be populated with successive data blocks of the data object). The data segments are encoded to produce a plurality of sets of encoded data slices. Additional data received for the data object is divided into additional data blocks that are allocated to data segments to create one or more new columns, which are encoded to produce a plurality of encoded data slice addendums. The encoded data slice addendums are then appended to existing encoded data slices corresponding to the data object.

Claims

1. A method comprises:

receiving, by a first and a second dispersed storage processing unit, a media broadcast for storage in dispersed storage network (DSN) memory;

dispersed storage error encoding, by the first dispersed storage processing unit, in accordance with dispersed storage error encoding parameters, the media broadcast to produce a first plurality of sets of encoded data slices;

dispersed storage error encoding, by the second dispersed storage processing unit, in accordance with the dispersed storage error encoding parameters, the media broadcast to produce a second plurality of sets of encoded data slices;

outputting, by the first dispersed storage processing unit, the first plurality of sets of encoded data slices to the DSN memory for storage therein;

outputting, by the second dispersed storage processing unit, the second plurality of sets of encoded data slices to the DSN memory for storage therein; and

retrieving, by a computing device of the DSN, a decode threshold number of non-redundant encoded data slices for a data segment of the media broadcast, wherein encoded data slices of the decode threshold number of non-redundant encoded data slices are from a first corresponding set of encoded data slices of the first plurality of sets of encoded data slices corresponding to a first at least one pillar of a plurality of pillars specified by the dispersed storage error encoding parameters and a second corresponding set of encoded data slices of the second plurality of sets of encoded data slices corresponding to a second at least one pillar that is non-overlapping with the first at least one pillar of the plurality of pillars specified by the dispersed storage error encoding parameters,

wherein the decode threshold number of non-redundant encoded data slices is a minimum number of encoded data slices required to recover the data segment, and wherein an individual encoded data slice of the decode threshold number of non-redundant encoded data slices yields substantially no information regarding the data segment.

2. The method of claim 1 further comprises:

generating a first plurality of sets of slice names for the first plurality of sets of encoded data slices;

generating a second plurality of sets of slice names for the second plurality of sets of encoded data slices;

storing the first and second pluralities of sets of encoded data slices in the DSN memory based on the first and second pluralities of sets of slice names; and

sending a set of read requests to retrieve the decode threshold number of non-redundant encoded data slices, wherein a read request includes a slice name from the first or the second pluralities of sets of slice names.

3. The method of claim 1 further comprises:

the media broadcast including one or more of audio streaming, video streaming, multimedia streaming, music streaming, video files, audio files, and multimedia files.

4. The method of claim 1, wherein the dispersed storage error encoding parameters comprises:

the decode threshold number (k); and

a pillar width number (n) that corresponds to a total number of encoded data slices in a set of encoded data slices in the first or second plurality of sets of encoded data slices.

5. The method of claim 4, wherein the retrieving the decode threshold number of non-redundant encoded data slices for the data segment comprises:

retrieving k-x encoded data slices from the first corresponding set of encoded data slices; and

retrieving x encoded data slices from the second corresponding set of encoded data slices.

6. The method of claim 1 further comprises:

outputting, by the first dispersed storage processing unit, the first plurality of sets of encoded data slices to a set of storage units of the DSN memory for storage therein; and

outputting, by the second dispersed storage processing unit, the second plurality of sets of encoded data slices to the set of storage units for storage therein.

7. A non-transitory computer readable storage device comprises:

a first storage section that stores operational instructions that, when executed by a first computing device, causes the first computing device to;

receive a media broadcast for storage in dispersed storage network (DSN) memory;

dispersed storage error encode, in accordance with dispersed storage error encoding parameters, the media broadcast to produce a first plurality of sets of encoded data slices;

output the first plurality of sets of encoded data slices to the DSN memory for storage therein;

a second storage section that stores operational instructions that, when executed by a second computing device, causes the second computing device to;

receive the media broadcast;

dispersed storage error encode, in accordance with the dispersed storage error encoding parameters, the media broadcast to produce a second plurality of sets of encoded data slices;

output the second plurality of sets of encoded data slices to the DSN memory for storage therein;

a third storage section that stores operational instructions that, when executed by the first, the second, or a third computing device, causes the first, the second, or the third computing device to;

retrieve a decode threshold number of non-redundant encoded data slices for a data segment of the media broadcast, wherein encoded data slices of the decode threshold number of non-redundant encoded data slices are from a first corresponding set of encoded data slices of the first plurality of sets of encoded data slices corresponding to a first at least one pillar of a plurality of pillars specified by the dispersed storage error encoding parameters and a second corresponding set of encoded data slices of the second plurality of sets of encoded data slices corresponding to a second at least one pillar that is non-overlapping with the first at least one pillar of the plurality of pillars specified by the dispersed storage error encoding parameters, wherein the decode threshold number of non-redundant encoded data slices is a minimum number of encoded data slices required to recover the data segment, and wherein an individual encoded data slice of the decode threshold number of non-redundant encoded data slices yields substantially no information regarding the data segment.

8. The computer readable storage device of claim 7 further comprises:

the first storage section further stores operational instructions that, when executed by the first computing device, causes the first computing device to;

generate a first plurality of sets of slice names for the first plurality of sets of encoded data slices;

the second storage section further stores operational instructions that, when executed by the second computing device, causes the second computing device to;

generate a second plurality of sets of slice names for the second plurality of sets of encoded data slices;

a fourth storage section that stores operational instructions that, when executed by storage units of the DSN memory, causes the storage units to;

store the first and second pluralities of sets of encoded data slices in the storage units based on the first and second pluralities of sets of slice names; and

the third storage section further stores operational instructions that, when executed by the first, the second, or the third computing device, causes the first, the second, or the third computing device to;

send a set of read requests to retrieve the decode threshold number of non-redundant encoded data slices, wherein a read request includes a slice name from the first or the second pluralities of sets of slice names.

9. The computer readable storage device of claim 7, wherein the dispersed storage error encoding parameters comprises:

the decode threshold number (k); and

a pillar width number (n) that corresponds to a total number of encoded data slices in a set of encoded data slices in the first or second plurality of sets of encoded data slices.

10. The computer readable storage device of claim 9, wherein the retrieving the decode threshold number of non-redundant encoded data slices for the data segment comprises:

retrieving k-x encoded data slices from the first corresponding set of encoded data slices; and

retrieving x encoded data slices from the second corresponding set of encoded data slices.