Streaming content storage

• US 9,043,548 B2
• Filed: 08/01/2014
• Issued: 05/26/2015
• Est. Priority Date: 01/28/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Grube, Gary W., Markison, Timothy W., Resch, Jason K.
• Primary Examiner(s): Faal, Baboucarr
• Application Number:
  US14/449,549
• Publication Number:
  US 20140344227A1
• Time in Patent Office: 298 Days
• US Class Current: 711/114
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A computing system includes a plurality of dispersed storage (DS) processing units operable to receive a continuous data stream, simultaneously disperse storage error encode the continuous data stream to produce a plurality of encoded data slices and store the plurality of encoded data slices in a DS memory.

Claims

1. A computing system, comprises:

a dispersed storage (DS) memory; and

a plurality of DS processing units operable to receive a continuous data stream, each of the plurality of DS processing units being further operable to;

process a respective portion of the continuous data stream, the respective portion of the continuous data stream being segmented into a respective set of a plurality of data segments;

simultaneously disperse storage error encode the respective set of the plurality of data segments to produce a respective set of a plurality of encoded data slices, a number of encoded data slices within each of the data segments corresponding to a number of pillars per data segment for the data stream, the respective set of the plurality of encoded data slices corresponding to a respective set of pillars, the respective set of the plurality of encoded data slices for a data segment of the plurality of data segments being produced by;

producing a data matrix corresponding to the data segment;

determining a respective portion of an encoding matrix associated with the respective set of pillars; and

matrix multiplying the respective portion of the encoding matrix with the data matrix to produce the respective set of the plurality of encoded data slices for the data segment; and

store the respective set of the plurality of encoded data slices corresponding to the respective set of pillars in the DS memory.

2. The computing system of claim 1, wherein the respective portion of the continuous data stream encoded by at least two of the plurality of processing units at least partially overlaps.

3. The computing system of claim 2, further comprising:

a storage integrity processing unit operable to;

retrieve the respective sets of the plurality of encoded data slices from the DS memory;

identify duplicate encoded data slices of the plurality of encoded data slices; and

delete at least some of the duplicate encoded data slices from the DS memory.

4. The computing system of claim 3, wherein the storage integrity processing unit is further operable to:

compare the duplicate encoded data slices to identify incorrect data slices; and

delete the incorrect data slices from the DS memory.

5. The computing system of claim 3, wherein the storage integrity processing unit is further operable to:

re-create respective copies of the data stream from the respective sets of the plurality of encoded data slices;

compare the respective copies of the data stream to identify incorrect data slices; and

delete the incorrect data slices from the DS memory.

6. The computing system of claim 1, further comprising:

a data stream bus coupled to the plurality of DS processing units; and

a master DS processing unit coupled to the data stream bus, the master DS processing unit operable to;

receive the continuous data stream;

segment the continuous data stream into the plurality of data segments; and

provide the respective set of the data segments to each of the plurality of DS processing units via the data stream bus.

7. A method, comprises:

receiving a continuous data stream at a plurality of dispersed storage (DS) processing units;

processing, by each of the plurality of DS processing units, a respective portion of the continuous data stream, the respective portion of the continuous data stream being segmented into a respective set of a plurality of data segments;

simultaneously disperse storage error encoding, by each of the plurality of DS processing units, the respective set of the plurality of data segments to produce a respective set of a plurality of encoded data slices, a number of encoded data slices within each of the data segments corresponding to a number of pillars per data segment for the data stream, the respective set of the plurality of encoded data slices corresponding to a respective set of pillars, the respective set of the plurality of encoded data slices for a data segment of the plurality of data segments being produced by;

producing a data matrix corresponding to the data segment;

determining a respective portion of an encoding matrix associated with the respective set of pillars; and

matrix multiplying the respective portion of the encoding matrix with the data matrix to produce the respective set of the plurality of encoded data slices for the data segment; and

storing, by each of the plurality of DS processing units, the respective set of the plurality of encoded data slices corresponding to the respective set of pillars in a DS memory.

8. The method of claim 7, wherein the respective portion of the continuous data stream encoded by at least two of the plurality of DS processing units at least partially overlaps and further comprising:

retrieving, by a storage integrity processing unit, the respective sets of the plurality of encoded data slices from the DS memory;

identifying, by the storage integrity processing unit, duplicate encoded data slices of the plurality of encoded data slices; and

deleting, by the storage integrity processing unit, at least some of the duplicate encoded data slices from the DS memory.

9. The method of claim 8, further comprises:

comparing, by the storage integrity processing unity, the duplicate encoded data slices to identify incorrect data slices; and

deleting, by the storage integrity processing unit, the incorrect data slices from the DS memory.

10. The method of claim 8, further comprises:

re-creating, by the storage integrity processing unit, respective copies of the data stream from the respective sets of the plurality of encoded data slices;

comparing, by the storage integrity processing unit, the respective copies of the data stream to identify incorrect data slices; and

deleting, by the storage integrity processing unit, the incorrect data slices from the DS memory.

11. The method of claim 7, further comprises:

receiving, by a master DS processing unit, the continuous data stream;

segmenting, by the master DS processing unit, the continuous data stream into the plurality of data segments; and

providing, by the master DS processing unit, the respective set of the data segments to each of the plurality of DS processing units.

12. A non-transitory computer readable storage medium having accessible therefrom a set of instructions interpretable by a processing module, the set of instructions being configured to cause the processing module to carry out operations for:

receiving a continuous data stream;

determining a set of operational parameters for storing the continuous data stream; and

providing the continuous data stream to a plurality of DS processing units for simultaneous disperse storage error encoding, by the plurality of DS processing units, the continuous data stream, in accordance with the set of operational parameters, to produce a plurality of encoded data slices for storage in a DS memory, the providing further including;

determining an encoding matrix for encoding the continuous data stream; and

providing a respective portion of the encoding matrix to each of the plurality of DS processing units for matrix multiplying, by each of the plurality of DS processing units, the respective portion of the encoding matrix with a data matrix produced from the continuous data stream to produce respective sets of encoded data slices.

13. The storage medium of claim 12, wherein the set of instructions further causes the processing module to carry out operations for:

segmenting the continuous data stream into a plurality of data segments; and

providing a respective set of the data segments to each of the plurality of DS processing units.