METHOD AND APPARATUS FOR DISPERSED STORAGE OF STREAMING DATA

• US 20110083049A1
• Filed: 06/09/2010
• Published: 04/07/2011
• Est. Priority Date: 10/05/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: GRUBE, GARY W., MARKISON, TIMOTHY W.
• Granted Patent: US 8,438,456 B2
• Time in Patent Office: Days
• US Class Current: 714/702
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 21/6272 by registering files or documents with a third party

Abstract

A method begins by a processing module receiving streaming data and dispersed storage resource configuration information. The method continues with the processing module allocating a plurality of sets of dispersed storage resources, obtaining error coding dispersed storage function parameters, and partitioning the streaming data into a plurality of data streams in accordance with the dispersed storage resource configuration information when the dispersed storage resource configuration information requires a plurality of sets of dispersed storage resources. In addition, the method continues with the processing module converting, via the plurality of sets of dispersed storage resources, the plurality of data streams into pluralities of sets of error coded data slices in accordance with the error coding dispersed storage function parameters.

Claims

1. A dispersed storage processing method comprises:

receiving streaming data;

receiving dispersed storage resource configuration information;

when the dispersed storage resource configuration information requires a plurality of sets of dispersed storage resources;

allocating the plurality of sets of dispersed storage resources;

obtaining error coding dispersed storage function parameters;

partitioning the streaming data into a plurality of data streams in accordance with the dispersed storage resource configuration information; and

converting, via the plurality of sets of dispersed storage resources, the plurality of data streams into pluralities of sets of error coded data slices in accordance with the error coding dispersed storage function parameters.

2. The method of claim 1 further comprises:

interpreting the dispersed storage resource configuration information to determine capacity requirements and capability requirements;

determining whether a set of the plurality of sets of dispersed storage resources has capacity and capabilities corresponding to the capacity requirements and the capability requirements;

when the set does have the capacity and capabilities corresponding to the capacity requirements and the capability requirements, enabling the set to generate the pluralities of sets of error coded data slices; and

when the set does not have the capacity and capabilities corresponding to the capacity requirements and the capability requirements, enabling the plurality of sets of dispersed storage resources to generate the pluralities of sets of error coded data slices.

3. The method of claim 1 further comprises:

when the dispersed storage resource configuration information indicates that the plurality of sets of dispersed storage resources is required, enabling the plurality of sets of dispersed storage resources to generate the pluralities of sets of error coded data slices.

4. The method of claim 1, wherein each of the plurality of sets of dispersed storage resources further comprises:

an encoder; and

a slicer.

5. The method of claim 4, wherein each of the plurality of sets of dispersed storage resources further comprises at least one of:

a pre-slice data manipulator; and

a post-slice data manipulator.

6. The method of claim 1, wherein the streaming data comprises one or more of high definition video, standard definition video, multi-media, music, broadcast, telephony, internet data, intranet data, communications assistance for law enforcement act (CALEA) information, and communication system link information, wherein the properties of the streaming data include one or more of file type, timestamp, data stream rate, resolution, aspect ratio, protocol indication, source location, user entered data, user ID, and pattern matching information.

7. The method of claim 1 further comprises:

receiving first and second error coding dispersed storage function parameters;

partitioning the streaming data into first and second data streams;

converting, via a first set of the plurality of sets of dispersed storage resources, the first data stream into first sets of error coded data slices in accordance with the first error coding dispersed storage function parameters; and

converting, via a second set of the plurality of sets of dispersed storage resources, the second data stream into second sets of error coded data slices in accordance with the second error coding dispersed storage function parameters.

8. The method of claim 1 further comprises:

receiving first and second error coding dispersed storage function parameters;

partitioning the streaming data into first and second data streams;

converting, via a first set of the plurality of sets of dispersed storage resources, the first data stream into first sets of error coded data slices in accordance with the first error coding dispersed storage function parameters;

converting, via a second set of the plurality of sets of dispersed storage resources, the first data stream into second sets of error coded data slices in accordance with the second error coding dispersed storage function parameters;

converting, via a third set of the plurality of sets of dispersed storage resources, the second data stream into third sets of error coded data slices in accordance with the first error coding dispersed storage function parameters; and

converting, via a fourth set of the plurality of sets of dispersed storage resources, the second data stream into fourth sets of error coded data slices in accordance with the second error coding dispersed storage function parameters.

9. The method of claim 1 further comprises:

replicating a data stream to produce the streaming data;

receiving first and second error coding dispersed storage function parameters;

partitioning the streaming data into a first copy of the data stream and a second copy of the data stream;

converting, via a first set of the plurality of sets of dispersed storage resources, the first copy of the data stream into first sets of error coded data slices in accordance with the first error coding dispersed storage function parameters; and

converting, via a second set of the plurality of sets of dispersed storage resources, the second copy of the data stream into second sets of error coded data slices in accordance with the second error coding dispersed storage function parameters.

10. A dispersed storage processing module comprises:

an interface;

sets of dispersed storage resources; and

a processing module operable to;

receive, via the interface, streaming data;

receive dispersed storage resource configuration information;

when the dispersed storage resource configuration information requires a plurality of the sets of dispersed storage resources;

allocate the plurality of the sets of dispersed storage resources;

obtain error coding dispersed storage function parameters;

enable partitioning of the streaming data into a plurality of data streams in accordance with the dispersed storage resource configuration information; and

enable the plurality of sets of dispersed storage resources to convert the plurality of data streams into pluralities of sets of error coded data slices in accordance with the error coding dispersed storage function parameters.

11. The dispersed storage processing module of claim 10, wherein the processing module further functions to:

interpret the dispersed storage resource configuration information to determine capacity requirements and capability requirements;

determine whether a set of the plurality of sets of dispersed storage resources has capacity and capabilities corresponding to the capacity requirements and the capability requirements;

when the set does have the capacity and capabilities corresponding to the capacity requirements and the capability requirements, enable the set to generate the pluralities of sets of error coded data slices; and

when the set does not have the capacity and capabilities corresponding to the capacity requirements and the capability requirements, enable the plurality of sets of dispersed storage resources to generate the pluralities of sets of error coded data slices.

12. The dispersed storage processing module of claim 10, wherein the processing module further functions to:

when the dispersed storage resource configuration information indicates that the plurality of sets of dispersed storage resources is required, enable the plurality of sets of dispersed storage resources to generate the pluralities of sets of error coded data slices.

13. The dispersed storage processing module of claim 10, wherein each of the plurality of sets of dispersed storage resources further comprises:

an encoder; and

a slicer.

14. The dispersed storage processing module of claim 13, wherein each of the plurality of sets of dispersed storage resources further comprises at least one of:

a pre-slice data manipulator; and

a post-slice data manipulator.

15. The dispersed storage processing module of claim 10, wherein the streaming data comprises one or more of high definition video, standard definition video, multi-media, music, broadcast, telephony, internet data, intranet data, communications assistance for law enforcement act (CALEA) information, and communication system link information, wherein the properties of the streaming data include one or more of file type, timestamp, data stream rate, resolution, aspect ratio, protocol indication, source location, user entered data, user ID, and pattern matching information.

16. The dispersed storage processing module of claim 10, wherein the processing module further functions to:

receive, via the interface, first and second error coding dispersed storage function parameters;

enable partitioning of the streaming data into first and second data streams;

enable a first set of the plurality of sets of dispersed storage resources to convert the first data stream into first sets of error coded data slices in accordance with the first error coding dispersed storage function parameters; and

enable a second set of the plurality of sets of dispersed storage resources to convert the second data stream into second sets of error coded data slices in accordance with the second error coding dispersed storage function parameters.

17. The dispersed storage processing module of claim 10, wherein the processing module further functions to:

receive, via the interface, first and second error coding dispersed storage function parameters;

enable partitioning of the streaming data into first and second data streams;

enable a first set of the plurality of sets of dispersed storage resources to convert the first data stream into first sets of error coded data slices in accordance with the first error coding dispersed storage function parameters;

enable a second set of the plurality of sets of dispersed storage resources to convert the first data stream into second sets of error coded data slices in accordance with the second error coding dispersed storage function parameters;

enable a third set of the plurality of sets of dispersed storage resources to convert the second data stream into third sets of error coded data slices in accordance with the first error coding dispersed storage function parameters; and

enable a fourth set of the plurality of sets of dispersed storage resources to convert the second data stream into fourth sets of error coded data slices in accordance with the second error coding dispersed storage function parameters.

18. The dispersed storage processing module of claim 10, wherein the processing module further functions to:

enable replicating of a data stream to produce the streaming data;

receive, via the interface, first and second error coding dispersed storage function parameters;

enable partitioning of the streaming data into a first copy of the data stream and a second copy of the data stream;

enable a first set of the plurality of sets of dispersed storage resources to convert the first copy of the data stream into first sets of error coded data slices in accordance with the first error coding dispersed storage function parameters; and

enable a second set of the plurality of sets of dispersed storage resources to convert the second copy of the data stream into second sets of error coded data slices in accordance with the second error coding dispersed storage function parameters.