DISPERSED STORAGE NETWORK DATA MANIPULATION

• US 20110026842A1
• Filed: 05/07/2010
• Published: 02/03/2011
• Est. Priority Date: 08/03/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: CILFONE, BART, LEGGETTE, WESLEY, RESCH, JASON K.
• Granted Patent: US 9,167,277 B2
• Time in Patent Office: Days
• US Class Current: 382/232
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04N 21/23116 involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/25875 involving end-user authentication restricting access to computer systems by authenticating users using a predetermined code G06F21/33; arrangements for secret or secure communication including means for verifying the identity or authority of a user of the system H04L9/32; networks authentication protocols H04L63/08; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/2747 Remote storage of video programs received via the downstream path, e.g. from the server

Abstract

A method begins by a processing module receiving data for dispersed storage, wherein the data has an associated user identification (ID), and obtaining a codec flag based on the associated user ID. The codec flag may indicate one or more codec types and a codec execution order. The codec types may include two or more of: a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and/or one or more versions of an encryption function. The method continues with the processing module manipulating the data to produce manipulated data utilizing the one or more codec types in the codec execution order, encoding the manipulated data using an error coding dispersal storage function to produce a plurality of encoded data slices, and sending the slices and codec flag to a plurality of DS storage units for storage.

Claims

1. A method for execution by a dispersed storage (DS) processing unit, the method comprises:

receiving data for dispersed storage, wherein the data has an associated user identification (ID);

obtaining a codec flag based on the associated user ID, wherein the codec flag indicates one or more codec types and a codec execution order, wherein the codec types include two or more of;

a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

manipulating the data using the one or more codec types in the codec execution order to produce manipulated data;

encoding the manipulated data using an error coding dispersal storage function to produce a plurality of encoded data slices; and

outputting the plurality of encoded data slices and the codec flag to a plurality of DS storage units for storage therein.

2. The method of claim 1, wherein the encoding further comprises:

appending the codec flag to the manipulated data to produce appended manipulated data; and

encoding the appended manipulated data using the error coding dispersal storage function to produce the plurality of encoded data slices.

3. The method of claim 1, wherein the outputting further comprises:

appending the codec flag to at least one of the plurality of encoded data slices to produce an appended plurality of encoded data slices; and

outputting the appended plurality of encoded data slices to the plurality of DS storage units for storage therein.

4. The method of claim 1 further comprises:

obtaining a second codec flag based on the associated user ID, wherein the second codec flag indicates a second one or more codec types and a second codec execution order, wherein the second codec types include two or more of;

a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

manipulating at least one of the plurality of encoded data slices using the second one or more codec types in the second codec execution order to produce a plurality of post data manipulated slices; and

outputting the plurality of post data manipulated slices, the codec flag, and the second codec flag to the plurality of DS storage units for storage therein.

5. The method of claim 1, wherein the codec flag further comprises at least one of:

a current state indicator; and

a codec stack that includes at least one codec indicator arranged in accordance with the codec execution order, wherein the codec indicator is associated with the one or more codec types.

6. The method of claim 1, wherein the obtaining further comprises:

identifying a vault identity (ID) based on the associated user ID;

identifying a registry descriptor based on the vault ID;

accessing the registry descriptor to retrieve vault information, wherein the vault information includes one or more of a codec stack, a data manipulation table, the codec types, the codec execution order; and

determining the codec flag based on the vault information.

7. A method for execution by a dispersed storage (DS) processing unit, the method comprises:

receiving a plurality of encoded data slices and a codec flag from a plurality of DS storage units, wherein the codec flag indicates one or more codec types and a codec execution order, wherein the codec types include two or more of;

a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

decoding the plurality of encoded data slices using an error coding dispersal storage function to produce manipulated data; and

de-manipulating the manipulated data using the one or more codec types in the codec execution order to produce recovered data.

8. The method of claim 7 further comprises:

receiving the plurality of encoded data slices from the plurality of DS storage units;

decoding the plurality of encoded data slices using the error coding dispersal storage function to produce appended manipulated data; and

separating the appended manipulated data to produce the manipulated data and the codec flag.

9. The method of claim 7 further comprises:

receiving an appended plurality of encoded data slices from the plurality of DS storage units; and

separating the appended plurality of encoded data slices to produce the plurality of encoded data slices and the codec flag.

10. The method of claim 7 further comprises:

receiving a plurality of post data manipulated slices, the codec flag, and a second codec flag from a plurality of DS storage units wherein the second codec flag indicates a second one or more codec types and a second codec execution order, wherein the second codec types include two or more of;

a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

de-manipulating at least one of the plurality of post data manipulated slices using the second one or more codec types in the second codec execution order to produce the plurality of encoded data slices.

11. A dispersed storage (DS) processing unit comprises:

a network interface; and

a processing module operable to;

obtain data for dispersed storage, wherein the data has an associated user identification (ID);

obtain a codec flag based on the associated user ID, wherein the codec flag indicates one or more codec types and a codec execution order, wherein the codec types include two or more of;

a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

manipulate the data using the one or more codec types in the codec execution order to produce manipulated data;

encode the manipulated data using an error coding dispersal storage function to produce a plurality of encoded data slices; and

output, via the network interface, the plurality of encoded data slices and the codec flag to a plurality of DS storage units for storage therein.

12. The DS processing unit of claim 11, wherein the processing module further functions to encode by:

appending the codec flag to the manipulated data to produce appended manipulated data; and

encoding the appended manipulated data using the error coding dispersal storage function to produce the plurality of encoded data slices.

13. The DS processing unit of claim 11, wherein the processing module further functions to output by:

appending the codec flag to at least one of the plurality of encoded data slices to produce an appended plurality of encoded data slices; and

outputting, via the network interface, the appended plurality of encoded data slices to the plurality of DS storage units for storage therein.

14. The DS processing unit of claim 11, wherein the processing module further functions to:

obtain a second codec flag based on the associated user ID, wherein the second codec flag indicates a second one or more codec types and a second codec execution order, wherein the second codec types include two or more of;

a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

manipulate at least one of the plurality of encoded data slices using the second one or more codec types in the second codec execution order to produce a plurality of post data manipulated slices; and

output, via the network interface, the plurality of post data manipulated slices, the codec flag, and the second codec flag to the plurality of DS storage units for storage therein.

15. The DS processing unit of claim 11, wherein the codec flag further comprises at least one of:

a current state indicator; and

a codec stack that includes at least one codec indicator arranged in accordance with the codec execution order, wherein the codec indicator is associated with the one or more codec types.

16. The DS processing unit of claim 11, wherein the processing module further functions to obtain the codec flag by:

identifying a vault identity (ID) based on the associated user ID;

identifying a registry descriptor based on the vault ID;

accessing the registry descriptor to retrieve vault information, wherein the vault information includes one or more of a codec stack, a data manipulation table, the codec types, the codec execution order; and

determining the codec flag based on the vault information.

17. A dispersed storage (DS) processing unit comprises:

a network interface; and

a processing module operable to;

receive, via the network interface, a plurality of encoded data slices and a codec flag from a plurality of DS storage units, wherein the codec flag indicates one or more codec types and a codec execution order, wherein the codec types include two or more of;

a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

decode the plurality of encoded data slices using an error coding dispersal storage function to produce manipulated data; and

de-manipulate the manipulated data using the one or more codec types in the codec execution order to produce recovered data.

18. The DS processing unit of claim 17, wherein the processing module further functions to:

receive, via the network interface, the plurality of encoded data slices from the plurality of DS storage units;

decode the plurality of encoded data slices using the error coding dispersal storage function to produce appended manipulated data; and

separate the appended manipulated data to produce the manipulated data and the codec flag.

19. The DS processing unit of claim 17, wherein the processing module further functions to:

receive, via the network interface, an appended plurality of encoded data slices from the plurality of DS storage units; and

separate the appended plurality of encoded data slices to produce the plurality of encoded data slices and the codec flag.

20. The DS processing unit of claim 17, wherein the processing module further functions to:

receive a plurality of post data manipulated slices, the codec flag, and a second codec flag from a plurality of DS storage units wherein the second codec flag indicates a second one or more codec types and a second codec execution order, wherein the second codec types include two or more of;

a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

de-manipulate at least one of the plurality of post data manipulated slices using the second one or more codec types in the second codec execution order to produce the plurality of encoded data slices.