Dispersed storage network data manipulation

• US 9,167,277 B2
• Filed: 05/07/2010
• Issued: 10/20/2015
• Est. Priority Date: 08/03/2009
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Cilfone, Bart, Leggette, Wesley, Resch, Jason K.
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): MCINNISH, KEVIN K
• Application Number:
  US12/776,098
• Publication Number:
  US 20110026842A1
• Time in Patent Office: 1,992 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04N 21/23116 involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/25875 involving end-user authentication restricting access to computer systems by authenticating users using a predetermined code G06F21/33; arrangements for secret or secure communication including means for verifying the identity or authority of a user of the system H04L9/32; networks authentication protocols H04L63/08; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/2747 Remote storage of video programs received via the downstream path, e.g. from the server

Abstract

A method begins by a processing module receiving data for dispersed storage, wherein the data has an associated user identification (ID), and obtaining a codec flag based on the associated user ID. The codec flag may indicate one or more codec types and a codec execution order. The codec types may include two or more of: a null data manipulation, one or more versions of a data integrity function, one or more versions of a compression function, and/or one or more versions of an encryption function. The method continues with the processing module manipulating the data to produce manipulated data utilizing the one or more codec types in the codec execution order, encoding the manipulated data using an error coding dispersal storage function to produce a plurality of encoded data slices, and sending the slices and codec flag to a plurality of DS storage units for storage.

Claims

1. A method for execution by a dispersed storage (DS) processing unit, the method comprises:

receiving a data segment of a plurality of data segments of a data object for dispersed storage, wherein the data object has an associated user identification (ID) of a user;

obtaining a codec flag based on the associated user ID, wherein the codec flag includes a current state field and a codec stack field, in which the current state field is initialized in a null state and in which the codec stack field contains numerical values in execution order to access respective numerical entries in a data manipulation table where the numerical entries correspond to different codec functions for use in manipulating the data segment, and wherein the codec functions listed in the data manipulation table include one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

manipulating the data segment using selected codec functions by selecting a first numerical value of the numerical values contained in the codec stack field in the execution order and placing the first numerical value in the current state field to perform a first codec function corresponding to a respective numerical entry in the data manipulation table and further manipulating the data segment by selecting subsequent numerical value or values contained in the codec stack field, one at a time in the execution order, for placing into the current state field to perform a respective codec function, until codec functions indicated in the codec stack field are completed to generate a manipulated data;

encoding the manipulated data using an error coding dispersal storage function to generate a plurality of encoded data slices for storage, wherein a read threshold number of encoded data slices of the plurality of encoded data slices are needed to recover the manipulated data, where the read threshold number of encoded data slices is less than a number corresponding to the plurality of encoded data slices; and

outputting the plurality of encoded data slices and the codec stack field of the codec flag to a plurality of DS storage units for storage therein.

2. The method of claim 1, wherein the encoding further comprises:

appending the codec stack field of the codec flag to the manipulated data to produce appended manipulated data; and

encoding the appended manipulated data using the error coding dispersal storage function to produce the plurality of encoded data slices.

3. The method of claim 1, wherein the outputting further comprises:

appending the codec stack field of the codec flag to the plurality of encoded data slices to produce an appended plurality of encoded data slices; and

outputting the appended plurality of encoded data slices to the plurality of DS storage units for storage therein.

4. The method of claim 1 further comprises:

obtaining a second codec flag based on the associated user ID, wherein the codec stack field of the second codec flag indicates a second execution order of codec functions to perform in manipulating the plurality of encoded data slices;

manipulating the plurality of encoded data slices using one or more codec functions corresponding to the codec stack field of the second codec flag in the second codec execution order to generate manipulated encoded data slices; and

outputting the plurality of encoded data slices as the manipulated encoded data slices, along with the codec stack field of the second codec flag to the plurality of DS storage units for storage therein.

5. The method of claim 1, wherein the codec stack field of the codec flag further has repeat numerical values to perform a same codec function, but at a different period in the execution order.

6. The method of claim 1, wherein the obtaining further comprises:

identifying a vault identity (ID) based on the associated user ID;

identifying a registry descriptor based on the vault ID; and

accessing the registry descriptor from a plurality of registry descriptors to retrieve vault information, wherein the vault information includes a codec stack for the codec stack field and identifies the data manipulation table.

7. A method for execution by a dispersed storage (DS) processing unit, the method comprises:

providing access to at least a threshold number of a plurality of encoded data slices of a data segment of a plurality of data segments of a data object stored within a plurality of DS storage units, wherein the threshold number of encoded data slices of the plurality of encoded data slices are needed to recover the data segment, the threshold number of encoded data slices being less than the plurality of encoded data slices;

receiving a codec stack field of a codec flag and the threshold number of encoded data slices of the plurality of encoded data slices from the plurality of DS storage units, wherein the codec stack field contains numerical values in execution order to access respective numeric entries in a data manipulation table where the numerical entries correspond to different codec functions for use in recovering the data segment from the threshold number of encoded data slices, and wherein the codec functions listed in the data manipulation table include one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

decoding the threshold number of encoded data slices using an error coding dispersal storage function to produce manipulated data; and

de-manipulating the manipulated data using selected codec functions by selecting a first numerical value of the numerical values contained in the codec stack field in the execution order and placing the first numerical value in a current state field of the codec flag to perform a first codec function corresponding to a respective numerical entry in the data manipulation table and further de-manipulating the manipulated data by selecting subsequent numerical value or values contained in the codec stack field, one at a time in the execution order, for placing into the current state field to perform a respective codec function, until codec functions indicated in the codec stack field are completed to recover the data segment.

8. The method of claim 7 further comprises:

receiving the threshold number of encoded data slices from the plurality of DS storage units;

decoding the threshold number of encoded data slices using the error coding dispersal storage function to produce appended manipulated data; and

separating the appended manipulated data to produce the manipulated data and the codec stack field.

9. The method of claim 7 further comprises:

receiving an appended threshold number of encoded data slices from the plurality of DS storage units; and

separating the appended encoded data slices to produce the threshold number of encoded data slices and the codec stack field.

10. The method of claim 7 further comprises:

receiving the threshold number of encoded data slices, the codec stack field, and a second codec field from the plurality of DS storage units, wherein the second codec stack field indicates a second execution order of codec functions to perform in de-manipulating the manipulated data; and

de-manipulating at least one of the plurality of encoded data slices using the second execution order.

11. A dispersed storage (DS) processing unit comprises:

a network interface; and

a processing module operable to;

obtain a data segment of a plurality of data segments of a data object for dispersed storage, wherein the data object has an associated user identification (ID) of a user;

obtain a codec flag based on the associated user ID, wherein the codec flag includes a current state field and a codec stack field, in which the current state field is initialized in a null state and in which the codec stack field contains numerical values in execution order to access respective numerical entries in a data manipulation table where the numerical entries correspond to different codec functions for use in manipulating the data segment, and wherein the codec functions listed in the data manipulation table include one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

manipulate the data segment using selected codec functions by selecting a first numerical value of the numerical values contained in the codec stack field in the execution order and placing the first numerical value in the current state field to perform a first codec function corresponding to a respective numerical entry in the data manipulation table and further manipulating the data segment by selecting subsequent numerical value or values contained in the codec stack field, one at a time in the execution order, for placing into the current state field to perform a respective codec function, until codec functions indicated in the codec stack field are completed to generate a manipulated data;

encode the manipulated data using an error coding dispersal storage function to generate a plurality of encoded data slices for storage, wherein a read threshold number of encoded data slices of the plurality of encoded data slices are needed to recover the manipulated data, where the read threshold number of encoded data slices is less than a number corresponding to the plurality of encoded data slices; and

output, via the network interface, the plurality of encoded data slices and the codec stack field of the codec flag to a plurality of DS storage units for storage therein.

12. The DS processing unit of claim 11, wherein the processing module further functions to encode by:

appending the codec stack field of the codec flag to the manipulated data to produce appended manipulated data; and

encoding the appended manipulated data using the error coding dispersal storage function to produce the plurality of encoded data slices.

13. The DS processing unit of claim 11, wherein the processing module further functions to output by:

appending the codec stack field of the codec flag to the plurality of encoded data slices to produce an appended plurality of encoded data slices; and

outputting, via the network interface, the appended plurality of encoded data slices to the plurality of DS storage units for storage therein.

14. The DS processing unit of claim 11, wherein the processing module further functions to:

obtain a second codec flag based on the associated user ID, wherein the codec stack field of the second codec flag indicates a second execution order of coded functions to perform in manipulating the plurality of encoded data slices;

manipulate the plurality of encoded data slices using one or more codec functions corresponding to the codec stack field of the second codec flag in the second codec execution order to generate manipulated encoded data slices; and

output, via the network interface, the plurality of encoded data slices as the manipulated encoded data slices, along with the codec stack field of the second codec flag to the plurality of DS storage units for storage therein.

15. The DS processing unit of claim 11, wherein the codec stack field of the codec flag further has repeat numerical values to perform a same codec function, but at a different period in the execution order.

16. The DS processing unit of claim 11, wherein the processing module further functions to obtain the codec flag by:

identifying a vault identity (ID) based on the associated user ID;

identifying a registry descriptor based on the vault ID; and

accessing the registry descriptor from a plurality of registry descriptors to retrieve vault information, wherein the vault information includes a codec stack for the codec stack field and identifies the data manipulation table.

17. A dispersed storage (DS) processing unit comprises:

a network interface; and

a processing module operable to;

provide, via the network interface, access to at least a threshold number of a plurality of encoded data slices of a data segment of a plurality of data segments of a data object stored within a plurality of DS storage units, wherein the threshold number of encoded data slices of the plurality of encoded data slices are needed to recover the data segment, the threshold number of encoded data slices being less than the plurality of encoded data slices;

receive, via the network interface, a codec stack field of a codec flag and the threshold number of encoded data slices of the plurality of encoded data slices from the plurality of DS storage units, wherein the codec stack field contains numerical values in execution order to access respective numeric entries in a data manipulation table where the numerical entries correspond to different codec functions for use in recovering the data segment from the threshold number of encoded data slices, and wherein the codec functions listed in the data manipulation table include one or more versions of a data integrity function, one or more versions of a compression function, and one or more versions of an encryption function;

decode the threshold number of encoded data slices using an error coding dispersal storage function to produce manipulated data; and

de-manipulate the manipulated data using selected codec functions by selecting a first numerical value of the numerical values contained in the codec stack field in the execution order and placing the first numerical value in a current state field of the codec flag to perform a first codec function corresponding to a respective numerical entry in the data manipulation table and further de-manipulating the manipulated data by selecting subsequent numerical value or values contained in the codec stack field, one at a time in the execution order, for placing into the current state field to perform a respective codec function, until codec functions indicated in the codec stack field are completed to recover the data segment.

18. The DS processing unit of claim 17, wherein the processing module further functions to:

receive, via the network interface, the threshold number of encoded data slices from the plurality of DS storage units;

decode the threshold number of encoded data slices using the error coding dispersal storage function to produce appended manipulated data; and

separate the appended manipulated data to produce the manipulated data and the codec stack field.

19. The DS processing unit of claim 17, wherein the processing module further functions to:

receive, via the network interface, an appended threshold number of encoded data slices from the plurality of DS storage units; and

separate the appended encoded data slices to produce the threshold number of encoded data slices and the codec stack field.

20. The DS processing unit of claim 17, wherein the processing module further functions to:

receive the threshold number of encoded data slices, the codec stack field, and a second codec stack field from the plurality of DS storage units wherein the second codec stack field indicates a second execution order of codec functions to perform in de-manipulating the manipulated data; and

de-manipulate at least one of the plurality of encoded data slices using the second execution order.