Securing encoding data slices using an integrity check value list

• US 10,387,063 B2
• Filed: 02/20/2018
• Issued: 08/20/2019
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: McShane, Niall J., Resch, Jason K., Volvovski, Ilya
• Primary Examiner(s): Tu, Christine T.
• Application Number:
  US15/899,626
• Publication Number:
  US 20180188992A1
• Time in Patent Office: 546 Days
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/616 Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561

Abstract

A method includes retrieving a read threshold number of integrity check value list (ICVL) encoded data slices of a set of ICVL encoded data slices. The method further includes determining whether an appended ICVL of each ICVL encoded data slice of the read threshold number of ICVL encoded data slices substantially match. When the appended ICVL of one of the ICVL encoded does not substantially match the appended ICVL of other ICVL encoded data slices, the method further includes determining a likely cause for the mismatch. When the likely cause is missing a revision update, the method further includes initiate rebuilding of the encoded data slice portion. The method further includes generating an integrity check value for the rebuilt encoded data slice and updating the integrity check value list. The method further includes appending the updated integrity check value list to the rebuilt encoded data slice.

Claims

1. A computing device comprising:

an interface configured to interface and communicate with a dispersed storage network (DSN);

memory that stores operational instructions; and

processing circuitry operably coupled to the interface and to the memory, wherein the processing circuitry is configured to execute the operational instructions to;

retrieve a read threshold number of integrity check value list (ICVL) encoded data slices (EDSs) of a set of ICVL EDSs from at least some storage units (SUs) of a set of SUs based on a request;

determine whether an appended ICVL of each ICVL EDS of the read threshold number of ICVL EDSs substantially match;

based on determination that the appended ICVL of one of the ICVL EDSs of the read threshold number of ICVL EDSs does not substantially match the appended ICVL of other ICVL EDSs of the read threshold number of ICVL EDSs;

determine a likely cause for the appended ICVL of the one of the ICVL EDSs not substantially matching the appended ICVL of the other ICVL EDSs; and

based on determination that the likely cause is an inaccurate ICVL appended to the one of the ICVL EDSs;

calculate a new integrity check value for an EDS portion of the one of the ICVL EDSs;

extract an integrity check value from the appended ICVL of one of the other ICVL EDSs to produce an extracted integrity check value;

compare the new integrity check value with the extracted integrity check value; and

based on determination that the new integrity check value substantially matches the extracted integrity check value, utilize the EDS in decoding of the read threshold number of EDSs to recover a data segment of a data object.

2. The computing device of claim 1, wherein the processing circuitry is further configured to execute the operational instructions to:

based on determination that the likely cause is not the inaccurate ICVL appended to the one of the ICVL EDSs and also is not missing a revision update, take at least one corrective measure for an unauthorized modification that includes at least one of sending a delete message to a SU storing the one of the ICVL EDSs, initiating a rebuild process, or initiating at least one of a decommissioning of the SU, a quarantine of the SU, or a higher level of security regarding access to the SU.

3. The computing device of claim 1, wherein the processing circuitry is further configured to execute the operational instructions to:

based on determination that the new integrity check value does not substantially matches the extracted integrity check value;

initiate rebuilding of the EDS portion of the one of the ICVL EDSs to produce a rebuilt EDS;

generate an integrity check value for the rebuilt EDS;

generate an updated ICVL including the integrity check value for the rebuilt EDS and integrity check values of the EDSs of the other ICVL EDSs;

append the updated ICVL to the rebuilt EDS to produce a rebuilt ICVL EDS; and

send the rebuilt ICVL EDS to one of the SUs of the set of SUs for storage therein.

4. The computing device of claim 1, wherein the processing circuitry is further configured to execute the operational instructions to:

based on determination that the likely cause is not missing a revision update;

initiate rebuilding of the EDS portion of the one of the ICVL EDSs to produce a rebuilt EDS;

generate an integrity check value for the rebuilt EDS;

generate an updated ICVL including the integrity check value for the rebuilt EDS and integrity check values of the EDSs of the other ICVL EDSs;

append the updated ICVL to the rebuilt EDS to produce a rebuilt ICVL EDS; and

send the rebuilt ICVL EDS to one of the SUs of the set of SUs for storage therein.

5. The computing device of claim 1, wherein the processing circuitry is further configured to execute the operational instructions to:

based on determination that the appended ICVL of one of the ICVL EDSs of the read threshold number of ICVL EDSs substantially matches the appended ICVL of other ICVL EDSs of the read threshold number of ICVL EDSs, process the request.

6. The computing device of claim 1, wherein the processing circuitry is further configured to execute the operational instructions to:

generate a set of integrity check values for a set of EDSs, wherein the data segment of the data object is dispersed storage error encoded to produce the set of EDSs;

generate an ICVL from the set of integrity check values;

append the ICVL to each EDS of the set of EDSs to produce the set of ICVL EDSs; and

send the set of ICVL EDSs to the set of SUs of the DSN for storage therein.

7. The computing device of claim 1, wherein the computing device is one of the SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computer (PC), a workstation, or a video game device.

8. The computing device of claim 1, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

9. A computing device comprising:

an interface configured to interface and communicate with a dispersed storage network (DSN);

memory that stores operational instructions; and

processing circuitry operably coupled to the interface and to the memory, wherein the processing circuitry is configured to execute the operational instructions to;

retrieve a read threshold number of integrity check value list (ICVL) encoded data slices (EDSs) of a set of ICVL EDSs from at least some storage units (SUs) of a set of SUs based on a request;

determine whether an appended ICVL of each ICVL EDS of the read threshold number of ICVL EDSs substantially match;

generate a set of integrity check values for a set of EDSs, wherein a data segment of a data object is dispersed storage error encoded to produce the set of EDSs;

generate an ICVL from the set of integrity check values;

append the ICVL to each EDS of the set of EDSs to produce the set of ICVL EDSs;

send the set of ICVL EDSs to the set of SUs of the DSN for storage therein;

based on determination that the appended ICVL of one of the ICVL EDSs of the read threshold number of ICVL EDSs does not substantially match the appended ICVL of other ICVL EDSs of the read threshold number of ICVL EDSs;

determine a likely cause for the appended ICVL of the one of the ICVL EDSs not substantially matching the appended ICVL of the other ICVL EDSs; and

based on determination that the likely cause is an inaccurate ICVL appended to the one of the ICVL EDSs;

calculate a new integrity check value for an EDS portion of the one of the ICVL EDSs;

extract an integrity check value from the appended ICVL of one of the other ICVL EDSs to produce an extracted integrity check value;

compare the new integrity check value with the extracted integrity check value; and

based on determination that the new integrity check value substantially matches the extracted integrity check value, utilize the EDS in decoding of the read threshold number of EDSs to recover the data segment of the data object; and

based on determination that the appended ICVL of one of the ICVL EDSs of the read threshold number of ICVL EDSs substantially matches the appended ICVL of other ICVL EDSs of the read threshold number of ICVL EDSs, process the request.

10. The computing device of claim 9, wherein the processing circuitry is further configured to execute the operational instructions to:

based on determination that the likely cause is not the inaccurate ICVL appended to the one of the ICVL EDSs and also is not missing a revision update, take at least one corrective measure for an unauthorized modification that includes at least one of sending a delete message to a SU storing the one of the ICVL EDSs, initiating a rebuild process, or initiating at least one of a decommissioning of the SU, a quarantine of the SU, or a higher level of security regarding access to the SU.

11. The computing device of claim 9, wherein the processing circuitry is further configured to execute the operational instructions to:

based on determination that the new integrity check value does not substantially matches the extracted integrity check value;

initiate rebuilding of the EDS portion of the one of the ICVL EDSs to produce a rebuilt EDS;

generate an integrity check value for the rebuilt EDS;

generate an updated ICVL including the integrity check value for the rebuilt EDS and integrity check values of the EDSs of the other ICVL EDSs;

append the updated ICVL to the rebuilt EDS to produce a rebuilt ICVL EDS; and

send the rebuilt ICVL EDS to one of the SUs of the set of SUs for storage therein.

12. The computing device of claim 9, wherein the processing circuitry is further configured to execute the operational instructions to:

based on determination that the likely cause is not missing a revision update;

initiate rebuilding of the EDS portion of the one of the ICVL EDSs to produce a rebuilt EDS;

generate an integrity check value for the rebuilt EDS;

generate an updated ICVL including the integrity check value for the rebuilt EDS and integrity check values of the EDSs of the other ICVL EDSs;

append the updated ICVL to the rebuilt EDS to produce a rebuilt ICVL EDS; and

send the rebuilt ICVL EDS to one of the SUs of the set of SUs for storage therein.

13. The computing device of claim 9, wherein the computing device is one of the SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computer (PC), a workstation, or a video game device.

14. The computing device of claim 9, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

15. A method for execution by a computing device, the method comprising:

retrieving, via an interface configured to interface and communicate with a dispersed storage network (DSN), a read threshold number of integrity check value list (ICVL) encoded data slices (EDSs) of a set of ICVL EDSs from at least some storage units (SUs) of a set of SUs within the DSN based on a request;

determining whether an appended ICVL of each ICVL EDS of the read threshold number of ICVL EDSs substantially match;

based on determination that the appended ICVL of one of the ICVL EDSs of the read threshold number of ICVL EDSs does not substantially match the appended ICVL of other ICVL EDSs of the read threshold number of ICVL EDSs;

determining a likely cause for the appended ICVL of the one of the ICVL EDSs not substantially matching the appended ICVL of the other ICVL EDSs; and

based on determination that the likely cause is an inaccurate ICVL appended to the one of the ICVL EDSs;

calculating a new integrity check value for an EDS portion of the one of the ICVL EDSs;

extracting an integrity check value from the appended ICVL of one of the other ICVL EDSs to produce an extracted integrity check value;

comparing the new integrity check value with the extracted integrity check value; and

based on determination that the new integrity check value substantially matches the extracted integrity check value, utilizing the EDS in decoding of the read threshold number of EDSs to recover a data segment of a data object.

16. The method of claim 15 further comprising:

based on determination that the likely cause is not the inaccurate ICVL appended to the one of the ICVL EDSs and also is not missing a revision update, taking at least one corrective measure for an unauthorized modification that includes at least one of sending a delete message to a SU storing the one of the ICVL EDSs, initiating a rebuild process, or initiating at least one of a decommissioning of the SU, a quarantine of the SU, or a higher level of security regarding access to the SU.

17. The method of claim 15 further comprising:

based on determination that the new integrity check value does not substantially matches the extracted integrity check value;

initiating rebuilding of the EDS portion of the one of the ICVL EDSs to produce a rebuilt EDS;

generating an integrity check value for the rebuilt EDS;

generating an updated ICVL including the integrity check value for the rebuilt EDS and integrity check values of the EDSs of the other ICVL EDSs;

appending the updated ICVL to the rebuilt EDS to produce a rebuilt ICVL EDS; and

sending the rebuilt ICVL EDS to one of the SUs of the set of SUs for storage therein.

18. The method of claim 15 further comprising:

based on determination that the likely cause is not missing a revision update;

initiating rebuilding of the EDS portion of the one of the ICVL EDSs to produce a rebuilt EDS;

generating an integrity check value for the rebuilt EDS;

generating an updated ICVL including the integrity check value for the rebuilt EDS and integrity check values of the EDSs of the other ICVL EDSs;

appending the updated ICVL to the rebuilt EDS to produce a rebuilt ICVL EDS; and

sending the rebuilt ICVL EDS to one of the SUs of the set of SUs for storage therein.

19. The method of claim 15, wherein the computing device is one of the SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computer (PC), a workstation, or a video game device.

20. The method of claim 15, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).