Method and apparatus for identifying data inconsistency in a dispersed storage network

• US 9,411,810 B2
• Filed: 04/02/2010
• Issued: 08/09/2016
• Est. Priority Date: 08/27/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Mark, Zachary J.
• Primary Examiner(s): Hershley, Mark E
• Application Number:
  US12/753,569
• Publication Number:
  US 20110055170A1
• Time in Patent Office: 2,321 Days
• Field of Search: 707/691
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

A method for execution by a DS processing unit begins with the DS processing unit receiving a plurality of integrity data, which was generated by a plurality of DS storage units regarding one or more data slices of one or more pluralities of data slices. The method continues with the DS processing unit evaluating the received integrity data. The method continues with the DS processing unit requesting naming information of the one or more pluralities of data slices when the evaluation of the received integrity data yields unfavorable results. The method continues with the DS processing unit receiving the naming information from the plurality of DS storage units. The method continues with the DS processing unit evaluating the received naming information to identify at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

Claims

1. A method for execution by a processing module, the method comprises:

receiving, from at least some of a plurality of distributed storage (DS) storage units, integrity data of slice names of one or more pluralities of data slices without retrieving the one or more pluralities of data slices and without retrieving the slice names to produce a plurality of received integrity data, wherein a plurality of data slices of the one or more pluralities of data slices corresponds to a dispersed storage error encoded data segment;

evaluating the plurality of received integrity data without using any of the one or more pluralities of data slices;

when the evaluating of the plurality of received integrity data yields an inconsistency between the plurality of received integrity data, requesting, from the at least some of the plurality of DS storage units, the slice names of the one or more pluralities of data slices;

receiving, from the at least some of the plurality of DS storage units, the slice names without receiving the one or more pluralities of data slices to produce a plurality of received slice names; and

evaluating a revision level of the plurality of received slice names to identify at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

2. The method of claim 1, wherein the receiving the integrity data further comprises:

requesting the integrity data from at least some of the plurality of DS storage units, wherein the requesting identifies the one or more pluralities of data slices.

3. The method of claim 1, the integrity data comprises one or more of:

a cyclic redundancy check of at least a portion of each of the slice names;

a cyclic redundancy check of at least a portion of each group of the slice names wherein a group of slice names corresponds to a group of the pluralities of data slices;

a hash of at least a portion of each of the slice names;

a hash of at least a portion of each group of the slice names;

a digest list of at least a portion of at least one of the slice names; and

a digest list of at least a portion of each group of the slice names.

4. The method of claim 1, wherein the evaluating the plurality of received integrity data comprises:

comparing one of the plurality of received integrity data with at least some of the plurality of the received integrity data to produce a comparison; and

indicating unfavorable results when the comparison is unfavorable.

5. The method of claim 1, wherein the slice names comprises:

at least a portion of the slice name, wherein the slice name includes a source name.

6. The method of claim 1, wherein the data inconsistency comprises at least one of:

outdated data slice;

missing data slice; and

corrupted slice names.

7. The method of claim 1, wherein the evaluating the plurality of received slice names comprises one or more of:

comparing one of the plurality of received slice names with at least some of the plurality of the received slice names to identify at least one data slice of the one or more of pluralities of data slices having a revision level data inconsistency;

comparing one of the plurality of received slice names with at least some of the plurality of the received slice names to identify at least one data slice of the one or more of pluralities of data slices having a missing slice name data inconsistency; and

comparing one of the plurality of received slice names with at least some of the plurality of the received slice names to identify at least one data slice of the one or more of pluralities of data slices having a corrupted slice name data inconsistency.

8. The method of claim 1 further comprises:

rebuilding the at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

9. A method for execution in a distributed storage (DS) storage system, the method comprises:

generating integrity data of one or more slice names of one or more data slices of one or more pluralities of data slices without retrieving the one or more pluralities of data slices and without retrieving the slice names, wherein a plurality of data slices of the one or more pluralities of data slices corresponds to a dispersed storage error encoded data segment;

evaluating the integrity data without using any of the one or more data slices;

when the evaluating of the integrity data yields an inconsistency between a plurality of received integrity data, retrieving one or more slice names for the one or more data slices without retrieving the one or more data slices; and

evaluating a revision level of the one or more slice names to identify at least one data slice of the one or more data slices having a data inconsistency.

10. The method of claim 9, wherein the generating the integrity data further comprises at least one of:

generating the integrity data in response to a request; and

generating the integrity data in accordance with a predetermined pattern.

11. The method of claim 9 further comprises:

receiving updated slice names for the data slice; and

storing the updated slice names.

12. The method of claim 9 further comprises:

transmitting the integrity data via a network to a computing core.

13. The method of claim 9, the slice names comprises:

at least a portion of a slice name, wherein the slice name includes a source name.

14. A distributed storage (DS) processing unit comprises:

a network interface; and

a processing module operable to;

receive, via the network interface, integrity data from at least some of a plurality of DS storage units of slice names of one or more pluralities of data slices without receiving the one or more pluralities of data slices and without retrieving the slice names to produce a plurality of received integrity data, wherein a plurality of data slices of the one or more pluralities of data slices corresponds to a dispersed storage error encoded data segment;

evaluate the plurality of received integrity data without using any of the one or more pluralities of data slices;

request, via the network interface, the slice names of the one or more pluralities of data slices from the at least some of the plurality of DS storage units when the evaluation of the plurality of received integrity data yields an inconsistency between the plurality of received integrity data;

receive, via the network interface, the slice names from the at least some of the plurality of DS storage units without receiving the one or more pluralities of data slices to produce a plurality of received slice names; and

evaluate a revision level of the plurality of received slice names to identify at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

15. The DS processing unit of claim 14, wherein the processing module further functions to receive the integrity data by:

requesting the integrity data from the at least some of the plurality of DS storage units, wherein the requesting identifies the one or more pluralities of data slices.

16. The DS processing unit of claim 14, wherein the integrity data comprises one or more of:

a cyclic redundancy check of at least a portion of each of the slice names;

a cyclic redundancy check of at least a portion of each group of the slice names wherein a group of slice names corresponds to a group of the pluralities of data slices;

a hash of at least a portion of each of the slice names;

a hash of at least a portion of each group of the slice names;

a digest list of at least a portion of at least one of the slice names; and

a digest list of at least a portion of each group of the slice names.

17. The DS processing unit of claim 14, wherein the processing module further functions to evaluate the plurality of received integrity data by:

comparing one of the plurality of received integrity data with at least some of the plurality of the received integrity data to produce a comparison; and

indicating unfavorable results when the comparison is unfavorable.

18. The DS processing unit of claim 14, wherein the slice names comprises:

at least a portion of a slice name, wherein the slice name includes a source name.

19. The DS processing unit of claim 14, wherein the data inconsistency comprises at least one of:

outdated data slice;

missing data slice; and

corrupted slice names.

20. The DS processing unit of claim 14, wherein the processing module further functions to evaluate the plurality of received slice names by:

comparing one of the plurality of received slice names with at least some of the plurality of the received slice names to identify at least one data slice of the one or more of pluralities of data slices having a revision level data inconsistency;

comparing one of the plurality of received slice names with at least some of the plurality of the received slice names to identify at least one data slice of the one or more of pluralities of data slices having a missing slice names data inconsistency; and

comparing one of the plurality of received slice names with at least some of the plurality of the received slice names to identify at least one data slice of the one or more of pluralities of data slices having a corrupted slice names data inconsistency.

21. The DS processing unit of claim 14, wherein the processing module further functions to:

rebuild the at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

22. A distributed storage (DS) storage system comprises:

a network interface;

at least one processing module operable to;

generate integrity data of one or more slices names of one or more data slices of one or more pluralities of data slices without retrieving the one or more pluralities of data slices and without retrieving the slice names, wherein a plurality of data slices of the one or more pluralities of data slices corresponds to a dispersed storage error encoded data segment;

evaluate the integrity data without using any of the one or more data slices;

when the evaluating of the integrity data yields an inconsistency between a plurality of received integrity data, retrieve one or more slice names for the one or more data slices without retrieving the one or more data slices; and

evaluate a revision level of the one or more slice names to identify at least one data slice of the one or more data slices having a data inconsistency.

23. The DS storage system of claim 22, wherein the at least one processing module further functions to generate integrity data by at least one of:

generating the integrity data in response to a request; and

generating the integrity data in accordance with a predetermined pattern.

24. The DS storage system of claim 22, wherein the at least one processing module further functions to:

receive updated slice names for the data slice; and

store the updated slice names.

25. The DS storage system of claim 22, wherein the at least one processing module further functions to:

transmit the integrity data via a network and the network interface to a computing core.

26. The DS storage system of claim 22, wherein the slice names comprises:

at least a portion of a slice name, wherein the slice name includes a source name.