METHOD AND APPARATUS FOR IDENTIFYING DATA INCONSISTENCY IN A DISPERSED STORAGE NETWORK

• US 20110055170A1
• Filed: 04/02/2010
• Published: 03/03/2011
• Est. Priority Date: 08/27/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: MARK, ZACHARY J.
• Granted Patent: US 9,411,810 B2
• Time in Patent Office: Days
• US Class Current: 707/691
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

A method for execution by a DS processing unit begins with the DS processing unit receiving a plurality of integrity data, which was generated by a plurality of DS storage units regarding one or more data slices of one or more pluralities of data slices. The method continues with the DS processing unit evaluating the received integrity data. The method continues with the DS processing unit requesting naming information of the one or more pluralities of data slices when the evaluation of the received integrity data yields unfavorable results. The method continues with the DS processing unit receiving the naming information from the plurality of DS storage units. The method continues with the DS processing unit evaluating the received naming information to identify at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

Claims

1. A method for execution by a processing module, the method comprises:

receiving, from a plurality of distributed storage (DS) storage units, integrity data regarding one or more pluralities of data slices to produce a plurality of received integrity data, wherein a plurality of data slices of the one or more pluralities of data slices corresponds to a data segment;

evaluating the plurality of received integrity data;

when the evaluating of the plurality of received integrity data yields unfavorable results, requesting, from the plurality of DS storage units, naming information of the one or more pluralities of data slices;

receiving, from the plurality of DS storage units, the naming information to produce a plurality of received naming information; and

evaluating the plurality of received naming information to identify at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

2. The method of claim 1, wherein the receiving the integrity data further comprises:

requesting the integrity data from the plurality of DS storage units, wherein the requesting identifies the one or more pluralities of data slices.

3. The method of claim 1, the integrity data comprises one or more of:

a cyclic redundancy check of at least a portion of a slice name for one of the pluralities of data slices;

a cyclic redundancy check of at least a portion of each of a group of slice names wherein the group of slice names corresponds to a group of the pluralities of data slices;

a hash of at least a portion of a slice name for one of the pluralities of data slices;

a hash of at least a portion of each of a group of slice names wherein the group of slice names corresponds to a group of the pluralities of data slices;

a digest list of at least a portion of a slice name for one of the pluralities of data slices; and

a digest list of at least a portion of each of a group of slice names wherein the group of slice names corresponds to a group of the pluralities of data slices.

4. The method of claim 1, wherein the evaluating the plurality of received integrity data comprises:

comparing one of the plurality of received integrity data with at least some of the plurality of the received integrity data to produce a comparison; and

indicating the unfavorable results when the comparison is unfavorable.

5. The method of claim 1, the naming information comprises:

at least a portion of a slice name, wherein the slice name includes a source name.

6. The method of claim 1, the data inconsistency comprises at least one of:

outdated data slice;

missing data slice; and

corrupted naming information.

7. The method of claim 1, wherein the evaluating the plurality of received naming information comprises one or more of:

comparing one of the plurality of received naming information with at least some of the plurality of the received naming information to identify at least one data slice of the one or more of pluralities of data slices having a revision level data inconsistency;

comparing one of the plurality of received naming information with at least some of the plurality of the received naming information to identify at least one data slice of the one or more of pluralities of data slices having a missing naming information data inconsistency; and

comparing one of the plurality of received naming information with at least some of the plurality of the received naming information to identify at least one data slice of the one or more of pluralities of data slices having a corrupted naming information data inconsistency.

8. The method of claim 1 further comprises:

rebuilding the at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

9. A method for execution by a distributed storage (DS) storage unit, the method comprises:

generating integrity data regarding one or more data slices of one or more pluralities of data slices, wherein a plurality of data slices of the one or more pluralities of data slices corresponds to a data segment;

transmitting the integrity data;

receiving a request for naming information for the one or more data slices;

retrieving the naming information for the one or more data slices in response to receiving the request; and

transmitting the naming information for the one or more data slices.

10. The method of claim 9, wherein the generating the integrity data further comprises at least one of:

generating the integrity data in response to a request; and

generating the integrity data in accordance with a predetermined pattern.

11. The method of claim 9 further comprises:

receiving updated naming information for a data slice of the one or more data slices; and

storing the updated naming information.

12. The method of claim 9, wherein the transmitting the integrity data comprises:

transmitting the integrity data via a network to a computing core.

13. The method of claim 9, the naming information comprises:

at least a portion of a slice name, wherein the slice name includes a source name.

14. A distributed storage (DS) processing unit comprises:

a network interface; and

a processing module operable to;

receive, via the network interface, integrity data from a plurality of DS storage units regarding one or more pluralities of data slices to produce a plurality of received integrity data, wherein a plurality of data slices of the one or more pluralities of data slices corresponds to a data segment;

evaluate the plurality of received integrity data;

request, via the network interface, naming information of the one or more pluralities of data slices from the plurality of DS storage units when the evaluation of the plurality of received integrity data yields unfavorable results;

receive, via the network interface, the naming information from the plurality of DS storage units to produce a plurality of received naming information; and

evaluate the plurality of received naming information to identify at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

15. The DS processing unit of claim 14, wherein the processing module further functions to receive the integrity data by:

requesting the integrity data from the plurality of DS storage units, wherein the requesting identifies the one or more pluralities of data slices.

16. The DS processing unit of claim 14, wherein the integrity data comprises one or more of:

a cyclic redundancy check of at least a portion of a slice name for one of the pluralities of data slices;

a cyclic redundancy check of at least a portion of each of a group of slice names wherein the group of slice names corresponds to a group of the pluralities of data slices;

a hash of at least a portion of a slice name for one of the pluralities of data slices;

a hash of at least a portion of each of a group of slice names wherein the group of slice names corresponds to a group of the pluralities of data slices;

a digest list of at least a portion of a slice name for one of the pluralities of data slices; and

a digest list of at least a portion of each of a group of slice names wherein the group of slice names corresponds to a group of the pluralities of data slices.

17. The DS processing unit of claim 14, wherein the processing module further functions to evaluate the plurality of received integrity data by:

comparing one of the plurality of received integrity data with at least some of the plurality of the received integrity data to produce a comparison; and

indicating the unfavorable results when the comparison is unfavorable.

18. The DS processing unit of claim 14, wherein the naming information comprises:

at least a portion of a slice name, wherein the slice name includes a source name.

19. The DS processing unit of claim 14, wherein the data inconsistency comprises at least one of:

outdated data slice;

missing data slice; and

corrupted naming information.

20. The DS processing unit of claim 14, wherein the processing module further functions to evaluate the plurality of received naming information by:

comparing one of the plurality of received naming information with at least some of the plurality of the received naming information to identify at least one data slice of the one or more of pluralities of data slices having a revision level data inconsistency;

comparing one of the plurality of received naming information with at least some of the plurality of the received naming information to identify at least one data slice of the one or more of pluralities of data slices having a missing naming information data inconsistency; and

comparing one of the plurality of received naming information with at least some of the plurality of the received naming information to identify at least one data slice of the one or more of pluralities of data slices having a corrupted naming information data inconsistency.

21. The DS processing unit of claim 14, wherein the processing module further functions to:

rebuild the at least one data slice of the one or more of pluralities of data slices having a data inconsistency.

22. A distributed storage (DS) storage unit comprises:

a network interface;

a processing module operable to;

generate integrity data regarding one or more data slices of one or more pluralities of data slices, wherein a plurality of data slices of the one or more pluralities of data slices corresponds to a data segment;

transmit, via the network interface, the integrity data;

receive, via the network interface, a request for naming information for the one or more data slices;

retrieve the naming information for the one or more data slices in response to receiving the request; and

transmit, via the network interface, the naming information for the one or more data slices.

23. The DS storage unit of claim 22, wherein the processing module further functions to generate integrity data by at least one of:

generating the integrity data in response to a request; and

generating the integrity data in accordance with a predetermined pattern.

24. The DS storage unit of claim 22, wherein the processing module further functions to:

receive updated naming information for a data slice of the one or more data slices; and

store the updated naming information.

25. The DS storage unit of claim 22, wherein the processing module further functions to transmit the integrity data by:

transmitting the integrity data via a network to a computing core.

26. The DS storage unit of claim 22, wherein the naming information comprises:

at least a portion of a slice name, wherein the slice name includes a source name.