Resolving a performance issue within a dispersed storage network

• US 10,447,767 B2
• Filed: 06/18/2013
• Issued: 10/15/2019
• Est. Priority Date: 04/26/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Baptist, Andrew, Leggette, Wesley, Resch, Jason K.
• Primary Examiner(s): Gillis, Brian J.
• Assistant Examiner(s): Guzman, Javier O
• Application Number:
  US13/920,635
• Publication Number:
  US 20130282865A1
• Time in Patent Office: 2,310 Days
• Field of Search: 709217
• CPC Class Codes: H04L 1/24 Testing correct operation
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/123 received data contents, e.g. message integrity
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services

Abstract

A method begins by a processing module identifying one or more devices of a dispersed storage network (DSN) potentially contributing to a DSN performance issue. For a device of the identified one or more devices, the method continues where the processing module determines a potential performance issue of the device and determines a performance test based on the potential performance issue. The method continues where the processing module issues a message to the device that includes test information specific for the device to execute the performance test and receives a response message that includes a specific test result data generated based on the test information. The method continues where the processing module determines, based on the specific test result data, whether the device has the potential performance issue and is contributing to the DSN performance issue.

Claims

1. A method for execution by one or more processing modules of one or more requesting devices of a dispersed storage network (DSN), the method comprises:

when a DSN performance issue is detected that will affect a data access request, identifying a distributed storage unit (SU) of a set of distributed SUs of the DSN potentially contributing to the DSN performance issue, wherein a requesting device of the DSN sends a set of messages regarding the data access request to the set of SUs; and

for the SU;

determining a potential performance issue of the SU based on how the SU is potentially contributing to the DSN performance issue;

determining a performance test based on the potential performance issue;

generating a message that includes a protocol header and a payload, wherein the protocol header includes an indication to identify the message as a test request message and the payload includes test information specific for the SU to execute the performance test;

sending the message to the SU;

receiving, from the SU, a response message that includes a response header and a response payload, wherein the response header includes an indication to identify the response message as a test response message and the response payload includes a specific test result data generated based on the test information; and

determining, based on the specific test result data, whether the SU has the potential performance issue and is contributing to the DSN performance issue.

2. The method of claim 1 further comprises:

for a second SU of the identified set of SUs;

determining a second potential performance issue of the second SU based on how the second SU is potentially contributing to the DSN performance issue;

determining a second performance test based on the second potential performance issue;

generating a second message that includes a second protocol header and a second payload, wherein the second protocol header includes an indication to identify the second message as another test request message and the second payload includes second test information specific for the second SU to execute the second performance test;

sending the second message to the second SU;

receiving, from the second SU, a second response message that includes a second response header and a second response payload, wherein the second response header includes an indication to identify the second response message as another test response message and the second response payload includes a second specific test result data generated based on the second test information; and

determining, based on the second specific test result data, whether the second SU has the second potential performance issue and is contributing to the DSN performance issue.

3. The method of claim 2 further comprises:

when the potential performance issue and the second potential performance issue are not individually contributing to the DSN performance issue, determining whether a combination of the potential performance issue and the second potential performance issues are collectively contributing to the DSN performance issue.

4. The method of claim 1 further comprises:

detecting the DSN performance issue based on one or more of;

detecting a communication error to, from, or between one or more SUs of the set of SUs;

detecting a storage error indication regarding the one or more SUs;

detecting a reading error indication regarding the one or more SUs;

in response to performing a general test in accordance with a DSN test schedule;

detecting a reliability issue regarding the one or more SUs; and

receiving a potential failing performance indicator regarding the one or more SUs.

5. The method of claim 1, wherein the identifying the SU comprises one or more of:

when the DSN performance issue is based on a communication error, identifying the SU based on a failure of the SU to send a communication or to acknowledge receipt of the communication;

when the DSN performance issue is based on a storage error, identifying the SU based on a failure to provide confirmation of successfully completing a storage request;

when the DSN performance issue is based on a read error, identifying the SU based on a failure to provide retrieved data in accordance with a read request;

receiving a message identifying the SU as having a potential reliability issue; and

receiving a message identifying the SU as having the potential performance issue.

6. The method of claim 1, wherein the requesting device comprises one of:

a DS processing unit;

a DS processing module;

a DS managing module;

a DS rebuilding module; and

a user device.

7. The method of claim 1, wherein the generating the message further comprises:

generating the protocol header to include;

an operation code field to identify the message as a test message;

a request/response field to identify the message as a request message;

a request number field to identify the message as a particular test request message; and

a payload length field.

8. The method of claim 1, wherein the generating the message further comprises at least one of:

generating the payload to test data transfer speed, wherein the payload includes;

a response length field to indicate a desired length of the response message or portion thereof; and

the test information specific for the SU that includes an indication of the data transfer speed test and minimal data for transferring between the SU and the one or more requesting devices to test speed;

generating the payload to test data transfer bandwidth, wherein the payload includes;

the response length field to indicate the desired length of the response message or the portion thereof; and

the test information specific for the SU that includes an indication of the data transfer bandwidth test and a large amount of data for transferring between the SU and the one or more requesting devices to test bandwidth;

generating the payload to test data transfer latency, wherein the payload includes;

the response length field to indicate the desired length of the response message or the portion thereof; and

the test information specific for the SU that includes an indication of the data transfer latency test and data for transferring between the SU and the one or more requesting devices to test latency; and

generating the payload to test error rate of data transfer, wherein the payload includes;

the response length field to indicate the desired length of the response message or the portion thereof; and

the test information specific for the SU that includes an indication of the data transfer error rate test and control data for transferring between the SU and the one or more requesting devices to test error rate.

9. A computer of a dispersed storage network (DSN) comprises:

an interface;

a memory; and

a processing module operably coupled to the interface and to the memory, where the processing module is operable to;

when a DSN performance issue is detected that will affect a data access request, identify a distributed storage units (SUs) of a set of SUs of the DSN potentially contributing to the DSN performance issue, wherein a requesting device of the DSN sends a set of messages regarding the data access request to the set of storage units; and

for the SU;

determine a potential performance issue of the SU based on how the SU is potentially contributing to the DSN performance issue;

determine a performance test based on the potential performance issue;

generate a message that includes a protocol header and a payload, wherein the protocol header includes an indication to identify the message as a test request message and the payload includes test information specific for the SU to execute the performance test;

send, via the interface, the message to the SU;

receive, from the SU via the interface, a response message that includes a response header and a response payload, wherein the response header includes an indication to identify the response message as a test response message and the response payload includes a specific test result data generated based on the test information; and

determine, based on the specific test result data, whether the SU has the potential performance issue and is contributing to the DSN performance issue.

10. The computer of claim 9 further comprises:

the processing module is further operable to, for a second SU of the set of SUs;

determine a second potential performance issue of the second SU based on how the second SU is potentially contributing to the DSN performance issue;

determine a second performance test based on the second potential performance issue;

generate a second message that includes a second protocol header and a second payload, wherein the second protocol header includes an indication to identify the second message as another test request message and the second payload includes second test information specific for the second SU to execute the second performance test;

send, via the interface, the second message to the second SU;

receive, from the second SU via the interface, a second response message that includes a second response header and a second response payload, wherein the second response header includes an indication to identify the second response message as another test response message and the second response payload includes a second specific test result data generated based on the second test information; and

determine, based on the second specific test result data, whether the SU has the second potential performance issue and is contributing to the DSN performance issue.

11. The computer of claim 10 further comprises:

the processing module is further operable to, when the potential performance issue and the second potential performance issue are not individually contributing to the DSN performance issue, determine whether a combination of the potential performance issue and the second potential performance issue are collectively contributing to the DSN performance issue.

12. The computer of claim 9 further comprises:

the processing module is further operable to detect the DSN performance issue based on one or more of;

detecting a communication error to, from, or between one or more SUs of the set of SUs;

detecting a storage error indication regarding the one or more SUs of the set of SUs;

detecting a reading error indication regarding the one or more SUs of the set of SUs;

in response to performing a general test in accordance with a DSN test schedule;

detecting a reliability issue regarding the one or more responding devices; and

receiving, via the interface, a potential failing performance indicator regarding the one or more SUs of the set of SUs.

13. The computer of claim 9, wherein the processing module identifies the SU by one or more of:

when the DSN performance issue is based on a communication error, identifying the SU based on a failure to send a communication or to acknowledge receipt of the communication;

when the DSN performance issue is based on a storage error, identifying the SU based on a failure to provide confirmation of successfully completing a storage request;

when the DSN performance issue is based on a read error, identifying the SU based on a failure to provide retrieved data in accordance with a read request;

receiving, via the interface, a message identifying the SU as having a potential reliability issue; and

receiving, via the interface, a message identifying the SU as having the potential performance issue.

14. The computer of claim 9, wherein the requesting device comprises at least one of:

a DS processing unit;

a DS processing module;

a DS managing module;

a DS rebuilding module; and

a user device.

15. The computer of claim 9, wherein the processing module further functions to generate the message by:

generating the protocol header to include;

an operation code field to identify the message as a test message;

a request/response field to identify the message as a request message;

a request number field to identify the message as a particular test request message; and

a payload length field.

16. The computer of claim 9, wherein the processing module further functions to generate the message by at least one of:

generating the payload to test data transfer speed, wherein the payload includes;

a response length field to indicate a desired length of the response message or portion thereof; and

the test information specific for the SU that includes an indication of the data transfer speed test and minimal data for transferring between the SU and one or more requesting devices to test speed;

generating the payload to test data transfer bandwidth, wherein the payload includes;

the response length field to indicate the desired length of the response message or the portion thereof; and

the test information specific for the SU that includes an indication of the data transfer bandwidth test and a large amount of data for transferring between the SU and the one or more requesting devices to test bandwidth;

generating the payload to test data transfer latency, wherein the payload includes;

the response length field to indicate the desired length of the response message or the portion thereof; and

the test information specific for the SU that includes an indication of the data transfer latency test and data for transferring between the SU and the one or more requesting devices to test latency; and

generating the payload to test error rate of data transfer, wherein the payload includes;

the response length field to indicate the desired length of the response message or the portion thereof; and

the test information specific for the SU that includes an indication of the data transfer error rate test and control data for transferring between the SU and the one or more requesting devices to test error rate.