Monitoring inter-site bandwidth for rebuilding

• US 10,277,490 B2
• Filed: 07/19/2016
• Issued: 04/30/2019
• Est. Priority Date: 07/19/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Iljazi, Ilir
• Primary Examiner(s): Donabed, Ninos
• Application Number:
  US15/213,432
• Publication Number:
  US 20180026865A1
• Time in Patent Office: 1,015 Days
• CPC Class Codes: H04L 41/0686 Additional information in the notification, e.g. enhancement of specific meta-data
• CPC Class Codes: H04L 43/0829 Packet loss
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/0894 Packet rate
• CPC Class Codes: H04L 43/16 Threshold monitoring

Abstract

A dispersed storage network (DSN) includes a DSN memory employing multiple distributed storage (DS) units, e.g., memory devices, operating at multiple different physical sites, with each site having one or more of the memory devices. A monitoring entity can monitor the sites to determine a data loss rate and a threshold communication bandwidth, e.g., a bandwidth available for rebuilding encoded data slices, associated with particular sites. If the data loss rate of a particular site exceeds the threshold communication bandwidth, the monitoring unit can send an alert message notifying, for example, a DSN controller, an integrity processing unit, the DSN memory, about the determination, so that corrective action can be taken.

Claims

1. A method for use in a dispersed storage network (DSN), the method comprising:

monitoring a plurality physical sites communicatively coupled via a data link having a data link bandwidth, each of the plurality of physical sites hosting a plurality of distributed storage (DS) units hosted by a DSN memory;

determining a data loss rate associated with a particular physical site of the plurality of physical sites, wherein the data loss rate associated with the particular physical site includes data loss attributable to the plurality of DS units hosted at the particular physical site;

determining a threshold communication bandwidth associated with the particular physical site, the threshold communication bandwidth indicating a maximum portion of the data link bandwidth allocated to the particular physical site for slice rebuilding;

determining that the data loss rate associated with the particular individual physical site exceeds the threshold communication bandwidth associated with the particular physical site, wherein determining the data loss rate includes;

determining an average DS unit failure rate for DS units operating at the particular physical site;

multiplying capacities of the DS units operating at the particular physical site by the average DS unit failure rate; and

transmitting an alert message in response to determining that the data loss rate exceeds the threshold communication bandwidth.

2. The method of claim 1, wherein the alert message includes a request to augment a communication bandwidth of the particular physical site by reallocating a portion of the data link bandwidth otherwise allocated to another physical site.

3. The method of claim 1, wherein determining the data loss rate includes determining a number of DS units operating at the particular physical site.

4. The method of claim 1, further comprising:

transmitting the alert message from an external monitoring device to the DSN memory in response to a change in storage capacity of the particular physical site.

5. The method of claim 1, wherein the alert message indicates that the data loss rate associated with the particular physical site exceeds an amount of bandwidth available for rebuilding data stored in the DS units operating at the particular physical site.

6. A processing system configured to implement a managing unit, the managing unit comprising:

a computing core including associated memory;

a network interface configured to couple the managing unit to a dispersed storage network (DSN) including one or more DSN memory devices;

the computing core configured to;

monitor a plurality physical sites communicatively coupled via a data link having a data link bandwidth, each of the plurality of physical sites hosting a plurality of distributed storage (DS) units hosted by a particular DSN memory device;

determine a data loss rate associated with a particular physical site, wherein the data loss rate associated with the particular physical site includes data loss attributable to the plurality of DS units hosted at the particular physical site, wherein determining the data loss rate includes;

determining an average DS unit failure rate for DS units operating at the particular physical site;

multiplying capacities of the DS units operating at the particular physical site by the average DS unit failure rate;

compare the data loss rate associated with a particular physical site to a threshold communication bandwidth associated with the particular physical site, the threshold communication bandwidth indicating a maximum portion of the data link bandwidth allocated to the particular physical site for slice rebuilding; and

transmit an alert message in response to determining that the data loss rate exceeds the threshold communication bandwidth.

7. The processing system of claim 6, wherein the alert message includes a request to augment the threshold communication bandwidth by increasing the data link bandwidth.

8. The processing system of claim 6, wherein determining the data loss rate includes determining a number of DS units at the particular physical site.

9. The processing system of claim 6, further comprising:

transmitting the alert message in response to an increase in either or both of input and output requests to the plurality of DS units at the particular physical site.

10. The processing system of claim 6, wherein alert message indicates that the data loss rate associated with the particular physical site exceeds an amount of bandwidth available for rebuilding data stored in the DS units at the particular physical site.

11. A dispersed storage network (DSN) comprising:

a DSN memory including a plurality of memory devices located at a particular physical site;

at least one processing core implementing an integrity processing unit configured to rebuild encoded data slices;

at least one processing core implementing a managing unit configured to;

monitor a plurality physical sites communicatively coupled via a data link having a data link bandwidth, each of the plurality of physical sites hosting a plurality of distributed storage (DS) units hosted by a particular DSN memory device;

determine a data loss rate associated with a particular physical site, wherein the data loss rate associated with the particular physical site includes data loss attributable to the plurality of DS units hosted at the particular physical site, wherein determining the data loss rate includes;

determining an average DS unit failure rate for DS units operating at the particular physical site;

multiplying capacities of the DS units operating at the particular physical site by the average DS unit failure rate;

compare the data loss rate associated with a particular physical site to a threshold communication bandwidth associated with the particular physical site, the threshold communication bandwidth indicating a maximum portion of the data link bandwidth allocated to the particular physical site for slice rebuilding; and

transmit an alert message in response to determining that the data loss rate exceeds the threshold communication bandwidth.

12. The dispersed storage network of claim 11, wherein the alert message includes a request to augment the threshold communication bandwidth.

13. The dispersed storage network of claim 11, wherein determining the data loss rate includes determining a number of memory devices operating at the particular physical site.

14. The dispersed storage network of claim 11, further comprising:

transmitting the alert message in response to an increase in a failure rate associated with the plurality of memory devices.