Smart Load Balancing Replication When Adding or Removing Storage Disks in a Distributed Storage System

• US 20160321008A1
• Filed: 04/29/2015
• Published: 11/03/2016
• Est. Priority Date: 04/29/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Wu, Jie P., Zhang, Xiao D., Fang, Liang, Zhang, Jun W.
• Granted Patent: US 9,569,136 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A mechanism is provided for balancing workload to one or more storage disks in a plurality of storage disks during redistribution or replication associated with adding or removing a storage disk to the plurality of storage disks. Historical information in collected information from the plurality of storage disks is analyzed to identify I/O operation patterns on a per storage disk level. An average amount of I/O operations that occur within each storage disk for a given time period are identified. For each storage disk that is impacted, a disk on/off-hoarding plan is generated that identifies a subset of I/O operations from a set of I/O operations to execute in the given time period using the average amount of I/O operations that historically occur within the storage disk that is impacted during the given time period. The subset of I/O operations are then executed in the given time period.

Claims

1. A method, in a data processing system, for balancing workload to one or more storage disks in a plurality of storage disks during redistribution or replication associated with adding or removing a storage disk to the plurality of storage disks, the method comprising:

utilizing collected information for the plurality of storage disks, analyzing historical information in the collected information to identify I/O operation patterns on a per storage disk level;

determining an average amount of I/O operations that occur within each storage disk in the plurality of storage disks for a given time period; and

for each storage disk that is impacted based on the adding or removing of the storage disk to the plurality of storage disks;

generating a disk on/off-boarding plan that identifies a subset of I/O operations from a set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in the given time period using the average amount of I/O operations that historically occur within the storage disk that is impacted during the given time period; and

executing the subset of I/O operations in the given time period.

2. The method of claim 1, further comprising:

executing external I/O operations from one or more applications alongside the set of I/O operations during the given time period.

3. The method of claim 1, wherein the set of I/O operations comprises level 1 object operations (Level1IO) that are urgent to be moved or replicated and level 2 object operations (Level2IO) that are lower priority for replication or migration.

4. The method of claim 3, wherein the level 1 object operations (Level1IO) that are part of the set of I/O operations for the given time period T_i are identified using:

Level1IO_Ti=max(Weightlevel1*MaxBandwidth_Ti, MaxBandwidth_Ti−

DeltaIO_Ti-1−

AvgIO_Ti)

where Weightlevel1 is a configurable number, AvgIO_Ti is the historical average I/O operations performed on the storage disk that is impacted during the given time period T_i, MaxBandwidth_Ti is the maximum workload bandwidth of the storage disk that is impacted during the time period T_i, and DeltaIO_Ti-1 are the delta external I/O operations that did not finish in a previous time period on the storage disk that is impacted.

5. The method of claim 3, wherein the level 2 object operations (Level2IO) that are part of the set of I/O operations for the given time period T_i are identified using:

Level2IO_Ti=max(Weightlevel2*MaxBandwidth_Ti, MaxBandwidth_Ti−

Level1IO_Ti−

DeltaIO_Ti-1−

AvgIO_Ti)

where Weightlevel2 is a configurable number, AvgIO_Ti is the historical average I/O operations performed on the storage disk that is impacted during the given time period T_i, MaxBandwidth_Ti is the maximum workload bandwidth of the storage disk that is impacted during the time period T_i, DeltaIO_Ti-1 are the delta external I/O operations that did not finish in a previous time period on the storage disk that is impacted, and Level1IO_Ti are the level 1 operations that are to be performed in the given time period.

6. The method of claim 1, further comprising:

for each storage disk that is impacted based on the adding or removing of the storage disk to the plurality of storage disks;

responsive to a failure to complete all of the set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in the given time period, generating a revised disk on/off-boarding plan that identifies another subset of I/O operations from a set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in a next time period using the average amount of I/O operations that occur historically within the storage disk that is impacted during the next time period; and

executing the other subset of I/O operations in the given time period.

7. The method of claim 1, wherein the collected information comprises one or more of whether an I/O operation is from an external user of an internal component, whether the I/O operation is a read I/O operation or a write I/O operation;

a time that the I/O operation was initialized;

a length of time required to execute the I/O operation;

or a type of storage disk upon which the I/O operation is being performed on a per storage disk basis for the plurality of storage disks.

8. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

utilizing collected information for a plurality of storage disks, analyze historical information in the collected information to identify I/O operation patterns on a per storage disk level;

determine an average amount of I/O operations that occur within each storage disk in the plurality of storage disks for a given time period; and

for each storage disk that is impacted based on the adding or removing of the storage disk to the plurality of storage disks;

generate a disk on/off-boarding plan that identifies a subset of I/O operations from a set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in the given time period using the average amount of I/O operations that historically occur within the storage disk that is impacted during the given time period; and

execute the subset of I/O operations in the given time period.

9. The computer program product of claim 8, wherein the computer readable program further causes the computing device to:

execute external I/O operations from one or more applications alongside the set of operations during the given time period.

10. The computer program product of claim 8, wherein the set of I/O operations comprises level 1 object operations (Level1IO) that are urgent to be moved or replicated and level 2 object operations (Level2IO) that are lower priority for replication or migration.

11. The computer program product of claim 10, wherein the level 1 object operations (Level1IO) that are part of the set of I/O operations for the given time period T_i are identified using:

Level1IO_Ti=max(Weightlevel1*MaxBandwidth_Ti, MaxBandwidth_Ti−

DeltaIO_Ti-1−

AvgIO_Ti)

where Weightlevel1 is a configurable number, AvgIO_Ti is the historical average I/O operations performed on the storage disk that is impacted during the given time period T_i, MaxBandwidth_Ti is the maximum workload bandwidth of the storage disk that is impacted during the time period T_i, and DeltaIO_Ti-1 are the delta external I/O operations that did not finish in a previous time period on the storage disk that is impacted.

12. The computer program product of claim 10, wherein the level 2 object operations (Level2IO) that are part of the set of I/O operations for the given time period T_i are identified using:

Level2IO_Ti=max(Weightlevel2*MaxBandwidth_Ti, MaxBandwidth_Ti−

Level1IO_Ti−

DeltaIO_Ti-1−

AvgIO_Ti)

where Weightlevel2 is a configurable number, AvgIO_Ti is the historical average I/O operations performed on the storage disk that is impacted during the given time period T_i, MaxBandwidth_Ti is the maximum workload bandwidth of the storage disk that is impacted during the time period T_i, DeltaIO_Ti-1 are the delta external I/O operations that did not finish in a previous time period on the storage disk that is impacted, and Level1IO_Ti are the level 1 operations that are to be performed in the given time period.

13. The computer program product of claim 8, wherein the computer readable program further causes the computing device to:

for each storage disk that is impacted based on the adding or removing of the storage disk to the plurality of storage disks;

responsive to a failure to complete all of the set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in the given time period, generate a revised disk on/off-boarding plan that identifies another subset of I/O operations from a set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in a next time period using the average amount of I/O operations that occur historically within the storage disk that is impacted during the next time period; and

execute the other subset of I/O operations in the given time period.

14. The computer program product of claim 8, wherein the collected information comprises one or more of whether an I/O operation is from an external user of an internal component, whether the I/O operation is a read I/O operation or a write I/O operation;

a time that the I/O operation was initialized;

a length of time required to execute the I/O operation;

or a type of storage disk upon which the I/O operation is being performed on a per storage disk basis for the plurality of storage disks.

15. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

utilizing collected information for a plurality of storage disks, analyze historical information in the collected information to identify I/O operation patterns on a per storage disk level;

determine an average amount of I/O operations that occur within each storage disk in the plurality of storage disks for a given time period; and

for each storage disk that is impacted based on the adding or removing of the storage disk to the plurality of storage disks;

generate a disk on/off-boarding plan that identifies a subset of I/O operations from a set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in the given time period using the average amount of I/O operations that historically occur within the storage disk that is impacted during the given time period; and

execute the subset of I/O operations in the given time period.

16. The apparatus of claim 15, wherein the instructions further cause the processor to:

execute external I/O operations from one or more applications alongside the set of I/O operations during the given time period.

17. The apparatus of claim 15, wherein the set of I/O operations comprises level 1 object operations (Level1IO) that are urgent to be moved or replicated and level 2 object operations (Level2IO) that are lower priority for replication or migration.

18. The apparatus of claim 17, wherein the level 1 object operations (Level1IO) that are part of the set of I/O operations for the given time period T_i are identified using:

Level1IO_Ti=max(Weightlevel1*MaxBandwidth_Ti, MaxBandwidth_Ti−

DeltaIO_Ti-1−

AvgIO_Ti)

where Weightlevel1 is a configurable number, AvgIO_Ti is the historical average I/O operations performed on the storage disk that is impacted during the given time period T_i, MaxBandwidth_Ti is the maximum workload bandwidth of the storage disk that is impacted during the time period T_i, and DeltaIO_Ti-1 are the delta external I/O operations that did not finish in a previous time period on the storage disk that is impacted.

19. The apparatus of claim 17, wherein the level 2 object operations (Level2IO) that are part of the set of I/O operations for the given time period T_i are identified using:

Level2IO_Ti=max(Weightlevel2*MaxBandwidth_Ti, MaxBandwidth_Ti−

Level1IO_Ti−

DeltaIO_Ti-1−

AvgIO_Ti)

where Weightlevel2 is a configurable number, AvgIO_Ti is the historical average I/O operations performed on the storage disk that is impacted during the given time period T_i, MaxBandwidth_Ti is the maximum workload bandwidth of the storage disk that is impacted during the time period T_i, DeltaIO_Ti-1 are the delta external I/O operations that did not finish in a previous time period on the storage disk that is impacted, and Level1IO_Ti are the level 1 operations that are to be performed in the given time period.

20. The apparatus of claim 15, wherein the instructions further cause the processor to:

for each storage disk that is impacted based on the adding or removing of the storage disk to the plurality of storage disks;

responsive to a failure to complete all of the set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in the given time period, generate a revised disk on/off-boarding plan that identifies another subset of I/O operations from a set of I/O operations associated with the adding or removing of the storage disk to the plurality of storage disks to execute in a next time period using the average amount of I/O operations that occur historically within the storage disk that is impacted during the next time period; and

execute the other subset of I/O operations in the given time period.