Automatic rebalancing of a data storage system

• US 20070016726A1
• Filed: 07/15/2005
• Published: 01/18/2007
• Est. Priority Date: 07/15/2005
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: XIV Ltd. (International Business Machines Corporation)
• Inventors: Helman, Haim, Zohar, Ofir, Schwartz, Shemer, Revah, Yaron, Cohen, Dror
• Granted Patent: US 7,836,250 B2
• Time in Patent Office: Days
• US Class Current: 711/114
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 2003/0697 device management, e.g. handlers, drivers, I/O schedulers
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 2201/885 Monitoring specific for caches
• CPC Class Codes: G06F 2206/1012 Load balancing
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A method for operating a data storage system that includes a plurality of mass storage devices, which are configured to store data redundantly, the method including determining a characteristic service level of one or more of the mass storage devices and defining a reduced service level, which is less than the characteristic service level. The method further includes performing the following steps automatically: detecting the reduced service level on a first mass storage device in the data storage system; determining that data on the first mass storage device is stored redundantly on a second mass storage device in the data storage system; and in response to detecting the reduced service level, diverting an input/output (IO) request for the data directed to the first mass storage device to the second mass storage device, while operating the first mass storage device at the reduced service level.

Claims

1. A method for operating a data storage system that includes a plurality of mass storage devices, which are configured to store data redundantly, the method comprising:

determining a characteristic service level of one or more of the mass storage devices;

defining a reduced service level, which is less than the characteristic service level; and

performing the following steps automatically, without intervention by a human operator;

detecting the reduced service level on a first mass storage device in the data storage system;

determining that data on the first mass storage device is stored redundantly on a second mass storage device in the data storage system; and

in response to detecting the reduced service level, diverting an input/output (IO) request for the data directed to the first mass storage device to the second mass storage device, while operating the first mass storage device at the reduced service level.

2. The method according to claim 1, wherein the characteristic service level is determined with respect to a rate of completion of IO requests.

3. The method according to claim 1, wherein the reduced service level is a preset fraction of the characteristic service level.

4. The method according of claim 1, and comprising:

performing iteratively the following steps;

checking after diverting the IO request if the reduced service level exists; and

if the reduced service level does exist, diverting a subsequent IO request for the data directed to the first mass storage device to the second mass storage device, and if the reduced service level does not exist, servicing the further IO request at the first mass storage device.

5. The method according of claim 1, and comprising copying the data to a third mass storage device different from the first and the second devices.

6. The method according to claim 5, and comprising conveying a subsequent IO request for the data to the third mass storage device.

7. The method according to claim 1, wherein IO requests to the plurality of mass storage devices are stored in respective IO request queues for the devices prior to being serviced therein, and wherein detecting the reduced service level comprises determining that an imbalance exists in a length of one of the IO request queues compared to the other IO request queues.

8. The method according to claim 1, wherein the plurality of mass storage devices comprise sub-groups of mass storage devices, each sub-group having a respective cache communicating with its sub-group, each respective cache comprising IO request queues corresponding to the sub-group of mass storage devices communicating with the cache, each IO request queue storing IO requests for its device prior to the IO requests being serviced therein, and wherein detecting the reduced service level comprises determining that an imbalance exists in a length of one of the IO request queues of one of the respective caches compared to the other IO request queues of the cache.

9. The method according to claim 1, wherein the IO request is stored in one of an IO read request queue and an IO write request queue, and wherein detecting the reduced service level comprises detecting an imbalance in a length of at least one of the read and write request queues.

10. The method according to claim 1, wherein diverting the IO request comprises selecting one of a plurality of corrective actions, each of which actions divert the IO request.

11. The method according to claim 10, and comprising assigning a respective priority to each of the corrective actions prior to selecting the one corrective action, and wherein selecting the one corrective action comprises selecting the one corrective action according to the respective priorities.

12. The method according to claim 1, wherein the reduced service level comprises a transient reduction in service level.

13. The method according to claim 1, wherein the reduced service level comprises an invariant reduction in service level.

14. Apparatus for operating a data storage system, comprising:

a plurality of mass storage devices which are configured to store data redundantly; and

a system manager, which is adapted to;

determine a characteristic service level of one or more of the mass storage devices,

define a reduced service level, which is less than the characteristic service level, and

perform the following steps automatically, without intervention by a human operator;

detect the reduced service level on a first mass storage device in the data storage system,

determine that data on the first mass storage device is stored redundantly on a second mass storage device in the data storage system, and

in response to detecting the reduced service level, divert an input/output (IO) request for the data directed to the first mass storage device to the second mass storage device, while operating the first mass storage device at the reduced service level.

15. The apparatus according to claim 14, wherein the characteristic service level is determined with respect to a rate of completion of IO requests.

16. The apparatus according to claim 14, wherein the reduced service level is a preset fraction of the characteristic service level.

17. The apparatus according to claim 14, and comprising the system manager being adapted to iteratively perform the following steps:

checking after diverting the IO request if the reduced service level exists; and

if the reduced service level does exist, diverting a subsequent IO request for the data directed to the first mass storage device to the second mass storage device, and if the reduced service level does not exist, servicing the further IO request at the first mass storage device.

18. The apparatus according to claim 14, wherein the system manager is adapted to copy the data to a third mass storage device different from the first and the second devices.

19. The apparatus according to claim 18, and comprising the system manager conveying a subsequent IO request for the data to the third mass storage device.

20. The apparatus according to claim 14, wherein IO requests to the plurality of mass storage devices are stored in respective IO request queues for the devices prior to being serviced therein, and wherein the system manager detecting the reduced service level comprises the system manager determining that an imbalance exists in a length of one of the IO request queues compared to the other IO request queues.

21. The apparatus according to claim 14, wherein the plurality of mass storage devices comprise sub-groups of mass storage devices, each sub-group having a respective cache communicating with its sub-group, each respective cache comprising IO request queues corresponding to the sub-group of mass storage devices communicating with the cache, each IO request queue storing IO requests for its device prior to the IO requests being serviced therein, and wherein the system manager detecting the reduced service level comprises one of the caches determining that an imbalance exists in a length of one of the IO request queues of the cache compared to the other IO request queues of the cache.

22. The apparatus according to claim 14, wherein the IO request is stored in one of an IO read request queue and an IO write request queue, and wherein detecting the reduced service level comprises detecting an imbalance in a length of at least one of the read and write request queues.

23. The apparatus according to claim 14, wherein diverting the IO request comprises selecting one of a plurality of corrective actions, each of which actions divert the IO request.

24. The apparatus according to claim 23, and comprising assigning a respective priority to each of the corrective actions prior to selecting the one corrective action, and wherein selecting the one corrective action comprises selecting the one corrective action according to the respective priorities.

25. The apparatus according to claim 14, wherein the reduced service level comprises a transient reduction in service level.

26. The apparatus according to claim 14, wherein the reduced service level comprises an invariant reduction in service level.