Automatic rebalancing of a data storage system

• US 7,836,250 B2
• Filed: 07/15/2005
• Issued: 11/16/2010
• Est. Priority Date: 07/15/2005
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cohen, Dror, Helman, Haim, Zohar, Ofir, Schwartz, Shemer, Revah, Yaron
• Primary Examiner(s): Kim; Matt
• Assistant Examiner(s): Alsip; Michael
• Application Number:
  US11/182,338
• Publication Number:
  US 20070016726A1
• Time in Patent Office: 1,950 Days
• Field of Search: 711/114
• US Class Current: 711/114
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 2003/0697 device management, e.g. handlers, drivers, I/O schedulers
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 2201/885 Monitoring specific for caches
• CPC Class Codes: G06F 2206/1012 Load balancing
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A method for operating a data storage system that includes a plurality of mass storage devices, which are configured to store data redundantly, the method including determining a characteristic service level of one or more of the mass storage devices and defining a reduced service level, which is less than the characteristic service level. The method further includes performing the following steps automatically: detecting the reduced service level on a first mass storage device in the data storage system; determining that data on the first mass storage device is stored redundantly on a second mass storage device in the data storage system; and in response to detecting the reduced service level, diverting an input/output (IO) request for the data directed to the first mass storage device to the second mass storage device, while operating the first mass storage device at the reduced service level.

Claims

1. A method for operating a data storage system that includes a plurality of mass storage memory disks that are configured to store data redundantly, the method comprising:

storing in the data storage system a first mapping between a given logical address, a primary mass storage memory disk, and a secondary mass storage memory disk;

storing in the primary mass storage memory disk a second mapping between the given logical address, a physical address within the primary mass storage memory disk, and the secondary mass storage memory disk;

configuring the data storage system to use the first mapping to direct a read request for data stored at the given logical address to the primary mass storage memory disk and not to the secondary mass storage memory disk;

determining a characteristic service level of one or more of the mass storage memory disk;

defining a reduced service level, which is less than the characteristic service level; and

performing the following steps automatically, without intervention by a human operator;

detecting the reduced service level on the primary mass storage memory disk in the data storage system;

in response to detecting the reduced service level, rebalancing read requests using the second mapping to divert the read request for the data from the primary mass storage memory disk to the secondary mass storage memory disk, while servicing previously received read requests, previously received write requests, and currently received write requests in the primary mass storage memory disk at the reduced service level;

wherein for each mass storage memory disk, write requests are stored in a first queue only including write requests and read requests are stored in a second queue only including read requests;

wherein detecting the reduced service level comprises determining that the second queue for the primary mass storage memory disk includes an amount of read requests greater than a predetermined amount of read requests.

2. The method according to claim 1, wherein the characteristic service level is determined with respect to a rate of completion of read and write requests.

3. The method according to claim 1, wherein the reduced service level is a preset fraction of the characteristic service level.

4. The method according to claim 1, further comprising:

performing iteratively the following steps;

checking, after diverting the read request, if the reduced service level exists; and

if the reduced service level exists, diverting a subsequent read request for the data directed to the primary mass storage memory disk to the secondary mass storage memory disk, and if the reduced service level does not exist, servicing the further read request at the primary mass storage memory disk.

5. The method according to claim 1, further comprising copying the data to a third mass storage memory disk different from the primary and the secondary mass storage memory disks.

6. The method according to claim 5, further comprising conveying a subsequent request for the data to the third mass storage memory disk.

7. The method according to claim 1, wherein the plurality of mass storage memory disks comprise sub-groups of mass storage memory disks, each sub-group including a respective cache communicating with its sub-group, each respective cache comprising the first and second queues corresponding to the sub-group of mass storage memory disks communicating with the cache.

8. The method according to claim 1, wherein diverting the read request comprises selecting one of a plurality of corrective actions, each of which actions diverts the read request.

9. The method according to claim 8, further comprising assigning a respective priority to each of the corrective actions prior to selecting the one corrective action, and wherein selecting the one corrective action comprises selecting the one corrective action according to the respective priorities.

10. The method according to claim 1, wherein the reduced service level comprises a transient reduction in service level.

11. The method according to claim 1, wherein the reduced service level comprises an invariant reduction in service level.

12. The method according to claim 1, wherein the queues are managed according to a First In First Out (FIFO) scheme.

13. The method according to claim 1, wherein the predetermined quantity is a second length of the second queue of read requests.

14. The method according to claim 1, wherein the plurality of mass storage memory disks comprise respective first queues including at all times exclusively of write requests, and wherein the predetermined quantity is an average length of the respective first queues.

15. The method according to claim 1, further comprising providing an indication showing that the read request is being diverted to the secondary mass storage memory disk from the primary mass storage memory disk, and using the indication to prevent the read request being returned to the primary mass storage memory disk.

16. A method for operating a data storage system that includes a plurality of mass storage memory disks that are configured to store data redundantly, the method comprising:

storing in the data storage system a first mapping between a given logical address, a primary mass storage memory disk, and a secondary mass storage memory disk;

storing in the primary mass storage memory disk a second mapping between the given logical address, a physical address within the primary mass storage memory disk, and the secondary mass storage memory disk;

configuring the data storage system to use the first mapping to direct a read request for data stored at the given logical address to the primary mass storage disk and not to the secondary mass storage memory disk;

determining a characteristic service level of one or more of the mass storage memory disk;

defining a reduced service level, which is less than the characteristic service level; and

performing the following steps automatically, without intervention by a human operator;

detecting the reduced service level on the primary mass storage memory disk in the data storage system;

in response to detecting the reduced service level, copying the data to a different mass storage memory disk from the primary mass storage memory disk and the secondary mass storage memory disk, and diverting the read request to the different mass storage memory disk, while servicing previously received read requests, previously received write requests, and currently received write requests in the primary mass storage memory disk at the reduced service level;

wherein for each mass storage memory disk, write requests are stored in a first queue only including write requests and read requests are stored in a second queue only including read requests;

wherein detecting the reduced service level comprises determining that an imbalance exists in a first length of the first queue of write requests of the primary mass storage memory disk compared to a predetermined quantity.

17. A method for operating a data storage system that includes a plurality of mass storage memory disks that are configured to store data redundantly, the method comprising:

storing in the data storage system a first mapping between a given logical address, a primary mass storage memory disk, and a secondary mass storage memory disk;

storing in the primary mass storage disk a second mapping between the given logical address, a physical address within the primary mass storage memory disk, and the secondary mass storage memory disk;

configuring the data storage system to use the first mapping to direct a read request for data stored at the given logical address to the primary mass storage memory disk and not to the secondary mass storage memory disk;

determining a characteristic service level of one or more of the mass storage memory disks;

defining a reduced service level, which is less than the characteristic service level; and

performing the following steps automatically, without intervention by a human operator;

detecting the reduced service level on the primary mass storage memory disk in the data storage system;

in response to detecting the reduced service level, altering the first mapping to a reconfigured mapping, and using the reconfigured mapping to direct the read request for data stored at the given logical address to a different mass storage memory disk from the primary mass storage memory disk and not to the primary mass storage memory disk while servicing previously received read requests, previously received write requests, and currently received write requests in the primary mass storage memory disk at the reduced service level;

wherein for each mass storage memory disk, write requests are stored in a first queue only including write requests and read requests are stored in a second queue only including read requests;

wherein detecting the reduced service level comprises determining that an imbalance exists in a first length of the first queue of write requests of the primary mass storage memory disk compared to a predetermined quantity.