Storage system with capability to allocate virtual storage segments among a plurality of controllers

• US 7,305,520 B2
• Filed: 01/30/2004
• Issued: 12/04/2007
• Est. Priority Date: 01/30/2004
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Voigt, Doug, Woods, Hal
• Primary Examiner(s): Peugh; Brian R.
• Assistant Examiner(s): Rutz; Jared I
• Application Number:
  US10/769,380
• Publication Number:
  US 20050172097A1
• Time in Patent Office: 1,404 Days
• Field of Search: 711111-114, 711161-162, 718/105, 710/15, 709/226
• US Class Current: 711/112
• CPC Class Codes: G06F 11/201 between storage system components
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A storage system comprises a plurality of storage arrays configurable into a plurality of storage device groups, a plurality of controllers selectively coupled to ones of the storage arrays, a network fabric arranged as a linkage of multiple sets of associated controllers and storage devices, and a processor. The processor is capable of associating the plurality of storage device groups among controllers based on a distribution of performance demand according to controller processor utilization of the individual storage device groups and bandwidth utilization pro-ratable among storage device groups within a set.

Claims

1. A method of allocating control of virtual storage segments in a storage system among a plurality of controllers comprising:

arranging the storage system as multiple sets of associated controllers and storage devices that communicate with full cross-sectional bandwidth between controllers and associated storage devices internal to ones of the multiple sets and a reduced inter-set bandwidth in inter-set links between controllers and storage devices in different sets of the multiple sets;

determining performance demand among a plurality of storage device groups;

distributing performance demand values according to controller processor utilization of the individual storage device groups and inter-set bandwidth utilization; and

associating the plurality of storage device groups with controllers based on the performance demand value distributions.

2. The method according to claim 1 further comprising:

measuring performance demand including measurements of data transfer bandwidth and/or operations executed per second.

3. The method according to claim 1 further comprising:

estimating performance demand including enumerating performance demand as a proportion of per-storage device throughput and/or evaluating implemented upstream workload throttling parameters.

4. The method according to claim 1 further comprising:

establishing utilization guidelines for controller processor and inter-set communication bandwidth linking storage devices in different sets of the multiple sets.

5. The method according to claim 1 further comprising:

establishing utilization guidelines for controller processor and inter-set communication at a storage device level including guidelines for maximum utilization targets and/or equalization across controller processors.

6. The method according to claim 1 further comprising:

administering controller memory resources as virtual slots whereby disk group related control and data are placed; and

optimizing the allocation of controllers and storage device groups using three-dimensional bin-packing techniques with the dimensions defined as virtual slot utilization, controller processor utilization, and inter-set link utilization.

7. The method according to claim 1 further comprising:

optimizing the allocation of controllers and storage device groups using a custom heuristic algorithm.

8. The method according to claim 1 further comprising:

optimizing the allocation of controllers and storage device groups using selection from among a set of pre-planned controller to storage device group association patterns.

9. The method according to claim 1 further comprising;

determining whether a storage device group to controller allocation meets preselected criteria;

if so, allocating the storage devices groups and controllers; and

if not, indicating allocation failure and responding with one or more actions selected from among a group of actions including adding controllers, adding inter-set bandwidth, adding storage devices, and reconfiguring storage device groupings by moving data.

10. The method according to claim 1 further comprising:

optimizing the allocation of controllers and storage device groups;

determining whether the optimized allocation of storage device groups to controllers improves performance;

if so, allocating the storage devices groups and controllers; and

if not, indicating failure to improve allocation and responding with no action or one or more actions selected from among a group actions including adding controllers, adding inter-set bandwidth, adding storage devices, and reconfiguring storage device groupings by moving data.

11. The method according to claim 1 further comprising:

optimizing the allocation of controllers and storage device groups; and

planning a sequence of virtual storage segment movements that advance from a current state to the optimized state.

12. The method according to claim 1 further comprising:

moving storage device-related activity between controllers by transferring management information rather than data comprising;

identifying a storage device group to be moved from a first controller to a second controller;

eliminating data structures associated with the storage device group in the first controller;

queuing commands directed to the storage device group at a level sufficiently high to include both the first controller and the second controller;

waiting for commands directed to the storage device group to complete;

posting any remaining storage device group state information to the storage device group;

enabling both the first controller and the second controller to host the storage device group;

uploading the storage device group state information to the second controller;

when the storage device group state information is uploaded, disabling the first controller from hosting the storage device group; and

resuming delivery of commands to the storage device group.

13. A storage system comprising:

a plurality of storage arrays configurable into a plurality of storage device groups;

a plurality of controllers selectively coupled to ones of the storage arrays;

a network fabric arranged as a linkage of multiple sets of associated controllers and storage devices that communicate with full cross-sectional bandwidth between controllers and associated storage devices internal to ones of the multiple sets and a reduced inter-set bandwidth in inter-set links between controllers and storage devices in different sets of the multiple sets; and

a processor configured to associate the plurality of storage device groups among controllers based on a distribution of performance demand according to controller processor utilization of the individual storage device groups and inter-set bandwidth utilization.

14. The storage system according to claim 13 further comprising:

a process executable in the processor that determines performance demand among a plurality of storage device groups by measuring performance demand including measurements of data transfer bandwidth and/or operations executed per second.

15. The storage system according to claim 13 further comprising:

a process executable in the processor that determines performance demand among a plurality of storage device groups by estimating performance demand including enumerating performance demand as a proportion of per-storage device throughput and/or evaluating implemented upstream workload throttling parameters.

16. The storage system according to claim 13 further comprising:

a process executable in the processor that administers controller memory resources as virtual slots whereby disk group related control and data are placed; and

a process executable in the processor that optimizes the allocation of controllers and storage device groups using three-dimensional bin-packing techniques with the dimensions defined as virtual slot utilization, controller processor utilization, and inter-set link utilization.

17. The storage system according to claim 13 further comprising:

a process executable in the processor that optimizes the allocation of controllers and storage device groups using a custom heuristic algorithm.

18. The storage system according to claim 13 further comprising:

a process executable in the processor that optimizes the allocation of controllers and storage device groups using selection from among a set of pre-planned controller to storage device group association patterns.

19. The storage system according to claim 13 wherein:

the plurality of controllers are configured as a plurality of controller pairs with a storage array having a restrictive physical association with one controller pair.

20. The storage system according to claim 13 wherein:

the plurality of storage arrays are grouped for Redundant Array of Independent Disks (RAID) style redundancy algorithms with group membership determined according to algorithms resident in a controller that is currently in control of the group.

21. The storage system according to claim 13 further comprising:

controller memory resources administered as virtual slots for placement of group related control and data.

22. The storage system according to claim 21 further comprising:

battery-backed memories included in the virtual slots; and

a process that mirrors information in the battery-backed memories from a primary controller to a secondary controller to maintain continuous data access in conditions of a controller failure event without storage device group movement.

23. The storage system according to claim 21 wherein:

at least one virtual slot sufficiently large to accommodate the largest storage device group is available when no storage device group movement is in progress.

24. The storage system according to claim 13 wherein:

a controller'"'"'s internal bandwidth is sufficient to accommodate full utilization of all external bandwidth impinging on the controller.

25. The storage system according to claim 13 further comprising:

a network configured as multiple sets of associated controller pairs and storage shelves, the individual sets having a bandwidth adequate for accessing all storage arrays in the set, bandwidth between sets being limited.

26. The storage system according to claim 25 wherein:

all interaction with a storage element other than support of access to data or metadata contained on the storage element involves a controller in the same set as the storage element, support of access.

27. The storage system according to claim 13 further comprising:

a process executable in the processor that moves storage device-related activity between controllers by transferring management information rather than data comprising;

identifying a storage device group to be moved from a first controller to a second controller;

eliminating data structures associated with the storage device group in the first controller;

queuing commands directed to the storage device group at a level sufficiently high to include both the first controller and the second controller;

waiting for commands directed to the storage device group to complete;

posting any remaining storage device group state information to the storage device group;

enabling both the first controller and the second controller to host the storage device group;

uploading the storage device group state information to the second controller;

when the storage device group state information is uploaded, disabling the first controller from hosting the storage device group; and

resuming delivery of commands to the storage device group.

28. An article of manufacture comprising:

a tangible computer-readable medium having a program code embodied therein for execution on a controller for moving a virtual storage segment in a storage system from a previous location to a target location, the program code further comprising;

a code causing the controller to arrange the storage system as multiple sets of associated storage controllers and storage devices that communicate with full cross-sectional bandwidth between controllers and associated storage devices internal to ones of the multiple sets and a reduced inter-set bandwidth in inter-set links between controllers and storage devices in different sets of the multiple sets;

a code causing the controller to determine performance demand among a plurality of storage device groups;

a code causing the controller to distribute performance demand values according to controller processor utilization of the individual storage device groups and inter-set bandwidth utilization; and

a code causing the controller to associate the plurality of storage device groups with controllers based on the performance demand value distributions.

29. A storage system comprising:

a processor;

means executable on the processor for allocating control of virtual storage segments in a storage system among a plurality of controllers, the storage system being arranged as multiple sets of associated controllers and storage devices that communicate with full cross-sectional bandwidth between controllers and associated storage devices internal to ones of the multiple sets and a reduced inter-set bandwidth in inter-set links between controllers and storage devices in different sets of the multiple sets;

means executable on the processor for determining performance demand among a plurality of storage device groups;

means executable on the processor for distributing performance demand values according to controller processor utilization of the individual storage device groups and inter-set bandwidth utilization; and

means executable on the processor for associating the plurality of storage device groups with controllers based on the performance demand value distributions.

30. A storage system comprising:

a plurality of storage arrays configurable into a plurality of storage device groups;

a plurality of controllers selectively coupled to ones of the storage arrays;

a network fabric arranged as a linkage of multiple sets of associated controllers and storage devices that communicate with full cross-sectional bandwidth between controllers and associated storage devices internal to ones of the multiple sets and a reduced inter-set bandwidth in inter-set links between controllers and storage devices in different sets of the multiple sets; and

an inter-set communication linkage among the plurality of storage arrays across set boundaries.