Integrated-circuit implementation of a storage-shelf router and a path controller card for combined use in high-availability mass-storage-device shelves that may be incorporated within disk arrays

• US 20040139260A1
• Filed: 06/23/2003
• Published: 07/15/2004
• Est. Priority Date: 01/13/2003
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Sierra Logic (Broadcom, Inc.)
• Inventors: Kompella, Murthy, Wakeley, Matthew Paul, Steinmetz, Joseph Harold, Scotten, Jeffrey Douglas
• Granted Patent: US 7,353,321 B2
• Time in Patent Office: Days
• US Class Current: 710/269
• CPC Class Codes: G06F 11/201 between storage system components
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0658 Controller construction arrangements
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 12/66 Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
• CPC Class Codes: H04L 45/60 Router architectures
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

An integrated circuit implementing a storage-shelf router used alone, or in combination with other storage-shelf routers, and in combination with path controller cards, to interconnect the disks within a storage shelf or disk array to a high-bandwidth communications medium, such as an FC arbitrated loop, through which data is exchanged between the individual disk drives of the storage shelf and a disk-array controller. A set of interconnected storage-shelf routers within a storage shelf can be accessed through a single port of an FC arbitrated loop or other high-bandwidth communications medium. Because, in one implementation, eight storage-shelf routers can be interconnected within a storage shelf to provide highly available interconnection of sixty-four disk drives within the storage shelf to an FC arbitrated loop via a single FC-arbitrated-loop port, a single FC arbitrated loop including a disk-array controller, may interconnect 8,000 individual disk drives to the disk-array controller within a disk array. The storage-shelf router can serve to translate FC-based communications protocols into one or more communication protocols appropriate to the internal links, providing for use of less expensive, non-FC-compatible disk drives within the storage shelf.

Claims

1. A storage shelf that contains a number of data-storage devices interconnected to a communications medium, the storage shelf including:

a first storage-shelf-router integrated circuit and a last storage-shelf-router integrated circuit, each storage-shelf router integrated circuit including

a first communications-medium port,

a second communications-medium port,

one or more processors,

a number of disk-drive-link-port components that transmit data and commands to the number of data-storage devices through disk-drive links, and

routing logic for routing commands received through the first and second communications-medium ports to the one or more processors and for routing data received through the two or more communications-medium ports to the number of data-storage-device-link-port components; and

a number of path controller cards, each path controller card

receiving data and commands transmitted through disk-drive links from the number of data-storage-device-link-port components of one of the two storage-shelf-router integrated circuits, and, following a failure of a disk-drive link or data-storage-device-link port, receiving data and commands transmitted from one or more of the number of data-storage-device-link-port components of the other of the storage-shelf-router integrated circuits, and

transmiting the received data and commands to a data-storage device.

2. The storage shelf of claim 1 wherein the storage shelf is interconnected to a first communications medium and to a second communications medium, and wherein the number of storage-shelf-router integrated circuits are linked together in a first series, the first series comprising:

the first communications medium;

the first storage-shelf-router integrated circuit connected to the first communications medium through the first communications-medium port of the first storage-shelf-router and connected to the last storage-shelf-router integrated circuit through the second communications-medium port of the first storage-shelf-router integrated circuit, the first communications-medium port of the last storage-shelf-router integrated circuit, and an internal communications medium; and

the last storage-shelf-router integrated circuit.

3. The storage shelf of claim 2 wherein the number of storage-shelf-router integrated circuits are linked together in a second series, the second series comprising:

the second communications medium;

the last storage-shelf-router integrated circuit connected to the second communications medium through the second communications-medium port of the last storage-shelf-router and connected to the first storage-shelf-router integrated circuit through the first communications-medium port of the last storage-shelf-router integrated circuit, the second communications-medium port of the last storage-shelf-router integrated circuit, and the internal communications medium; and

the first storage-shelf-router integrated circuit.

4. The storage shelf of claim 3 further including additional storage-shelf-router integrated circuits, each storage-shelf router integrated circuit linked together in the first series and the second series in between the first storage-shelf-router integrated circuit and the last storage-shelf-router integrated circuit, the storage-shelf-router integrated circuits each assigned a unique number, the first storage-shelf-router integrated circuit assigned a lowest unique number, the last storage-shelf-router integrated circuit assigned a highest unique number, the unique numbers assigned to the additional storage-shelf-router integrated circuits increasing along the first series and decreasing along the second series.

5. The storage shelf of claim 4 wherein each additional storage-shelf-router integrated circuit having an assigned unique number is linked to a storage-shelf-router integrated circuit with a lower unique number than the assigned unique number through the first communications-medium port of the additional storage-shelf-router, the second communications-medium port of the storage-shelf-router integrated circuit with a lower unique number than the assigned unique number, and an internal communications medium connecting the additional storage-shelf-router integrated circuit with the storage-shelf-router integrated circuit with a lower unique number than the assigned unique number.

6. The storage shelf of claim 4 wherein each additional storage-shelf-router integrated circuit having an assigned unique number is linked to a storage-shelf-router integrated circuit with a higher unique number than the assigned unique number through the second communications-medium port of the additional storage-shelf-router, the first communications-medium port of the storage-shelf-router integrated circuit with a higher unique number than the assigned unique number, and an internal communications medium connecting the additional storage-shelf-router integrated circuit with the storage-shelf-router integrated circuit with a higher unique number than the assigned unique number.

7. The storage shelf of claim 1

wherein the communications medium is a fibre channel arbitrated loop, and

wherein each of the number of data-storage devices is an Advanced Technology Attachment disk drive.

8. The storage shelf of claim 1

wherein the communications medium is a fibre channel arbitrated loop,

wherein each of the number of data-storage devices is a Serial Advanced Technology Attachment disk drive, and

wherein the number of data-storage-device-link-port components are Serial Advanced Technology Attachment ports.

9. A storage-shelf-router integrated circuit employed within a storage shelf that contains a number of data-storage devices interconnected to two communications media, the storage-shelf-router integrated circuit including:

a first communications-medium port;

a second communications-medium port;

one or more processors;

a number of data-storage-device-link-port components that transmit data and commands to the number of data-storage devices through disk-drive links; and

routing logic for routing commands received through the first and second communications-medium ports to the one or more processors and for routing data received through the two or more communications-medium ports to the number of data-storage-device-link-port components.

10. The storage-shelf-router integrated circuit of claim 9 wherein each of the two communications-medium ports include a first-in-first-out buffer into which commands and data received by the communications-medium port are written, and from which command and data received by the communications-medium port are accessed by the routing logic.

11. The storage-shelf-router integrated circuit of claim 10 wherein the routing logic may access an initial portion of a command or data from the first-in-first-out buffer while the communications-medium port is writing a latter portion of the command or data into the first-in-first-out buffer.

12. The storage-shelf-router integrated circuit of claim 10 wherein the routing logic routes commands accessed from the first-in-first-out buffer within the two communications-medium ports to the one or more processors by directing the commands to a storage-shelf-router-integrated-circuit module that writes the commands to a shared memory, from which the commands can be accessed by the one or more processors.

13. The storage-shelf-router integrated circuit of claim 10 wherein the routing logic routes data accessed from the first-in-first-out buffer within the two communications-medium ports to the number of data-storage-device-link-port components by directing the data to a storage-shelf-router-integrated-circuit module that writes the data to a virtual queue within a global-shared-memory switch, from which the data can be accessed by the one of the one or more number of data-storage-device-link-port components.

14. The storage-shelf-router integrated circuit of claim 9 wherein the storage-shelf-router integrated circuit is assigned a unique number and is linked through the first communications-medium port and a first communications medium to a first entity and is linked through the second communications-medium port and a second communications medium to a second entity, the first entity one of

a remote device external to the storage shelf, and

a storage-shelf-router integrated circuit having a unique number less than the assigned unique number, and

the second entity one of

a remote device external to the storage shelf, and

a storage-shelf-router integrated circuit having a unique number greater than the assigned unique number.

15. The storage-shelf-router integrated circuit of claim 14 wherein the storage-shelf-router integrated circuit further includes a routing table that lists, for each data-storage device interconnected through the number of data-storage-device-link-port components to the storage-shelf-router integrated circuit, a first-communications-medium address associated with the data-storage device, a second-communications-medium address associated with the data-storage device, and additional information related to the data-storage-device addresses supported by the data-storage device.

16. The storage-shelf-router integrated circuit of claim 15 wherein, when the routing logic accesses a command message received through the first communications-medium port, the routing logic

routes the command message to one of the one or more processors when a destination address of the command message matches a first-communications-medium address associated with a data-storage device in the routing table,

routes the command message to the second communications-medium port when the destination address of the command message does not match a first-communications-medium address associated with a data-storage device in the routing table, and the second entity is not a remote device external to the storage shelf,

routes the command message to the first communications-medium port when the destination address of the command message does not match a first-communications-medium address associated with a data-storage device in the routing table, and the second entity is a remote device external to the storage shelf, and

routes the command message to one of the one or more processors when the routing logic determines that the command message needs error handling.

17. The storage-shelf-router integrated circuit of claim 16 wherein the routing logic determines that the command message needs error handling when the routing logic accesses additional tables within the storage-shelf-router integrated circuit and determines that the entity which sent the command message is not authorized to direct a command to a data-storage device interconnected to the storage-shelf router.

18. The storage-shelf-router integrated circuit of claim 15 wherein, when the routing logic accesses a data message received through the first communications-medium port, the routing logic

routes the data message to one of the number of data-storage-device-link-port components when a destination address of the command message matches a first-communications-medium address associated with a data-storage device in the routing table,

routes the data message to the second communications-medium port when the destination address of the command message does not match a first-communications-medium address associated with a data-storage device in the routing table, and the second entity is not a remote device external to the storage shelf,

routes the data message to the first communications-medium port when the destination address of the command message does not match a first-communications-medium address associated with a data-storage device in the routing table, and the second entity is a remote device external to the storage shelf, and

routes the data message to one of the one or more processors when the routing logic determines that the data message needs error handling.

19. The storage-shelf-router integrated circuit of claim 18 wherein the routing logic determines that the data message needs error handling when the routing logic accesses additional tables within the storage-shelf-router integrated circuit and determines that no context has been created within shared memory during processing of a previous command message for a data transfer operation, all or a portion of which involves the data message and when the routing logic accesses additional tables within the storage-shelf-router integrated circuit and determines that the entity which sent the command message is not authorized to direct data to a data-storage device interconnected to the storage-shelf router.

20. The storage-shelf-router integrated circuit of claim 15 wherein, when the routing logic accesses a status message received through the first communications-medium port, the routing logic

routes the status message to the second communications-medium port when the second entity is not a remote device external to the storage shelf, and

routes the status message to the first communications-medium port when the second entity is a remote device external to the storage shelf.

21. The storage-shelf-router integrated circuit of claim 15 wherein, when the routing logic accesses a storage-shelf-internal-management message received through the first communications-medium port, the routing logic

routes the storage-shelf-internal-management message to one of the one or more processors when a destination address of the storage-shelf-internal-management message matches the unique number assigned to the storage-shelf-router integrated circuit,

routes the storage-shelf-internal-management message to the second communications-medium port when the destination address of the storage-shelf-internal-management message is greater than the unique number assigned to the storage-shelf-router integrated circuit and the second entity is not a remote device external to the storage shelf, and

routes the storage-shelf-internal-management message to one of the one or more processors when the routing logic determines that the storage-shelf-internal-management message needs error handling.

22. The storage-shelf-router integrated circuit of claim 15 wherein, when the routing logic accesses a command message received through the second communications-medium port, the routing logic

routes the command message to one of the one or more processors when a destination address of the command message matches a second-communications-medium address associated with a data-storage device in the routing table,

routes the command message to the first communications-medium port when the destination address of the command message does not match a second-communications-medium address associated with a data-storage device in the routing table, and the first entity is not a remote device external to the storage shelf,

routes the command message to the second communications-medium port when the destination address of the command message does not match a second-communications-medium address associated with a data-storage device in the routing table, and the first entity is a remote device external to the storage shelf, and

routes the command message to one of the one or more processors when the routing logic determines that the command message needs error handling.

23. The storage-shelf-router integrated circuit of claim 22 wherein the routing logic determines that the command message needs error handling when the routing logic accesses additional tables within the storage-shelf-router integrated circuit and determines that the entity which sent the command message is not authorized to direct a command to a data-storage device interconnected to the storage-shelf router.

24. The storage-shelf-router integrated circuit of claim 15 wherein, when the routing logic accesses a data message received through the second communications-medium port, the routing logic

routes the data message to one of the number of data-storage-device-link-port components when a destination address of the command message matches a second-communications-medium address associated with a data-storage device in the routing table,

routes the data message to the first communications-medium port when the destination address of the command message does not match a second-communications-medium address associated with a data-storage device in the routing table, and the first entity is not a remote device external to the storage shelf,

routes the data message to the second communications-medium port when the destination address of the command message does not match a second-communications-medium address associated with a data-storage device in the routing table, and the first entity is a remote device external to the storage shelf, and

routes the data message to one of the one or more processors when the routing logic determines that the data message needs error handling.

25. The storage-shelf-router integrated circuit of claim 24 wherein the routing logic determines that the data message needs error handling when the routing logic accesses additional tables within the storage-shelf-router integrated circuit and determines that no context has been created within shared memory during processing of a previous command message for a data transfer operation, all or a portion of which involves the data message and when the routing logic accesses additional tables within the storage-shelf-router integrated circuit and determines that the entity which sent the command message is not authorized to direct data to a data-storage device interconnected to the storage-shelf router

26. The storage-shelf-router integrated circuit of claim 15 wherein, when the routing logic accesses a status message received through the second communications-medium port, the routing logic

routes the status message to the first communications-medium port when the first entity is not a remote device external to the storage shelf, and

routes the status message to the second communications-medium port when the first entity is a remote device external to the storage shelf.

27. The storage-shelf-router integrated circuit of claim 15 wherein, when the routing logic accesses a storage-shelf-internal-management message received through the second communications-medium port, the routing logic

routes the storage-shelf-internal-management message to one of the one or more processors when a destination address of the storage-shelf-internal-management message matches the unique number assigned to the storage-shelf-router integrated circuit,

routes the storage-shelf-internal-management message to the first communications-medium port when the destination address of the storage-shelf-internal-management message is less than the unique number assigned to the storage-shelf-router integrated circuit and the first entity is not a remote device external to the storage shelf, and

routes the storage-shelf-internal-management message to one of the one or more processors when the routing logic determines that the storage-shelf-internal-management message needs error handling.

28. A storage-shelf-router integrated circuit of claim 9

wherein each of the number of data-storage devices is an Advanced Technology Attachment disk drive,

wherein each of the communications media is a fibre channel arbitrated loop.

29. A storage-shelf-router integrated circuit of claim 9

wherein each of the communications media is a fibre channel arbitrated loop,

wherein each of the number of data-storage devices is a Serial Advanced Technology Attachment disk drive, and

wherein the number of data-storage-device-link-port components are Serial Advanced Technology Attachment ports.

30. A routing logic component within a local storage-shelf router, included within a storage shelf, that includes a first port to a first communications medium, a second port to a second communications medium, a command and error processing component, and a data-storage-link-port layer, the routing logic component comprising:

destination logic that determines whether a message received from one of the first port and the second port is directed to the local storage-shelf-router, to a remote storage shelf router intercommunicating with the local storage router, or to a remote entity external to the storage shelf; and

routing logic that routes a message received from one of the first port and the second port to one of the first port and second port in order to forward the message to a remote storage-shelf router when the destination logic determines that the message is directed to the remote storage-shelf router, that routes a message received from one of the first port and the second port to one of the first port and second port in order to forward the message to a remote entity external to the storage shelf when the destination logic determines that the message is directed to the remote entity external to the storage shelf, and that routes the message received from one of the first port and the second port to one of the command and error processing component or to the data-storage-link-port layer when the destination logic determines that the message is directed to the local storage-shelf router.

31. The routing logic component of claim 30

wherein the routing logic component accesses a routing table within the storage-shelf router that lists, for each of a number of data-storage devices interconnected through the data-storage-link-port layer to the storage-shelf router, a first-communications-medium address associated with the data-storage device, a second-communications-medium address associated with the data-storage device, and additional information related to data-storage-device addresses supported by the data-storage device,

wherein the storage-shelf router is assigned a unique number and is linked through the first port to a first entity and is linked through the second port to a second entity,

wherein the first entity is one of

a remote device external to the storage shelf, and

a storage-shelf router having a unique number less than the assigned unique number, and

wherein the second entity is one of

a remote device external to the storage shelf, and

a storage-shelf router having a unique number greater than the assigned unique number.

32. The routing logic component of claim 31 wherein, when the routing logic component accesses a command message received through the first port, the routing logic component

routes the command message to the command and error processing component when a destination address within the command message matches a first-communications-medium address in the routing table,

routes the command message to the second port when the destination address within the command message does not match a first-communications-medium address in the routing table, and the second entity is not a remote device external to the storage shelf,

routes the command message to the first port when the destination address of the command message does not match a first-communications-medium address in the routing table, and the second entity is a remote device external to the storage shelf, and

routes the command message to the command and error processing component when the routing logic component determines that the command message needs error handling.

33. The routing logic component of claim 32 wherein the routing logic component determines that the command message needs error handling when the routing logic component accesses additional tables within the storage-shelf-router integrated circuit and determines that the entity which sent the command message is not authorized to direct a command to a data-storage device interconnected with the storage-shelf router.

34. The routing logic component of claim 31 wherein, when the routing logic component accesses a data message received through the first port, the routing logic component

routes the data message to the data-storage-link-port layer when a destination address within the data message matches a first-communications-medium address in the routing table,

routes the data message to the second port when the destination address within the data message does not match a first-communications-medium address in the routing table, and the second entity is not a remote device external to the storage shelf,

routes the data message to the first port when the destination address within the data message does not match a first-communications-medium address in the routing table, and the second entity is a remote device external to the storage shelf, and routes the data message to the command and error processing component when the routing logic component determines that the data message needs error handling.

35. The routing logic component of claim 34 wherein the routing logic component determines that the data message needs error handling when the routing logic component accesses additional tables within the storage-shelf router and determines that no context has been created within shared memory during processing of a previous command message for a data transfer operation, all or a portion of which involves the data message and when the routing logic component accesses additional tables within the storage-shelf router and determines that the entity which sent the command message is not authorized to direct data to a data-storage device interconnected to the storage-shelf router.

36. The routing logic component of claim 31 wherein, when the routing logic component accesses a status message received through the first port, the routing logic component

routes the status message to the second port when the second entity is not a remote device external to the storage shelf, and

routes the status message to the first port when the second entity is a remote device external to the storage shelf.

37. The routing logic component of claim 31 wherein, when the routing logic component accesses a storage-shelf-internal-management message received through the first port, the routing logic component

routes the storage-shelf-internal-management message to the command and error processing component when a destination address within the storage-shelf-internal-management message matches the unique number assigned to the storage-shelf router,

routes the storage-shelf-internal-management message to the second port when the destination address within the storage-shelf-internal-management message is greater than the unique number assigned to the storage-shelf router and the second entity is not a remote device external to the storage shelf, and

routes the storage-shelf-internal-management message to the command and error processing component when the routing logic determines that the storage-shelf-internal-management message needs error handling.

38. The routing logic component of claim 31 wherein, when the routing logic component accesses a command message received through the second port, the routing logic component

routes the command message to the command and error processing component when a destination address within the command message matches a second-communications-medium address in the routing table,

routes the command message to the first port when the destination address within the command message does not match a second-communications-medium address in the routing table, and the first entity is not a remote device external to the storage shelf,

routes the command message to the second port when the destination address of the command message does not match a second-communications-medium address in the routing table, and the first entity is a remote device external to the storage shelf, and

routes the command message to the command and error processing component when the routing logic component determines that the command message needs error handling.

39. The routing logic component of claim 38 wherein the routing logic component determines that the command message needs error handling when the routing logic component accesses additional tables within the storage-shelf-router integrated circuit and determines that the entity which sent the command message is not authorized to direct a command to a data-storage device interconnected with the storage-shelf router.

40. The routing logic component of claim 31 wherein, when the routing logic component accesses a data message received through the second port, the routing logic component

routes the data message to the data-storage-link-port layer when a destination address within the data message matches a second-communications-medium address in the routing table,

routes the data message to the first port when the destination address within the data message does not match a second-communications-medium address in the routing table, and the first entity is not a remote device external to the storage shelf,

routes the data message to the second port when the destination address within the data message does not match a second-communications-medium address in the routing table, and the first entity is a remote device external to the storage shelf, and

routes the data message to the command and error processing component when the routing logic component determines that the data message needs error handling.

41. The routing logic component of claim 34 wherein the routing logic component determines that the data message needs error handling when the routing logic component accesses additional tables within the storage-shelf router and determines that no context has been created within shared memory during processing of a previous command message for a data transfer operation, all or a portion of which involves the data message and when the routing logic component accesses additional tables within the storage-shelf router and determines that the entity which sent the command message is not authorized to direct data to a data-storage device interconnected to the storage-shelf router.

42. The routing logic component of claim 31 wherein, when the routing logic component accesses a status message received through the second port, the routing logic component

routes the status message to the first port when the first entity is not a remote device external to the storage shelf, and

routes the status message to the second port when the first entity is a remote device external to the storage shelf.

43. The routing logic component of claim 31 wherein, when the routing logic component accesses a storage-shelf-internal-management message received through the second port, the routing logic component

routes the storage-shelf-internal-management message to the command and error processing component when a destination address within the storage-shelf-internal-management message matches the unique number assigned to the storage-shelf router,

routes the storage-shelf-internal-management message to the first port when the destination address within the storage-shelf-internal-management message is greater than the unique number assigned to the storage-shelf router and the first entity is not a remote device external to the storage shelf, and

routes the storage-shelf-internal-management message to the command and error processing component when the routing logic determines that the storage-shelf-internal-management message needs error handling.