High performance storage array interconnection fabric using multiple independent paths

• US 7,401,161 B2
• Filed: 12/18/2000
• Issued: 07/15/2008
• Est. Priority Date: 12/18/2000
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Wu, Chia Y., Lee, Whay S., Chong, Fay Jr., Talagala, Nisha D., Rettberg, Randall
• Primary Examiner(s): Cardone; Jason
• Assistant Examiner(s): Kianersi; Mitra
• Application Number:
  US09/739,924
• Publication Number:
  US 20030221018A1
• Time in Patent Office: 2,766 Days
• Field of Search: 709/249
• US Class Current: 709/249
• CPC Class Codes: H04L 12/56 Packet switching systems
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A storage array interconnection fabric may be configured using multiple independent paths. A storage system including a plurality of communication paths is configured for connecting each node of a plurality of nodes forming an interconnection fabric. Each of the communications paths is an independent communications path. In addition, a first portion of the plurality of nodes is configured to communicate with a plurality of mass storage devices such as disk drives. A second portion of the plurality of nodes may be configured to communicate with a host.

Claims

1. A storage system, comprising:

a plurality of nodes, wherein different ones of said plurality of nodes perform different functional roles in the storage system;

wherein a first portion of said plurality of nodes are storage nodes each comprising one or more mass storage devices;

wherein a second portion of said plurality of nodes are one or more host interface nodes each configured to provide an interface for the storage system to one or more host computers; and

an interconnection fabric configured for connecting each one of said plurality of nodes to every other one of said plurality of nodes;

wherein the interconnection fabric comprises a plurality of point-to-point connections between said plurality of nodes, wherein said interconnection fabric is configured to provide a plurality of independent communication paths between each one of said plurality of nodes to every other one of said plurality of nodes, wherein the interconnection fabric comprises at least one of a torus interconnection fabric, a mesh interconnection fabric, a hypercube interconnection fabric, a multiple path butterfly interconnection fabric, a complete graph interconnection fabric, a hex network interconnection fabric, or a fat tree interconnection fabric.

2. The storage system as recited in claim 1, wherein each node is configured to communicate with each other node by routing messages bi-directionally.

3. The storage system as recited in claim 1, wherein said mass storage devices comprise disk drives.

4. The storage system as recited in claim 1, wherein a third portion of said plurality of nodes comprise random access memories configured as storage cache.

5. The storage system as recited in claim 1, wherein said mass storage devices comprise tape drives.

6. The storage system as recited in claim 1, wherein said mass storage devices comprise optical storage drives.

7. The storage system as recited in claim 1, wherein each node is configured to communicate with each other node by routing messages uni-directionally.

8. A method of interconnecting a plurality of nodes in a storage system, said method comprising:

connecting each node to each other node using a plurality of point-to-point connections;

forming an interconnection fabric comprising the nodes and said point-to-point connections, wherein the interconnection fabric comprises at least one of a torus interconnection fabric, a mesh interconnection fabric, a hypercube interconnection fabric, a multiple path butterfly interconnection fabric, a complete graph interconnection fabric, a hex network interconnection fabric, or a fat tree interconnection fabric;

a source node sending a first message to a destination node over a first communication path in said interconnection fabric;

said source node sending a second message to said destination node over a second communication path in said interconnection fabric, wherein said second communication path is independent from said first communication path;

said destination node interfacing to a mass storage device to respond to said first and second communications;

wherein different ones of said plurality of nodes perform different functional roles within the storage system, wherein a first portion of said plurality of nodes are storage nodes each comprising one or more mass storage devices, wherein said destination node is one of said first portion of said plurality of nodes, and wherein a second portion of said plurality of nodes are one or more host interface nodes each configured to provide an interface for the storage system to a host computer.

9. The method as recited in claim 8 further comprising nodes on said first and second communication path routing messages bi-directionally.

10. The method as recited in claim 8, wherein said mass storage devices comprise disk drives.

11. The method as recited in claim 8, wherein said mass storage devices comprise tape drives.

12. The method as recited in claim 8, wherein said mass storage devices comprise optical storage drives.

13. The method as recited in claim 8, wherein a third portion of said plurality of nodes comprise random access memories configured as cache memories for caching data stored in one or more mass storage devices.

14. The method as recited in claim 8 further comprising said source node interfacing to a host, wherein said source node is one of said second portion of said plurality of nodes.

15. The method as recited in claim 8 further comprising nodes on said first and second communication path routing messages uni-directionally.

16. A method for routing communications within a storage system comprising a plurality of nodes interconnected by an interconnection fabric, the method comprising:

sending a communication from a source node to a destination node using a first communication path comprising one or more point-to-point connections between said source node, any intervening nodes, and said destination node;

detecting a failure in said first communication path; and

resending said communication from said source node to said destination node using a second communication path which is independent from said first communication path, wherein said second communication path comprises one or more point-to-point connections between said source node, any intervening nodes, and said destination node;

wherein different ones of said plurality of nodes perform different functional roles within the storage system, wherein a first portion of said plurality of nodes are storage nodes each comprising one or more mass storage devices, and wherein a second portion of said plurality of nodes are one or more host interface nodes each configured to provide an interface for the storage system to a host computer;

wherein said source node and said destination node are part of the plurality of nodes interconnected by the interconnection fabric, wherein the interconnection fabric comprises at least one of a torus interconnection fabric, a mesh interconnection fabric, a hypercube interconnection fabric, a multiple path butterfly interconnection fabric, a complete graph interconnection fabric, a hex network interconnection fabric, or a fat tree interconnection fabric.

17. The method as recited in claim 16, further comprising:

detecting a failure in said second communication path; and

resending said communication from said source node to said destination node using a third communication path which is independent from said first and said second communication paths, wherein said third communication path comprises one or more point-to-point connections between said source node, any intervening nodes, and said destination node.

18. The method as recited in claim 17, further comprising:

detecting a failure in said third communication path; and

resending said communication from said source node to said destination node using a fourth communication path which is independent from said first, said second and said third communication paths, wherein said fourth communication path comprises one or more point-to-point connections between said source node, any intervening nodes, and said destination node.

19. The method as recited in claim 16 further comprising said destination node interfacing to a plurality of mass storage devices, wherein said destination node is one of said first portion of said plurality of nodes.

20. The method as recited in claim 16 further comprising said source node interfacing to a plurality of mass storage devices, wherein said source node is one of said first portion of said plurality of nodes.