Creation and migration of distributed streams in clusters of networked computers

• US 6,047,323 A
• Filed: 06/27/1996
• Issued: 04/04/2000
• Est. Priority Date: 10/19/1995
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Company (HP Inc.)
• Inventors: Krause, Michael R.
• Primary Examiner(s): Asta, Frank J.
• Assistant Examiner(s): Patru, Daniel
• Application Number:
  US08/671,562
• Time in Patent Office: 1,377 Days
• Field of Search: 395/200.31, 395/200.46, 395/200.55, 395/200.59, 395/200.68, 395/200.57, 395/200.61, 395/200.6, 395/200.74, 395/200.78, 395/200.79 , 395/681, 709/201, 709/216, 709/225, 709/229, 709/238, 709/230, 709/244, 709/248, 709/249, 709/301, 709/231, 709/227
• US Class Current: 709/227
• CPC Class Codes: G06F 9/4486 Formation of subprogram jump address
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A distributed STREAMS process operates on a multicomputer system composed of a cluster of nodes of one or more processors running an operating system having a file system and a STREAMS message-passing mechanism implementing network protocols, client-server applications, and STREAMS-based pipes. A local node has a software application operative under the operating system. The distributed STREAMS process determines that it is operating within a cluster and transparently intercepts application open requests which are sent to a controlling thread (CT) created during node initialization. The CT determines whether the open is to occur on the local or a remote node and whether any cluster facility should be activated by examining major and minor numbers encoded within the file structure being opened. If the CT determines a local open is to occur, it performs a local open, as normal, and activates the indicated cluster facilities. If the CT determines a remote open is to occur, it uses the STREAMS mechanism to establish a connection to the remote node via a STREAMS software interconnect driver (S-ICS) operating on both nodes. The local node'"'"'s CT then communicates the open request to the remote node'"'"'s CT which performs an internal STREAMS open to create data structures and infrastructure to ensure proper communication and error recovery. If a failure occurs, the CT and S-ICS detect this failure and transparently initiate error recovery by migrating failed components, if possible, to new node(s) within the cluster. This migration capability can also be used to provide load balancing within the cluster of distributed STREAMS.

Claims

1. A multicomputer system capable of distributed STREAMS operation, the system comprising:

a cluster of two or more nodes connected via a data communications interconnect subsystem, each node including a computer having one or more system processor units, local memory, and input/output subsystem;

an operating system running on each of the system processor units in the cluster, the operating system including a STREAMS message-passing mechanism for use in implementing one or more of networking protocols, client/server applications and services;

a software application operative on the system processor unit of at least one node under control of the operating system to perform a task or solve a problem;

means for creating a distributed STREAMS instance on a first of said nodes (initiating node) which is independent of the application and the message-passing mechanism; and

means for migrating at least a portion of the distributed STREAMS instance to a second of said nodes (target node) for execution of a selected task of the software application on the second node transparently with respect to the networking protocols, client/server applications and services on the first node.

2. A multicomputer system according to claim 1 in which:

the STREAMS message-passing mechanism includes a data structure implemented in each computer which provides a bidirectional data path between a user process and a device driver via a module which is an intermediate element that can be dynamically added to or removed from the data path;

the device driver residing in a system kernel of the operating system and operative to control a peripheral device to transfer data between said kernel and said device so that data received by the device driver can travel upstream to the user process; and

the STREAM data structure including a stack stored in local memory and containing a set of function pointers for controlling execution of module and driver functions.

3. A multicomputer system according to claim 2 including:

controlling thread means in the operating system for communications with and control of the STREAMS message passing mechanism in accordance with a cluster parameter; and

means defining a set of preview functions including a dynamic function replacement function for multiplexing messages between components of the system;

the device driver including a physical cluster interconnect driver (P-ICS) that provides basic communications facilities between nodes in the system under control of the controlling thread and in accordance with the preview functions.

4. A multicomputer system according to claim 3 further including means defining a STREAMS software interconnect driver (S-ICS) for moving messages between two or more of the following system components:

the application'"'"'s STREAMS stack, the controlling thread, the P-ICS, and the preview functions.

5. A multicomputer system according to claim 4 including means defining a middleware thread operative to process and communicate messages between the operating system and the S-ICS software interconnect driver.

6. A multicomputer system according to claim 4 in which the means for creating a distributed STREAMS instance includes:

a file system in the operating system of each node operative upon performing an open function responsive to an open request from the application to initiate opening of the STREAMS mechanism;

means for mapping and saving a device number of the interconnect driver in a first of said nodes (initiating node) to a target device number;

means in the STREAMS mechanism for opening the selected driver on the first node;

means for remapping the saved device number into a node address for a second target node;

means in the interconnect driver for transmitting a message to the target node, the message including the application'"'"'s open request and the saved device number; and

the file system in the target node being responsive to the message to open the target driver.

7. A multicomputer system according to claim 6 in which the target node operating system includes means for returning a message to the initiating node'"'"'s interconnect driver including a stream head address for the target driver.

8. A multicomputer system according to claim 2 in which the means for migrating at least a portion of the distributed STREAMS instance to a second, target node includes means for communicating at least a portion of the stack on the first node to the second node.

9. A multicomputer system according to claim 8 including:

means for issuing a ioctl representing to the STREAMS mechanism that migration is to occur;

the STREAMS mechanism including means responsive to the ioctl to freeze the stack on the initiating node and forwards a message to the target node which includes information from which the stack can be re-created on the target node.

10. A multicomputer system according to claim 9 in which the STREAMS mechanism includes means for performing a marshalling function upon a portion of the data structure to assemble information needed to replicate a private data structure for a module represented in the stack on the target node.

11. In a multicomputer system capable of STREAMS operation, the system which including a cluster of two or more nodes including a local node and a remote node connected via a data communications interconnect subsystem, each node including a computer having one or more system processor units, local memory, and an input/output subsystem, an operating system running on each of the system processor units in the cluster, the operating system including a file system and a STREAMS message-passing mechanism having an open function for creating a stream head and a driver with bidirectional communications, client/server applications and services, a software application operative on the system processor unit of at least one node under control of the operating system to perform a task or solve a problem, a method of creating a distributed STREAMS data structure, the method comprising:

initiating a controlling thread on each node in the cluster, including determining where STREAMS drivers are located within each node and determining which facilities are associated with each driver;

setting a flag in the STREAMS mechanism on each node in the cluster indicating that clustering is enabled;

assigning a file name uniquely representing a specific driver on a node in the cluster, the file name encoding a major number table identification for each driver and a minor number parameter for selectably identifying at least one of a driver on a remote node and local facilities on the local node;

communicate the file names of the drivers on each node to the other nodes in the cluster through the file system;

running the STREAMS open function to open the driver for the file, looking at the file name to ascertain that it is a clustering facility and, if so, passing the initial major number and minor number to the controlling thread;

the controlling thread using the major and minor numbers to look up the represented devices and facilities and deriving a new major and minor number and passing those numbers along with a set of facilities identifiers back to the STREAMS mechanism; and

if the initial major number and minor number pertain to facilities on the remote node, the STREAMS mechanism in the local node opening a STREAMS software interconnect driver (S-ICS) on the local node and the S-ICS driver communicating an open request to the controlling thread on the remote node;

the controlling thread being operative upon initialization to establish a STREAMS software interconnect driver (S-ICS) on the local node;

the controlling thread on the remote node performing an internal STREAMS open to create on the remote node a distributed STREAMS instance of the STREAMS data structure on the local node.

12. A method according to claim 11 including, if the initial major number and minor number pertain to facilities on the local node, the STREAMS mechanism in the local node opening a designated local driver to enable the local facilities associated with that driver.

13. A method according to claim 11 in which:

the STREAMS message-passing mechanism includes a data structure implemented in each computer which provides a bidirectional data path between a user process and a device driver via a module which is an intermediate element that can be dynamically added to or removed from the data path;

the device driver residing in a system kernel of the operating system and operative to control a peripheral device to transfer data between said kernel and said device so that data received by the device driver can travel upstream to the user process; and

the STREAM data structure includes a stack stored in local memory and containing a set of function pointers for controlling execution of module and driver functions;

the method including migrating at least a portion of the distributed STREAMS instance to a second of said nodes by communicating at least a portion of the stack on the first node to the second node.

14. A method according to claim 13 in which the STREAMS stack executes on a different node from the node where the application accessing it is executing.

15. A method according to claim 13 in which the STREAM is broken into constituent components at the module/driver/stream head levels with each component being executed on different, individual nodes within the cluster.

16. A method according to claim 13 in which the distributed STEAMS data structure includes a STREAMS-based pipe having two pipe ends, each end executing on a different node within the cluster.

17. A method according to claim 13 in which the STREAMS stack is migrated, either in whole or part, from one node to another.

18. A method according to claim 13 including detecting a failure condition on the second node and initiating error recovery.

19. A method according to claim 13 including performing a marshalling function upon a portion of the data structure to assemble information needed to replicate a private data structure for a module represented in the stack on the second node.

20. A method for migrating at a portion of a stack on multicomputer system capable of distributed operation, from a first node to a second node, wherein the stack contains a set of function pointers for controlling execution of module and driver functions, the method comprising the steps of:

issuing a system call requesting migration;

freezing the stack to prevent message flow into and out of the stack;

marshaling all information necessary to replicate the portion;

delivering the marshaled information to a control thread on the second node;

recreating the portion on the second node;

releasing the remaining portion of the stack; and

activating a communications route between the portion on the second node and the remaining portion on the first node.

21. The method of claim 20, further comprising the step of:

informing the control thread on the second node that migration will occur.

22. The method of claim 20, wherein the step of freezing comprises the steps of:

preventing entities from queuing messages for the stack; and

preventing the stack from sending out messages.

23. The method of claim 20, wherein the step of marshaling comprises the step of:

forming a marshaling structure for each component of the stack which will be migrated.

24. The method of claim 23, wherein:

the marshaling structure includes state specific information.

25. The method of claim 24, wherein:

the state specific information includes information detailing queued messages awaiting processing, queued messages awaiting transmission, and timer values.

26. The method of claim 20, wherein:

the system call contains necessary information for the migration of the portion of the stack.

27. The method of claim 20, wherein:

the migration is occurring to facilitate load balancing of the nodes of the multicomputer system.

28. The method of claim 20, wherein:

the migration requires the stack to be split between a module and a driver.

29. The method of claim 20, wherein:

the migration requires the stack to be split between a multiplexor and a driver.

30. The method of claim 20, wherein the portion is a driver, the method further comprising the steps of:

installing a software interconnect driver in the first node; and

linking the software interconnect driver with the remaining portion of the stack in the first node to allow communications with the migrated driver in the second node.

31. The method of claim 30, further comprising the step of:

updating tables to allow inbound packets to be routed to the migrated driver on the second node through the software interconnect driver.

32. The method of claim 20, wherein the portion is a module, the method further comprising the steps of:

installing an upper level software interconnect driver in the first node; and

linking the upper level software interconnect driver with the remaining portion of the stack in the first node to allow communications with the migrated module in the second node.

33. The method of claim 32, further comprising the step of:

updating tables to allow inbound packets to be routed to the migrated module on the second node through the software interconnect driver.

34. The method of claim 20, further comprising the step of:

reallocating resources in the first node which were occupied by the migrated portion.

35. The method of claim 20, wherein:

the step of issuing a system call is performed by a third node.

36. The method of claim 20, wherein:

the distributed operation is STREAMS distributed operation, and the stack is a STREAMS stack.

37. A multicomputer system that is capable of distributed operations and includes a plurality of nodes connected via communications interconnects, the system is running an operating system which uses a messaging mechanism in a portion of its operations, the system comprising:

a plurality of controlling threads, with one controlling thread associated with each node of the plurality of nodes, wherein each controlling thread is resident on a kernel of its associated node, and each controlling thread managing the messaging mechanism resident within its node;

a plurality of stacks, wherein each stack is capable of containing a set of function pointers for controlling execution of module and driver functions useable by the plurality of nodes;

a plurality of a physical interconnection drivers (PID), with one PID associated with each node of the plurality of nodes, wherein each PID which provides communication facilities between its node and the remaining nodes of the plurality of nodes, each PID is under the control of the control thread of the same node;

a plurality of a software interconnection drivers (SID), with one SID associated with each node of the plurality of nodes, wherein each SID allows communications between the PID of its associated node and at least one other component of its node, each SID is under the control of the control thread of the same node; and

a plurality of preview functions, with at least one preview function associated with each node of the plurality of nodes, wherein each preview function is a policy regarding a system wide facility, the controlling thread of the associated node will use the at least one preview function to modify operations of the associated stack.

38. The system of claim 37, wherein:

each controlling thread acts as a third-party communication and control point for its associated node.

39. The system of claim 37, wherein:

the stack is created with assistance from the controlling thread of its associated node.

40. The system of claim 37, wherein:

a portion of the stack of one node of the plurality of nodes may be migrated to another node of the plurality of nodes.

41. The system of claim 37, wherein:

each stack may utilize components resident on other nodes of the plurality of nodes.

42. The system of claim 41, wherein:

a controlling thread may create components to allow the stack associated with its node to utilize components resident on the other nodes.

43. The system of claim 37, wherein:

a controlling thread creates the SID for its associated node.

44. The system of claim 37, wherein:

each node includes at least one SMP processor.

45. The system of claim 37, wherein:

the distributed operation is a STREAMS distributed operation, the messaging mechanism is a STREAMS messaging mechanism, and the stack is a STREAMS stack.

46. The system of claim 37, wherein:

a stack of the plurality of stacks includes a stream head and a driver.

47. The system of claim 37, wherein:

a stack of the plurality of stacks includes a stream head, at least one module, and at least one driver.