Fault-tolerant messaging system and methods

• US 8,191,078 B1
• Filed: 03/22/2005
• Issued: 05/29/2012
• Est. Priority Date: 03/22/2005
• Status: Active Grant

• Current Assignee: Aurea Software Incorporated (ESW Capital, LLC)
• Original Assignee: Progress Software Corporation
• Inventors: Cullen, William M., Theroux, Michael E., Easton, Peter S., Yin, Ling-Yun, Boschi, Giovanni, MacNaughton, Colin W., Rudis, Ronald F.
• Primary Examiner(s): Sough, S.
• Assistant Examiner(s): RONI, SYED A
• Application Number:
  US11/088,024
• Time in Patent Office: 2,625 Days
• Field of Search: 719/315
• US Class Current: 719/315
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A fault-tolerant messaging system comprises a primary broker, a first network, a back up broker, and a second network. The primary broker and the back up broker are coupled to the first network for communication with clients thus creating a messaging system. The primary broker and the back up broker are also coupled to the second network for replicating state from the primary broker and the back up broker, and also sending transaction events immediately to maintain synchronization. The brokers further comprise a replication module for communicating state between the primary broker and the back up broker, a recovery module for performing recovery on the back up broker upon failure of the primary broker, and a fault-tolerant connection module for establishing a fault-tolerant connection between the primary broker and the back up broker over the second network. The present invention also includes a number of novel methods for performing fault tolerance.

Claims

1. A method for providing messages on a fault tolerant basis, the method comprising the steps of:

establishing a connection between a first broker and a second broker;

replicating a state of the first broker to the second broker;

maintaining dynamic run-time synchronization of the state of the first broker in an active state and the second broker in a standby state by (1) receiving a message from the first broker that includes a copy of transactions and events that comprise the state of the first broker and (2) storing, to a volatile memory of the second broker, the transactions and the events included in the message;

recovering to the second broker upon a failure of the first broker using the transactions and events stored in the volatile memory of the second broker;

bi-directionally connecting the first broker to at least one client, wherein the at least one client is configured to send messages to and receive messages from the first broker;

wherein the step of connecting the first broker to at least one client comprises establishing a fault tolerant connection between the at least one client and the first broker that permits the at least one client to re-establish the connection to the first broker before failing over to the second broker,

aligning the state of the first broker and the at least one client if an attempt to re-establish the connection was successful; and

synchronizing connection and message protocols if the attempt to re-establish the connection was successful, and

wherein the state of the first broker reflects messages sent and received by the first broker.

2. The method of claim 1, wherein the step of establishing the first broker in an active mode comprises the steps of:

operating the first broker in a stand-alone mode;

operating the first broker in a synchronization mode for replicating the state of the first broker to the second broker; and

transitioning the first broker to the active state upon completion of synchronization.

3. The method of claim 1, wherein the step of establishing the second broker in the standby state comprises the steps of:

operating the second broker in a synchronization mode for replicating the state of the first broker to the second broker; and

transitioning the second broker to the standby state upon completion of synchronization.

4. The method of claim 1, further comprising the step of connecting the second broker to at least one client when in active mode.

5. The method of claim 1, wherein the step of establishing a fault tolerant connection, comprises the steps of:

establishing a connection between the client and the first broker;

providing reconnection information to the client; and

detecting a failure of the connection between the client and the first broker.

6. The method of claim 5, further comprising the steps of:

determining whether the attempt to re-establish the connection was successful;

connecting to the second broker if the attempt was not successful.

7. The method of claim 1, wherein the step of establishing a connection between the first broker and the second broker comprises the steps of:

establishing a first connection between the first broker and the second broker; and

establishing a second connection between the first broker and the second broker.

8. The method of claim 7, wherein the first connection and the second connection between the first broker and the second broker are made over a private network.

9. The method of claim 1, wherein the step of replicating the state of the first broker to the second broker comprises the steps of:

creating a second broker database at the second broker;

retrieving a portion of a first broker database; and

storing the retrieved portion of the first broker database in the second broker database.

10. The method of claim 9, further comprising the steps of:

determining whether the first broker database is equivalent to the second broker database; and

repeating the steps of retrieving and storing if the first broker database is not equivalent to the second broker database.

11. The method of claim 1, wherein the step of maintaining synchronization comprises the steps of:

receiving an event at the first broker;

processing the event by the first broker;

storing the event at the first broker;

sending the event to the second broker; and

storing the event at the second broker.

12. The method of claim 11, comprising the steps of:

generating an acknowledgment signal by the second broker after the event has been stored at the second broker; and

sending the acknowledgement signal from the second broker to the first broker.

13. The method of claim 11, comprising the step of:

waiting for an acknowledgment signal at the first broker;

sending the acknowledgment signal from the first broker to a client.

14. The method of claim 1, wherein upon failure of the first broker, the second broker operates in a stand-alone mode.

15. The method of claim 1, further comprising the steps of:

attempting to connect to the first broker using another connection;

determining whether another connection is available; and

using the other connection to communicate with the first broker if another connection is available.

16. The method of claim 1, wherein the step of recovering to the second broker comprises the steps of:

reconciling a client state with a stored state at the second broker;

recovering transactions and events stored at the second broker; and

queuing reloading of recovered transactions and events.

17. A fault tolerant messaging system, the system comprising:

a first broker having an input and output for processing messages to and from a client, the first broker having a state reflecting the processing of messages and events, the first broker adapted to maintain dynamic run-time synchronization of the state of the first broker by transmitting a synchronization message that includes a copy of transactions and events that comprise the state of the first broker;

a second broker having an input and output for processing messages, the second broker adapted to receive and store the synchronization message in a volatile memory of the second broker, the second broker also adapted to use the stored synchronization message from the volatile memory of the second broker in recovering and communicating with the client upon failure of the first broker; and

a first bi-directional connection coupled to the first broker and the second broker, the first connection being used to send and receive state information; and

a fault tolerant connection between the first broker and at least one client, the fault tolerant connection permitting at least one client to re-establish the connection to the first broker before failing over to the second broker,

wherein the state of the first broker reflects messages sent and received by the first broker, and

wherein if an attempt to re-establish the connection between the first broker and the at least one client is successful, the first broker and the client are configured to align states and to synchronize connection and message protocols.

18. The fault tolerant messaging system of claim 17, further comprising a second connection coupled to the first broker and the second broker, the second connection being used to send and receive state information.

19. The fault tolerant messaging system of claim 18, wherein the first connection and the second connection are over a private network.

20. The fault tolerant messaging system of claim 18, comprising a third connection coupled to the first broker and the client for sending and receiving messages and events.

21. The fault tolerant messaging system of claim 19, further comprising a fourth connection coupled to the first broker and the client for sending and receiving messages and events.

22. The fault tolerant messaging system of claim 17, wherein the first broker comprises a message database for storing state information.

23. The fault tolerant messaging system of claim 17, wherein the first broker comprises a recovery log for storing events.

24. The fault tolerant messaging system of claim 17, wherein the second broker comprises a message database for storing state information.

25. The fault tolerant messaging system of claim 17, wherein the second broker comprises a recovery log for storing events.

26. The fault tolerant messaging system of claim 17, wherein the first broker comprises a replication module for sending events received at the first broker to the second broker.

27. The fault tolerant messaging system of claim 26, wherein the second broker comprises a replication module for receiving events from the replication module of the first broker, the replication module of the second broker storing events received in recovery log at the second broker.

28. The fault tolerant messaging system of claim 27, wherein the replication module processes events received from the first broker to maintain the state of the second broker.

29. The fault tolerant messaging system of claim 17, wherein the first broker comprises a replication connection module for establishing a replication connection between the first broker and the second broker.

30. The fault tolerant messaging system of claim 17, wherein the first broker comprises a transaction manager, the transaction manager tracking a status of a transaction, and keeping the transaction open until complete.

31. The fault tolerant messaging system of claim 30, wherein the second broker comprises a transaction manager, the transaction manager of the second broker tracking status of transactions, the transaction manager of the second broker coupled for communication with the transaction manager of the first broker such that the transaction manager of the second broker tracks the transactions processed by the first broker.

32. The fault tolerant messaging system of claim 17, wherein the second broker comprises a recovery module, the recovery module responsive to the failure of the first broker, the recovery module of the second broker for making the second broker operational with the state of the first broker upon the failure of the first broker.

33. The fault tolerant messaging system of claim 32, wherein the failure of the first broker includes an inability to connect to the first broker by the client.

34. The fault tolerant messaging system of claim 17, wherein the client includes a fault detection module for detecting the failure of the first broker.

35. The fault tolerant messaging system of claim 17, comprising a third broker having an input and output for processing messages, the third broker adapted to receive and store the state of the first broker, the third broker communicating with the client upon failure of the first broker and the second broker.

36. The fault tolerant messaging system of claim 17 wherein the first broker, second broker, and a third broker simultaneously serve different clients,

the first broker configured to receive and store a state of the third broker in a volatile memory of the first broker, the first broker also configured to use information from the volatile memory of the first broker in recovering and communicating with a client of the third broker upon failure of the third broker;

and

the third broker configured to receive and store a state of the second broker in a volatile memory of the third broker, the third broker also configured to use information from the volatile memory of the third broker in recovering and communicating with a client of the second broker upon failure of the second broker.