Dynamic membership management in a distributed system

• US 8,719,227 B1
• Filed: 03/04/2013
• Issued: 05/06/2014
• Est. Priority Date: 04/13/2005
• Status: Active Grant

• Current Assignee: Progress Software Corporation
• Original Assignee: Progress Software Corporation
• Inventors: Rousseau, Benjamin A., Hu, Carl, Tompkins, Frederick A. Jr.
• Primary Examiner(s): VO, TRUONG V
• Application Number:
  US13/784,479
• Time in Patent Office: 428 Days
• Field of Search: 707/638
• US Class Current: 707/638
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 9/466 Transaction processing

Abstract

Transactional database replication techniques are disclosed that do not require altering of the database management system implementation. A replicator module includes a dynamic membership manager, which is configured to manage the persistent membership of a coordination group. The persistent membership can be, for example, a durable set of sites that can replicate changes amongst themselves. Changes from sites not in the persistent membership are discarded. The set of recovered members is a subset of the persistent membership. The persistent membership changes incrementally by either removing or adding members. Failed members may still be part of the persistent membership. The dynamic membership manager module manages the modification of the persistent membership, initialization of replicas, and propagation of membership information.

Claims

1. A method for dynamically managing a persistent membership of a coordination group in a distributed environment, the coordination group including a number of sites included in the persistent membership, with one or more of the sites including a database and a registrar, the method comprising:

receiving, by a registrar of a first site that is a member of the persistent membership, a Membership Request from an entrant seeking to become a member of the persistent membership, the first site having participated in at least one quorum;

modifying, by the registrar of the first site, the persistent membership for the entrant;

responsive to the registrar receiving the Membership Request, disconnecting the registrar of the first site from the coordination group and inhibiting, by a distributed lock manager, updates to a first database while cotemporaneous to the inhibition of the updates the registrar of the first site initializes an entrant database for the entrant; and

reconnecting, by the registrar of the first site, to the coordination group after initialization of the entrant database.

2. The method of claim 1 wherein should connection between the registrar of the first site and the entrant be severed, the entrant can reconnect to any available registrar via a group communication system providing an extended virtual synchrony service to the coordination group, and recover.

3. The method of claim 1 wherein in response to the registrar of the first site not containing historical update information for an action identifier uniquely identifying the persistent membership, the method further comprises refusing, by the registrar of the first site, to proceed.

4. The method of claim 1 further comprising:

acquiring, by the registrar of the first site, a distributed lock on the coordination group and a name for the entrant;

examining, by the registrar of the first site upon receiving the distributed lock, the persistent membership to ensure that no other site utilizes the name for the entrant; and

refusing, by the registrar of the first site, to proceed with initializing the entrant database in response to the name for the entrant being already in use.

5. The method of claim 4 wherein acquiring the distributed lock is carried out by a fault tolerant distributed lock manager associated with the registrar of the first site, the method further comprising:

aborting, by the fault tolerant distributed lock manager, any transactions in process;

releasing, by the fault tolerant distributed lock manager, any locks held by the registrar of the first site; and

disconnecting, by the registrar of the first site, from a group communication system providing an extended virtual synchrony service to the coordination group thereby allowing the registrar of the first site to provide a consistent snapshot to the entrant.

6. The method of claim 1 wherein the initialization of the entrant database is carried out using snapshot initialization, and the entrant database is initialized with contents of the first database.

7. The method of claim 1 wherein the initialization of the entrant database is carried out using verify/update initialization, the initializing further comprising:

comparing contents of the entrant database and the first database;

resolving discrepancies; and

in response to the entrant including new data, creating, by the entrant, actions corresponding to the new data and indicating total persisted order of those actions is not yet known.

8. The method of claim 1 wherein the initialization of the entrant database is carried out using backup/historical initialization, the initializing further comprising:

sending, by the registrar of the first site, Green actions after a last applied action at the entrant to the entrant, wherein the Green actions are those whose total persisted order is known and confirmed; and

applying, by the entrant, the Green actions.

9. The method of claim 1 wherein the registrar of the first site is allowed to service the Membership Request from the entrant only if the first site has participated in at least one quorum.

10. The method of claim 1 wherein the registrar services the entrant if the registrar has attained a quorum based on a quorum requirement, and the quorum requirement is loosened to allow more quorums by not counting previous quorum members that have been evicted.

11. A non-transitory computer-readable medium encoded with instructions that, when executed by a processor, cause the processor to carry out a process for dynamically managing a persistent membership of a coordination group in a distributed environment, the coordination group including a number of sites included in the persistent membership, with one or more of the sites including a database and a registrar, the process comprising:

receiving, by a registrar of a first site that is a member of the persistent membership, a Membership Request from an entrant seeking to become a member of the persistent membership, the first site having participated in at least one quorum;

modifying, by the registrar of the first site, the persistent membership for the entrant;

responsive to the registrar receiving the Membership Request, disconnecting the registrar of the first site from the coordination group and inhibiting, by a distributed lock manager, updates to a first database while cotemporaneous to the inhibition of the updates the registrar of the first site initializes an entrant database for the entrant; and

reconnecting, by the registrar of the first site, to the coordination group after initialization of the entrant database.

12. The computer-readable medium of claim 11 wherein should connection between the registrar of the first site and the entrant be severed, the entrant can reconnect to any available registrar via a group communication system providing an extended virtual synchrony service to the coordination group, and recover.

13. The computer-readable medium of claim 11 wherein in response to the registrar of the first site not containing historical update information for an action identifier uniquely identifying the persistent membership, the method further comprises refusing, by the registrar of the first site, to proceed.

14. The computer-readable medium of claim 11, the process further comprising:

acquiring, by the registrar of the first site, a distributed lock on the coordination group and a name for the entrant;

examining, by the registrar of the first site upon receiving the distributed lock, the persistent membership to ensure that no other site utilizes the name for the entrant; and

refusing, by the registrar of the first site, to proceed with initializing the entrant database in response to the name for the entrant being already in use.

15. The computer-readable medium of claim 14 wherein acquiring the distributed lock is carried out by a fault tolerant distributed lock manager associated with the registrar of the first site, the process further comprising:

aborting, by the fault tolerant distributed lock manager, any transactions in process;

releasing, by the fault tolerant distributed lock manager, any locks held by the registrar of the first site; and

disconnecting, by the registrar of the first site, from a group communication system providing an extended virtual synchrony service to the coordination group thereby allowing the registrar of the first site to provide a consistent snapshot to the entrant.

16. The computer-readable medium of claim 11 wherein the initialization of the entrant database is carried out using snapshot initialization, and the entrant database is initialized with contents of the first database.

17. The computer-readable medium of claim 11 wherein the initialization of the entrant database is carried out using verify/update initialization, the initializing further comprising:

comparing contents of the entrant database and the first database;

resolving discrepancies; and

in response to the entrant including new data, creating, by the entrant, actions corresponding to the new data and indicating total persisted order of those actions is not yet known.

18. The computer-readable medium of claim 11 wherein the initialization of the entrant database is carried out using backup/historical initialization, the initializing further comprising:

sending, by the registrar of the first site, Green actions after a last applied action at the entrant to the entrant, wherein the Green actions are those whose total persisted order is known and confirmed; and

applying, by the entrant, the Green actions.

19. The computer-readable medium of claim 11 wherein the registrar of the first site is allowed to service the Membership Request from the entrant only if the first site has participated in at least one quorum.

20. The computer-readable medium of claim 11 wherein the registrar services the entrant if the registrar has attained a quorum based on a quorum requirement, and the quorum requirement is loosened to allow more quorums by not counting previous quorum members that have been evicted.