SYSTEMS AND METHODS FOR SERVICE REPLICATION, VALIDATION, AND RECOVERY IN CLOUD-BASED SYSTEMS

• US 20190146884A1
• Filed: 11/15/2017
• Published: 05/16/2019
• Est. Priority Date: 11/15/2017
• Status: Active Grant

• Current Assignee: Zscaler Incorporated
• Original Assignee: Zscaler Incorporated
• Inventors: GANGADHARAPPA, Tejus, CHAN, Wai Leung, PADMANABHAN, Dinoja, UDUPA, Sivaprasad
• Granted Patent: US 10,379,966 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/182 based on mutual exchange of the output between redundant processing components
• CPC Class Codes: G06F 11/184 where the redundant components implement processing functionality
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Systems and methods for replication and validation between a plurality of nodes in a cloud-based system forming a cluster include replicating a database and a software state between the plurality of nodes utilizing a replication algorithm which elects a master of the plurality of nodes and remaining nodes comprising peers, wherein the master is configured to perform the replicating; validating database sequences in the database at all of the plurality of nodes utilizing the replication algorithm to ensure sequential order of the validating between the plurality of nodes; and, responsive to differences between a peer and the master, performing a failure recovery in the database at the peer.

Claims

1. A method of replication and validation between a plurality of nodes in a cloud-based system forming a cluster, the method comprising:

replicating a database and a software state between the plurality of nodes utilizing a replication algorithm which elects a master of the plurality of nodes and remaining nodes as peers, wherein the master is configured to perform the replicating;

validating database sequences in the database at all of the plurality of nodes utilizing the replication algorithm to ensure sequential order of the validating between the plurality of nodes;

responsive to differences between a peer and the master, performing a failure recovery in the database at the peer; and

verifying the database of the peer and the master utilizing the replication algorithm without closing an application interacting with the database and honoring requests from the application to the database during the verifying, wherein the verifying is performed through a replication operation using the replication algorithm to perform a database dump which is verified offline with a tool for comparison, wherein the comparison omits one or more fields in the database between the peer and the master, wherein the one or more fields are inherently different.

5. The method of claim 1, wherein the validating is performed after each database update.

6. The method of claim 1, wherein the validating comprises:

causing each node to determine a checksum of all sequence values by the master;

providing the master'"'"'s checksum to each node via the replication algorithm; and

comparing the master'"'"'s checksum with a local checksum at each node.

7. The method of claim 1, wherein the validating is performed through commands in the replication algorithm ensuring a sequential order of operations.

8. The method of claim 1, wherein the failure recovery comprises obtaining a full dump from the master.

9. The method of claim 1, wherein the replication algorithm comprises Raft.

2. (canceled)

3. (canceled)

4. (canceled)

10. A cloud-based system configured for replication and validation between a plurality of nodes in a cluster, the cloud-based system comprising:

a plurality of nodes each configured to replicate a database and a software state between one another utilizing a replication algorithm which elects a master of the plurality of nodes and remaining nodes as peers, wherein the master is configured to perform replication;

wherein each of the plurality of nodes is configured to validate database sequences utilizing the replication algorithm to ensure sequential order of the validation between the plurality of nodes, and

wherein, responsive to differences between a peer and the master in the validation, a failure recovery is performed for the database at the peer, wherein the database of the peer and the master are verified utilizing the replication algorithm without closing an application interacting with the database and while honoring requests from the application to the database during the verification, wherein the verification is performed through a replication operation using the replication algorithm to perform a database dump which is verified offline with a tool for comparison, and wherein the comparison omits one or more fields in the database between the peer and the master, wherein the one or more fields are inherently different.

14. The cloud-based system of claim 10, wherein the validation is performed after each database update.

15. The cloud-based system of claim 10, wherein the validation comprises:

a determination of a checksum of all sequence values at each node;

each node obtaining the master'"'"'s checksum via the replication algorithm; and

each node comparing the master'"'"'s checksum with a local checksum.

16. The cloud-based system of claim 10, wherein the validation is performed through commands in the replication algorithm ensuring a sequential order of operations.

17. The cloud-based system of claim 10, wherein the failure recovery comprises obtaining a full dump from the master.

18. The cloud-based system of claim 10, wherein the replication algorithm comprises Raft.

11. (canceled)

12. (canceled)

13. (canceled)

19. A non-transitory computer-readable medium comprising instructions that, when executed, cause a processor to perform steps of:

replicating a database and a software state between the plurality of nodes utilizing a replication algorithm which elects a master of the plurality of nodes and remaining nodes as peers, wherein the master is configured to perform the replicating;

validating database sequences in the database at all of the plurality of nodes utilizing the replication algorithm to ensure sequential order of the validating between the plurality of nodes;

responsive to differences between a peer and the master, performing a failure recovery in the database at the peer; and

verifying the database of the peer and the master utilizing the replication algorithm without closing an application interacting with the database and honoring requests from the application to the database during the verifying, wherein the verifying is performed through a replication operation using the replication algorithm to perform a database dump which is verified offline with a tool for comparison, wherein the comparison omits one or more fields in the database between the peer and the master, wherein the one or more fields are inherently different.

21. The non-transitory computer-readable medium of claim 19, wherein the validating is performed after each database update.

22. The non-transitory computer-readable medium of claim 19, wherein the validating comprises:

causing each node to determine a checksum of all sequence values by the master;

providing the master'"'"'s checksum to each node via the replication algorithm; and

comparing the master'"'"'s checksum with a local checksum at each node.

23. The non-transitory computer-readable medium of claim 19, wherein the validating is performed through commands in the replication algorithm ensuring a sequential order of operations.

24. The non-transitory computer-readable medium of claim 19, wherein the failure recovery comprises obtaining a full dump from the master.

25. The non-transitory computer-readable medium of claim 19, wherein the replication algorithm comprises Raft.

20. (canceled)