Failure resistant distributed computing system

• US 10,261,873 B2
• Filed: 12/12/2016
• Issued: 04/16/2019
• Est. Priority Date: 12/31/2014
• Status: Active Grant

• Current Assignee: ServiceNow Incorporated
• Original Assignee: ServiceNow Incorporated
• Inventors: Chandrashekar, Sridhar, Patel, Swapnesh, Shah, Viral, Garg, Anurag, Chablani, Anjali
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US15/376,184
• Publication Number:
  US 20170093615A1
• Time in Patent Office: 855 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035
• CPC Class Codes: H04L 41/0668 by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
• CPC Class Codes: H04L 61/4511 using domain name system [DNS]
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A failure resistant distributed computing system includes primary and secondary datacenters each comprising a plurality of computerized servers. A control center selects orchestrations from a predefined list and transmits the orchestrations to the datacenters. Transmitted orchestrations include less than all machine-readable actions necessary to execute the orchestrations. The datacenters execute each received orchestration by referencing a full set of actions corresponding to the received orchestration as previously stored or programmed into the computerized server and executing the referenced full set of actions. At least one of the orchestrations comprises a failover operation from the primary datacenter to the secondary datacenter. Failover shifts performance of task from a set of processing nodes of the primary datacenter to a set of processing nodes of the secondary datacenter, such tasks including managing storage accessible by one or more remote clients and running programs on behalf of remote clients.

Claims

1. A method for providing a failover between a primary datacenter and a standby datacenter performed by a control center, the method comprising:

detecting, via the control center, that a failover condition exists in the primary datacenter;

stopping, via the control center, processing on the primary datacenter;

ensuring, via the control center, that each database of the primary datacenter is in read-only mode;

ensuring, via the control center, that each database of the standby datacenter is in read/write mode;

redirecting, via the control center, communications from the primary datacenter to the standby datacenter, wherein the control center directs operations of the primary datacenter and the standby datacenter and controls shifting performance of tasks from the primary datacenter to the standby datacenter.

2. The method, as set forth in claim 1, wherein detecting that a failover condition exists in the primary datacenter comprises:

testing to determine if transfers to the primary datacenter can complete.

3. The method, as set forth in claim 1, wherein stopping processing on the primary datacenter comprises:

stopping all processing nodes being executed by computerized servers of the primary datacenter and of the standby datacenter.

4. The method, as set forth in claim 1, wherein ensuring that each database of the standby datacenter is in read/write mode comprises:

switching each database of the standby datacenter to read/write mode after replication lag is zero.

5. The method, as set forth in claim 1, wherein redirecting communications from the primary datacenter to the standby datacenter comprises:

changing connection strings in application nodes to point to the standby datacenter.

6. The method, as set forth in claim 5, wherein redirecting communications from the primary datacenter to the standby datacenter comprises:

switching DNS entries to point to a load balancer of the standby datacenter.

7. The method, as set forth in claim 1, comprising running a discovery process on the standby datacenter and updating a configuration management database based on the discovery process.

8. A non-transitory tangible computer readable medium comprising instructions that when executed by a processor of a control center cause the processor to:

determine, via the control center, that a failover condition exists in the primary datacenter;

stop, via the control center, processing on all active nodes of the primary datacenter and of the standby datacenter;

switch, via the control center, each database of the primary datacenter from read/write mode to read-only mode;

switch, via the control center, each database of the standby datacenter from read only mode to read/write mode;

route, via the control center, connections of processing nodes of the primary datacenter to processing nodes of the standby datacenter; and

restart, via the control center, all nodes of the primary datacenter and the standby datacenter, wherein the control center directs operations of the primary datacenter and the standby datacenter and controls shifting performance of tasks from the primary datacenter to the standby datacenter.

9. The method, as set forth in claim 8, wherein detecting that a failover condition exists in the primary datacenter comprises:

testing to determine whether transfers to the primary datacenter can complete.

10. The method, as set forth in claim 8, wherein switching each database of the standby datacenter from read only mode to read/write mode comprises:

switching each database of the standby datacenter to read/write mode after replication lag is zero.

11. The method, as set forth in claim 8, wherein routing connections of the processing nodes of the primary datacenter to processing nodes of the standby datacenter comprises:

changing connection strings in application nodes to point to the standby datacenter.

12. The method, as set forth in claim 11, wherein routing connections of processing nodes of the primary datacenter to processing nodes of the standby datacenter comprises:

switching DNS entries to point to a load balancer of the standby datacenter.

13. The method, as set forth in claim 8, comprising, after restarting all nodes of the standby datacenter, running a discovery process on the standby datacenter and updating a configuration management database based on the discovery process.

14. A distributed computing system comprising:

a primary datacenter having a plurality of computerized servers and a plurality of databases;

a secondary datacenter having a plurality of computerized servers and a plurality of databases; and

a control center having a data processing machine and being communicatively coupled to the primary datacenter and the secondary datacenter, wherein the data processing machine of the control center is configured to provide an instruction to perform a failover operation to at least one of the computerized servers of the primary datacenter and/or secondary datacenter to cause a failover operation from the primary datacenter to the secondary datacenter to be conducted, wherein the failover operation comprises;

stopping, via the control center, processing on the primary datacenter and the secondary datacenter;

switching, via the control center, each of the plurality of databases of the primary datacenter to read-only mode;

switching, via the control center, each of the plurality of databases of the secondary datacenter to read/write mode;

redirecting, via the control center, communications from the primary datacenter to the secondary datacenter; and

restarting, via the control center, the primary datacenter and the secondary datacenter, wherein the control center directs operations of the primary datacenter and the secondary datacenter and controls shifting performance of tasks from the primary datacenter to the secondary datacenter.

15. The system, as set forth in claim 14, wherein the control center is configured to detect that a failover condition exists in the primary datacenter and to provide the instruction to perform the failover operation in response to detecting that the failover condition exists.

16. The system, as set forth in claim 14, wherein stopping processing on the primary datacenter and the secondary datacenter comprises:

stopping all processing nodes being executed by the plurality of computerized servers of the primary datacenter and by the plurality of computerized servers of the secondary datacenter.

17. The system, as set forth in claim 14, wherein switching each of the plurality of databases of the secondary datacenter to read/write mode comprises:

switching each of the plurality of databases of the standby datacenter to read/write mode after replication lag is zero.

18. The system, as set forth in claim 14, wherein redirecting communications from the primary datacenter to the secondary datacenter comprises:

changing connection strings in application nodes to point to the secondary datacenter as a new primary datacenter.

19. The system, as set forth in claim 18, wherein redirecting communications from the primary datacenter to the secondary datacenter comprises:

switching DNS entries to point to a load balancer closer to the new primary datacenter.

20. The system, as set forth in claim 14, comprising, after restarting the secondary datacenter, running a discovery process on all processing nodes of the secondary datacenter and updating a configuration management database based on the discovery process.