Automated failure recovery of subsystems in a management system

• US 10,129,176 B2
• Filed: 04/13/2015
• Issued: 11/13/2018
• Est. Priority Date: 07/31/2009
• Status: Active Grant

• Current Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Original Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Inventors: Jaisinghani, Devendra Rajkumar
• Primary Examiner(s): Leibovich, Yair
• Application Number:
  US14/685,185
• Publication Number:
  US 20150220408A1
• Time in Patent Office: 1,310 Days
• Field of Search: 714 4
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 9/4401 Bootstrapping security arrangements therefor G06F21/57
• CPC Class Codes: H04L 12/14 Charging , metering or billing arrangements for data wireline or wireless communications
• CPC Class Codes: H04L 12/1428 Invoice generation, e.g. customization, lay-out, database processing, algorithms for calculating the bill or formatting invoices as WWW pages invoicing in general G06Q30/04
• CPC Class Codes: H04L 41/0803 Configuration setting
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/344 Out-of-band transfers
• CPC Class Codes: H04L 47/829 Topology based
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/30 Profiles

Abstract

Systems and methods for automated failure recovery of subsystems of a management system are described. The subsystems are built and modeled as services, and their management, specifically their failure recovery, is done in a manner similar to that of services and resources managed by the management system. The management system consists of a microkernel, service managers, and management services. Each service, whether a managed service or a management service, is managed by a service manager. The service manager itself is a service and so is in turn managed by the microkernel. Both managed services and management services are monitored via in-band and out-of-band mechanisms, and the performance metrics and alerts are transported through an event system to the appropriate service manager. If a service fails, the service manager takes policy-based remedial steps including, for example, restarting the failed service.

Claims

1. A system comprising:

at least one memory storage device to store a plurality of microkernel controllers and a plurality of pre-defined topologies; and

a service manager comprising one or more hardware processors, the plurality of microkernel controllers including at least one microkernel controller configured to deploy the service manager, the service manager configured to perform operations comprising;

parsing a first pre-defined topology from the plurality of pre-defined topologies, the first pre-defined topology specifying a plurality of logical types of resources to be used by a management service;

loading one or more controllers corresponding to the management service;

deploying the management service using the first pre-defined topology;

monitoring operation of the management service; and

dynamically re-deploying the management service using the first pre-defined topology responsive to a failure of the management service.

2. The system of claim 1, wherein the dynamically re-deploying the management service includes:

allocating a replacement node; and

moving the failed management service to the replacement node.

3. The system of claim 1, wherein the plurality of logical types of resources include at least one type of physical hardware resource and at least one type of software resource.

4. The system of claim 3, wherein the at least one type of physical hardware resource includes a server system.

5. The system of claim 3, wherein the at least one type of software resource includes an application program.

6. A computer-implemented method comprising:

receiving a pre-defined computing topology that specifies a plurality of logical types of computing resources to be used by a management service, the management service being one of a plurality of management services configured to collectively manage a plurality of domain services, the plurality of management services configured to collectively monitor the domain services and dynamically allocate computing resources to the plurality of domain services;

parsing the pre-defined computing topology;

loading one or more computing controllers corresponding to the management service;

deploying the management service using the pre-defined computing topology;

monitoring operation of the management service; and

dynamically re-deploying the management service using the pre-defined computing topology responsive to a failure of the management service, the dynamically re-deploying of the management service including;

allocating a replacement node; and

moving the failed management service to the replacement node.

7. The method of claim 6, further comprising deploying a service manager corresponding to the management service, wherein the service manager deploys and dynamically re-deploys the management service using the pre-defined computing topology.

8. The method of claim 6, further comprising detecting the failure of the management service.

9. The method of claim 6, wherein the dynamically re-deploying of the management service includes restarting the management service.

10. The method of claim 6, wherein the dynamically re-deploying of the management service includes identifying a recovery action to be performed according to a pre-defined policy.

11. A non-transitory computer-readable storage medium having instructions embodied thereon, the instructions executable by a processor for performing operations for managing one or more components of a management system, the operations comprising:

initiating a microkernel;

initiating a microkernel controller;

deploying, at the microkernel, a service manager using the microkernel controller;

parsing, at the service manager, a pre-defined topology that specifies a plurality of logical types of resources to be used by a management service, the management service being one of a plurality of management services configured to collectively manage a plurality of domain services, the plurality of management services configured to collectively monitor the domain services and dynamically allocate resources to the plurality of domain services;

loading, at the service manager, one or more controllers corresponding to the management service;

deploying the management service using the pre-defined topology;

detecting a failure of the management service; and

dynamically re-deploying the management service using the pre-defined topology responsive to the failure of the management service.

12. The non-transitory computer-readable storage medium of claim 11, wherein the operations further comprise, at the microkernel controller, registering the deployed service manager in the microkernel.

13. The non-transitory computer-readable storage medium of claim 11, wherein the pre-defined topology is received by the microkernel.

14. The non-transitory computer-readable storage medium of claim 11, wherein the pre-defined topology is a physical topology bound to a plurality of actual resources.

15. The non-transitory computer-readable storage medium of claim 11, wherein the operations further comprise identifying a recovery action to be performed according to a pre-defined policy associated with the management service.

16. The non-transitory computer-readable storage medium of claim 11, wherein each domain service of the plurality of domain services is managed by further service managers.