Automated failure recovery of subsystems in a management system

• US 9,009,521 B2
• Filed: 07/07/2010
• Issued: 04/14/2015
• Est. Priority Date: 07/31/2009
• Status: Active Grant

• Current Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Original Assignee: eBay Inc.
• Inventors: Jaisinghani, Devendra Rajkumar
• Primary Examiner(s): Leibovich, Yair
• Application Number:
  US12/831,929
• Publication Number:
  US 20110029810A1
• Time in Patent Office: 1,742 Days
• Field of Search: 714/4.1, 714/48, 711/154
• US Class Current: 714/4.1
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 9/4401 Bootstrapping security arrangements therefor G06F21/57
• CPC Class Codes: H04L 12/14 Charging , metering or billing arrangements for data wireline or wireless communications
• CPC Class Codes: H04L 12/1428 Invoice generation, e.g. customization, lay-out, database processing, algorithms for calculating the bill or formatting invoices as WWW pages invoicing in general G06Q30/04
• CPC Class Codes: H04L 41/0803 Configuration setting
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/344 Out-of-band transfers
• CPC Class Codes: H04L 47/829 Topology based
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/30 Profiles

Abstract

Systems and methods for automated failure recovery of subsystems of a management system are described. The subsystems are built and modeled as services, and their management, specifically their failure recovery, is done in a manner similar to that of services and resources managed by the management system. The management system consists of a microkernel, service managers, and management services. Each service, whether a managed service or a management service, is managed by a service manager. The service manager itself is a service and so is in turn managed by the microkernel. Both managed services and management services are monitored via in-band and out-of-band mechanisms, and the performance metrics and alerts are transported through an event system to the appropriate service manager. If a service fails, the service manager takes policy-based remedial steps including, for example, restarting the failed service.

Claims

1. A system comprising:

a processor; and

at least one memory storage device having stored thereon instructions executable by the processor to cause the system to perform operations comprising;

deploying a service manager using a microkernel controller of a plurality of microkernel controllers, the service manager corresponding to a management service that is one of a plurality of management services configured to collectively manage a plurality of domain services, the plurality of management services being configured to collectively monitor the domain services and dynamically allocate resources to the plurality of domain services such that each of the plurality of domain services maintains a predefined performance level;

receiving and parsing, at the service manager, a first pre-defined topology of a plurality of pre-defined topologies, the first pre-defined topology specifying a plurality of logical types of resources to be used by the management service;

loading, at the service manager, one or more controllers corresponding to the management service;

deploying the management service using first the pre-defined topology, including;

for a first one of the plurality of logical types of resources, binding a first respective quantity of instances of a type of physical hardware resource to the management service; and

for a second one of the plurality of logical types of resources, binding a second respective quantity of instances of a type of software resource to the management service;

in response to a failure of at least one of the first quantity of instances of a type of physical hardware resource or a failure of at least one of the second quantity of instances of a type of software resource, dynamically re-deploying the management service using the first pre-defined topology, including;

for the first logical type of resource, binding at least one different instance of the type of physical hardware to the management service;

or

for the second logical type of resource, binding at least one different instance of the type of software resource to the management service.

2. The system of claim 1, wherein the instructions executable by the processor to cause the system to instantiate the microkernel controller upon startup.

3. The system of claim 1, wherein the instructions are executable by the processor to cause the system to deploy further service managers corresponding to the plurality of domain services.

4. The system of claim 1, wherein instructions are executable by the processor to cause the system to detect a failure of the management service or the service manager.

5. The system of claim 4, wherein instructions are executable by the processor to cause the system to:

allocate a replacement node and move the failed management service or service manager to the replacement node.

6. The system of claim 1, wherein the type of physical hardware resource is a server system.

7. The system of claim 1, wherein the type of software resource is an application program.

8. A method comprising:

at a microkernel, deploying a service manager using a microkernel controller, the service manager corresponding to a management service, the management service being one of a plurality of management services configured to collectively manage a plurality of domain services, the plurality of management services configured to collectively monitor the domain services and dynamically allocate resources to the plurality of domain services such that each of the plurality of domain services maintains a predefined performance level;

receiving and parsing, at the service manager, a first pre-defined topology that specifies a plurality of logical types of resources to be used by the management service;

loading, at the service manager, one or more controllers corresponding to the management service;

deploying the management service using first the pre-defined topology, including;

for a first one of the plurality of logical types of resources, binding a first respective quantity of instances of a type of physical hardware resource to the management service; and

for a second one of the plurality of logical types of resources, binding a second respective quantity of instances of a type of software resource to the management service;

in response to a failure of at least one of the first quantity of instances of a type of physical hardware resource or a failure of at least one of the second quantity of instances of a type of software resource, dynamically re-deploying the management service using the first pre-defined topology, including;

for the first logical type of resource, binding at least one different instance of the type of physical hardware to the management service;

or

for the second logical type of resource, binding at least one different instance of the type of software resource to the management service.

9. The method of claim 8, further comprising deploying the plurality of domain services from further service managers.

10. The method of claim 8, further comprising detecting a failure of the management service or the service manager.

11. The method of claim 10, further comprising restarting the failed management service or the failed service manager.

12. The method of claim 10, further comprising identifying a recovery action to be performed according to a pre-defined policy.

13. A non-transitory computer readable storage medium having instructions embodied thereon, the instructions executable by a processor for performing a method for managing one or more components of a management system, the method comprising:

at a microkernel, deploying a service manager using a microkernel controller, the service manager corresponding to a management service, the management service being one of a plurality of management services configured to collectively manage a plurality of domain services, the plurality of management services configured to collectively monitor the domain services and dynamically allocate resources to the plurality of domain services such that each of the plurality of domain services maintains a predefined performance level;

receiving and parsing, at the service manager, a first pre-defined topology that specifies a plurality of logical types of resources to be used by the management service;

loading, at the service manager, one or more controllers corresponding to the management service;

deploying the management service using first the pre-defined topology, including;

for a first one of the plurality of logical types of resources, binding a first respective quantity of instances of a type of physical hardware resource to the management service; and

for a second one of the plurality of logical types of resources, binding a second respective quantity of instances of a type of software resource to the management service;

in response to a failure of at least one of the first quantity of instances of a type of physical hardware resource or a failure of at least one of the second quantity of instances of a type of software resource, dynamically re-deploying the management service using the first pre-defined topology, including;

for the first logical type of resource, binding at least one different instance of the type of physical hardware to the management service;

or

for the second logical type of resource, binding at least one different instance of the type of software resource to the management service.

14. The computer readable storage medium of claim 13, wherein the method further comprises:

initiating the microkernel; and

instantiating the microkernel controller.

15. The computer-readable storage medium of claim 13, wherein failure of the at least one of the first quantity of instances of the type of physical hardware resource occurs, and for the first logical type of resource, at least one different instance of the type of physical hardware is bound to the management service.

16. The computer-readable storage medium of claim 13, wherein each domain service of the plurality of domain services is managed by further service managers.

17. The computer-readable storage medium of claim 13, wherein the method further comprises, at the microkernel controller, registering the deployed service manager in the microkernel.