Container management and application ingestion engine

• US 10,389,598 B2
• Filed: 10/27/2016
• Issued: 08/20/2019
• Est. Priority Date: 10/29/2015
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Padala, Pradeep, Kadirvel, Selvi, Raj, Himanshu, Kamity, Kiran
• Primary Examiner(s): Neurauter, Jr., George C
• Application Number:
  US15/336,360
• Publication Number:
  US 20170126432A1
• Time in Patent Office: 1,027 Days
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H04L 12/4641 Virtual LANs, VLANs, e.g. virtual private networks [VPN] LAN interconnection over a bridge based backbone H04L12/462; encapsulation techniques H04L12/4633; routing of packets H04L45/00; packet switches H04L49/00; virtual private networks for security H04L63/0272
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0803 Configuration setting
• CPC Class Codes: H04L 41/0806 for initial configuration or provisioning, e.g. plug-and-play
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 41/0894 Policy-based network configuration management
• CPC Class Codes: H04L 41/0895 Configuration of virtualised networks or elements, e.g. virtualised network function or OpenFlow elements
• CPC Class Codes: H04L 41/14 Network analysis or design
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

In one embodiment, a system has host machines forming a cluster. Each host machine runs containers, where each container includes a segment of hardware resources associated with the host machine, a segment of an operating system utilized by the host machine, and at least one application. Host agents operate on the host machines. Each host agent collects operational parameters associated with the containers on each host machine. A management platform is operative to divide the cluster into container pools, where each container pool includes a sub-set of computation resources in the cluster and has associated container pool metrics including a priority level and computation resource limits. Operational parameters are collected from the host agents. The operational parameters are evaluated in accordance with the container pool metrics.

Claims

1. A method for managing containers in a cluster, the method comprising:

collecting operational parameters from a plurality of host agents operating on respective host machines in the cluster;

evaluating the operational parameters in accordance with metrics associated with container pools, wherein each container pool includes a plurality of containers;

issuing commands to the plurality of host agents, wherein the commands include operating system commands enforced at a container to facilitate operation of the cluster in accordance with the metrics associated with container pools;

determining whether a current imbalance value of the cluster is over a threshold;

identifying a first host machine from the host machines in the cluster for container-balancing;

selecting a first number of containers on the first host machine based on maximum resource usage, wherein the first number of containers is selected only from containers that have two or more replicas; and

stopping the first number of containers on the first host machine and restarting the first number of containers on a second host machine, wherein the second host machine is different than the first host machine.

2. The method of claim 1, wherein:

the cluster is divided into the container pools; and

each container pool includes a sub-set of computation resources in the cluster.

3. The method of claim 1, wherein:

each container includes a segment of hardware resources associated with a host machine, a segment of an operating system utilized by the host machine in the cluster, and at least one application.

4. The method of claim 1, wherein:

the metrics associated with a given container pool include a priority level and computation resource limits.

5. The method of claim 4, wherein:

the computation resource limits associated with all of the container pools in the cluster sums up to over 100 percent utilization of resources in the cluster.

6. The method of claim 1, further comprising:

responding to queries from the host agents for updated metrics associated with the container pools.

7. The method of claim 1, wherein issuing commands to the host agents comprises:

providing resource limits to a host agent to enable the host agent to ascertain a desired resource target based on current and past utilization and apply a corrective measure to achieve the desired resource target.

8. The method of claim 7, wherein the host agent is configured to apply hysteresis correction by varying refresh rates and/or alter an amount by which resource utilization is corrected on each iteration.

9. The method of claim 1, further comprising:

determining a subset of the first number of containers that have placement constraints satisfied by the second host machine.

10. The method of claim 9, wherein upon determining that the subset of the first number of containers is empty, selecting a new host machine on which the first number of containers are to be restarted.

11. The method of claim 1, wherein the current imbalance value is determined based on the operation parameters by determining resource usage deviation for each host machine, wherein deviation is defined by host usage minus an average host usage of the cluster, and determining the current imbalance value of the cluster, wherein imbalance is defined as 1 divided by a number of host machines in the cluster multiplied by a sum of squares of resource usage deviations.

12. The method of claim 11, wherein the second host machine has a lowest resource usage deviation.

13. The method of claim 1, further comprising:

evaluating a new imbalance value as if the first number of containers are stopped on the first host machine and restarted on the second host machine; and

determining whether the new imbalance value improves over the current imbalance value.

14. The method of claim 13, further comprising:

restarting one or more ones of the first number of containers on the second host machine.

15. The method of claim 13, further comprising:

selecting the first number of containers to be stopped on the first host machine and restarted on a third host machine.

16. The method of claim 13, further comprising:

selecting a second number of containers to be stopped on the first host machine and restarted on the second host machine, wherein the second number of containers is one less than the first number of containers.

17. The method of claim 1, wherein determining the current imbalance value does not factor in the cost to stop and restart containers.

18. A platform for managing containers in a cluster, the platform comprising:

one or more non-transient storage media for storing instructions for managing containers; and

one or more processors operable to execute the instructions to;

collect operational parameters from a plurality of host agents operating on respective host machines in a cluster;

evaluate the operational parameters in accordance with metrics associates with container pools, wherein each container pool includes a plurality of containers;

issue commands to the plurality of host agents, wherein the commands include operating system commands enforced at a container to facilitate operation of the cluster in accordance with the metrics defined for container pools;

determine whether a current imbalance value of the cluster is over a threshold;

identify a first host machine from the host machines in the cluster for container-balancing;

select a first number of containers on the first host machine based on maximum resource usage, wherein the first number of containers is selected only from containers that have two or more replicas; and

stop the first number of containers on the first host machine and restart the first number of containers on a second host machine, wherein the second host machine is different than the first host machine.

22. The platform of claim 18, wherein the one or more processors are further operable to execute the instructions to:

evaluate a new imbalance value as if the first number of containers are stopped on the first host machine and restarted on the second host machine; and

determine whether the new imbalance value improves over the current imbalance value.

23. The platform of claim 22, wherein the one or more processors are further operable to execute the instructions to:

select a second number of containers to be stopped on the first host machine and restarted on the second host machine, wherein the second number of containers is one less than the first number of containers.

19. One or more non-transient computer-readable media having instructions stored thereon, when executed by one or more processors the instructions implement the following operations:

collecting operational parameters from a plurality of host agents operating on respective host machines in a cluster;

evaluating the operational parameters in accordance with metrics associated with container pools, wherein each container pool includes a plurality of containers;

issuing commands to the plurality of host agents, wherein the commands include operating system commands enforced at a container to facilitate operation of the cluster in accordance with the metrics defined for container pool;

determining whether a current imbalance value of the cluster is over a threshold;

identifying a first host machine from the host machines in the cluster for container-balancing;

selecting a first number of containers on the first host machine based on maximum resource usage, wherein the first number of containers is selected only from containers that have two or more replicas; and

stopping the first number of containers on the first host machine and restarting the first number of containers on a second host machine, wherein the second host machine is different than the first host machine.

20. The one or more non-transient computer-readable media of claim 19, wherein:

the metrics associated with a given container pool includes computation resource limits; and

the computation resource limits associated with all of the container pools in the cluster sums up to over 100 percent utilization of resource in the cluster.

21. The one or more non-transient computer-readable media of claim 19, wherein the instructions further implement the following operations

evaluating a new imbalance value as if the first number of containers are stopped on the first host machine and restarted on the second host machine;

and

determining whether the new imbalance value improves over the current imbalance value.