Monitoring containers in a distributed computing system

• US 10,642,694 B2
• Filed: 01/12/2018
• Issued: 05/05/2020
• Est. Priority Date: 01/12/2018
• Status: Active Grant

• Current Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Original Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Inventors: Mageswaran, Manjunath, Alluboyina, Ravi Kumar, Kesavan, Giridharan
• Primary Examiner(s): Ko, Chae M
• Application Number:
  US15/870,296
• Publication Number:
  US 20190220361A1
• Time in Patent Office: 844 Days
• Field of Search: 714 381
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. An orchestration layer implements a multi-role application that is provisioned with virtualized storage and computation resources. A snapshot of the application may be created and used to rollback or clone the application. Containers may be loaded with data structures for encrypted and/or authenticated communication. Failed storage volumes may be detected and repaired. Repopulating repaired volumes may be managed by an application or a storage manager implementing a virtualized storage system. An autopilot invokes repair of failed containers and storage nodes in response to detected failures.

Claims

1. A method comprising:

deploying, by a first computing device, a bundled application on a plurality of computing devices of a distributed computing system, the bundled application including a plurality of containers executing on the plurality of computing devices, each container of the plurality of containers executing an application instance forming part of the bundled application;

monitoring, by the first computing device, a state of each container of the plurality of containers; and

invoking at least one of restarting, redeploying, and moving containers of the plurality of containers, by the first computing device, effective to maintain a state of the bundled application in response to any faults indicated by the monitoring;

(a) detecting, by the first computing device, failure of a first container of the plurality of containers executing on a second computing device of the plurality of computing devices;

(b) in response to (a), invoking restarting of the first container on the second computing device;

(c) determining that (b) did not resolve the failure of the first container; and

(d) in response to (c), invoking redeploying of the first container on the second computing device.

2. The method of claim 1, further comprising:

(e) determining that (d) did not resolve failure of the first container;

(f) in response to (e), invoking deployment of a second container on a third computing device of the plurality of computing devices and restoring a state of the first container in the second container.

3. The method of claim 1, further comprising:

deploying a plurality of first containers of the plurality of containers on a second computing device of the plurality of computing devices;

associating each container of the plurality of first containers with a port on the second computing device; and

periodically querying, by an agent, the port associated with each container of the plurality of first containers to determine a state of the each container, the agent being software executing on one of the second computing device and another computing device.

4. The method of claim 1, wherein a plurality of storage volumes are mounted to the plurality of containers.

5. The method of claim 4, further comprising:

monitoring, by the first computing device, a state of the plurality of storage volumes;

in response to failure of a first storage volume of the plurality of storage volumes;

invoking re-provisioning and remounting of a new storage volume to replace the first storage volume; and

instructing the bundled application to re-sync the new storage volume from one or more replicas of the first storage volume.

6. The method of claim 4, wherein the plurality of storage volumes each include one or more replicas, the method further comprising:

in response to failure of a first storage volume of the plurality of storage volumes;

invoking re-provisioning and remounting of a new storage volume to replace the first storage volume; and

restoring the new storage volume to include contents of the first storage volume using the one or more replicas of the first storage volume.

7. A method comprising:

deploying, by a first computing device, a bundled application on a plurality of computing devices of a distributed computing system, the bundled application including a plurality of containers executing on the plurality of computing devices, each container of the plurality of containers executing an application instance forming part of the bundled application;

monitoring, by the first computing device, a state of each container of the plurality of containers;

invoking at least one of restarting, redeploying, and moving containers of the plurality of containers, by the first computing device, effective to maintain a state of the bundled application in response to any faults indicated by the monitoring;

deploying a plurality of first containers of the plurality of containers on a second computing device of the plurality of computing devices;

associating each container of the plurality of first containers with a port on the second computing device;

periodically querying, by an agent, the port associated with each container of the plurality of first containers to determine a state of the each container, the agent being software executing on one of the second computing device and another computing device;

initiating a parent process on the second computing device;

starting, by the parent process a container daemon;

initiating, by the container daemon the plurality of first containers; and

implementing a socket on the second computing device by the container daemon;

wherein associating each container of the plurality of first containers with the port on the second computing device comprises associating, by the container daemon, each container of the plurality of first containers with a port of the socket.

8. The method of claim 7, further comprising:

obtaining, by the agent, a mapping of the port of the socket with the plurality of first containers from the container daemon; and

querying, by the agent, the port associated with each container of the plurality of first containers according to the mapping.

9. A system comprising:

a first computing device comprising one or more processing devices and one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to;

deploy a bundled application on a plurality of computing devices of a distributed computing system, the bundled application including a plurality of containers executing on the plurality of computing devices, each container of the plurality of containers executing an application instance forming part of the bundled application;

monitor a state of each container of the plurality of containers; and

invoke at least one of restarting, redeploying, and moving containers of the plurality of containers effective to maintain a state of the bundled application in response to any faults indicated by the monitoring;

wherein the executable code is further effective to cause the one or more processing devices to;

(a) detect failure of a first container of the plurality of containers executing on a second computing device of the plurality of computing devices;

(b) in response to (a), invoke restarting of the first container on the second computing device;

(c) determine that (b) did not resolve the failure of the first container; and

(d) in response to (c), invoke redeploying of the first container on the second computing device.

10. The system of claim 9, wherein the executable code is further effective to cause the one or more processing devices to:

(e) determine that (d) did not resolve failure of the first container;

(f) in response to (e), invoke deployment of a second container on a third computing device of the plurality of computing devices and restoring a state of the first container in the second container.

11. The system of claim 9, wherein the executable code is further effective to cause the one or more processing devices to:

deploy a plurality of first containers of the plurality of containers on a second computing device of the plurality of computing devices;

associate each container of the plurality of first containers with a port on the second computing device; and

invoke periodically querying, by an agent, the port associated with each container of the plurality of first containers to determine a state of the each container, the agent being software executing on one of the second computing device and another computing device.

12. The system of claim 9, further comprising a plurality of storage volumes mounted to the plurality of containers.

13. The system of claim 12, wherein the executable code is further effective to cause the one or more processing devices to:

monitor a state of the plurality of storage volumes;

in response to failure of a first storage volume of the plurality of storage volumes;

invoke re-provisioning and remounting of a new storage volume to replace the first storage volume; and

instruct the bundled application to re-sync the new storage volume from one or more replicas of the first storage volume.

14. The system of claim 12, wherein the plurality of storage volumes each include one or more replicas;

wherein the executable code is further effective to cause the one or more processing devices to;

in response to failure of a first storage volume of the plurality of storage volumes;

invoke re-provisioning and remounting of a new storage volume to replace the first storage volume; and

restore the new storage volume to include contents of the first storage volume using the one or more replicas of the first storage volume.

15. A system comprising:

a first computing device comprising one or more processing devices and one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to;

deploy a bundled application on a plurality of computing devices of a distributed computing system, the bundled application including a plurality of containers executing on the plurality of computing devices, each container of the plurality of containers executing an application instance forming part of the bundled application;

monitor a state of each container of the plurality of containers; and

invoke at least one of restarting, redeploying, and moving containers of the plurality of containers effective to maintain a state of the bundled application in response to any faults indicated by the monitoring;

deploy a plurality of first containers of the plurality of containers on a second computing device of the plurality of computing devices;

associate each container of the plurality of first containers with a port on the second computing device; and

invoke periodically querying, by an agent, the port associated with each container of the plurality of first containers to determine a state of the each container, the agent being software executing on one of the second computing device and another computing device; and

the second computing device, the second computing device further programmed to;

initiate a parent process on the second computing device;

start, by the parent process a container daemon;

initiate, by the container daemon the plurality of first containers; and

implement a socket on the second computing device by the container daemon;

associate each container of the plurality of first containers with the port on the second computing device by associating, by the container daemon, each container of the plurality of first containers with a port of the socket.

16. The system of claim 15, wherein the agent is further programmed to:

obtain a mapping of the port of the socket with the plurality of first containers from the container daemon; and

query the port associated with each container of the plurality of first containers according to the mapping.