Accessing log files of a distributed computing system using a simulated file system

• US 10,628,235 B2
• Filed: 01/11/2018
• Issued: 04/21/2020
• Est. Priority Date: 01/11/2018
• Status: Active Grant

• Current Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Original Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Inventors: Mageswaran, Manjunath, Alluboyina, Ravi Kumar
• Primary Examiner(s): Teets, Bradley A
• Application Number:
  US15/868,771
• Publication Number:
  US 20190213054A1
• Time in Patent Office: 831 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: G06F 9/485 Task life-cycle, e.g. stopping, restarting, resuming execution G06F9/4881 takes precedence
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. A snapshot of the application may be created and used to rollback or clone the application. Clones snapshots of storage volumes may be gradually populated with data from prior snapshots to reduce loading on a primary snapshot. Components of cloned applications may communicate with one another using addresses of these components in the parent application. Changes to application state may be written to a remote storage volume that may be mounted to a new instance or clone of the application to recreate its state. Jobs to create and execute a bundled application may be referenced with a simulated file system that generates reads to hosts only when the job log file is actually read. Storage nodes may implement interfaces to a SAN or cloud storage system.

Claims

1. A method comprising:

orchestrating, by a first computing device in a distributed computing system, deployment of a plurality of roles in a bundled application in a plurality of second computing devices of the distributed computing system by processing a plurality of jobs, each job of the plurality of jobs being at least one of started by another job of the plurality of jobs and invoking starting of another job of the plurality of jobs, each role of the plurality of roles being an executable program;

defining, by the first computing device, a job hierarchy such that each node of the job hierarchy represents a job and any descendants of the each node are one or more jobs started by the job represented by the each node, each node of the job hierarchy referencing a second computing device of the distributed computing system on which the job represented by the each node is executing;

generating, by the plurality of second computing devices, log files during processing of at least a portion of the plurality of jobs;

receiving file system navigation commands with respect to the log files; and

processing the file system navigation commands in order to present a view of the job hierarchy without generating any queries or read requests to the second computing devices;

wherein the method further comprises;

moving a first job of the plurality of jobs from an initial second computing device to a subsequent second computing device;

recording, by the first computing device, identifiers of the initial and subsequent second computing devices in a node of the job hierarchy representing the first job;

receiving, a file system read command for a log file for the first job;

identifying the node in the job hierarchy representing the first job by traversing the job hierarchy; and

transmitting the file system read command to both of the initial second computing device and the subsequent second computing device for processing.

2. The method of claim 1, wherein the file system navigation commands include a change directory command, the method further comprising processing, by the first computing device, the change directory command by changing a current directory to be a node of the job hierarchy indicated in the change directory command.

3. The method of claim 1, wherein the file system navigation commands include a list command, the method further comprising processing, by the first computing device, the list command by providing a listing of a log file and descendent jobs of a node of the job hierarchy indicated by the list command.

4. The method of claim 1, further comprising:

receiving, the file system read command for a log file for a first job of the plurality of jobs;

identifying a corresponding computing device of the plurality of second computing devices executing the first job by traversing the job hierarchy; and

transmitting the file system read command to the corresponding computing device for processing.

5. The method of claim 1, further comprising executing the plurality of jobs on the plurality of second computing devices by agents executing on the plurality of second computing devices and invoked by a first job of the plurality of jobs executing on the first computing device.

6. The method of claim 5, wherein the first job of the plurality of jobs processes an application manifest defining the plurality of roles of the bundled application.

7. A method comprising:

orchestrating, by a first computing device in a distributed computing system, deployment of a plurality of roles in a bundled application in a plurality of second computing devices of the distributed computing system by processing a plurality of jobs, each job of the plurality of jobs being at least one of started by another job of the plurality of jobs and invoking starting of another job of the plurality of jobs, each role of the plurality of roles being an executable program;

defining, by the first computing device, a job hierarchy such that each node of the job hierarchy represents a job and any descendants of the each node are one or more jobs started by the job represented by the each node, each node of the job hierarchy referencing a second computing device of the distributed computing system on which the job represented by the each node is executing;

generating, by the plurality of second computing devices, log files during processing of at least a portion of the plurality of jobs;

receiving file system navigation commands with respect to the log files; and

processing the file system navigation commands in order to present a view of the job hierarchy without generating any queries or read requests to the second computing devices;

wherein the method further comprises, on each second computing device of the plurality of second computing devices;

providing a pool of threads;

providing a thread map;

(a) assigning a selected thread from the pool of threads to a selected job of a plurality of jobs assigned to the each second computing device;

(b) updating the thread map to include a mapping of the selected thread to a job identifier of the selected job;

(c) executing the selected job by the selected thread;

stopping execution of the selected job;

releasing the selected thread; and

removing the mapping of the selected thread to the selected job in the thread map.

8. The method of claim 7, further comprising, during execution of the selected job by the selected thread:

receiving a write request from the selected thread to write log data;

determining the job identifier of the selected job from the mapping of the selected thread to the job identifier of the selected job in the thread map; and

writing the log data to a log file corresponding to job identifier of the selected job.

9. The method of claim 8, further comprising, after stopping execution of the selected job:

initiating another phase of execution of the selected job on the each second computing device; and

in response to initiating another phase of execution, repeating (a)-(c).

10. A system comprising:

a distributed computing system comprising a first computing device and a plurality of second computing devices;

wherein the first computing device programmed to;

orchestrate deployment of a plurality of roles in a bundled application in the plurality of second computing devices of the distributed computing system by processing a plurality of jobs, each job of the plurality of jobs being at least one of started by another job of the plurality of jobs and invoking starting of another job of the plurality of jobs, each role of the plurality of roles being an executable program;

define a job hierarchy such that each node of the job hierarchy represents a job and any descendants of the each node are one or more jobs started by the job represented by the each node, each node of the job hierarchy referencing a second computing device of the distributed computing system on which the job represented by the each node is executing;

wherein each second computing device of the plurality of computing devices programmed to generate log files during processing of at least a portion of the plurality of jobs; and

wherein the first computing device is further programmed to;

receive file system navigation commands with respect to the log files; and

process the file system navigation commands in order to present a view of the job hierarchy without generating any queries or read requests to the second computing devices;

wherein the first computing device is further programmed to;

invoke moving of a first job of the plurality of jobs from an initial second computing device to a subsequent second computing device;

record identifiers of the initial and subsequent second computing devices in a node of the job hierarchy representing the first job;

receive a file system read command for a log file for the first job;

identify the node in the job hierarchy representing the first job by traversing the job hierarchy; and

transmit the file system read command to both of the initial second computing device and the subsequent second computing device for processing.

11. The system of claim 10, wherein the file system navigation commands include a change directory command, the first computing device being further programmed to process the change directory command by changing a current directory to be a node of the job hierarchy indicated in the change directory command.

12. The system of claim 10, wherein the file system navigation commands include a list command, the first computing device being further programmed to process the list command by providing a listing of a log file and descendent jobs of a node of the job hierarchy indicated by the list command.

13. The system of claim 10, wherein the first computing device is further programmed to:

receive the file system read command for a log file for a first job of the plurality of jobs;

identify a corresponding computing device of the plurality of second computing devices executing the first job by traversing the job hierarchy; and

transmit the file system read command to the corresponding computing device for processing.

14. The system of claim 10, wherein the plurality of second computing devices are programmed to execute the plurality of jobs on the plurality of second computing devices by agents executing on the plurality of second computing devices and invoked by a first job of the plurality of jobs executing on the first computing device.

15. The system of claim 14, wherein the first computing device is programmed to execute the first job of the plurality of jobs to process an application manifest defining the plurality of roles of the bundled application.

16. A system comprising:

a distributed computing system comprising a first computing device and a plurality of second computing devices;

wherein the first computing device programmed to;

orchestrate deployment of a plurality of roles in a bundled application in the plurality of second computing devices of the distributed computing system by processing a plurality of jobs, each job of the plurality of jobs being at least one of started by another job of the plurality of jobs and invoking starting of another job of the plurality of jobs, each role of the plurality of roles being an executable program;

define a job hierarchy such that each node of the job hierarchy represents a job and any descendants of the each node are one or more jobs started by the job represented by the each node, each node of the job hierarchy referencing a second computing device of the distributed computing system on which the job represented by the each node is executing;

wherein each second computing device of the plurality of second computing devices programmed to generate log files during processing of at least a portion of the plurality of jobs; and

wherein the first computing device is further programmed to;

receive file system navigation commands with respect to the log files; and

process the file system navigation commands in order to present a view of the job hierarchy without generating any queries or read requests to the second computing devices;

wherein each second computing device of the plurality of second computing devices is programmed to;

provide a pool of threads;

provide a thread map;

(a) assign a selected thread from the pool of threads to a selected job of a plurality of jobs assigned to the each second computing device;

(b) update the thread map to include a mapping of the selected thread to a job identifier of the selected job;

(c) execute the selected job by the selected thread;

stop execution of the selected job;

release the selected thread; and

remove the mapping of the selected thread to the selected job in the thread map.

17. The system of claim 16, wherein each second computing device of the plurality of computing devices is programmed to, during execution of the selected job by the selected thread:

receive a write request from the selected thread to write log data;

determine the job identifier of the selected job from the mapping of the selected thread to the job identifier of the selected job in the thread map; and

write the log data to a log file corresponding to job identifier of the selected job.

18. The system of claim 17, wherein each second computing device of the plurality of second computing devices is programmed to, after stopping execution of the selected job:

initiate another phase of execution of the selected job on the each second computing device; and

in response to initiating another phase of execution, repeat (a)-(c).