Application bundle management across mixed file system types

• US 9,900,374 B2
• Filed: 06/27/2015
• Issued: 02/20/2018
• Est. Priority Date: 05/21/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Fawcett, Bradley W.
• Primary Examiner(s): Cheema, Umar
• Assistant Examiner(s): Williams, Tony
• Application Number:
  US14/752,849
• Publication Number:
  US 20160342607A1
• Time in Patent Office: 969 Days
• Field of Search: None
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Disclosed aspects include managing a file system having a cluster of compute nodes to process a stream of tuples using an application bundle. A cluster bundle directory is established on the cluster of compute nodes. A host bundle library is established on a first compute node of the cluster of compute nodes. The stream of tuples to be processed by a plurality of processing elements operating on the cluster of compute nodes including the first compute node is received. Using the cluster bundle directory, it is detected that the application bundle is not installed on the first compute node. Using both the cluster bundle directory and the host bundle library, a determination is made to initiate a sourcing operation with respect to the application bundle. In response to initiating the sourcing operation, the stream of tuples is processed using the application bundle.

Claims

1. A system for managing a file system having a cluster of compute nodes to process a stream of tuples using an application bundle, the system comprising:

a memory having a set of computer readable computer instructions, and

a processor for executing the set of computer readable instructions, the set of computer readable instructions including;

establishing, on the cluster of compute nodes, a cluster bundle directory, wherein the cluster bundle directory is a first level of a multilevel directory structure, and wherein the cluster bundle directory is a global directory that tracks which host has the application bundle installed;

establishing, on a first compute node of the cluster of compute nodes, a host bundle library, wherein the host bundle library is a second level of the multilevel directory structure, and wherein the host bundle library is a local library that tracks which host shares the application bundle;

receiving the stream of tuples to be processed by a plurality of processing elements operating on the cluster of compute nodes including the first compute node;

detecting, using the cluster bundle directory, that the application bundle is not installed on the first compute node;

determining, using both the cluster bundle directory and the host bundle library, to initiate a sourcing operation with respect to the application bundle; and

processing, in response to initiating the sourcing operation, the stream of tuples using the application bundle.

2. The system of claim 1, wherein the cluster bundle directory indicates installation of at least one application bundle on a group of the cluster of compute nodes.

3. The system of claim 1, wherein the host bundle library indicates a set of application bundle sharing states which map to the cluster of compute nodes.

4. The system of claim 1, wherein determining, using both the cluster bundle directory and the host bundle library, to initiate the sourcing operation with respect to the application bundle includes:

determining to install the application bundle on the first compute node; and

determining, based on the cluster bundle directory, to download the application bundle from a second compute node.

5. The system of claim 1, wherein the sourcing operation includes:

downloading the application bundle from a second compute node; and

installing the application bundle on the first compute node.

6. The system of claim 1, wherein determining, using both the cluster bundle directory and the host bundle library, to initiate the sourcing operation with respect to the application bundle includes:

determining, based on the host bundle library, an application bundle sharing state which maps to a second compute node; and

determining, based on the cluster bundle directory, the second compute node shares the application bundle.

7. The system of claim 1, wherein the sourcing operation includes:

comparing, by analyzing a latency element associated with a second compute node, the latency element with a threshold latency; and

accessing, without downloading the application bundle from the second compute node and without installing the application bundle on the first compute node, the application bundle on the second compute node in response to the threshold latency exceeding the latency element.

8. The system of claim 1, further comprising:

updating, in response to initiating the sourcing operation, the cluster bundle directory.

9. The system of claim 1, further comprising, in response to processing the stream of tuples using the application bundle:

detecting, based on both the cluster bundle directory and the host bundle library, a triggering event;

determining to remove the application bundle in response to detecting the triggering event; and

removing the application bundle in response to determining to remove the application bundle.

10. A computer program product for managing a file system having a cluster of compute nodes to process a stream of tuples using an application bundle, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:

establishing, on the cluster of compute nodes, a cluster bundle directory, wherein the cluster bundle directory is a first level of a multilevel directory structure, and wherein the cluster bundle directory is a global directory that tracks which host has the application bundle installed;

establishing, on a first compute node of the cluster of compute nodes, a host bundle library, wherein the host bundle library is a second level of the multilevel directory structure, and wherein the host bundle library is a local library that tracks which host shares the application bundle;

receiving the stream of tuples to be processed by a plurality of processing elements operating on the cluster of compute nodes including the first compute node;

detecting, using the cluster bundle directory, that the application bundle is not installed on the first compute node;

determining, using both the cluster bundle directory and the host bundle library, to initiate a sourcing operation with respect to the application bundle; and

processing, in response to initiating the sourcing operation, the stream of tuples using the application bundle.

11. The computer program product of claim 10, wherein the program instructions are stored in a computer readable storage medium in a data processing system, and the program instructions were downloaded over a network from a remote data processing system.

12. The computer program product of claim 10, wherein the program instructions are stored in a computer readable storage medium in a server data processing system, and the program instructions are downloaded over a network to a remote data processing system for use in a computer readable storage medium with the remote data processing system.