Optimized creation of distributed storage and distributed processing clusters on demand

• US 10,379,835 B2
• Filed: 12/19/2016
• Issued: 08/13/2019
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Agarwal, Anjali, Arora, Rachit, Gandhi, Deepashree, Jain, Dharmesh K., Kulkarni, Shrinivas S.
• Primary Examiner(s): Zhen, Wei Y
• Assistant Examiner(s): Wu, Junchun
• Application Number:
  US15/383,530
• Publication Number:
  US 20180173512A1
• Time in Patent Office: 967 Days
• Field of Search: 717175
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 8/63 Image based installation; Cloning; Build to order
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416

Abstract

A mechanism is provided in a data processing system for creating clusters on demand. The mechanism installs a cluster on the data processing system. The cluster comprises a master node and a managed node having distributed software installed thereon. The mechanism stores state of cluster on external volumes and removes references specific to the cluster from the state of the cluster stored on the external volumes. The mechanism takes a snapshot of the state of the cluster stored on the external volumes to form a set of volume templates and takes a snapshot of images of the master node and the managed node to form a set of node type images. Responsive to receiving a request to create a specified cluster, the mechanism creates a new set of nodes on the data processing system based on the set of node type images, clones the set of volume templates to form a new set of volumes in the data processing system, mounts the new set of volumes to the new set of nodes, and recreates configurations based on details in the request.

Claims

1. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

install a cluster on the data processing system, wherein the cluster comprises a master node and a managed node having distributed software installed thereon;

store state of cluster on external volumes;

remove references specific to the cluster from the state of the cluster stored on the external volumes;

take a snapshot of the state of the cluster stored on the external volumes to form a set of volume templates;

take a snapshot of images of the master node and the managed node to form a set of node type images; and

responsive to receiving a request to create a specified cluster, create a new set of nodes on the data processing system based on the set of node type images, clone the set of volume templates to form a new set of volumes in the data processing system, mount the new set of volumes to the new set of nodes, and recreate configurations based on details in the request,

wherein cloning the set of volume templates comprises allocating fixed space to each volume, setting a space quota for logs, data, and metadata, and creating a file system from a sparse file to serve as the volumes.

2. The computer program product of claim 1, wherein the computer readable program further causes the computing device to start cluster services for the specified cluster.

3. The computer program product of claim 1, wherein installing the cluster comprises installing binaries and libraries in the master node and the managed node.

4. The computer program product of claim 1, wherein the state of the cluster comprises configurations, metadata, and initial distributed file system state.

5. The computer program product of claim 1, wherein the request specifies roles of nodes, size of nodes, and number of nodes.

6. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

installing a cluster on the data processing system, wherein the cluster comprises a master node and a managed node having distributed software installed thereon;

storing state of cluster on external volumes;

removing references specific to the cluster from the state of the cluster stored on the external volumes;

taking a snapshot of the state of the cluster stored on the external volumes to form a set of volume templates;

taking a snapshot of images of the master node and the managed node to form a set of node type images; and

responsive to receiving a request to create a specified cluster, creating a new set of nodes on the data processing system based on the set of node type images, cloning the set of volume templates to form a new set of volumes in the data processing system, mounting the new set of volumes to the new set of nodes, and recreating configurations based on details in the request,

wherein cloning the set of volume templates comprises allocating fixed space to each volume, setting a space quota for logs, data, and metadata, and creating a file system from a sparse file to serve as the volumes.

7. The apparatus of claim 6, further comprising starting cluster services for the specified cluster.

8. The apparatus of claim 6, wherein installing the cluster comprises installing binaries and libraries in the master node and the managed node.

9. The apparatus of claim 6, wherein the state of the cluster comprises configurations, metadata, and initial distributed file system state.

10. The apparatus of claim 6, wherein the request specifies roles of nodes, size of nodes, and number of nodes.