Optimized creation of distributed storage and distributed processing clusters on demand

• US 10,379,836 B2
• Filed: 01/23/2018
• Issued: 08/13/2019
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Agarwal, Anjali, Arora, Rachit, Gandhi, Deepashree, Jain, Dharmesh K., Kulkarni, Shrinivas S.
• Primary Examiner(s): Zhen, Wei Y
• Assistant Examiner(s): Wu, Junchun
• Application Number:
  US15/877,717
• Publication Number:
  US 20180173513A1
• Time in Patent Office: 567 Days
• Field of Search: 717175
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 8/63 Image based installation; Cloning; Build to order
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416

Abstract

A mechanism is provided in a data processing system for creating clusters on demand. The mechanism installs a cluster on the data processing system. The cluster comprises a master node and a managed node having distributed software installed thereon. The mechanism stores state of cluster on external volumes and removes references specific to the cluster from the state of the cluster stored on the external volumes. The mechanism takes a snapshot of the state of the cluster stored on the external volumes to form a set of volume templates and takes a snapshot of images of the master node and the managed node to form a set of node type images. Responsive to receiving a request to create a specified cluster, the mechanism creates a new set of nodes on the data processing system based on the set of node type images, clones the set of volume templates to form a new set of volumes in the data processing system, mounts the new set of volumes to the new set of nodes, and recreates configurations based on details in the request.

Claims

1. A method, in a data processing system, for creating clusters on demand, the method comprising:

installing a cluster on the data processing system, wherein the cluster comprises a master node and a managed node having distributed software installed thereon;

storing state of the cluster on external volumes;

removing references specific to the cluster from the state of the cluster stored on the external volumes;

taking a snapshot of the state of the cluster stored on the external volumes to form a set of volume templates;

taking a snapshot of images of the master node and the managed node to form a set of node type images; and

responsive to receiving a request to create a specified cluster, creating a new set of nodes on the data processing system based on the set of node type images, cloning the set of volume templates to form a new set of volumes in the data processing system, mounting the new set of volumes to the new set of nodes, and recreating configurations based on details in the request,

wherein cloning the set of volume templates comprises allocating fixed space to each volume, setting a space quota for logs, data, and metadata, and creating a file system from a sparse file to serve as the volumes.

2. The method of claim 1, further comprising starting cluster services for the specified cluster.

3. The method of claim 1, wherein installing the cluster comprises installing binaries and libraries in the master node and the managed node.

4. The method of claim 1, wherein the state of the cluster comprises configurations, metadata, and initial distributed file system state.

5. The method of claim 1, wherein the request specifies roles of nodes, size of nodes, and number of nodes.