High performance hadoop with new generation instances

• US 10,606,478 B2
• Filed: 10/22/2015
• Issued: 03/31/2020
• Est. Priority Date: 10/22/2014
• Status: Active Grant

• Current Assignee: Qubole Inc.
• Original Assignee: Qubole Inc.
• Inventors: Ahuja, Mayank, Sarma, Joydeep Sen, Shankar, Shrikanth
• Primary Examiner(s): MacKall, Larry T
• Application Number:
  US14/920,640
• Publication Number:
  US 20160117107A1
• Time in Patent Office: 1,622 Days
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

The present invention is generally directed to a distributed computing system comprising a plurality of computational clusters, each computational cluster comprising a plurality of compute optimized instances, each instance comprising local instance data storage and in communication with reserved disk storage, wherein processing hierarchy provides priority to local instance data storage before providing priority to reserved disk storage.

Claims

1. A distributed computing system comprising a plurality of computational clusters, each computational cluster utilized in a MapReduce model and comprising a plurality of compute optimized instances, each instance comprising local instance data storage and in communication with reserved disk storage, wherein processing hierarchy is configured to use local instance data storage unless there is insufficient space on the local instance data storage, thereby providing priority to local instance data storage before providing priority to reserved disk storage,

wherein intermediate data files within the MapReduce model are stored at least in part on the compute optimized instances comprising a list of directories residing on the local instance data storage and a list of directories residing on the reserved disk storage, wherein the directories on the reserved disk storage are accessed when a processing request cannot be handled by the local instance data storage;

and

wherein the distributed computer system is configured to auto-scale;

upon adding an instance to a cluster, mounting a reserved disk storage associated with the instance is delayed until disk utilization on a cluster exceeds a predetermined threshold; and

upon terminating an instance from a cluster, terminating any reserved disk storage associated with the instance.

2. The system of claim 1, wherein the reserved disk storage provides persistent storage.

3. The system of claim 2, wherein the reserved disk storage comprises magnetic volume storage backed by magnetic drives.

4. The system of claim 1, wherein the reserved disk storage comprises solid state drives.

5. The system of claim 4, wherein the reserved disk storage comprises general purpose or provisioned input/output operations per second (TOPS) volumes using a web service that provides cloud-based resizable compute capacity.

6. The system of claim 1, wherein the computational cluster is utilized in a scalable distributed file system.

7. The system of claim 6, wherein the scalable distributed file system is configured to use reserved disk storage only if there is insufficient space on the local instance data storage.

8. The system of claim 1, wherein the processing request could not be handled due to the processing request taking longer than a predetermined amount of time.

9. The system of claim 1, wherein the processing request could not be handled due to insufficient space on the local instance data storage.

10. The system of claim 1, wherein the reserved disks are used in a round-robin scheme for allocation.

11. The system of claim 1, wherein the compute optimized instance is a C3 instance type provided by a cloud services platform.

12. An auto-scaling distributed computing system comprising:

a plurality of computational clusters, each computational cluster comprising a plurality of compute optimized instances, each instance comprising local instance data storage and in communication with persistent reserved disk storage, wherein;

the distributed computing system is configured to use local instance data storage unless there is insufficient space on the local instance data storage;

upon adding an instance to a cluster, mounting a reserved disk storage associated with the instance is delayed until disk utilization on a cluster exceeds a predetermined threshold; and

upon terminating an instance from a cluster, terminating any reserved disk storage associated with the instance.

13. The system of claim 12, wherein the computational cluster is utilized in a scalable distributed file system.

14. The system of claim 12, wherein the computational cluster is utilized in a MapReduce model.

15. The system of claim 14, wherein intermediate data files within the MapReduce model are stored at least in part on the compute optimized instances comprising a list of directories residing on the local instance data storage and a list of directories residing on the reserved disk storage, wherein the directories on the reserved disk storage are accessed when a processing request cannot be handled by the local instance data storage.

16. An auto-scaling distributed computing system comprising:

a plurality of computational clusters in a MapReduce and/or a scalable distributed file system model, each computational cluster comprising a plurality of C3 compute optimized instances offered by a cloud services platform, each instance comprising local instance data storage and in communication with persistent reserved disk storage, wherein;

the distributed computing system is configured to use local instance data storage unless there is insufficient space on the local instance data storage;

upon adding an instance to a cluster, mounting a reserved disk storage associated with the instance is delayed until disk utilization on a cluster exceeds a predetermined threshold; and

upon terminating an instance from a cluster, terminating any reserved disk storage associated with the instance.

17. A method of providing an auto-scaling distributed computing system comprising a plurality of computational clusters in a MapReduce and/or a scalable distributed file system model, each computational cluster comprising a plurality of C3 compute optimized instances offered by a cloud services platform, each instance comprising local instance data storage and in communication with persistent reserved disk storage, the method comprising:

receiving a processing request;

using local instance data storage for the processing request, unless;

there is insufficient space on the local instance data storage;

or

upon a determination that intermediate data files are to be stored at least in part on directories residing on the reserved disk storage; and

auto-scaling the system by;

upon adding an instance to a cluster, mounting a reserved disk storage associated with the instance is delayed until disk utilization on a cluster exceeds a predetermined threshold; and

upon terminating an instance from a cluster, terminating any reserved disk storage associated with the instance.