HYBRID LOCAL/REMOTE INFRASTRUCTURE FOR DATA PROCESSING WITH LIGHTWEIGHT SETUP, POWERFUL DEBUGGABILITY, CONTROLLABILITY, INTEGRATION, AND PRODUCTIVITY FEATURES

• US 20140123115A1
• Filed: 10/16/2013
• Published: 05/01/2014
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Jsmapreduce Corporation
• Original Assignee: Jsmapreduce Corporation
• Inventors: Peretz, Ervin
• Granted Patent: US 9,146,830 B2
• Time in Patent Office: Days
• US Class Current: 717/127
• CPC Class Codes: G06F 11/362 Software debugging
• CPC Class Codes: G06F 8/52 Binary to binary
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451

Abstract

The disclosed technology provides a hybrid local/remote hosted MapReduce framework and infrastructure comprising systems and methods for improving setup, configuration, controllability, debuggability, and integration of a compute job and systems and methods for increasing programmer productivity. The system applies an interpreted programming language for the programmer'"'"'s custom Map and Reduce algorithms, such that those algorithms can execute identically on both the hosted service as well as locally (e.g., on the programmer'"'"'s local computing system or device) for development and debugging purposes. Furthermore, the disclosed system delivers this service—a hosted MapReduce infrastructure—in a simple and transparent web service.

Claims

1. A method for providing an entirely hosted MapReduce data analytic infrastructure for executing a distributed large-scale compute job, the method comprising:

providing complete portability with respect to a programmer'"'"'s local machine, operating system, and browser;

continuous metering of consumed compute, storage, and network resources on a hosting system;

continuously reporting cost metering to a user;

for each compute instance of the distributed large-scale compute job, monitoring of input and output data for the compute instance, and logging of the input and output data for the compute instance;

generating input data from a kernel program, wherein the kernel program is smaller than the generated input data; and

data-sorting at least a portion of the output data of the distributed large-scale compute job during a Map phase, wherein the data-sorting comprises,

indexed writing of data sorting keys to shared storage by independently-executed Map jobs,

wherein the hosted infrastructure does not require an installed client software or software development kit and wherein the hosted infrastructure does not require the compilation of computer code.

2. The method of claim 1, further comprising:

pausing of a distributed large-scale compute job; and

resuming the execution of a distributed large-scale compute job, following a paused state.

3. The method of claim 1, further comprising:

use of data-sorting in the Map phase without performing a MapReduce shuffle phase.

4. The method of claim 1, further comprising:

extending a model of a transparently executing hosted first MapReduce job, to define data pipelines involving a plurality of hosted MapReduce jobs other than the first MapReduce job;

for each defined data pipeline,

creating at least one data source and compute stage, wherein each data source and compute stage is developed either by a hosting company, a user of a service, or by a third party; and

exposing a graphical programming environment.

5. The method of claim 1, further comprising:

exposing a graphical programming environment with parallelism between serial portions of the graphical programming environment.

6. A method for implementing a hybrid local/remote MapReduce large scale data analytic infrastructure, the method comprising:

exposing an identical execution environment within both a large-scale hosted service as on a developer'"'"'s local machine;

allowing for iterative local or in-browser development and testing of a programmer'"'"'s custom Map and Reduce algorithms; and

allowing the developer to debug any particular Map, Reduce, or related compute instance, by automatically fetching its execution environment and replicating it on the developer'"'"'s local machine or in the developer'"'"'s browser.

7. The method of claim 6, further comprising:

exposing a hybrid local/remote hosted MapReduce paradigm to each of a plurality of programming languages; and

remoting keyboard inputs and text I/O to and from a compilation and interpretation or execution environment on a server for inspection; and

single-stepping through the developer'"'"'s code within the developer'"'"'s browser or other client.

8. The method of claim 7, wherein the plurality of programming languages include Java, C, and C++.

9. A computer-readable memory storing instructions that, if executed by a computing system, cause the computing system to perform a method for providing an entirely hosted data analytic infrastructure for executing a distributed large-scale compute job, the method comprising:

providing complete portability with respect to a programmer'"'"'s local machine, operating system, and browser;

continuous metering of consumed compute, storage, and network resources on a hosting system;

continuously reporting cost metering to a user;

for each compute instance of the distributed large-scale compute job, monitoring of input and output data for the compute instance, and logging of the input and output data for the compute instance;

generating input data from a kernel program, wherein the kernel program is smaller than the generated input data; and

data-sorting at least a portion of the output data of the distributed large-scale compute job during a Map phase, wherein the data-sorting comprises,

indexed writing of data sorting keys to shared storage by independently-executed Map jobs,

wherein the hosted infrastructure does not require an installed client software or software development kit and wherein the hosted infrastructure does not require the compilation of computer code.

10. The computer-readable memory of claim 9 wherein the entirely hosted infrastructure is a MapReduce infrastructure.