Big data pipeline management within spreadsheet applications

• US 10,725,799 B2
• Filed: 02/22/2017
• Issued: 07/28/2020
• Est. Priority Date: 02/22/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zusman, Dmitry, Malyshev, Aleksandr
• Primary Examiner(s): Phantana-angkool, David
• Application Number:
  US15/439,452
• Publication Number:
  US 20180239617A1
• Time in Patent Office: 1,252 Days
• Field of Search: 715738
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 40/18 of spreadsheets form-filling G06F40/174
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/45512 Command shells
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G06F 9/547 Remote procedure calls [RPC]; Web services
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

The techniques described herein facilitate asynchronous management of external or Big Data processing pipelines from within client-side (or local) spreadsheet applications, e.g., Microsoft Excel®. In an implementation, a method of operating a spreadsheet application service to facilitate management of big data pipelines from within a client-side spreadsheet application is disclosed. The method includes receiving a processing request initiated by a spreadsheet application. The processing request includes instructions for generating one or more big data pipelines between one or more external distributed computing systems. The method further includes translating the instruction into one or more scripts corresponding to the one or more external distributed computing systems, and executing the one or more scripts to direct the one or more external distributed computing systems to automatically generate the big data pipelines.

Claims

1. A device-implemented method of operating a spreadsheet application service to facilitate asynchronous management of processing pipelines external to a system implementing the device-implemented method and a client device executing a client-side spreadsheet applications, the device-implemented method comprising:

receiving, from the client device, a data processing job request including an instructions that identifies external processing pipelines between

the client-side spreadsheet application and a distributed computing systems external to both the system implementing the device-implemented method and the client device, and

the distributed computing system and a location where output of the external processing pipelines is configured to be published;

translating the instruction into a script

that is compatible with the distributed computing system, and

includes a second instruction configured to direct the distributed computing system to automatically generate the external processing pipelines; and

communicating the script to the distributed computing system.

2. The device-implemented method of claim 1, wherein the instruction indicates a location of a dataset, a process to execute on the dataset, and the location to publish the output of the external processing pipelines.

3. The device-implemented method of claim 2, wherein the script further directs the distributed computing system to execute the process on the dataset.

4. The device-implemented method of claim 2, further comprising:

generating a data structure for receiving the output of the external processing pipelines; and

temporarily persisting the output in the data structure.

5. The device-implemented method of claim 2, further comprising:

receiving, from a client device executing a requesting spreadsheet application, a request for the output of the external processing pipelines,

wherein the request includes a job identifier (ID);

accessing the output of the external processing pipelines using the job ID; and

providing, to the client device executing the requesting spreadsheet application, the output of the external processing pipelines to the requesting spreadsheet application.

6. The device-implemented method of claim 1, further comprising:

registering the data processing job request with a job registry component; and

generating a job identifier (ID).

7. The device-implemented method of claim 6, further comprising:

persisting the job ID in association with the data processing job request; and

providing the job ID to a remote spreadsheet application.

8. The device-implemented method of claim 1, wherein translating the instruction comprises:

parsing the instruction to identify a processing step including a source data location, a destination data location, and a type field.

9. The device-implemented method of claim 8, wherein the processing step further includes a process to execute on a dataset.

10. A spreadsheet application system comprising:

one or more processors;

one or more computer readable storage media; and

a spreadsheet application service stored on the one or more computer readable storage media, the spreadsheet application service including program instructions that, when executed by the one or more processors, direct the one or more processors to at least;

identify, by processing a data processing job request received from a client device executing a client-side spreadsheet application, an instructions that identifies external processing pipelines between the client-side spreadsheet application, a distributed computing system external to the spreadsheet application system and the client device, and a location where output of the external processing pipelines is configured to be published;

translate the instruction into a script that is compatible with the distributed computing system; and

execute the script including directing the distributed computing systems to automatically generate the external processing pipelines.

11. The spreadsheet application system of claim 10, wherein the instruction indicates a location of a dataset, one or more processes to execute on the dataset, and the location to publish output of the external processing pipelines.

12. The spreadsheet application system of claim 11, wherein executing the script further includes directing the distributed computing systems to execute the one or more processes on the dataset.

13. The spreadsheet application system of claim 11, wherein the program instructions, when executed by the one or more processors, further direct the one or more processors to:

generate a data structure for receiving the output of the external processing pipelines; and

temporarily persist the output in the data structure.

14. The spreadsheet application system of claim 11, wherein the program instructions, when executed by the one or more processors, further direct the one or more processors to:

process a request for the output of the external processing pipelines received from a requesting spreadsheet application,

wherein the request includes a job identifier (ID);

access the output of the external processing pipelines using the job ID; and

provide the output of the external processing pipelines to the requesting spreadsheet application.

15. The spreadsheet application system of claim 10, wherein the program instructions, when executed by the one or more processors, further direct the one or more processors to:

register the data processing job request with a job registry component; and

generate a job identifier (ID).

16. The spreadsheet application system of claim 15, wherein the program instructions, when executed by the one or more processors, further direct the one or more processors to:

persist the job ID in association with the data processing job request; and

provide the job ID to a remote spreadsheet application.

17. The spreadsheet application system of claim 11, wherein to translate the instruction into a script, the program instructions, when executed by the one or more processors, further direct the one or more processors to:

parse the instruction to identify one or more processing steps,

wherein each processing step includes a source data location, a destination data location, and a type field, and wherein each processing step further includes one or more processes to execute on a dataset.

18. A spreadsheet application service comprising:

a messaging service component configured to process a data processing job request received from a client-side spreadsheet applications and responsively identify an instruction that identifies external processing pipelines between the client-side spreadsheet application, an external distributed computing system, and a location where output of the external processing pipelines is configured to be published;

a consumer component configured to translate the data processing job request into a script corresponding to the external distributed computing system; and

an interface configured to execute the script including directing the external distributed computing systems to automatically generate the external processing pipelines.

19. The spreadsheet application service of claim 18, wherein to translate the data processing request into the script, the consumer component is further configured to parse the instruction to identify one or more processing steps, wherein each processing job step includes a source data location, a destination data location, and a type field.

20. The spreadsheet application service of claim 19, wherein each processing step further includes one or more processes to execute on a dataset.