Behaviorally consistent cluster-wide data wrangling based on locally processed sampled data

• US 9,785,698 B2
• Filed: 12/31/2014
• Issued: 10/10/2017
• Est. Priority Date: 12/31/2014
• Status: Active Grant

• Current Assignee: Business Objects Incorporated (SAP SE)
• Original Assignee: Business Objects Incorporated (SAP SE)
• Inventors: Tsumura, Michael, Ivanov, Ivailo, Kumar, Viren Suresh
• Primary Examiner(s): Pham, Tuan A
• Application Number:
  US14/588,022
• Publication Number:
  US 20160188692A1
• Time in Patent Office: 1,014 Days
• Field of Search: None
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 16/986 Document structures and storage, e.g. HTML extensions
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Example embodiments involve a system, computer-readable storage medium storing at least one program, and computer-implemented method for behaviorally consistent data wrangling. A local client device selects a set of raw sample data from a remote datastore. A local execution engine then applies one or more local data wrangling operations to the raw sample data. If the results of the local data wrangling operations are satisfactory, the local data wrangling operations may then be transferred to a remote data wrangling cluster. A remote execution engine being executed by the remote data wrangling cluster then applies the data wrangling operations to the larger set of raw data from which the sample raw data was obtained. As the remote execution engine and the local execution engine are of the same type, the data wrangling behavior exhibited by the local execution engine is reflected in the data wrangling behavior of the remote execution engine.

Claims

1. A method comprising:

selecting, at a local client device, a first plurality of raw data from a second plurality of raw data, the second plurality of raw data being stored remote from the local client device and accessible by a remote device;

receiving the first plurality of raw data at the local client device;

selecting, from a library of data wrangling operations at the local client device, a plurality of data wrangling operations to perform on the first plurality of raw data;

instantiating, at the local client device, a first data wrangling module operating in a first mode selected from a first plurality of modes, the first mode being selected based on computing resources available to the local client device;

applying, using at least one hardware processor of the local client device, the plurality of data wrangling operations to the first plurality of raw data using the first mode of the first data wrangling module to obtain a first plurality of structured data;

causing presentation, via a user interface of the local client device, of the first plurality of structured data;

receiving, via the user interface of the local client device, after the causing of the presentation of the first plurality of structured data, an input indicating approval of the first plurality of structured data; and

sending, in response to receiving the input, the selection of the plurality of data wrangling operations to the remote device, the remote device being configured to;

instantiate a second data wrangling module operating in a second mode selected from a plurality of modes; and

apply the selected plurality of data wrangling operations to the second plurality of raw data using the second mode of the second data wrangling module to obtain a second plurality of structured data, the second plurality of structured data having an expected organization based on the first plurality of structured data.

2. The method of claim 1, further comprising:

receiving a notification from the remote device that the data wrangling operations have been applied to the second plurality of raw data.

3. The method of claim 1, wherein the first plurality of raw data is a subset of the second plurality of raw data.

4. The method of claim 1, wherein the first plurality of structured data is meaningfully organized relative to the first plurality of raw data.

5. The method of claim 1, wherein the remote device comprises a cluster of nodes, and at least one node is selected from the cluster of nodes to perform at least one data wrangling operation selected from the plurality of data wrangling operations on a subset of the second plurality of raw data.

6. A system comprising:

one or more processors at a local client device; and

a memory storing instructions that, when executed by at least one of the one or more processors, cause the local client device to perform operations comprising;

selecting a first plurality of raw data from a second plurality of raw data, the second plurality of raw data being stored remote from the local client device and accessible by a remote device;

receiving the first plurality of raw data;

selecting, from a library of data wrangling operations, a plurality of data wrangling operations to perform on the first plurality of raw data;

instantiating a first data wrangling module operating in a first mode selected from a first plurality of modes, the first mode being selected based on computing resources available to the local client device;

applying the plurality of data wrangling operations to the first plurality of raw data using the first mode of the first data wrangling module to obtain a first plurality of structured data;

causing presentation, via a user interface, of the first plurality of structured data;

receiving, via the user interface, after the causing of the presentation of the first plurality of structured data, an input indicating approval of the first plurality of structured data; and

sending, in response to receiving the input, the selection of the plurality of data wrangling operations to the remote device, the remote device being configured to;

instantiate a second data wrangling module operating in a second mode selected from a plurality of modes; and

apply the selected plurality of data wrangling operations to the second plurality of raw data using the second mode of the second data wrangling module to obtain a second plurality of structured data, the second plurality of structured data having an expected organization based on the first plurality of structured data.

7. The system of claim 6, the operations further comprising:

receiving a notification from the remote device that the data wrangling operations have been applied to the second plurality of raw data.

8. The system of claim 6, wherein the first plurality of raw data is a subset of the second plurality of raw data.

9. The system of claim 6, wherein the first plurality of structured data is meaningfully organized relative to the first plurality of raw data.

10. The system of claim 6, wherein the remote device comprises a cluster of nodes, and at least one node is selected from the cluster of nodes to perform at least one data wrangling operation selected from the plurality of data wrangling operations on a subset of the second plurality of raw data.

11. A non-transitory, computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a local client device, cause the local client device to perform operations comprising:

selecting a first plurality of raw data from a second plurality of raw data, the second plurality of raw data being stored remote from the local client device and accessible by a remote device;

receiving the first plurality of raw data at the local client device;

selecting, from a library of data wrangling operations, a plurality of data wrangling operations to perform on the first plurality of raw data;

instantiating a first data wrangling module operating in a first mode selected from a first plurality of modes, the first mode being selected based on computing resources available to the local client device;

applying the plurality of data wrangling operations to the first plurality of raw data using the first mode of the first data wrangling module to obtain a first plurality of structured data;

causing presentation, via a user interface, of the first plurality of structured data;

receiving, via the user interface, after the causing of the presentation of the first plurality of structured data, an input indicating approval of the first plurality of structured data; and

sending, in response to receiving the input, the selection of the plurality of data wrangling operations to the remote device, the remote device being configured to;

instantiate a second data wrangling module operating in a second mode selected from a plurality of modes; and

apply the selected plurality of data wrangling operations to the second plurality of raw data using the second mode of the second data wrangling module to obtain a second plurality of structured data, the second plurality of structured data having an expected organization based on the first plurality of structured data.

12. The non-transitory, computer-readable medium of claim 11, wherein the operations further comprise:

receiving a notification from the remote device that the data wrangling operations have been applied to the second plurality of raw data.

13. The non-transitory, computer-readable medium of claim 11, wherein the first plurality of raw data is a subset of the second plurality of raw data.

14. The non-transitory, computer-readable medium of claim 11, wherein the first plurality of structured data is meaningfully organized relative to the first plurality of raw data.