PREVIEW DATA AGGREGATION

• US 20180336230A1
• Filed: 05/16/2017
• Published: 11/22/2018
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Transier, Frederik, Stammerjohann, Kai, Bohnsack, Nico
• Granted Patent: US 10,621,153 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/287 Visualization; Browsing

Abstract

In one respect, there is provided a method. The method can include processing a first data chunk to generate a first intermediate result. A key map can be generated based on a determination that a quantity of the key-value pairs in the first intermediate result exceeds a threshold. The key map can be generated to include keys in the first intermediate result. A second data chunk can be processed to generate a second intermediate result. The second data chunk can be processed based on the key map. The processing of the second data chunk can include omitting a key-value pair in the second data chunk from being inserted into the second intermediate result based on a key associated with the key-value pair being absent from the key map. A preview of the processing of the dataset can be generated based on the first intermediate result and the second intermediate result.

Claims

1. A computer implemented method, comprising:

processing, at a first worker node, a first data chunk of a dataset to generate a first intermediate result, the processing of the first data chunk comprising inserting key-value pairs from the first data chunk into the first intermediate result, the dataset being partitioned into the first data chunk and a second data chunk;

generating, at a merger node, a key map based at least on a determination that a quantity of the key-value pairs in the first intermediate result exceeds a threshold value, the key map being generated to include one or more keys of the key-value pairs in the first intermediate result;

processing, at a second worker node, the second data chunk to generate a second intermediate result, the second data chunk being processed based at least on the key map, the processing of the second data chunk comprising omitting a key-value pair in the second data chunk from being inserted into the second intermediate result, the key-value pair being omitted based on a key associated with the key-value pair being absent from the key map; and

generating a preview of the processing of the dataset, the preview being generated based at least on the first intermediate result and the second intermediate result.

2. The computer-implemented method of claim 1, wherein the processing of the second data chunk further comprises inserting, into the second intermediate result, another key-value pair from the second data chunk, the other key-value pair being inserted based on another key associated with the other key-value pair being present in the key map.

3. The computer-implemented method of claim 1, wherein the threshold value corresponds to a quantity of key-value pairs required to be present in the preview.

4. The computer-implemented method of claim 1, wherein the inserting of the key-value pairs from the first data chunk into the first intermediate result comprises:

selecting a first key-value pair from the first data chunk; and

determining whether the first key-value pair is associated with a same key as a second key-value pair in the first intermediate result.

5. The computer-implemented method of claim 4, further comprising:

aggregating the first key-value pair and the second key-value pair based at least in part on a determination that the first key-value pair and the second key-value pair are associated with the same key.

6. The computer-implemented method of claim 5, wherein the aggregating comprises aggregating a first value of the first key-value pair and a second value of the second key-value pair, the first value and the second value being aggregated by addition, multiplication, division, subtraction, and/or comparison.

7. The computer-implemented method of claim 1, wherein the preview is generated by at least merging the first intermediate result and the second intermediate result.

8. The computer-implemented method of claim 7, wherein the first intermediate result and the second intermediate result are merged by at least identifying at least one key-value pair from the first intermediate result that share a same key as another key-value pair from the second intermediate result.

9. The computer-implemented method of claim 7, wherein the preview is further generated by merging a third intermediate result with the first intermediate result and/or the second intermediate result, the third intermediate result being generated based on the key map.

10. The computer-implemented method of claim 9, further comprising:

in response to determining that the third intermediate result is generated based on the key map, merging the third intermediate result without identifying at least one key-value pair from the third intermediate result that share a same key as another key-value pair from the first intermediate result and/or the second intermediate result.

11. A system, comprising:

at least one data processor; and

at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising;

processing, at a first worker node, a first data chunk of a dataset to generate a first intermediate result, the processing of the first data chunk comprising inserting key-value pairs from the first data chunk into the first intermediate result, the dataset being partitioned into the first data chunk and a second data chunk;

generating, at a merger node, a key map based at least on a determination that a quantity of the key-value pairs in the first intermediate result exceeds a threshold value, the key map being generated to include one or more keys of the key-value pairs in the first intermediate result;

processing, at a second worker node, the second data chunk to generate a second intermediate result, the second data chunk being processed based at least on the key map, the processing of the second data chunk comprising omitting a key-value pair in the second data chunk from being inserted into the second intermediate result, the key-value pair being omitted based on a key associated with the key-value pair being absent from the key map; and

generating a preview of the processing of the dataset, the preview being generated based at least on the first intermediate result and the second intermediate result.

12. The system of claim 11, wherein the processing of the second data chunk further comprises inserting, into the second intermediate result, another key-value pair from the second data chunk, the other key-value pair being inserted based on another key associated with the other key-value pair being present in the key map.

13. The system of claim 11, wherein the threshold value corresponds to a quantity of key-value pairs required to be present in the preview.

14. The system of claim 11, wherein the inserting of the key-value pairs from the first data chunk into the first intermediate result comprises:

selecting a first key-value pair from the first data chunk;

determining whether the first key-value pair is associated with a same key as a second key-value pair in the first intermediate result; and

aggregating the first key-value pair and the second key-value pair based at least in part on a determination that the first key-value pair and the second key-value pair are associated with the same key.

15. The system of claim 14, wherein the aggregating comprises aggregating a first value of the first key-value pair and a second value of the second key-value pair, the first value and the second value being aggregated by addition, multiplication, division, subtraction, and/or comparison.

16. The system of claim 11, wherein the preview is generated by at least merging the first intermediate result and the second intermediate result.

17. The system of claim 16, wherein the first intermediate result and the second intermediate result are merged by at least identifying at least one key-value pair from the first intermediate result that share a same key as another key-value pair from the second intermediate result.

18. The system of claim 16, wherein the preview is further generated by merging a third intermediate result with the first intermediate result and/or the second intermediate result, the third intermediate result being generated based on the key map.

19. The system of claim 18, further comprising:

in response to determining that the third intermediate result is generated based on the key map, merging the third intermediate result without identifying at least one key-value pair from the third intermediate result that share a same key as another key-value pair from the first intermediate result and/or the second intermediate result.

20. A non-transitory computer-readable storage medium including program code, which when executed by at least one data processor, cause operations comprising:

processing, at a first worker node, a first data chunk of a dataset to generate a first intermediate result, the processing of the first data chunk comprising inserting key-value pairs from the first data chunk into the first intermediate result, the dataset being partitioned into the first data chunk and a second data chunk;

generating, at a merger node, a key map based at least on a determination that a quantity of the key-value pairs in the first intermediate result exceeds a threshold value, the key map being generated to include one or more keys of the key-value pairs in the first intermediate result;

processing, at a second worker node, the second data chunk to generate a second intermediate result, the second data chunk being processed based at least on the key map, the processing of the second data chunk comprising omitting a key-value pair in the second data chunk from being inserted into the second intermediate result, the key-value pair being omitted based on a key associated with the key-value pair being absent from the key map; and

generating a preview of the processing of the dataset, the preview being generated based at least on the first intermediate result and the second intermediate result.