Global benchmarking and statistical analysis at scale

• US 10,511,659 B1
• Filed: 11/30/2017
• Issued: 12/17/2019
• Est. Priority Date: 04/06/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Florissi, Patricia Gomes Soares, Singer, Ido, Masad, Ofri
• Primary Examiner(s): Tokuta, Shean
• Application Number:
  US15/827,663
• Time in Patent Office: 747 Days
• CPC Class Codes: H04L 67/1089 Hierarchical topologies
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal

Abstract

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to receive results of intermediate statistical computations performed on respective ones of a plurality of datasets in respective ones of a plurality of distributed processing nodes configured to communicate over at least one network. The processing device is further configured to perform at least one global statistical computation based at least in part on the results of the intermediate statistical computations, and to utilize a result of the global statistical computation to perform one or more benchmarking operations for specified parameters relating to the plurality of datasets. The distributed processing nodes are associated with respective distinct data zones in which the respective datasets are locally accessible to the respective distributed processing nodes. At least a subset of the receiving, performing and utilizing are repeated in each of a plurality of iterations.

Claims

1. A method comprising:

receiving results of intermediate statistical computations performed on respective ones of a plurality of datasets in respective ones of a plurality of distributed processing nodes configured to communicate over at least one network;

performing at least one global statistical computation based at least in part on the results of the intermediate statistical computations; and

utilizing a result of the global statistical computation to perform one or more benchmarking operations for specified parameters relating to the plurality of datasets;

wherein the distributed processing nodes are associated with respective distinct data zones in which the respective datasets are locally accessible to the respective distributed processing nodes;

wherein the global statistical computation comprises at least one of;

computing a global standard deviation of values for a specified parameter based at least in part on sums of differences of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations and wherein the intermediate statistical computations determine the sums of differences relative to a global average of values for the specified parameter as determined in another global statistical computation performed in a previous iteration; and

computing a global histogram of values for a specified parameter based at least in part on histogram pair lists of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations wherein a given one of the histogram pair lists comprises a list of histogram slices with corresponding numbers of items in those histogram slices and wherein the intermediate statistical computations determine the histogram pair lists based at least in part on inputs including a minimum value, a maximum value and a number of histogram slices to be included in the corresponding histogram; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 further comprising:

repeating at least a subset of the receiving, performing and utilizing in each of a plurality of iterations; and

passing a result of the global statistical computation in a first one of the iterations as an input to the intermediate statistical computations in a second one of the iterations.

3. The method of claim 1 wherein the intermediate statistical computations are initiated by an initiating distributed processing node.

4. The method of claim 3 wherein the initiating distributed processing node is configured to perform the global statistical computation.

5. The method of claim 1 wherein the global statistical computation is performed at a same one of the distributed processing nodes that performs one of the intermediate statistical computations.

6. The method of claim 1 wherein the global statistical computation comprises computing a global average of values for a specified parameter based at least in part on sums of the values and number of values summed for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations.

7. The method of claim 1 wherein the global statistical computation comprises computing at least one of a global minimum and a global maximum of values for a specified parameter based at least in part on at least one of minimums and maximums of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations.

11. The method of claim 1 wherein at least one of the distributed processing nodes further performs in a subsequent iteration an additional intermediate statistical computation that receives as an input the global histogram and characterizes local placement of one or more values from its corresponding dataset within the global histogram.

12. The method of claim 1 wherein one or more of the minimum value and the maximum value are determined in at least one other global statistical computation performed in at least one previous iteration.

13. The method of claim 1 wherein the intermediate statistical computations performed on respective ones of a plurality of datasets are performed on respective subsets of the datasets and wherein each of the subsets is determined by application of at least one of an extraction operation and a transformation operation to the corresponding dataset.

14. The method of claim 1 wherein at least a subset of the distributed processing nodes are implemented in respective data processing clusters corresponding to respective ones of the data zones and wherein the clusters comprise respective cloud-based data centers each configured to store locally accessible datasets of its corresponding data zone in a manner that satisfies one or more specified policies relating to at least one of privacy, security, governance, risk and compliance.

8. A method comprising:

receiving results of intermediate statistical computations performed on respective ones of a plurality of datasets in respective ones of a plurality of distributed processing nodes configured to communicate over at least one network;

performing at least one global statistical computation based at least in part on the results of the intermediate statistical computations; and

utilizing a result of the global statistical computation to perform one or more benchmarking operations for specified parameters relating to the plurality of datasets;

wherein the distributed processing nodes are associated with respective distinct data zones in which the respective datasets are locally accessible to the respective distributed processing nodes;

wherein the global statistical computation comprises computing at least one of a global minimum set and a global maximum set of values for a specified parameter based at least in part on at least one of minimum sets and maximum sets of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations; and

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

9. The method of claim 8 wherein the global statistical computation further comprises computing a global standard deviation of values for a specified parameter based at least in part on sums of differences of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations and wherein the intermediate statistical computations determine the sums of differences relative to a global average of values for the specified parameter as determined in another global statistical computation performed in a previous iteration.

10. The method of claim 8 wherein the global statistical computation further comprises computing a global histogram of values for a specified parameter based at least in part on histogram pair lists of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations wherein a given one of the histogram pair lists comprises a list of histogram slices with corresponding numbers of items in those histogram slices and wherein the intermediate statistical computations determine the histogram pair lists based at least in part on inputs including a minimum value, a maximum value and a number of histogram slices to be included in the corresponding histogram.

15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to receive results of intermediate statistical computations performed on respective ones of a plurality of datasets in respective ones of a plurality of distributed processing nodes configured to communicate over at least one network;

to perform at least one global statistical computation based at least in part on the results of the intermediate statistical computations; and

to utilize a result of the global statistical computation to perform one or more benchmarking operations for specified parameters relating to the plurality of datasets;

wherein the distributed processing nodes are associated with respective distinct data zones in which the respective datasets are locally accessible to the respective distributed processing nodes; and

wherein the global statistical computation comprises at least one of;

computing a global standard deviation of values for a specified parameter based at least in part on sums of differences of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations and wherein the intermediate statistical computations determine the sums of differences relative to a global average of values for the specified parameter as determined in another global statistical computation performed in a previous iteration; and

computing a global histogram of values for a specified parameter based at least in part on histogram pair lists of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations wherein a given one of the histogram pair lists comprises a list of histogram slices with corresponding numbers of items in those histogram slices and wherein the intermediate statistical computations determine the histogram pair lists based at least in part on inputs including a minimum value, a maximum value and a number of histogram slices to be included in the corresponding histogram.

16. The computer program product of claim 15 wherein at least a subset of the receiving, performing and utilizing are repeated in each of a plurality of iterations and wherein a result of the global statistical computation in a first one of the iterations is passed as an input to the intermediate statistical computations in a second one of the iterations.

17. The computer program product of claim 15 wherein at least a subset of the distributed processing nodes are implemented in respective data processing clusters corresponding to respective ones of the data zones and wherein the clusters comprise respective cloud-based data centers each configured to store locally accessible datasets of its corresponding data zone in a manner that satisfies one or more specified policies relating to at least one of privacy, security, governance, risk and compliance.

18. An apparatus comprising:

at least one processing device having a processor coupled to a memory;

wherein said at least one processing device is configured;

to receive results of intermediate statistical computations performed on respective ones of a plurality of datasets in respective ones of a plurality of distributed processing nodes configured to communicate over at least one network;

to perform at least one global statistical computation based at least in part on the results of the intermediate statistical computations; and

to utilize a result of the global statistical computation to perform one or more benchmarking operations for specified parameters relating to the plurality of datasets;

wherein the distributed processing nodes are associated with respective distinct data zones in which the respective datasets are locally accessible to the respective distributed processing nodes; and

wherein the global statistical computation comprises at least one of;

computing a global standard deviation of values for a specified parameter based at least in part on sums of differences of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations and wherein the intermediate statistical computations determine the sums of differences relative to a global average of values for the specified parameter as determined in another global statistical computation performed in a previous iteration; and

computing a global histogram of values for a specified parameter based at least in part on histogram pair lists of the values for the specified parameter determined for respective ones of the datasets as part of respective ones of the intermediate statistical computations wherein a given one of the histogram pair lists comprises a list of histogram slices with corresponding numbers of items in those histogram slices and wherein the intermediate statistical computations determine the histogram pair lists based at least in part on inputs including a minimum value, a maximum value and a number of histogram slices to be included in the corresponding histogram.

19. The apparatus of claim 18 wherein at least a subset of the receiving, performing and utilizing are repeated in each of a plurality of iterations and wherein a result of the global statistical computation in a first one of the iterations is passed as an input to the intermediate statistical computations in a second one of the iterations.

20. The apparatus of claim 18 wherein at least a subset of the distributed processing nodes are implemented in respective data processing clusters corresponding to respective ones of the data zones and wherein the clusters comprise respective cloud-based data centers each configured to store locally accessible datasets of its corresponding data zone in a manner that satisfies one or more specified policies relating to at least one of privacy, security, governance, risk and compliance.