Multivariate Insight Discovery Approach

• US 20160103902A1
• Filed: 10/09/2014
• Published: 04/14/2016
• Est. Priority Date: 10/09/2014
• Status: Active Grant

• Current Assignee: Business Objects Incorporated (SAP SE)
• Original Assignee: Julian Gosper
• Inventors: Moser, Flavia, Gosper, Julian, MacAulay, Alexander Kennedy
• Granted Patent: US 10,255,345 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs

Abstract

A raw dataset including measures and dimensions is processed, by a preprocessing module, using an algorithm that produces a preprocessed dataset such that at least one type of statistical analysis of the preprocessed dataset yields equal results to the same type of statistical analysis of the raw dataset. The preprocessed dataset is then analyzed by a statistical analysis module to identify subsets of the preprocessed dataset that include a non-random structure or pattern. The analysis of the preprocessed dataset includes the at least one type of statistical analysis that produces the same results for both the preprocessed and raw datasets. The identified subsets are then ranked by a statistical ranker based on the analysis of the preprocessed dataset and a subset is selected for visualization based on the rankings A visualization module then generates a visualization of the selected identified subset that highlights a non-random structure of the selected subset.

Claims

1. A method comprising:

accessing a dataset including measures and dimensions by a preprocessing module including at least one processor;

processing the dataset, by the preprocessing module, to generate a preprocessed dataset such that at least one type of statistical analysis of the preprocessed dataset yields equal results to the same type of statistical analysis of the dataset;

analyzing the preprocessed dataset, by a statistical analysis module including at least one processor, to identify subsets of the preprocessed dataset that include a non-random structure, the analyzing including the at least one type of statistical analysis;

generating a score for each of the identified subsets, by the statistical analysis module, based on the non-random structures included in each of the identified subsets;

ranking each of the identified subsets, by a statistical ranking module including at least one processor, based on the score generated for each of the identified subsets and selecting an identified subset based on the ranking of the identified subset; and

generating, by a visualization module including at least one processor, a visualization that highlights a non-random structure of the selected identified subset.

2. The method of claim 1, wherein the preprocessed dataset includes at least one online analytical processing (OLAP) cube and processing the dataset includes:

discarding a measure of the dataset based on more than half of the values of the measure being zero;

determining hierarchical relationships between the attributes of the dataset;

storing “

Sum” and

“

Count”

values for corresponding measures of an attribute based on an aggregation type of the attribute being “

Average”

; and

aggregating over at least one attribute of the dataset.

3. The method of claim 2, wherein processing the dataset includes at least one of:

aggregating over attributes of the dataset 99% of the same value;

aggregating over attributes of the dataset with a cardinality greater than a threshold value;

or

aggregating over all of the attributes of the dataset in order of decreasing cardinality until the dataset has a threshold size.

4. The method of claim 3, wherein processing the dataset includes:

determining equivalencies between the attributes and between the measures of the dataset;

for equivalent attributes, discarding all but one of the equivalent attributes; and

for equivalent measures, discarding all but one of the equivalent measures.

5. The method of claim 4, wherein the at least one type of statistical analysis includes an analysis of variance (ANOVA) test, the method further comprising:

analyzing, by the statistical analysis module, only subsets of the preprocessed dataset that consist of an attribute and a measure and subsets that consist of two attributes and a measure;

generating, by the statistical analysis module, a score for each identified subset based on an effect size of ANOVA for the identified subset.

6. The method of claim 5, wherein the visualization includes at least one of:

a mark representing different values of an attribute, a mark type for each type of data point representation, a mark color property associated with a measure or an attribute, or a mark size property associated with a value of a measure;

the method further comprising;

selecting an attribute of the selected identified subset for the color property based on attribute hierarchies; and

selecting a mark type based on a cardinality of an attribute of the selected identified subset.

7. The method of claim 6, wherein the determining an attribute for the color property based on the attribute hierarchies includes one of:

determining that an attribute is one of included as a mark and included on an x-axis with a cardinality less than 10;

or

determining that an attribute is at a higher hierarchy level than an attribute used as a mark with a cardinality less than 10.

8. A system comprising:

a preprocessing module including a processor and configured to access a dataset including measures and dimensions and process the dataset to generate a preprocessed dataset such that at least one type of statistical analysis of the preprocessed dataset yields equal results to the same type of statistical analysis of the dataset;

a statistical analysis module including a processor and configured to;

analyze the preprocessed dataset to identify subsets of the preprocessed dataset that include a non-random structure, the analyzing including the at least one type of statistical analysis; and

generate a score for each of the identified subsets based on the non-random structures included in each of the identified subsets;

a statisitical ranking module including a processor and configured to;

rank each of the identified subsets based on the score generated for each of the identified subsets; and

select an identified subset based on the ranking of the identified subset; and

a visualization module including a processor and configured to generate a visualization of the selected identified subset that highlights a non-random structure of the selected identified subset.

9. The system of claim 8, wherein the preprocessed dataset includes at least one online analytical processing (OLAP) cube and the preprocessing module is configured to:

discard a measure of the dataset based on more than half of the values of the measure being zero;

determine hierarchical relationships between the attributes of the dataset;

store “

Sum” and

“

Count”

values for corresponding measures of an attribute based on an aggregation type of the attribute being “

Average”

; and

aggregate over at least one attribute of the dataset.

10. The system of claim 9, wherein the preprocessing module is configured to perform at least one of:

aggregate over attributes of the dataset 99% of the same value;

aggregate over attributes of the dataset with a cardinality greater than a threshold value;

or

aggregate over all of the attributes of the dataset in order of decreasing cardinality until the dataset has a threshold size.

11. The system of claim 10, wherein the preprocessing module is configured to:

determining equivalencies between the attributes and between the measures of the dataset;

for equivalent attributes, discarding all but one of the equivalent attributes; and

for equivalent measures, discarding all but one of the equivalent measures.

12. The system of claim 11, wherein:

the at least one type of statistical analysis includes an analysis of variance (ANOVA) test; and

the statistical analysis module is configured to;

analyze only subsets of the preprocessed dataset that consist of an attribute and a measure and subsets that consist of two attributes and a measure; and

generate a score for each identified subset based on an effect size of ANOVA for the identified subset.

13. The system of claim 12, wherein the visualization includes at least one of:

a mark representing different values of an attribute, a mark type for each type of data point representation, a mark color property associated with a measure or an attribute, or a mark size property associated with a value of a measure; and

the visualization module is further configured to;

select an attribute of the selected identified subset for the color property based on attribute hierarchies; and

select a mark type based on a cardinality of an attribute of the selected identified subset.

14. The system of claim 13, wherein the determining an attribute for the color property based on the attribute hierarchies includes one of:

determining that an attribute is one of included as a mark or included on an x-axis with a cardinality less than 10;

or

determining that an attribute is at a higher hierarchy level than an attribute used as a mark with a cardinality less than 10.

15. A non-transitory machine-readable storage medium including instructions that, when executed on at least one processor of a machine, cause the machine to perform operations comprising:

accessing a dataset including measures and dimensions by a preprocessing module;

processing the dataset, by the preprocessing module, to generate a preprocessed dataset such that at least one type of statistical analysis of the preprocessed dataset yields equal results to the same type of statistical analysis of the dataset;

analyzing the preprocessed dataset, by a statistical analysis module, to identify subsets of the preprocessed dataset that include a non-random structure, the analyzing including the at least one type of statistical analysis;

generating a score for each of the identified subsets, by the statistical analysis module, based on the non-random structures included in each of the identified subsets;

ranking each of the identified subsets, by a statistical ranking, based on the score generated for each of the identified subsets and selecting an identified subset based on the ranking of the identified subset; and

generating, by a visualization module, a visualization that highlights a non-random structure of the selected identified subset.

16. The machine-readable storage medium of claim 15, wherein the preprocessed dataset includes at least one online analytical processing (OLAP) cube and processing the dataset includes:

discarding a measure of the dataset based on more than half of the values of the measure being zero;

determining hierarchical relationships between the attributes of the dataset;

storing “

Sum” and

“

Count”

values for corresponding measures of an attribute based on an aggregation type of the attribute being “

Average”

; and

aggregating over at least one attribute of the dataset.

17. The machine-readable storage medium of claim 16, wherein processing the dataset includes:

determining equivalencies between the attributes and between the measures of the dataset;

for equivalent attributes, discarding all but one of the equivalent attributes; and

for equivalent measures, discarding all but one of the equivalent measures.

18. The machine-readable storage medium of claim 17, wherein the at least one type of statistical analysis includes an analysis of variance (ANOVA) test, the operations further comprising:

analyzing, by the statistical analysis module, only subsets of the preprocessed dataset that consist of an attribute and a measure and subsets that consist of two attributes and a measure;

generating, by the statistical analysis module, a score for each identified subset based on an effect size of ANOVA for the identified subset.

19. The machine-readable storage medium of claim 18, wherein the visualization includes at least one of:

a mark representing different values of an attribute, a mark type for each type of data point representation, a mark color property associated with a measure or an attribute, or a mark size property associated with a value of a measure;

the operations further comprising;

selecting an attribute of the selected identified subset for the color property based on attribute hierarchies; and

selecting a mark type based on a cardinality of an attribute of the selected identified subset.

20. The machine-readable storage medium of claim 19, wherein the determining an attribute for the color property based on attribute hierarchies includes one of:

determining that an attribute is one of included as a mark and included on an x-axis with a cardinality less than 10;

or

determining that an attribute is at a higher hierarchy level than an attribute used as a mark with a cardinality less than 10.