Cloud process for rapid data investigation and data integrity analysis

• US 10,367,888 B2
• Filed: 09/20/2017
• Issued: 07/30/2019
• Est. Priority Date: 10/03/2014
• Status: Active Grant

• Current Assignee: Fair Isaac Corporation
• Original Assignee: Fair Isaac Corporation
• Inventors: Zoldi, Scott M., Murray, Joseph F., Carlson, Jeffrey D.
• Primary Examiner(s): Nguyen, Thu Ha T
• Application Number:
  US15/709,868
• Publication Number:
  US 20180013829A1
• Time in Patent Office: 678 Days
• Field of Search: 709213, 709212, 709214, 709217, 709223, 709224
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A system and method for rapid data investigation and data integrity analysis is disclosed. A data set is received by a server computer from one or more client computers connected with the server computer via a communications network, and the data set is stored in a distributed storage memory. One or more analytical processes are executed on the data set from the distributed storage memory to generate statistics based on each of the analytical processes, and the statistics are stored in a random access memory, the random access memory being accessible by one or more compute nodes, which generate a graphical representation of at least some statistics stored in the random access memory. The graphical representation of at least some statistics is then formatted for transmission to and display by the one or more client computers.

Claims

1. A method comprising:

receiving summary statistics computed by at least executing one or more analytical processes on a dataset stored in parts across a set of memory based compute nodes, each compute node finding partial statistics of a data part stored on the respective compute node, the partial statistics representative of a respective data part;

storing the summary statistics in a random access memory associated with a server computer, the random access memory being accessible by at least one of the compute nodes, the summary statistics being a combination of the partial statistics and representative of a full dataset;

identifying, for pre-model building data understanding, outlier data by comparing subsets of data in the dataset, the identified outlier data accessible to a predictive model;

generating a graphical representation of at least some summary statistics stored in the random access memory; and

formatting the graphical representation of at least some summary statistics for transmission to and display by one or more client computers.

2. The method in accordance with claim 1, further comprising providing pattern detection for differentiated candidate model feature creation during predictive model development.

3. The method in accordance with claim 1, further comprising automatically identifying anomalous or outlier behavior in data to efficiently investigate large numbers of data subpopulations.

4. The method in accordance with claim 1, further comprising identifying suspected data integrity issues in data subpopulations by collecting data integrity checks into a report type, the report type being run and repeatable for each data integrity issue identification.

5. The method in accordance with claim 1, further comprising interactively exploring high-dimensional data distributions, the high-dimensional data being grouped into one or more statistics distribution arrays allowing users to interactively explore detected data patterns of interest.

6. The method in accordance with claim 1, further comprising maintaining the dataset within a distributed storage memory for future accessing and processing to occur in response to interaction with the graphical representation of the at least some summary statistics.

7. The method in accordance with claim 1, further comprising:

receiving a request for additional data analysis from the one or more client computers;

determining whether the dataset is present in the distributed storage memory; and

computing additional summary statistics in parallel on the compute nodes, the compute nodes accessing the dataset from memory without accessing distributed disk storage.

8. A system comprising:

at least one data processor and memory storing instructions, which when executed, cause the at least one data processor to perform operations comprising;

receiving summary statistics computed by at least executing one or more analytical processes on a dataset stored in parts across a set of memory based compute nodes, each compute node finding partial statistics of a data part stored on the respective compute node, the partial statistics representative of a respective data part;

storing the summary statistics in a random access memory associated with a server computer, the random access memory being accessible by at least one of the compute nodes, the summary statistics being a combination of the partial statistics and representative of a full dataset;

identifying, for pre-model building data understanding, outlier data by comparing subsets of data in the dataset, the identified outlier data accessible to a predictive model;

generating a graphical representation of at least some summary statistics stored in the random access memory; and

formatting the graphical representation of at least some summary statistics for transmission to and display by one or more client computers.

9. The system as in claim 8, the operations further comprising providing pattern detection for differentiated candidate model feature creation during predictive model development.

10. The system as in claim 8, the operations further comprising automatically identifying anomalous or outlier behavior in data to efficiently investigate large numbers of data subpopulations.

11. The system as in claim 8, the operations further comprising identifying suspected data integrity issues in data subpopulations by collecting data integrity checks into a report type, the report type being run and repeatable for each data integrity issue identification.

12. The system as in claim 8, the operations further comprising interactively exploring high-dimensional data distributions, the high-dimensional data being grouped into one or more statistics distribution arrays allowing users to interactively explore detected data patterns of interest.

13. The system as in claim 8, the operations further comprising maintaining the dataset within a distributed storage memory for future accessing and processing to occur in response to interaction with the graphical representation of the at least some summary statistics.

14. The system as in claim 13, the operations further comprising:

receiving a request for additional data analysis from the one or more client computers;

determining whether the dataset is present in the distributed storage memory; and

computing additional summary statistics in parallel on the compute nodes, the compute nodes accessing the dataset from memory without accessing distributed disk storage.

15. A computer program product comprising a non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

receiving summary statistics computed by at least executing one or more analytical processes on a dataset stored in parts across a set of memory based compute nodes, each compute node finding partial statistics of a data part stored on the respective compute node, the partial statistics representative of a respective data part;

storing the summary statistics in a random access memory associated with a server computer, the random access memory being accessible by at least one of the compute nodes, the summary statistics being a combination of the partial statistics and representative of the full dataset;

identifying, for pre-model building data understanding, outlier data by comparing subsets of data in the dataset, the identified outlier data accessible to a predictive model;

generating, a graphical representation of at least some summary statistics stored in the random access memory; and

formatting, the graphical representation of at least some summary statistics for transmission to and display by one or more client computers.

16. The computer program product as in claim 15, the operations further comprising providing pattern detection for differentiated candidate model feature creation during predictive model development.

17. The computer program product as in claim 15, the operations further comprising identifying suspected data integrity issues in data subpopulations by collecting data integrity checks into a report type, the report type being run and repeatable for each data integrity issue identification.

18. The computer program product as in claim 15, the operations further comprising interactively exploring high-dimensional data distributions, the high-dimensional data being grouped into one or more statistics distribution arrays allowing users to interactively explore detected data patterns of interest.

19. The computer program product as in claim 15, the operations further comprising maintaining the dataset within a distributed storage memory for future accessing and processing to occur in response to interaction with the graphical representation of the at least some summary statistics.

20. The computer program product as in claim 19, the operations further comprising:

receiving a request for additional data analysis from the one or more client computers;

determining whether the dataset is present in the distributed storage memory; and

computing additional summary statistics in parallel on the compute nodes, the compute nodes accessing the dataset from memory without accessing distributed disk storage.