Data Analysis Computer System and Method for Organizing, Presenting, and Optimizing Predictive Modeling

• US 20140279794A1
• Filed: 03/17/2014
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Alexander Statnikov, Konstantinos F. Aliferis, Lawrence Fu, Yin Aphinyanaphongs
• Original Assignee: Alexander Statnikov, Konstantinos F. Aliferis, Lawrence Fu, Yin Aphinyanaphongs
• Inventors: Aliferis, Konstantinos (Constantin) F., Aphinyanaphongs, Yin, Fu, Lawrence, Statnikov, Alexander
• Granted Patent: US 9,772,741 B2
• Time in Patent Office: Days
• US Class Current: 706/46
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 50/18 Legal services

Abstract

Predictive modeling is an important class of data analytics with applications in numerous fields. Once a predictive model is built, validated, and applied on a set of objects, by a data analytics system (or even by manual modeling), consumers of the model information need assistance to navigate through the results. This is because both regression and classification models that output continuous values (eg, probability of belonging to a class) are often used to rank objects and then a thresholding of the ranked scores needs to be used to separate objects into a “positive” and a “negative” class. The choice of threshold greatly affects the true positive, false positive, true negative, and false negative results of the model'"'"'s application. An ideal data analytics system should allow the user to understand the tradeoffs of different threshold values for different thresholds. The user interface should convey this information in an intuitive manner and provide the ability to vary the threshold interactively while simultaneously presenting the effects of thresholding on predictivity. This is precisely the function of the present invention. In addition to manual thresholding, the invention also allows for the thresholding to be performed by fully automated means (via standard statistical optimization methods) once a user has identified the desired balance of false positives and false negatives (or other predictivity metrics of interest). The invention can be applied to any application field of predictive modeling.

Claims

1. A computer-implemented method and system for visually-assisted, thresholding of predictive models:

a) incorporating a memory device, that contains objects (“

examples”

) and one or more predictive models;

b) incorporating a processing device which applies the model to the examples to generate a score for each;

c) the processing device outputting graphical displays on a display device;

d) the graphical displays containing a series of icons, each denoting an individual object to be classified;

e) the graphical displays including strips and vertical bars denoting the score of each object and the value of the currently chosen threshold while random vertical jittering is used to separate the presented examples;

f) the graphical displays containing tables showing expected predictive performance metrics at each threshold and for the currently chosen threshold value, calculated from performance statistics of the predictive model;

g) the tables being dynamically updated once a user changes values of score threshold;

h) the tables being configurable so that they show only metrics of interest to the system user;

i) the graphical displays allowing simultaneous depiction one or more thresholds, one or more classes and one or more predictive models;

j) the system allowing a user input preferences of trade-offs among false positives, false negatives, true positives and true negatives;

k) the system allowing a user input preferences for maximum number of objects classified as positive that can be examined by the user; and

l) outputting a final threshold that satisfies user preferences and is consistent with data and model operating characteristics.

2. The computer implemented method and system of claim 1 where:

a) users can “

freeze” and

save or overlay a threshold line and corresponding performance table so that multiple thresholding scenarios can be considered and compared;

b) graphical display can be organized horizontally or vertically;

c) scoring scales can be transformed from unit scale to logarithmic, standard deviation scale, or other standard statistical transformations;

d) graphical display allows making points semitransparent, binning the data into rectangles, binning the data into hexagons, using box plots, and using alternative colors;

e) representing points in various sizes, shapes, and colors to denote additional oject information;

f) making points “

clickable”

or “

hoverable” and

triggering another event such as displaying contents or other aspects of the object;

g) displaying baseline performance measures set by a third party or default trade-off among false negatives, false positives, and true positives; and

h) the strip can display labels for each point.

3. The computer implement method and system of claim 1 where once users input preferences about the acceptable ratio of false positives to true positives and ratio of true positives to false negatives, and then the system automatically identifies the threshold that maximizes true positives and minimizes false negatives in accordance with the user'"'"'s preferences.

4. The computer implement method and system of claim 2 where once users input preferences about the acceptable ratio of false positives to true positives and ratio of true positives to false negatives, and then the system automatically identifies the threshold that maximizes true positives and minimizes false negatives in accordance with the user'"'"'s preferences.