Data analysis computer system and method for organizing, presenting, and optimizing predictive modeling

• US 9,772,741 B2
• Filed: 03/17/2014
• Issued: 09/26/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Alexander Statnikov, Konstantinos F. Aliferis, Lawrence Fu, Yin Aphinyanaphongs
• Original Assignee: Alexander Statnikov, Konstantinos F. Aliferis, Lawrence Fu, Yin Aphinyanaphongs
• Inventors: Aliferis, Konstantinos (Constantin) F., Aphinyanaphongs, Yin, Fu, Lawrence, Statnikov, Alexander
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US14/216,585
• Publication Number:
  US 20140279794A1
• Time in Patent Office: 1,289 Days
• Field of Search: 706 46
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 50/18 Legal services

Abstract

Predictive modeling is an important class of data analytics with applications in numerous fields. Once a predictive model is built, validated, and applied on a set of objects, by a data analytics system (or even by manual modeling), consumers of the model information need assistance to navigate through the results. This is because both regression and classification models that output continuous values (eg, probability of belonging to a class) are often used to rank objects and then a thresholding of the ranked scores needs to be used to separate objects into a “positive” and a “negative” class. The choice of threshold greatly affects the true positive, false positive, true negative, and false negative results of the model'"'"'s application. An ideal data analytics system should allow the user to understand the tradeoffs of different threshold values for different thresholds. The user interface should convey this information in an intuitive manner and provide the ability to vary the threshold interactively while simultaneously presenting the effects of thresholding on predictivity. This is precisely the function of the present invention. In addition to manual thresholding, the invention also allows for the thresholding to be performed by fully automated means (via standard statistical optimization methods) once a user has identified the desired balance of false positives and false negatives (or other predictivity metrics of interest). The invention can be applied to any application field of predictive modeling.

Claims

1. A computer aided technique for visually analyzing the results of the application of predictive models to data sets, including big data, comprising the following steps:

a) storing data and predictive models;

b) selecting a predictive model;

c) applying the predictive model to the data to generate a score for each data object;

d) selecting an analytical threshold to employ with the scores;

e) outputting a graphical display that shows the data objects arranged along an axis according to their predictive model score along with an indication of the value of a currently chosen threshold;

f) outputting a table with a row associated with the threshold and columns showing expected predictive performance metrics calculated from performance statistics of the predictive model; and

g) repeating steps d) through f) for a different threshold and dynamically updating the display and tables with the data for each threshold

wherein the density of values in the graphical display and the tabular output informs the user choice in selecting an appropriate threshold.

2. The method of claim 1 further comprising:

h) repeating steps b) through g) for a different predictive model; and

i) repeating step h) until all predictive models have been utilized or an acceptable threshold has been found

wherein the graphical display and tables allow simultaneous depiction of one or more thresholds associated with one or more predictive models.

3. The method of claim 1 in which the user may input preferences or trade-offs among false positives, false negatives, true positives and true negatives.

4. The method of claim 2 in which the user may input preferences or trade-offs among false positives, false negatives, true positives and true negatives.

5. The method of claim 1 in which the user may input preferences for maximum number of objects classified as positive that can be examined by the user.

6. The method of claim 2 in which the user may input preferences for maximum number of objects classified as positive that can be examined by the user.

7. The computer implemented method of claim 1 further comprising:

A) users can “

freeze” and

save or overlay a threshold line and corresponding performance table so that multiple thresholding scenarios can be considered and compared;

B) graphical display can be organized horizontally or vertically;

C) scoring scales can be transformed from unit scale to logarithmic, standard deviation scale, or other standard statistical transformations;

D) graphical display allows making points semitransparent, binning the data into rectangles, binning the data into hexagons, using box plots, and using alternative colors;

E) representing points in various sizes, shapes, and colors to denote additional object information;

F) making points “

clickable”

or “

hoverable” and

triggering another event such as displaying contents or other aspects of the object;

G) displaying baseline performance measures set by a third party or default trade-off among false negatives, false positives, and true positives; and

H) the strip can display labels for each point.

8. The computer implemented method of claim 2 further comprising:

A) users can “

freeze” and

save or overlay a threshold line and corresponding performance table so that multiple thresholding scenarios can be considered and compared;

B) graphical display can be organized horizontally or vertically;

C) scoring scales can be transformed from unit scale to logarithmic, standard deviation scale, or other standard statistical transformations;

D) graphical display allows making points semitransparent, binning the data into rectangles, binning the data into hexagons, using box plots, and using alternative colors;

E) representing points in various sizes, shapes, and colors to denote additional object information;

F) making points “

clickable”

or “

hoverable” and

triggering another event such as displaying contents or other aspects of the object;

G) displaying baseline performance measures set by a third party or default trade-off among false negatives, false positives, and true positives; and

H) the strip can display labels for each point.

9. The method of claim 3 where once users input preferences about the acceptable ratio of false positives to true positives and ratio of true positives to false negatives, the system automatically identifies the threshold that maximizes true positives and minimizes false negatives in accordance with the user'"'"'s preferences.

10. The method of claim 4 where once users input preferences about the acceptable ratio of false positives to true positives and ratio of true positives to false negatives, the system automatically identifies the threshold that maximizes true positives and minimizes false negatives in accordance with the user'"'"'s preferences.

11. The method of claim 5 where, once users input preferences for maximum number of objects classified as positive that can be examined by the user, the system automatically identifies the threshold that maximizes true positives and minimizes false negatives in accordance with the user'"'"'s preferences.

12. The method of claim 6 where once users input preferences for maximum number of objects classified as positive that can be examined by the user, the system automatically identifies the threshold that maximizes true positives and minimizes false negatives in accordance with the user'"'"'s preferences.