Method and apparatus for creating and evaluating strategies

• US 20050096950A1
• Filed: 10/29/2003
• Published: 05/05/2005
• Est. Priority Date: 10/29/2003
• Status: Abandoned Application

• Current Assignee: Fair Isaac Corporation
• Original Assignee: Fair Isaac Corporation
• Inventors: Hua, Jun, Fahner, Gerald, Perlis, John, Cohen, Michael Raymond, Sullivan, Gary, Shikaloff, Nina, Thaker, Aush, Hoadley, Arthur Bruce, Chang, Yen Fook, Favero, Brendan Del, Caplan, Scott Malcolm, Fung, Robert Mun-Cheong, Crawford, Stuart, Lyons, Chisoo S., Wells, Eric C.
• Application Number:
  US10/697,907
• Publication Number:
  US 20050096950A1
• Time in Patent Office: Days
• US Class Current: 705/7.24
• CPC Class Codes: G06Q 10/06314 Calendaring for a resource
• CPC Class Codes: G06Q 10/06316 Sequencing of tasks or work
• CPC Class Codes: G06Q 10/0635 Risk analysis of enterprise or organisation activities
• CPC Class Codes: G06Q 10/0637 Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A method and apparatus for strategy science methodology involving computer implementation is provided. The invention includes a well-defined set of procedures for carrying out a full range of projects to develop strategies for clients. One embodiment of the invention produces custom consulting projects that are found at one end of the full range of projects. At the other end of the range are, for example, projects developing strategies from syndicated models. The strategies developed are for single decisions or for sequences of multiple decisions. Some parts of the preferred embodiment of the invention are categorized into the following areas: Team Development, Strategy Situation Analysis, Quantifying the Objective Function, Data Request and Reception, Data Transformation and Cleansing, Decision Key and Intermediate Variable Creation, Data Exploration, Decision Model Structuring, Decision Model Quantification, An Exemplary Score Tuner, Strategy Creation, An Exemplary Strategy Optimizer, An Exemplary Uncertainty Estimator, and Strategy Testing. Each of the sub-categories are described and discussed in detail under sections of the same headings. The invention uses judgment in addition to data for developing strategies for clients.

Claims

1. An iterative method for creating and evaluating strategies, comprising the steps of:

providing any of;

a team development module for developing a strategy modeling team;

a strategy situation analysis module for framing a decision situation;

a data request and reception module for designing and executing logistics of specifying, acquiring, and loading data required for decision and strategy modeling;

a data transformation and cleansing module for verifying, cleansing, and transforming data;

a decision key and intermediate variable creation module for computing additional variables from data and constructing a data dictionary;

a data exploration module for determining characteristics that are effective decision keys and intermediate variables;

a decision model structuring module for formalizing relationships between decisions, decision keys, intermediate variables, and value of a decision model;

a decision model quantification module for encoding information into a decision model;

a strategy creation module for determining strategies that a client can test; and

a strategy testing module for testing strategies to guide refinement of strategies and refinement of a decision model and to select a best strategy for deployment;

wherein each of said modules has capability to interact with an expert task manager, wherein said expert task manager provides expert knowledge about strategy modeling processes and sub-processes.

2. The iterative method of claim 1, the step of providing said team development module further comprising:

said strategy modeling team executing analysis to allow a leader of said strategy modeling team to convince a decision maker to implement a strategy favored by said analysis.

3. The iterative method of claim 1, the step of providing said strategy situation analysis module further comprising:

identifying the values of the organization; and

ensuring that the right decisions and strategies are considered in an analysis.

4. The iterative method of claim 1, the step of providing said data request and reception module further comprising:

designing and executing logistics of specifying, acquiring, and loading data required for decision and strategy modeling.

5. The iterative method of claim 1, the step of providing said data transformation and cleansing module further comprising:

verifying, cleansing, and transforming data.

6. The iterative method of claim 1, the step of providing said decision key and intermediate variable creation further comprising:

computing intermediate variables from said data, said intermediate variables dependent on decision keys; and

constructing a data dictionary.

7. The iterative method of claim 1, the step of providing said data exploration module further comprising:

providing insight into said data by determining which decision keys are most relevant for predicting said intermediate variables; and

gaining insight into a customer'"'"'s business and business processes.

8. The iterative method of claim 1, the step of providing said decision model structuring module further comprising:

formalizing relationships between decisions, decision keys, intermediate variables, and value by connecting such in a model.

9. The iterative method of claim 1, the step of providing said decision model quantification module further comprising:

encoding information into a decision model.

10. The iterative method of claim 1, the step of providing said strategy creation module further comprising:

applying optimization methods to a decision model to determine an optimal strategy for a set of cases.

11. The iterative method of claim 1, the step of providing said strategy creation module further comprising:

evolving using results from a decision model being enriched and from strategies tested.

12. The iterative method of claim 1, the step of providing said strategy testing module further comprising:

providing means for evaluating each strategy based on simulation; and

providing means for evaluating a strategy in the field.

13. The iterative method of claim 1, further comprising the steps of:

beginning with a simplified value model having less than eight drivers;

wherein each of said drivers is modeled crudely by one or two decision keys;

initially including no constraints;

using said simplified value model for beginning said strategy creation module and said strategy testing module, said strategy creation module and said strategy testing module indicating areas of said decision model where refinement adds particular value; and

after interaction between said decision model and strategies is acceptable, iteratively adding details reflecting limitations of a business process.

14. The iterative method of claim 1, wherein said team development module comprises a team creation component and a decision quality component.

15. The iterative method of claim 1, further comprising the step of:

providing a decision quality process for enabling an organization to systematically identify, understand, and track views of quality of decision making.

16. The iterative method of claim 1, further comprising the step of:

providing any of six dimensions associated with any of six links in a decision quality chain, said any of six links comprising;

appropriate frame;

creative-feasible alternatives;

meaningful-reliable Information;

clear values and tradeoffs;

logically-correct reasoning; and

commitment to action;

wherein said chain supports an organization'"'"'s value.

17. The iterative method of claim 1, said step of providing a strategy situation analysis module further comprising the steps of:

framing a problem by;

identifying issues;

developing a decision hierarchy;

understanding an organization'"'"'s values; and

brainstorming and clarifying alternatives;

further understanding said organization'"'"'s values by;

developing value metrics and prototyping metric results; and

planning for data acquisition by;

identifying intermediate variables; and

developing a plan for assessment;

wherein for clarification;

optionally returning to said framing a problem step after said further understanding said organization'"'"'s values step; and

optionally returning to said further understanding said organization'"'"'s values step after said planning for data acquisition step.

18. The iterative method of claim 1, the step of providing said data request and reception module further comprising the steps of:

developing data parameters, including;

determining data elements;

designing a performance period;

determining data records; and

constructing an initial data dictionary;

determining transfer parameters, including;

determining transfer format; and

determining transfer method;

preparing data, including;

assembling transfer data; and

transferring data; and

loading data on a target system.

19. The iterative method of claim 1, said step of providing a data transformation and cleansing module further comprising the steps of:

validating original data sets, comprising;

investigating original data sets; and

cleaning original data sets;

creating analysis data sets, comprising; and

transforming data; and

computing additional variables;

validating analysis data sets, comprising;

transforming data; and

computing additional variables;

wherein while creating analysis data sets and problems are uncovered in original data sets, then original data sets are further cleaned and retransformed; and

wherein while validating analysis data sets and problems in said transformation, or in original data sets, are uncovered, then such tasks are revisited.

20. The iterative method of claim 1, said step of providing a decision key and intermediate variable creation module further comprising the steps of:

first creating dependent variables useful for decision models, comprising;

identifying concepts;

triaging concepts; and

defining dependent variables; and

creating independent variables useful for decision models, comprising

identifying concepts;

triaging concepts; and

defining dependent variables;

wherein intermediate variables depend on decision keys, other intermediate variables, or decisions; and

wherein each intermediate variable encapsulates a predictive model with a dependent variable and independent variables.

21. The iterative method of claim 1, said step of providing a data exploration module further comprising the steps of:

applying basic statistical analysis, comprising;

analyzing continuous variables; and

analyzing discrete variables;

applying variable reduction techniques, comprising;

applying human and business judgment; and

applying computational methods;

applying advanced statistical analysis;

verifying results; and

presenting said results.

22. The iterative method of claim 1, said step of providing a decision model structuring module further comprising the steps of:

conceptualizing, comprising the steps of;

selecting intermediate variables that drive value;

building coarse models of intermediate variables; and

verifying constraints; and

drawing a decision model structure;

wherein said conceptualizing step is iteratively available for use after said drawing step.

23. The iterative method of claim 1, said step of providing a decision model quantification module further comprising the steps of:

modeling intermediate variables;

filling in nodes with models, functions, and/or constants; and

validating said decision model;

wherein said modeling step is iteratively available from said filling in step, and

wherein said filling in step is iteratively available from said validating said decision model step.

24. The iterative method of claim 1, further comprising the step of providing a score tuner component for automating decision model updating and reporting, said score tuner component comprising any of:

data awareness capability;

triggering rules;

model history retention;

self-guided model development;

connection to a decision engine; and

execution and analytic audit trails;

wherein when a tuning run is triggered, results are reviewed and either accepted and an update is deployed, or rejected.

25. The iterative method of claim 1, said step of providing a strategy creation module further comprising the steps of:

performing model optimization, comprising;

identifying metric variables;

determining optimization parameters; and

running optimization;

analyzing optimization results, comprising

viewing optimization results; and

performing sensitivity analysis on constraints; and

developing strategies, comprising;

building strategies; and

refining strategies;

wherein the performing model optimization step and the analyzing optimization results step are available to be used iteratively from either the analyzing optimization results step or the developing strategies step.

26. The iterative method of claim 1, further comprising the step of:

providing a non-linear constrained optimization tool for improving test designs and optimizing strategies.

27. The iterative method of claim 1, said step of providing a strategy testing module further comprising the steps of:

testing strategies, comprising;

performing strategy simulation; and

performing field testing;

evaluating strategies; and

performing active data collection;

wherein said testing strategies step is available for being used iteratively from said evaluating strategies step.

28. An apparatus for iteratively creating and evaluating strategies in an iterative, comprising:

means for providing any of;

a team development module for developing a strategy modeling team;

a strategy situation analysis module for framing a decision situation;

a data request and reception module for designing and executing logistics of specifying, acquiring, and loading data required for decision and strategy modeling;

a data transformation and cleansing module for verifying, cleansing, and transforming data;

a decision key and intermediate variable creation module for computing additional variables from data and constructing a data dictionary;

a data exploration module for determining characteristics that are effective decision keys and intermediate variables;

a decision model structuring module for formalizing relationships between decisions, decision keys, intermediate variables, and value of a decision model;

a decision model quantification module for encoding information into a decision model;

a strategy creation module for determining strategies that a client can test; and

a strategy testing module for testing strategies to guide refinement of strategies and refinement of a decision model and to select a best strategy for deployment;

wherein each of said modules has capability to interact with an expert task manager, wherein said expert task manager provides expert knowledge about strategy modeling processes and sub-processes.

29. The apparatus of claim 28, said team development module further comprising:

means for said strategy modeling team executing analysis to allow a leader of said strategy modeling team to convince a decision maker to implement a strategy favored by said analysis.

30. The apparatus of claim 28, said strategy situation analysis module further comprising:

means for identifying the values of the organization; and

means for ensuring that the right decisions and strategies considered in an analysis.

31. The apparatus of claim 28, said data request and reception module further comprising:

means for designing and executing logistics of specifying, acquiring, and loading data required for decision and strategy modeling.

32. The apparatus of claim 28, said data transformation and cleansing module comprising:

means for verifying, cleansing, and transforming data.

33. The apparatus of claim 28, said decision key and intermediate variable creation further comprising:

means for computing intermediate variables from said data, said intermediate variables dependent on decision keys; and

means for constructing a data dictionary.

34. The apparatus of claim 28, said data exploration module further comprising:

means for providing insight into said data by determining which decision keys are most relevant for predicting said intermediate variables; and

means for gaining insight into a customer'"'"'s business and business processes.

35. The apparatus of claim 28, further comprising:

means for said decision model structuring module formalizing relationships between decisions, decision keys, intermediate variables, and value by connecting such in a model.

36. The apparatus of claim 28, further comprising:

means for said decision model quantification module encoding information into a decision model.

37. The apparatus of claim 28, further comprising:

means for said strategy creation module applying optimization methods to a decision model to determine an optimal strategy for a set of cases.

38. The apparatus of claim 28, further comprising:

means for said strategy creation module evolving using results from a decision model being enriched and from strategies tested.

39. The apparatus of claim 28, further comprising:

means for said strategy testing module;

providing means for evaluating each strategy based on simulation; and

providing means for evaluating a strategy in the field.

40. The apparatus of claim 28, further comprising:

means for beginning with a simplified value model having less than eight drivers

wherein each of said drivers is modeled crudely by one or two decision keys;

means for initially including no constraints;

means for using said simplified value model for beginning said strategy creation module and said strategy testing module, said strategy creation module and said strategy testing module indicating areas of said decision model where refinement adds particular value; and

means for after interaction between said decision model and strategies is acceptable, iteratively adding details reflecting limitations of a business process.

41. The apparatus of claim 28, wherein said team development module comprises:

a team creation component; and

a decision quality component.

42. The apparatus of claim 28, further comprising:

means for providing a decision quality process for enabling an organization to systematically identify, understand, and track views of quality of decision making.

44. The apparatus of claim 28, said means for providing a strategy situation analysis module further comprises:

means for framing a problem by;

identifying issues;

developing a decision hierarchy;

understanding an organization'"'"'s values; and

brainstorming and clarifying alternatives;

means for further understanding said organization'"'"'s values by

developing value metrics and prototyping metric results; and

means for planning for data acquisition by;

identifying intermediate variables; and

developing a plan for assessment;

wherein for clarification;

optional means for returning to said framing a problem step after said further understanding said organization'"'"'s values step; and

optional means for returning to said further understanding said organization'"'"'s values step after said planning for data acquisition step.

45. The apparatus of claim 28, said data request and reception module further comprising:

means for developing data parameters, comprising any of;

determining data elements;

designing a performance period;

determining data records; and

constructing an initial data dictionary;

means for determining transfer parameters, comprising;

determining transfer format; and

determining transfer method;

means for preparing data, comprising;

assembling transfer data; and

transferring data; and

means for loading data on a target system.

46. The apparatus of claim 28, said means for providing a data transformation and cleansing module further comprising:

means for validating original data sets, comprising;

investigating original data sets; and

cleaning original data sets;

means for creating analysis data sets, comprising; and

transforming data; and

computing additional variables;

means for validating analysis data sets, comprising;

transforming data; and

computing additional variables;

wherein while creating analysis data sets and problems are uncovered in original data sets, then original data sets are further cleaned and retransformed; and

wherein while validating analysis data sets and problems in said transformation, or in original data sets, are uncovered, then such tasks are revisited.

47. The apparatus of claim 28, said means for providing a decision key and intermediate variable creation module further comprising:

means for first creating dependent variables useful for decision models, comprising;

identifying concepts;

triaging concepts; and

defining dependent variables; and

means for creating independent variables useful for decision models, comprising

identifying concepts;

triaging concepts; and

defining dependent variables;

wherein intermediate variables depend on decision keys, other intermediate variables, or decisions; and

wherein each intermediate variable encapsulates a predictive model with a dependent variable and independent variables.

48. The apparatus of claim 28, said means for providing a data exploration module further comprising:

means for applying basic statistical analysis, comprising;

analyzing continuous variables; and

analyzing discrete variables;

means for applying variable reduction techniques, comprising;

applying human and business judgment; and

applying computational methods;

means for applying advanced statistical analysis;

verifying results; and

presenting said results.

49. The apparatus of claim 28, said means for providing a decision model structuring module further comprising:

means for conceptualizing, comprising the steps of;

selecting intermediate variables that drive value;

building coarse models of intermediate variables; and

verifying constraints; and

means for drawing a decision model structure;

wherein said conceptualizing step is iteratively available for use after said drawing step.

50. The apparatus of claim 28, said means for providing a decision model quantification module further comprising:

means for modeling intermediate variables;

means for filling in nodes with models, functions, and/or constants; and

means for validating said decision model;

wherein said modeling step is iteratively available from said filling in step, and wherein said filling in step is iteratively available from said validating said decision model step.

51. The apparatus of claim 28, further comprising:

means for providing a score tuner component for automating decision model updating and reporting, said score tuner component comprising any of;

data awareness capability;

triggering rules;

model history retention;

self-guided model development;

connection to a decision engine; and

execution and analytic audit trails;

wherein when a tuning run is triggered, results are reviewed and either accepted and an update is deployed, or rejected.

52. The apparatus of claim 28, said means for providing a strategy creation module further comprising:

means for performing model optimization, comprising;

identifying metric variables;

determining optimization parameters; and

running optimization;

means for analyzing optimization results, comprising

viewing optimization results; and

performing sensitivity analysis on constraints; and

means for developing strategies, comprising;

building strategies; and

refining strategies;

wherein the performing model optimization step and the analyzing optimization results step are available to be used iteratively from either the analyzing optimization results step or the developing strategies step.

53. The apparatus of claim 28, further comprising:

a non-linear constrained optimization tool for improving test designs and optimizing strategies.

54. The apparatus of claim 28, said means for providing a strategy testing module further comprising:

testing strategies, comprising;

performing strategy simulation; and

performing field testing; and

evaluating strategies; and

performing active data collection;

wherein said testing strategies step is available for being used iteratively from said evaluating strategies step.

43. The apparatus of claim 90, further comprising:

means for providing any of six dimensions associated with any of six links in a decision quality chain, said six links comprising;

appropriate frame;

creative-feasible alternatives;

meaningful-reliable Information;

clear values and tradeoffs;

logically-correct reasoning; and

commitment to action;

wherein said chain supports an organization'"'"'s value.

55. An apparatus for automating decision model updating and reporting, comprising:

at least one tuning apparatus, comprising any of;

data awareness capability;

triggering rules;

model history retention;

self-guided model development;

connection to a decision engine; and

means for triggering a parameter tuning run execution and analytic audit trails; and

means for reviewing results, wherein said results are either accepted and an update is deployed, or rejected.

56. The apparatus of claim 55, further comprising:

means for interacting with a server that handles tuning parameters, and running a scripted model optimization engine for generating new models and evaluation reports;

wherein said tuning parameters are any of sample sizes, population definition, and whether tuning is manually initiated or triggered on a set schedule.

57. A decisioning client apparatus, comprising:

a decisioning client application processing system for;

supplying data associated with a customer to a decision engine; and

requesting a decision; and

wherein said decision engine comprises a score generation module;

means for said decision engine, using said score generation module, generating needed transformations of said data and generating at least one score, said at least one score based on at least one score weight of at least one scorecard at a time;

means for said decision engine applying pre-specified decision rules and strategies using said data and said transformed data, and at least one score for generating a vector of recommended decision actions;

means for said decision engine returning requested data, said transformed data, said at least one score, information about said at least one scorecard, and said recommended actions to said decisioning client application processing system;

means for said decisioning client application processing system optionally implementing said recommended actions, and storing results into a data store.

58. The decisioning client apparatus of claim 57, further comprising any of:

means for said decisioning client application processing system optionally taking additional non-score-based decisions over time;

means for said decisioning client application processing system monitoring and recording periodic signals from customers and general environment;

means for said decisioning client application processing system gathering data over time about a customer for helping determine one or more outcomes of interest; and

an asynchronous process periodically triggering preparation of a matched data set from information about a customer, said information from a predetermined time, wherein said results are appended to a growing store of predictive plus performance data records; and

said asynchronous process further comprising means for a score tuner component having a triggering mechanism, using said triggering mechanism for periodically taking said matched data set and producing, if appropriate, score weight updates of at least one active scorecard, wherein said scorecard is installed into said score generation module after a review.

59. A score tuner method, comprising the steps of:

providing a score tuning broker module for performing administrative tasks associated with updating of score weights, said score tuning broker module comprising the steps of;

determining which scorecards are candidates for tuning;

checking any operating scorecards are flagged for updates; and

at a pre-specified and parameterized time frequency, determining from a rule database which scorecards are up for score weight re-tuning;

extracting needed data set sub-population based on rules determining what sampling window and stratification a current scorecard needs;

for a scorecard that is a candidate for re-tuning for the current time stamp;

requesting generation of a data set to be used for said tuning; and

determining what score weight engine project is associated with said scorecard;

passing a reference to said data set and a project id to said score weight engine, and requesting metrics of scorecard performance from said score weight engine; and

determining whether updated version is better or not; and

providing a score weight engine module for performing activities related to scorecard results and score weights, said score weight engine module comprising the steps of;

reporting on an existing scorecard'"'"'s development measures;

computing a scorecard'"'"'s performance measures on a new sample;

auditing new predictive data set to ensure that settings are adequate to cover data values encountered in said new data;

creating a new scorecard version of said scorecard being tuned;

converting raw records in said new predictive data set into coarse classed records needed for building weights;

building and scaling score weights of said newly created scorecard given said new predictive data; and

archiving said newly built scorecard and its performance measures.

60. The score tuner method of claim 59, wherein said score weight engine module is script-driven.

61. A score tuner method, comprising the steps of:

providing rapid weights tuning for modifying score weights of a scorecard; and

/or

providing rapid score alignment for aligning parameters of said scorecard;

wherein said underlying structure of said scorecard'"'"'s data is not different from original implementation definition.

62. The score tuner method of claim 61, further comprising any of the steps of:

providing a range from a null set of weights to automated and intelligent variable selection, classing, model building, scaling, and evaluation;

providing automated validation of newly developed weights for a fixed set of characteristics against a set of previously developed weights on said same characteristic set; and

providing automatic re-alignment of a set of scorecards to scale to a previous set of odds.

63. The score tuner method of claim 61, further comprising any of the steps of:

providing capability of specifying updating or re-scaling of many models at once; and

providing capability of specifying a schedule for automatic scorecard updates and scaling, implying integration into current decision support systems.

64. The score tuner method of claim 61, further comprising the step of:

providing modeling functionality comprising the steps of;

importing of existing scorecards from decision support software;

auditing for legal values for scorecard characteristics in a new data set;

generating summarized data in preparation of the tuning process including;

classing of values of data records variables into those expected by the scorecard characteristics;

generating all summarization needed to run proprietary algorithms from a newly provided predictive data set and previously summarized results from past tuning runs; and

displaying some summary statistics of records encountered;

providing specification of expected scaling parameters;

running an algorithm to generate new score weights for scorecard characteristics;

running evaluation procedures on newly tuned weights;

displaying a scorecard and its evaluation results;

fitting of log of odds vs. score to determine expected odds by score;

adjusting alignment parameters to match user supplied expectation;

exporting of said tuned alignment parameters in a format acceptable to decision support software, and while maintaining version control for said scorecard; and

providing ability to sequence any of above mentioned steps.

65. The score tuner method of claim 61, further comprising the step of:

providing reporting and visualization capabilities, comprising summarized views of new score variable and scorecard characteristics;

wherein each view includes a comparison of old weights versus new, if applicable, and wherein data is divided by defined bins of scorecard characteristics.

66. A score tuner apparatus, comprising:

a database manager component for managing collection of cases used in analysis, and for providing a bridge to multiple possible input data files and/or database management systems;

a data manager component for providing data records to other data analysis components, one case at a time in the event that said data analysis components are processing cases in a sample point loop, for exposing a data dictionary to other components, and for allowing posting variables generated in said data analysis components back to said database manager for future recall;

a modeler component for providing score weight re-optimization and for logging odds to score alignment functionality;

a report collection component for providing viewing, printing, and limited editing of a standard set of model evaluation reports generated by said modeler;

a workflow controller for controlling flow of multiple business components performing a set of actions that are implied by user specifications and eventually fulfilling desired data preparation, analysis, and/or presentation steps; and

an intelligence agent for performing background checks on results from user actions and for providing suggestions if a query against its rule base returns a recommended intelligent action to take.

67. The score tuner apparatus of claim 66, wherein said intelligence agent comprises:

means for guiding specification of analytic steps;

means for reacting to interactive analytic actions with suggestions, via agents, for possible changes; and

means for automating intelligence-assisted decision-making in a sequence of analytic actions.

68. A system for estimating an uncertainty interval around at least one estimate of at least one expected outcome, comprising:

an input device operable to allow entering and transferring input data to a processor;

an output device for displaying human readable results of manipulation of said input data;

one or more communications buses between said input device and said processor and said output device and said processor, respectively; and

said processor comprising a memory, wherein said memory stores at least one program for quantifying said uncertainty interval due to variation based on case-level variation, model variation, and portfolio composition, said program performing a sequence of instructions, the sequences of instructions, which, when executed by said processor, cause the processor to perform the steps of;

causing a decision model to encapsulate case-level variation;

implementing non-parametric bootstrapping techniques to capture model variation;

using analysis of historic data on holdout samples to describe case-level error distributions; and

capturing portfolio composition variation as an integral element of said quantifying said uncertainty interval process.

69. The system of claim 68, wherein said process of quantifying said uncertainty interval comprises two stages:

wherein said first stage is repeated for each component model making up said decision model, resulting in estimating all necessary parameters, and

wherein said second stage uses said estimated parameters for rolling said up variations into aggregated measures and presenting a range of said at least one expected outcome.

70. A method for estimating an uncertainty interval around at least one estimate of at least one expected outcome, comprising the steps of:

providing an input device operable to allow entering and transferring input data to a processor;

providing an output device for displaying human readable results of manipulation of said input data;

providing communications buses between said input device and said processor and said output device and said processor, respectively; and

said processor comprising a memory, wherein said memory stores at least one program for quantifying said uncertainty interval due to variation based on case-level variation, model variation, and portfolio composition, said program performing a sequence of instructions, the sequences of instructions, which, when executed by said processor, cause the processor to perform the steps of;

providing a decision model to encapsulate case-level variation;

implementing non-parametric bootstrapping techniques to capture model variation;

using analysis of historic data on holdout samples to describe case-level error distributions; and

capturing portfolio composition variation as an integral element of said quantifying said uncertainty interval process.

71. The method of claim 70, wherein said process of quantifying said uncertainty interval comprises two stages:

wherein said first stage is repeated for each component model making up said decision model, resulting in estimating all necessary parameters, and

wherein said second stage uses said estimated parameters for rolling said up variations into aggregated measures and presenting a range of said at least one expected outcome.