Computer-implemented systems and methods for testing large scale automatic forecast combinations

• US 9,047,559 B2
• Filed: 04/05/2012
• Issued: 06/02/2015
• Est. Priority Date: 07/22/2011
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Brzezicki, Jerzy Michal, Apte, Dinesh P., Leonard, Michael J., Chipley, Michael Ryan, Mainkar, Sagar Arun, Blair, Edward Tilden
• Primary Examiner(s): Rivas, Omar Fernandez
• Assistant Examiner(s): CALLE, ANGEL J.
• Application Number:
  US13/440,045
• Publication Number:
  US 20130024173A1
• Time in Patent Office: 1,153 Days
• Field of Search: 703/2, 703/6
• US Class Current: 1/1
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 2111/08 Probabilistic or stochastic CAD
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G16B 40/00 ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding

Abstract

Systems and methods are provided for evaluating performance of forecasting models. A plurality of forecasting models may be generated using a set of in-sample data. Two or more forecasting models from the plurality of forecasting models may be selected for use in generating a combined forecast. An ex-ante combined forecast may be generated for an out-of-sample period using the selected two or more forecasting models. The ex-ante combined forecast may then be compared with a set of actual out-of-sample data to evaluate performance of the combined forecast.

Claims

1. A computer-program product tangibly embodied in a non-transitory, machine-readable storage medium having instructions stored thereon, the instructions operable to cause a data processing apparatus to perform operations including:

displaying representations of multiple models operable to provide ex-post forecasts and ex-ante forecasts with regard to a forecast variable;

accessing an input that represents a selection of at least two of the models;

accessing a historical time series that includes multiple historical observations of the forecast variable;

defining multiple distinct holdout time series within the historical time series, wherein each of the holdout time series includes a portion of the historical observations of the forecast variable and spans a corresponding time period;

generating a combination model by combining the selected models;

evaluating multi-step forecasting characteristics of the combination model by using ex-post forecasting at multiple different points of origin, wherein evaluating includes, with respect to each of the time periods, performing operations including;

using the combination model to generate ex-post forecasted values of the forecast variable at multiple times during the time period;

comparing the ex-post forecasted values to coinciding historical observations included in the holdout time series that spans the time period;

calculating ex-post forecast errors based on the comparison; and

displaying the ex-post forecast errors.

2. The computer-program product of claim 1, wherein the ex-post forecast errors calculated with respect to each of the time periods are displayed with reference to the time periods spanned by the respective holdout time series.

3. The computer-program product of claim 1, wherein each of the time periods differs in duration from every other one of the time periods.

4. The computer-program product of claim 1, wherein the operations further include:

displaying an indication of multiple selectable holdout time series durations; and

receiving an input that represents a selection of at least two of the holdout time series durations, wherein the multiple holdout time series are defined based on the selected holdout time series durations.

5. The computer-program product of claim 1, wherein comparing the ex-post forecasted values to coinciding historical observations includes displaying the ex-post forecasted values and the coinciding historical observations on a same graph.

6. The computer-program product of claim 5, further comprising:

using the combination model to generate an ex-ante forecast of the forecasting variable, wherein the ex-ante forecast includes forecasted values of the forecasting variable with respect to at least one future time or future time period.

7. The computer-program product of claim 6, wherein the operations further include:

receiving an input representing an ex-ante forecasting step-ahead parameter, wherein generating the ex-ante forecast further includes setting a step-ahead time of the ex-ante forecast based on the input.

8. The computer-program product of claim 1, wherein defining multiple holdout time series within the historical time series is done such that no two of the time periods begin at a same time.

9. A method comprising:

displaying representations of multiple models operable to provide ex-post forecasts and ex-ante forecasts with regard to a forecast variable;

accessing an input that represents a selection of at least two of the models;

accessing a historical time series that includes multiple historical observations of the forecast variable;

defining multiple distinct holdout time series within the historical time series, wherein each of the holdout time series includes a portion of the historical observations of the forecast variable and spans a corresponding time period;

generating a combination model by combining the selected models;

evaluating multi-step forecasting characteristics of the combination model by using ex-post forecasting at multiple different points of origin, wherein evaluating includes, with respect to each of the time periods, performing operations including;

using the combination model to generate ex-post forecasted values of the forecast variable at multiple times during the time period;

comparing the ex-post forecasted values to coinciding historical observations included in the holdout time series that spans the time period;

calculating ex-post forecast errors based on the comparison; and

displaying the ex-post forecast errors.

10. The method of claim 9, wherein the ex-post forecast errors calculated with respect to each of the time periods are displayed with reference to the time periods spanned by the respective holdout time series.

11. The method of claim 9, wherein each of the time periods differs in duration from every other one of the time periods.

12. The method of claim 9, further comprising:

displaying an indication of multiple selectable holdout time series durations; and

receiving an input that represents a selection of at least two of the holdout time series durations, wherein the multiple holdout time series are defined based on the selected holdout time series durations.

13. The method of claim 9, wherein comparing the ex-post forecasted values to coinciding historical observations includes displaying the ex-post forecasted values and the coinciding historical observations on a same graph.

14. The method of claim 13, further comprising:

using the combination model to generate an ex-ante forecast of the forecasting variable, wherein the ex-ante forecast includes forecasted values of the forecasting variable with respect to at least one future time or future time period.

15. The method of claim 14, further comprising:

receiving an input representing an ex-ante forecasting step-ahead parameter, wherein generating the ex-ante forecast further includes setting a step-ahead time of the ex-ante forecast based on the input.

16. The method of claim 9, wherein defining multiple holdout time series within the historical time series is done such that no two of the time periods begin at a same time.

17. A computerized system comprising:

a processor configured to perform operations including;

displaying representations of multiple models operable to provide ex-post forecasts and ex-ante forecasts with regard to a forecast variable;

accessing an input that represents a selection of at least two of the models;

accessing a historical time series that includes multiple historical observations of the forecast variable;

defining multiple distinct holdout time series within the historical time series, wherein each of the holdout time series includes a portion of the historical observations of the forecast variable and spans a corresponding time period;

generating a combination model by combining the selected models;

evaluating multi-step forecasting characteristics of the combination model by using ex-post forecasting at multiple different points of origin, wherein evaluating includes, with respect to each of the time periods, performing operations including;

using the combination model to generate ex-post forecasted values of the forecast variable at multiple times during the time period;

comparing the ex-post forecasted values to coinciding historical observations included in the holdout time series that spans the time period;

calculating ex-post forecast errors based on the comparison; and

displaying the ex-post forecast errors.

18. The system of claim 17, wherein the ex-post forecast errors calculated with respect to each of the time periods are displayed with reference to the time periods spanned by the respective holdout time series.

19. The system of claim 17, wherein each of the time periods differs in duration from every other one of the time periods.

20. The system of claim 17, wherein the operations further include:

displaying an indication of multiple selectable holdout time series durations; and

receiving an input that represents a selection of at least two of the holdout time series durations, wherein the multiple holdout time series are defined based on the selected holdout time series durations.

21. The system of claim 17, wherein comparing the ex-post forecasted values to coinciding historical observations includes displaying the ex-post forecasted values and the coinciding historical observations on a same graph.

22. The system of claim 21, wherein the operations further include:

using the combination model to generate an ex-ante forecast of the forecasting variable, wherein the ex-ante forecast includes forecasted values of the forecasting variable with respect to at least one future time or future time period.

23. The system of claim 22, wherein the operations further include:

receiving an input representing an ex-ante forecasting step-ahead parameter, wherein generating the ex-ante forecast further includes setting a step-ahead time of the ex-ante forecast based on the input.

24. The system of claim 17, wherein defining multiple holdout time series within the historical time series is done such that no two of the time periods begin at a same time.