Adaptive modeling of data streams

• US 10,565,331 B2
• Filed: 07/27/2017
• Issued: 02/18/2020
• Est. Priority Date: 07/27/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Varughese, Melvin Mathew
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Crabb, Steven W
• Application Number:
  US15/660,988
• Publication Number:
  US 20190034565A1
• Time in Patent Office: 936 Days
• CPC Class Codes: G06F 17/11 for solving equations , e.g. nonlinear equations, general mathematical optimization problems optimization specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30

Abstract

A computer-implemented method for adaptive modeling of a data stream is provided. The method may comprise receiving a plurality of data points forming part of a data stream and providing a predictive model to be fitted to the data stream. The predictive model may be a diffusion model having a plurality of diffusion parameters. A parameter estimate for each one of the diffusion parameters may be updated by obtaining a sample of at least one transition from the data stream and calculating an updated parameter estimate for the diffusion parameter by using a stochastic gradient descent algorithm on the sample. The updating of the parameter estimate may be repeated periodically or in response to one or more further data points being added to the data stream, thereby to permit adaptive estimation of the diffusion parameters of the diffusion model based on dynamics of the data stream.

Claims

1. A computer-implemented method comprising:

receiving, by a receiving module of a computer processor, a plurality of data points forming part of a data stream;

providing a predictive model to be fitted to the data stream, wherein the predictive model is a diffusion model having a plurality of diffusion parameters;

updating, by an updating module of the computer processor, a parameter estimate for each of the diffusion parameters by;

obtaining a sample of at least one transition from the data stream, wherein each transition is defined by at least two data points; and

calculating an updated parameter estimate for the diffusion parameter using a stochastic gradient descent algorithm on the sample, wherein a partial derivative associated with the diffusion parameter at each transition is calculated while holding values associated with the other diffusion parameters at smoothed or weighted average estimates, and

repeating, by the updating module, the updating periodically, or in response to one or more further data points being added to the data stream, of the parameter estimate for each one of the diffusion parameters, thereby to permit adaptive estimation of the diffusion parameters of the diffusion model based on dynamics of the data stream.

2. The method of claim 1, which further comprises receiving, by the receiving module, an indication of a sampling distribution associated with each one of the diffusion parameters and/or an indication of a sample size associated with each one of the diffusion parameters, and wherein the obtaining a sample comprises obtaining the sample in accordance with the sampling distribution associated with the diffusion parameter and/or including in the sample a number of transitions corresponding to the sample size associated with the diffusion parameter.

3. The method of claim 1, wherein the obtaining a sample comprises obtaining the sample in accordance with a sampling distribution, wherein the sampling distribution is defined by a sliding and/or decaying probability distribution that allocates higher selection probabilities to more recent data points in the data stream.

4. The method of claim 3, wherein the obtaining a sample comprises, in addition to obtaining the sample in accordance with the sampling distribution, obtaining a most recent transition defining a transition to a most recent data point in the data stream, such that the sample includes at least one transition obtained in accordance with the sampling distribution as well as the most recent transition.

5. The method of claim 3, wherein the sampling distribution is truncated so as to cause only transitions within a sliding effective time window associated with the data stream to be capable of being sampled.

6. The method of claim 1, wherein the obtaining a sample comprises obtaining a most recent transition defining a transition to a most recent data point in the data stream.

7. The method of claim 1, wherein the calculating an updated parameter estimate comprises, for each transition in the sample:

calculating or approximating a transitional density for the transition;

evaluating the transitional density at a most recent data point of the transition;

calculating the partial derivative of the log of the transitional density with respect to the diffusion parameter while holding values associated with the other diffusion parameters at smoothed or weighted average estimates; and

using the calculated partial derivate to update the parameter estimate for the diffusion parameter, wherein, if the sample includes a plurality of transitions, an average of or smoothed value for the partial derivatives calculated for the transitions is used to update the parameter estimate.

8. The method of claim 1, wherein the repeating the updating is carried out in response to each new data point of the data stream received by the receiving module.

9. The method of claim 1, which further comprises forecasting, by a forecasting module of the computer processor, future data points associated with the data stream using the parameter estimates for the diffusion parameters.

10. The method of claim 1, further comprising:

detecting, by a monitoring module of the computer processor, a risk event, an opportunity event, a regime shift or an anomaly associated with the data stream; and

providing, by an alert module of the computer processor, in response to detecting the risk event, opportunity event, regime shift or anomaly, an alert message or risk indication to a user.

11. A computer system comprising:

a computer processor; and

a computer readable storage medium having stored thereon program instructions executable by the computer processor to direct the operation of the processor in fitting a predictive model to a data stream, wherein the predictive model is a diffusion model having a plurality of diffusion parameters, and wherein the computer processor, when executing the program instructions, comprises;

a receiving module configured to receive a plurality of data points forming part of the data stream; and

an updating module configured to update a parameter estimate for each of the diffusion parameters by

obtaining a sample of at least one transition from the data stream, wherein each transition is defined by at least two data points; and

calculating an updated parameter estimate for the diffusion parameter by using a stochastic gradient descent algorithm on the sample, wherein a partial derivative associated with the diffusion parameter is calculated while holding values associated with the other diffusion parameters at smoothed or weighted average estimates,

wherein the updating module is configured to repeat the updating of the parameter estimate for each one of the diffusion parameters periodically or in response to one or more further data points being added to the data stream, thereby to permit adaptive estimation of the diffusion parameters of the diffusion model based on dynamics of the data stream.

12. The computer system of claim 11, wherein the receiving module is configured to receive an indication of a sampling distribution associated with each one of the diffusion parameters and/or an indication of a sample size associated with each one of the diffusion parameters, and wherein the updating module is configured to obtain the sample of at least one transition from the data stream for each one of the diffusion parameters in accordance with the sampling distribution associated with the diffusion parameter and/or to include in the sample a number of transitions corresponding to the sample size associated with the diffusion parameter.

13. The computer system of claim 11, wherein the updating module is configured to obtain the sample of at least one transition from the data stream in accordance with a sampling distribution, wherein the sampling distribution is defined by a sliding and/or decaying probability distribution that allocates higher selection probabilities to more recent data points in the data stream.

14. The computer system of claim 13, wherein, in addition to obtaining the sample in accordance with the sampling distribution, the updating module is configured to obtain a most recent transition defining a transition to a most recent data point in the data stream, such that the sample includes at least one transition obtained in accordance with the sampling distribution as well as the most recent transition.

15. The computer system of claim 11, wherein the updating module is configured to obtain the sample of at least one transition from the data stream by obtaining at least a most recent transition defining a transition to a most recent data point in the data stream.

16. The computer system of claim 11, wherein the updating module is configured to calculate the updated parameter estimate for the diffusion parameter by, for each transition in the sample:

calculating or approximating a transitional density for the transition;

evaluating the transitional density at a most recent data point of the transition;

calculating the partial derivative of the log of the transitional density with respect to the diffusion parameter while holding values associated with the other diffusion parameters at smoothed or weighted average estimates; and

using the calculated partial derivate to update the parameter estimate for the diffusion parameter, wherein, if the sample includes a plurality of transitions, an average of or smoothed value for the partial derivatives calculated for the transitions is used to update the parameter estimate.

17. The computer system of claim 11, wherein the updating module is configured to repeat the updating of the parameter estimate for each one of the diffusion parameters in response to each new data point of the data stream received by the receiving module.

18. The computer system of claim 11, wherein the computer processor, when executing the program instructions, comprises a forecasting module configured to forecast future data points associated with the data stream using the parameter estimates for the diffusion parameters.

19. The computer system of claim 11, wherein the computer processor, when executing the program instructions, comprises:

a monitoring module configured to detect a risk event, an opportunity event, a regime shift or an anomaly associated with the data stream; and

an alert module configured to provide an alert message or risk indication to a user in response to detecting the risk event, opportunity event, regime shift or anomaly.

20. A computer program product for adaptive modeling of a data stream, the computer program product comprising:

a computer readable storage medium having stored thereon;

first program instructions executable by a computer processor to cause the computer processor to receive a plurality of data points forming part of a data stream, wherein a predictive model is to be fitted to the data stream, and wherein the predictive model is a diffusion model having a plurality of diffusion parameters; and

second program instructions executable by the computer processor to cause the computer processor to update a parameter estimate for each one of the diffusion parameters by

obtaining a sample of at least one transition from the data stream, wherein each transition is defined by at least two data points; and

calculating an updated parameter estimate for the diffusion parameter by using a stochastic gradient descent algorithm on the sample, wherein a partial derivative associated with the diffusion parameter is calculated while holding values associated with the other diffusion parameters at smoothed or weighted average estimates,

wherein the second program instructions are executable by the computer processor to cause the computer processor to repeat the updating of the parameter estimate for each one of the diffusion parameters periodically or in response to one or more further data points being added to the data stream, thereby to permit adaptive estimation of the diffusion parameters of the diffusion model based on dynamics of the data stream.