Gradient learning for probabilistic ARMA time-series models

US 7,421,380 B2
Filed: 12/14/2004
Issued: 09/02/2008
Est. Priority Date: 12/14/2004
Status: Expired due to Fees

First Claim

Patent Images

1. A computer readable storage medium having stored thereon components that facilitates statistical modeling, the components comprising:

a gradient determination component that determines a conditional log-likelihood model gradient for tied parameters by employing a Recursive Exponential Mixed Model (REMM) in a continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model;

a gradient search component that determines optimal parameters to generate the stochastic ARMA^xptime series model by employing the conditional log-likelihood model gradient of the tied parameters;

a receiving component that receives a query; and

a statistical modeling component that applies the query to the stochastic ARMA^xptime series model and generates data predictions for the received query.

View all claims

2 Assignments

Timeline View

Assignment View

0 Petitions

Accused Products

Abstract

The subject invention leverages the conditional Gaussian (CG) nature of a continuous variable stochastic ARMA^xptime series model to efficiently determine its parametric gradients. The determined gradients permit an easy means to construct a parametric structure for the time series model. This provides a gradient-based alternative to the expectation maximization (EM) process for learning parameters of the stochastic ARMA^xptime series model. Thus, gradients for parameters can be computed and utilized with a gradient-based learning method for estimating the parameters. This allows values of continuous observations in a time series to be predicted utilizing the stochastic ARMA^xptime series model, providing efficient and accurate predictions.

61 Citations

View as Search Results

27 Claims

1. A computer readable storage medium having stored thereon components that facilitates statistical modeling, the components comprising:
- a gradient determination component that determines a conditional log-likelihood model gradient for tied parameters by employing a Recursive Exponential Mixed Model (REMM) in a continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model;
  
  a gradient search component that determines optimal parameters to generate the stochastic ARMA^xptime series model by employing the conditional log-likelihood model gradient of the tied parameters;
  
  a receiving component that receives a query; and
  
  a statistical modeling component that applies the query to the stochastic ARMA^xptime series model and generates data predictions for the received query.
- View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
- - 2. The medium of claim 1 further comprising:
    - the statistical modeling component employs a gradient-based learning process to estimate parameters of the model.
  - 3. The medium of claim 2,the gradient-based learning process comprising a gradient-based optimization process that utilizes the gradient to iteratively adapt the parameters of the model to improve the incomplete-data log-likelihood in order to identify the maximum likelihood estimate and/or local maxima of the incomplete-data log-likelihood.
  - 4. The medium of claim 3,the gradient-based optimization process comprising a conjugate gradient process.
  - 5. The medium of claim 1, the time series model comprising a model wherein conditional variance of each continuous time tube variable is fixed to a small positive value.
  - 6. The medium of claim 5, the small positive value approximately equal to a value of 0.01.
  - 7. The medium of claim 1, the gradient determination component utilizes at least one fixed parameter to facilitate in determining the model gradient.
  - 8. The medium of claim 7, the fixed parameter having a value of one.
  - 9. The medium of claim 7, the fixed parameter comprising a regression parameter.
  - 10. The medium of claim 7, the fixed parameter comprising a variance parameter.
  - 11. A device employing the system of claim 1 comprising at least one selected from the group consisting of a computer, a server , or a handheld electronic device.

12. A method for facilitating statistical modeling, comprising:
- obtaining an incomplete data set for a continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model; and
  
  determining a conditional log-likelihood model gradient, for parameters tied across time steps by employing a Recursive Exponential Mixed Model (REMM) in the continuous variable, stochastic autoregressive moving average cross-predicting time series model utilizing the data set;
  
  selecting optimal parameters based on the determined model gradient for the parameters;
  
  constructing the continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model utilizing the selected parameters; and
  
  generating data predictions for a given query from the time series model.
- View Dependent Claims (13, 14, 15, 16, 17, 18, 19, 20, 27)
- - 13. The method of claim 12 further comprising:
    - utilizing at least one fixed parameter to facilitate in determining the model gradient.
  - 14. The method of claim 13,the fixed parameter having a value of one.
  - 15. The method of claim 13,the fixed parameter comprising a regression parameter.
  - 16. The method of claim 13,the fixed parameter comprising a variance parameter.
  - 17. The method of claim 12 further comprising:
    - employing a plurality of processes for determining the gradients for the time series model;
      
      each process, respectively, utilizing sum and sum of squares statistics and gradient determination expressions based on process variations for a particular time series “
      
      s”
      
      of the time series model.
  - 18. The method of claim 12 further comprising:
    - employing a propagation scheme for Bayesian networks with conditional Gaussian distributions to determine a mean vector and a covariance matrix associated with a posterior marginal distribution for all families (X_v,X_pa(v)), where X_vis a continuous variable in the stochastic ARMA^xpmodel and X_pa(v)represents the parents of that variable;
      
      utilizing posterior conditional means and variances to determine expected values (X_v*,X_pa(v)*) and ((X_v,X_pa(v))′
      
      (X_v,X_pa(v)))* for each of these families;
      
      For all time series s=1, . . . ,S;
      
      setting a sum and a sum of squares expected posterior statistics for E variables to zero;
      
      setting a sum and a sum of squares expected posterior statistics for Y variables to zero;
      
      for each t=R+1, . . . ,T;
      
      adding e_s,t* and (e_s,te_s,t)* to the sum and sum of squares statistics, respectively, associated with the E variables for time series s;
      
      adding (y_s,t*,a_s,t*) and ((y_s,t,a_s,t) ′
      
      (y _s,t,a_s,t))* to the sum and sum of squares statistics, respectively, associated with the Y variables for time series s; and
      
      determining the gradient for tied parameters in the continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model utilizing;
      
      $\begin{matrix} \frac{\partial \log p (y ⋆,_{R + 1}, \dots, y ⋆_{, T} \langle y ⋆_{, 1} \dots, y ⋆_{, T}, θ)}{\partial γ_{s}} = \sum_{t = R + 1}^{T} \frac{{(e_{s, t} e_{s, t})}^{*} - γ_{s}}{2 γ_{s}^{2}}; and, & (Eq . 6) \\ \frac{\partial \log p (y ⋆,_{R + 1}, \dots, y ⋆_{, T} \langle y ⋆_{, 1} \dots, y ⋆_{, T}, θ)}{\partial (c_{s}, β_{s})} = \sum_{t = R + 1}^{T} [\begin{matrix} (y_{s, t}^{*} - β_{s} z_{s, t}^{*} - c_{s}) / σ_{s} \\ ({(y_{s, t} a_{s, t})}^{*} - c_{s} a_{s, t}^{*} - {β (a_{s, t}^{'} a_{s, t})}^{*} / σ_{s} \end{matrix}] . & (Eq . 7) \end{matrix}$
  - 19. The method of claim 12 further comprising:
    - setting a variance parameter, γ
      
      _s, to a value of one for all time series s=1, . . . ,S in the model;
      
      employing a propagation scheme for Bayesian networks with conditional Gaussian distributions to determine a mean vector and a covariance matrix associated with a posterior marginal distribution for all families (X_v,X_pa(v)), where X_vis a continuous variable in the stochastic ARMA^xpmodel and X_pa(v)represents the parents of that variable;
      
      utilizing posterior conditional means and variances to determine expected values (X_v* ,X_pa(v)*) and ((X_v,X_pa(v))′
      
      (X_v,X_pa(v)))* for each of these families;
      
      For all time series s=1, . . . ,T;
      
      setting a sum and a sum of squares expected posterior statistics for Y variables to zero;
      
      for each t=R+1, . . . ,T;
      
      adding (y_s,t*, a_s,t*) and ((y_s,t,a_s,t) ′
      
      (y_s,t,a_s,t))* to the sum and sum of squares statistics, respectively, associated with the Y variables for time series s; and
      
      determining the gradient for tied parameters in the continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model utilizing;
      
      $\begin{matrix} \frac{\partial \log p (y ⋆,_{R + 1}, \dots, y ⋆_{, T} \langle y ⋆_{, 1} \dots, y ⋆_{, T}, θ)}{\partial (c_{s}, β_{s})} = \sum_{t = R + 1}^{T} [\begin{matrix} (y_{s, t}^{*} - β_{s} a_{s, t}^{*} - c_{s}) / σ_{s} \\ ({(y_{s, t} a_{s, t})}^{*} - c_{s} a_{s, t}^{*} - {β (a_{s, t}^{'} a_{s, t})}^{*} / σ_{s} \end{matrix}] . & (Eq . 7) \end{matrix}$
  - 20. The method of claim 12 further comprising:
    - setting a regression parameter, β
      
      _s,o, to a value of one for all time series s=1, . . . ,S in the model;
      
      employing a propagation scheme for Bayesian networks with conditional Gaussian distributions to determine a mean vector and a covariance matrix associated with a posterior marginal distribution for all families (X_v,X_pa(v)), where X_vis a continuous variable in the stochastic ARMA^xpmodel and X_pa(v)represents the parents of that variable;
      
      utilizing posterior conditional means and variances to determine expected values (X_v*,X_pa(v)*) and ((X_v,X_pa(v))′
      
      (X_v,X_pa(v)))* for each of these families;
      
      For all time series s=1, . . . ,S;
      
      setting a sum and a sum of squares expected posterior statistics for E variables to zero;
      
      setting a sum and a sum of squares expected posterior statistics for Y variables to zero;
      
      for each t=R+1, . . . , T;
      
      adding e_s,t* and (e_s,te_s,t)* to the sum and sum of squares statistics, respectively, associated with the E variables for time series s;
      
      adding (y_s,t*,z_s,t*) and ((y_s,t,z_s,t) ′
      
      (y_s,t,z_s,t))* to the sum and sum of squares statistics, respectively, associated with the Y variables for time series s; and
      
      determining the gradients for tied parameters in the continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model utilizing;
      
      $\begin{matrix} \frac{\partial \log p (y ⋆,_{R + 1}, \dots, y ⋆_{, T} \langle y ⋆_{, 1} \dots, y ⋆_{, T}, θ)}{\partial γ_{s}} = \sum_{t = R + 1}^{T} \frac{{(e_{s, t} e_{s, t})}^{*} - γ_{s}}{2 γ_{s}^{2}}; and, & (Eq . 6) \\ \frac{\partial \log p (y ⋆,_{R + 1}, \dots, y ⋆_{, T} \langle y ⋆_{, 1} \dots, y ⋆_{, T}, θ)}{\partial (c_{s}, β_{s}^{z_{t}})} = \sum_{t = R + 1}^{T} [\begin{matrix} (y_{s, t}^{*} - β_{s} z_{s, t}^{*} - c_{s}) / σ_{s} \\ ({(y_{s, t} z_{s, t})}^{*} - c_{s} z_{s, t}^{*} - {β_{s}^{z_{t}} (z_{s, t}^{'} z_{s, t})}^{*} / σ_{s} \end{matrix}] . & (Eq . 8) \end{matrix}$
  - 27. A device employing the method of claim 12 comprising at least one selected from the group consisting of a computer, a server ,or a handheld electronic device.

21. A system that facilitates statistical modeling, comprising:
- means for storing a data set for a continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model;
  
  means for employing a Recursive Exponential Mixed Model (REMM) to facilitate in determining a model gradient;
  
  means for determining the conditional log-likelihood model gradient for tied parameters in the continuous variable, stochastic autoregressive moving average, cross-predicting time series model utilizing the data set;
  
  means for selecting optimal parameters for constructing the continuous variable, stochastic autoregressive moving average, cross-predicting (stochastic ARMA^xp) time series model based on the determined model gradients; and
  
  means for making data predictions from the stochastic ARMA^xptime series in response to a received query.
- View Dependent Claims (22, 23, 24, 25, 26)
- - 22. The system of claim 21 further comprising:
    - means for utilizing a gradient-based learning process to estimate parameters of the time series model via employment of the determined gradient.
  - 23. The system of claim 22, the gradient-based learning process comprising a conjugate gradient process.
  - 24. The system of claim 21 further comprising:
    - means for utilizing at least one fixed parameter to facilitate in determining the model gradient.
  - 25. The system of claim 24, the fixed parameter comprising a regression parameter.
  - 26. The system of claim 24, the fixed parameter comprising a variance parameter.

Specification

Resources

Litigation Campaign Assessment

Current Assignee
Microsoft Technology Licensing LLC (Microsoft Corporation)
Original Assignee
Microsoft Corporation
Inventors
Meek, Christopher A., Thiesson, Bo
Primary Examiner(s)
PHAN, THAI Q

Application Number

US11/011,864
Publication Number

US 20060129395A1
Time in Patent Office

1,358 Days
Field of Search

703/2, 703/6, 702/184, 702/182, 702/183, 706/14, 706/21
US Class Current

703/2
CPC Class Codes

G06F 18/00 Pattern recognition

G06F 2218/08 Feature extraction

Gradient learning for probabilistic ARMA time-series models

First Claim

2 Assignments

0 Petitions

Accused Products

Abstract

61 Citations

27 Claims

Specification

Solutions

Use Cases

Quick Links

Gradient learning for probabilistic ARMA time-series models

First Claim

2 Assignments

Subscription Required

Subscription Required

0 Petitions

Subscription Required

Accused Products

Subscription Required

Abstract

61 Citations

27 Claims

Specification

Subscription Required

Solutions

Use Cases

Quick Links