This section deals with the estimation of identified SARIMA models, i.e. we want to estimate
the unknown parameters of the multiplicative SARIMA model. If the model contains only
AR terms, the unknown coefficients can be estimated using ordinary least squares. However, if
the model contains MA terms too, the task becomes more complicated, because the lagged values
of the innovations are unobservable. Consequently, it is not possible to derive explicit
expressions to estimate the unknown coefficients and therefore one has to use maximum
likelihood for estimation purposes. In the next subsection 5.4.1 the
theoretical background of maximizing a likelihood function is briefly outlined.
In order to convey an idea of what follows, we will shortly outline the procedure: first, one
sets up the multiplicative SARIMA model--in the following also called original
model-- with some initial values for the unknown parameters . In subsection
5.4.2 it is explained how to set the original SARIMA model using the quantlet
msarimamodel. Restrictions can be imposed on the coefficients. The simplest restriction
is that some of the coefficients are zero. Then the value of the likelihood function--given
the initial parameters--is evaluated.
Unfortunately, in most cases the original SARIMA model cannot be estimated directly. If one
looks at the SARIMA(3,1,1)(12,1,0,0) model in section 5.4.3--equation
(5.18)--one recognizes on the left hand side the product of two expressions. Both of
them contain lag-operators. Such expressions have to be telescoped out first. XploRe
provides a very convenient tool to do so: msarimaconvert. This quantlet is explained in
detail in subsection 5.4.3. The result you get from msarimaconvert is an
ordinary ARMA(p,q) model which can be estimated.
Under the assumption that an ARMA model is stationary and invertible and that the
observations are normally distributed, it can be estimated using the maximum likelihood
approach. By making suitable assumptions about initial conditions, the maximum likelihood
estimators can be obtained by minimizing the conditional sum of squares. In subsection
5.4.4 the quantlet msarimacond is presented. It calculates the conditional
sum of squares function and allows for zero restrictions on the coefficients. Given this
function, numerical methods have to be applied to maximize the likelihood function with
respect to .
To evaluate the fit of the estimated model, the quantlet msarimacond also delivers several criteria for diagnostic checking. The residuals of the model should be white noise. The quantlet eacf provides an easy way to check the behavior of the residuals.
However, the conditional sum of squares is not always very satisfactory for seasonal series. In that case the calculation of the exact likelihood function becomes necessary (Box and Jenkins; 1976, p. 211). One approach is to set up the likelihood function via Kalman filter techniques. We briefly discuss how to set up the airline model in state space form and how to use the Kalman filter to evaluate the exact likelihood function. Once again, numerical methods are necessary to maximize the exact likelihood function.
The approach of maximum likelihood (ML) requires the specification of a particular
distribution for a sample of observations
. Let
![]() |
denote the probability density of the sample given the unknown
parameters
. It can be interpreted as the probability of having observed the given sample
(Hamilton; 1994, p. 117).
With the sample
at hand, the above given probability can be rewritten
as a function of the unknown parameters given the sample
. Following the notation of
Box and Jenkins (1976), we use the notation
. In most cases it is easier to
work with the log likelihood function
.
The maximum likelihood estimate
is the parameter vector that maximizes the
probability for the observed sample
. Thus, the MLE satisfies the so-called likelihood
equations, which are obtained by differentiating
with respect to each of the
unknown parameters of the vector
and setting the derivatives to zero
(Harvey; 1993). Using vector notation and suppressing
, we obtain
As a rule, the likelihood equations are non-linear. Therefore, the ML estimates must be found in the course of an iterative procedure. This is true for the exact likelihood function of every Gaussian ARMA(p,q) process (see Hamilton; 1994, Chapter 5).
As already mentioned above, there are two different likelihood functions in use: the conditional and the exact likelihood function. Both alternatives can be estimated using XploRe.
In many applications of ARMA models the conditional likelihood function is an
alternative to the exact likelihood function. In that case, one assumes that the first
observations of a Gaussian ARMA(p,q) process are deterministic and are equal to its observed
values
. The initial residuals
for
are set
to its expected values 0. The log likelihood function for this setting is of the following
form with
and denoting the sum of squares function
The notation
emphasizes that
is no longer a disturbance, but a
residual which depends on the value taken by the variables in
.
Note, that the parameter is an additional one, that is not included in vector
. It is easy to see that (5.9) is maximized with respect to
if the sum of squares
is minimized. Using the condition
(5.8), this leads to
Thus, the ML estimate for can be obtained by minimizing (5.10).
Furthermore, we obtain from (5.9) and (5.8) that
Thus, given the parameter vector
that maximizes the sum of squares--and
thus the conditional log likelihood function (5.9)--one divides
the sum of squares by
to obtain the ML estimate
. Another
estimator for the variance of the innovations corrects furthermore for the number of
estimated coefficients
.
As already mentioned, one approach to calculate the exact log likelihood is to use the Kalman filter. We want to show this for the airline model. This model is rewritable in state space form (SSF) as
where is a
identity matrix (Koopman, Shephard and Doornik; 1999). Here,
is the so-called transition matrix.
Given Gaussian error terms , the value of the log likelihood function
for
the above given state space form is
Here,
![]() ![]() |
are the innovations of the Kalman filtering procedure and
is the
information set up to
.
is the matrix from the above given state space form that
contains the identity matrix. The matrix
is the covariance matrix of the innovations in
period
and it is a by-product of the Kalman filter. The above log likelihood is known as
the prediction error decomposition form (Harvey; 1989).
Given some initial values for , the above exact log likelihood is evaluated with
the Kalman filter. Using numerical methods, the function is maximized with respect to
.
Once the ML estimate
is calculated, one wants to have standard errors for
testing purposes. If
is sufficiently large, the ML estimate
is
approximately normally distributed with the inverse of the information matrix divided by
as covariance matrix. The inverse of the Hessian for
is one way to estimate
the covariance matrix (Hamilton; 1994, Section 5.8). One can calculate the Hessian applying
numerical methods.
|
The original model is specified by means of the quantlet
msarimamodel
. The three
arguments are lists that give the difference orders
, the ordinary ARMA parts
and
, and the seasonal AR and MA polynomials
and
. If the model has no seasonal difference, one just omits
.
The arma list has at most four elements: the first element is a vector that
specifies the lags of the AR polynomial that are not zero. The second element
is a vector that specifies the lags of the MA polynomial
. If the model has
only one polynomial, one sets the lags of the other one to 0.
The third element of the arma list is a vector with
If the model has both an AR and a MA polynomial then the fourth argument is necessary. It is a list that contains the coefficients of the MA polynomial. For example,
arma = list((1|3),0,(0.1|-0.25))
specifies an ARMA(3,0) part with ,
and
.
The last argument season is a list that contains the information concerning the seasonal
AR and MA polynomials. This list has at most five elements: the first element specifies the
season . If the data show no seasonal pattern, one sets
as the only argument of the
list season. The second element is the lag structure of the seasonal AR polynomial. You
have to fill in the lags that are different from zero. The third element is the lag structure
of the seasonal MA polynomial. The last two elements are for the coefficient vectors of the
polynomials. As explained for the arma list, one can omit the respective vector if the
model has only one polynomial. For example,
season = list(12,0,(1|4),(-0.3|0.1))
gives a model with no seasonal AR polynomial and with the seasonal MA(4) polynomial
![]() |
(5.16) |
To understand what
msarimamodel
does, let us assume that the multiplicative
SARIMA model is given as
![]() |
(5.17) |
Here and
, so that we can set d=1. The lists for the ordinary and seasonal
polynomials are given above. To have a look at the output of msarimamodel, one just
compiles the following piece of XploRe code
arma = list((1|3),0,(0.1|-0.25)) ; ordinary ARMA part season = list(12,0,(1|4),(-0.3|0.1)) ; seasonal ARMA part msarimamodelOut = msarimamodel(1,arma,season) msarimamodelOut ; shows the elements of msarimamodelOut
The output is
Contents of msarimamodelOut.d [1,] 1 Contents of msarimamodelOut.arlag [1,] 1 [2,] 3 Contents of msarimamodelOut.malag [1,] 0 Contents of msarimamodelOut.s [1,] 12 Contents of msarimamodelOut.sarlag [1,] 0 Contents of msarimamodelOut.smalag [1,] 1 [2,] 4 Contents of msarimamodelOut.phi [1,] 0.1 [2,] -0.25 Contents of msarimamodelOut.theta [1,] 0 Contents of msarimamodelOut.Phi [1,] 0 Contents of msarimamodelOut.Theta [1,] -0.3 [2,] 0.1
and it resembles our example in general notation (see equation (5.4)). The list msarimamodelOut is an easy way to check the correct specification of the model.
|
If you want to estimate the coefficients in (5.4), you have to telescope out the
original model. Given the specification by the list msarimaOut (see Subsection
5.4.2) and the time series
, the quantlet
msarimaconvert
telescopes out the original model automatically.
Let us consider the following SARIMA(3,1,1)(12,1,0,0) model with
:
Telescoping out the polynomials on the left-hand side leads to an ordinary ARMA model:
![]() |
(5.19) |
with
,
,
,
,
,
, and
.
The superscript denotes the coefficients of the expanded model. The output of the
quantlet is thus self-explaining: the series
is just the differenced original series
and the other two outputs are the vector
(with
) and the
vector
(with
). The first vector has the dimension
and second one has the dimension
. The scalar
gives
the number of coefficients in the original model. For the above given example, we have
,
whereas the number of coefficients of the expanded model is 6. Later on, we need
for the
calculation of some regression diagnostics.
|
The sum of squares is a criterion that can be used to identify the coefficients of the best model. The output of the quantlet is the conditional sum of squares for a given model specification (Box and Jenkins; 1976, Chapter 7).
For an ARMA(p,q) model this sum is just
![]() |
(5.20) |
Here denotes the number of observations
. Recall that the first entries of the
lag-polynomials are for
.
denotes the MA-polynomial without the
first entry. The first
residuals are set to zero.
The arguments of the quantlet are given by the output of
msarimaconvert
. mu is
the mean of the series
.
is the number of coefficients in the original
model and will be used to calculate some regression diagnostics. This argument is optional.
If you do not specify
, the number of coefficients in the expanded model is used instead.
Furthermore, the quantlet
msarimacond
gives the list dia that contains several
diagnostics. After the maximization of the conditional sum of squares, one can use these
diagnostics to compare different specifications. In the ongoing
denotes the number of
ARMA parameters that are different from zero. In our example we have
for both
specifications.
The first element of the list--that is dia.s2--is the estimated variance of the residuals
![]() |
(5.21) |
The second element dia.R2 gives the coefficient of determination
![]() |
(5.22) |
The variance of the dependent variables is calculated for the observations starting
with
. It is possible in our context that
becomes negative.
The adjusted coefficient of determination is calculated as
![]() |
(5.23) |
It is the third argument of the list and is labeled dia.aR2.
The fourth element dia.logl gives the values of the log likelihood function evaluated
at
. Given the likelihood function (5.9),
is a function of
. To take this into account, we plug
(5.12) into (5.9) and obtain
![]() |
(5.24) |
This expression is the value of the log likelihood function.
The fifth element dia.AIC gives the Akaike Information Criteria (AIC)
AIC![]() |
(5.25) |
The sixth element dia.SIC gives the Schwarz Information Criteria (SIC)
SIC![]() |
(5.26) |
For both criteria see Shumway and Stoffer (2000). These criteria can be used for model
selection (Durbin and Koopman (2001)). Eventually, the last element dia.a gives the
vector of the residuals
.
Now we can come back to our example of the airline data: recall that we have identified
two possible specifications for this data set. The first specification is a
SARIMA(0,1,1)(12,0,1,1) with
. The
second is a SARIMA(0,1,12)
(12,0,1,0) with
.
We maximize the conditional sum of squares for both specifications using the BFGS algorithm.
Given the estimates
, the standard errors are obtained by means of the Hessian
matrix for the log likelihood. The Hessian is calculated for this function using numerical
methods. The square roots of the diagonal elements of the inverse Hessian are the standard
errors of the estimates
.
The results of the first specification are obtained using the the quantlet
and the results of the second specification can be also obtained with the quantlet
It is obvious that both specifications deliver good results. However, the sum of squared resids is smaller for the specification with a seasonal MA term. Additionally, both information criteria indicate that this specification is slightly better.
|
After estimating the unknown parameters
for competing specifications, one
should have a look at the residual series
. They should behave
like a white noise process and should exhibit no autocorrelation. In order to check for
autocorrelation, one could use the ACF and PACF. However, the extended autocorrelation
function (EACF) is also a convenient tool for inspecting time series
(Peña, Tiao and Tsay; 2001, Chapter 3) that show no seasonal pattern.
In general, the EACF allows for the identification of ARIMA models (differencing is not
necessary). The quantlet
eacf
generates a table of the sample EACF for a time series.
You have to specify the maximal number of AR lags (p) and MA lags (q). Every row of the
output table gives the ACF up to q lags for the residuals of an AR regression with
lags. Furthermore, the simplified EACF is tabulated. If an autocorrelation is
significant according to Bartlett's formula the entry is 1. Otherwise the entry is 0.
Bartlett's formula for an MA(q) is given as
![]() |
where is the number of observations (Peña, Tiao and Tsay; 2001). For identification, look for the
vertex of a triangle of zeros. You can immediately read off the order of the series from the
table.
We use the EACF to explore the behavior of the residuals of both
specifications. The next quantlet computes the EACF of the
residuals that come from the SARIMA(0,1,1) (12,0,1,1)
specification.
It is obvious, that the vertex of zeros is at position and
. Thus we conclude that
the residuals are white noise. Notice, that the first line in the above table at
just
gives the ACF of the residual series. According to Bartlett's formula, all autocorrelation
coefficients are not significantly different from zero.
The next quantlet computes the EACF of the residuals of the
SARIMA(0,1,12) (12,0,1,0) specification.
We can conclude that the vertex of zeros is at position and
. Thus the residuals
are white noise. According to Bartlett's formula, once again all autocorrelation coefficients
are not significantly different from zero.
|
As already mentioned in the introductory part of this section, the Kalman filter can be used to evaluate the exact log likelihood function. For the estimation of the unknown parameters the evaluated log likelihood function 5.15 is required. The second element of the quantlet provides the value of the exact log likelihood function.
We now shortly describe the procedure of the Kalman filter and the implementation with
gkalfilter
. Good references for the Kalman filter are--in addition to
Harvey (1989)--Hamilton (1994), Gourieroux and Monfort (1997) and
Shumway and Stoffer (2000). The first argument is an array with the observed time series.
The vector mu specifies the initial conditions of the filtering procedure with
corresponding covariance matrix Sig. Due to the fact that our SSF (5.13)
contains no constants, we set ca and da to zero. Furthermore, we have no
disturbance in the measurement equation--that is the equation for
in
(5.13)--so we also set Ha to zero. The covariance matrix for the disturbance
in the state equation is given as
![]() |
Eventually, Za is an array for the matrix given in the measurement equation.
The state space form of the airline model is given in (5.13). It is a well known fact
that the product of eigenvalues for a square matrix is equal to its determinant. The
determinant of the transition matrix for our model--given in equation
(5.14)--is zero and so all eigenvalues are also zero. Thus our system is
stable (Harvey; 1989, p. 114). In that case, we should set the initial values to the
unconditional mean and variance of the state vector (Koopman, Shephard and Doornik; 1999). We easily obtain
for our model (5.13) that
and
![]() |
A way to solve for the elements of is
Here, vec denotes the vec-operator that places the columns of a matrix below each other
and denotes the Kronecker product.
For the estimation, we use the demeaned series of the growth rates of the airline data.
The standard errors of the estimates are given by the square roots of the diagonal elements of
the inverse Hessian evaluated at
. The following table shows the results:
"===================================================================" " Estimation results for the SARIMA(0,1,12)x(12,0,1,0) specification" " Exact Log Likelihood function is maximized " "===================================================================" " Convergence achieved after 12 iterations " " 131 observations included " " " " Variable Coefficient t-stat p-value " " ------------------------------------------------------ " " theta_1 -0.3998 -4.4726 0.00 " " theta_12 -0.5545 -7.5763 0.00 " " sigma2 0.0014 8.0632 0.00 " " ------------------------------------------------------ " " AIC -3.6886 SIC -3.5111 " "==================================================================="
The estimators are only slightly different from the estimators we have calculated with
the conditional likelihood. The variance
is identical.