# 4.2 Linear Stationary Models for Time Series

A stochastic process is a model that describes the probability structure of a sequence of observations over time. A time series is a sample realization of a stochastic process that is observed only for a finite number of periods, indexed by .

Any stochastic process can be partially characterized by the first and second moments of the joint probability distribution: the set of means, , and the set of variances and covariances . In order to get consistent forecast methods, we need that the underlying probabilistic structure would be stable over time. So a stochastic process is called weak stationary or covariance stationary when the mean, the variance and the covariance structure of the process is stable over time, that is:

 (4.1) (4.2) (4.3)

Given condition (4.3), the covariance between and depends only on the displacement and it is called autocovariance at lag , . The set of autocovariances , , is called the autocovariance function of a stationary process.

The general Autoregressive Moving Average model is a linear stochastic model where the variable is modelled in terms of its own past values and a disturbance. It is defined as follows:

 (4.4)

where the random variable is called the innovation because it represents the part of the observed variable that is unpredictable given the past values .

The general model (4.4) assumes that is the output of a linear filter that transforms the past innovations , that is, is a linear process. This linearity assumption is based on the Wold's decomposition theorem (Wold; 1938) that says that any discrete stationary covariance process can be expressed as the sum of two uncorrelated processes,

 (4.5)

where is purely deterministic and is a purely indeterministic process that can be written as a linear sum of the innovation process :

 (4.6)

where is a sequence of serially uncorrelated random variables with zero mean and common variance . Condition is necessary for stationarity.

The formulation (4.4) is a finite reparametrization of the infinite representation (4.5)-(4.6) with constant. It is usually written in terms of the lag operator defined by , that gives a shorter expression:

 (4.7)

where the lag operator polynomials and are called the polynomial and the polynomial, respectively. In order to avoid parameter redundancy, we assume that there are not common factors between the and the components.

Next, we will study the plot of some time series generated by stationary models with the aim of determining the main patterns of their temporal evolution. Figure 4.2 includes two series generated from the following stationary processes computed by means of the genarma quantlet:

 Series 1: *[2mm] Series 2:

As expected, both time series move around a constant level without changes in variance due to the stationary property. Moreover, this level is close to the theoretical mean of the process, , and the distance of each point to this value is very rarely outside the bounds . Furthermore, the evolution of the series shows local departures from the mean of the process, which is known as the mean reversion behavior that characterizes the stationary time series.

Let us study with some detail the properties of the different processes, in particular, the autocovariance function which captures the dynamic properties of a stochastic stationary process. This function depends on the units of measure, so the usual measure of the degree of linearity between variables is the correlation coefficient. In the case of stationary processes, the autocorrelation coefficient at lag , denoted by , is defined as the correlation between and :

Thus, the autocorrelation function (ACF) is the autocovariance function standarized by the variance . The properties of the ACF are:

 (4.8) (4.9) (4.10)

Given the symmetry property (4.10), the ACF is usually represented by means of a bar graph at the nonnegative lags that is called the simple correlogram.

Another useful tool to describe the dynamics of a stationary process is the partial autocorrelation function (PACF). The partial autocorrelation coefficient at lag measures the linear association between and adjusted for the effects of the intermediate values . Therefore, it is just the coefficient in the linear regression model:

 (4.11)

The properties of the PACF are equivalent to those of the ACF (4.8)-(4.10) and it is easy to prove that (Box and Jenkins; 1976). Like the ACF, the partial autocorrelation function does not depend on the units of measure and it is represented by means of a bar graph at the nonnegative lags that is called partial correlogram.

The dynamic properties of each stationary model determine a particular shape of the correlograms. Moreover, it can be shown that, for any stationary process, both functions, ACF and PACF, approach to zero as the lag tends to infinity. The models are not always stationary processes, so it is necessary first to determine the conditions for stationarity. There are subclasses of models which have special properties so we shall study them separately. Thus, when and , it is a white noise process, when , it is a pure moving average process of order , , and when it is a pure autoregressive process of order , .

## 4.2.1 White Noise Process

The simplest model is a white noise process, where is a sequence of uncorrelated zero mean variables with constant variance . It is denoted by . This process is stationary if its variance is finite, , since given that:

verifies conditions (4.1)-(4.3). Moreover, is uncorrelated over time, so its autocovariance function is:

And its ACF and PACF are as follows:

To understand the behavior of a white noise, we will generate a time series of size 150 from a gaussian white noise process . Figure 4.3 shows the simulated series that moves around a constant level randomly, without any kind of pattern, as corresponds to the uncorrelation over time. The economic time series will follow white noise patterns very rarely, but this process is the key for the formulation of more complex models. In fact, it is the starting point of the derivation of the properties of processes given that we are assuming that the innovation of the model is a white noise.

## 4.2.2 Moving Average Model

The general (finite-order) moving average model of order , is:

 (4.12)

It can be easily shown that processes are always stationary, given that the parameters of any finite processes always verify condition (4.6). Moreover, we are interested in invertible processes. When a process is invertible, it is possible to invert the process, that is, to express the current value of the variable in terms of a current shock and its observable past values . Then, we say that the model has an autoregressive representation. This requirement provides a sensible way of associating present events with past happenings. A model is invertible if the roots of the characteristic equation lie outside the unit circle. When the root is real, this condition means that the absolute value must be greater than unity, . If there are a pair of complex roots, they may be written as , where are real numbers and , and then the invertibility condition means that its moduli must be greater than unity, .

Let us consider the moving average process of first order, :

It is invertible when the root of lies outside the unit circle, that is, . This condition implies the invertibility restriction on the parameter, .

Let us study this simple process in detail. Figure 4.4 plots simulated series of length 150 from two processes where the parameters take the values (0, 0.8) in the first model and (4, -0.5) in the second one. It can be noted that the series show the general patterns associated with stationary and mean reversion processes. More specifically, given that only a past innovation affects the current value of the series (positively for and negatively for ), the process is known as a very short memory process and so, there is not a 'strong' dynamic pattern in the series. Nevertheless, it can be observed that the time evolution is smoother for the positive value of .

The ACF for models is derived from the following moments:

given that, for all and for all , the innovations are uncorrelated with . Then, the autocorrelation function is:

That is, there is a cutoff in the ACF at the first lag. Finally, the partial autocorrelation function shows an exponential decay. Figure 4.5 shows typical profiles of this ACF jointly with the PACF.

It can be shown that the general stationary and invertible process has the following properties (Box and Jenkins; 1976):

• The mean is equal to and the variance is given by .

• The ACF shows a cutoff at the lag, that is, .

• The PACF decays to zero, exponentially when has real roots or with sine-cosine wave fluctuations when the roots are complex.

Figure 4.6 shows the simple and partial correlograms for two different processes. Both ACF exhibit a cutoff at lag two. The roots of the polynomial of the first series are real, so the PACF decays exponentially while for the second series with complex roots the PACF decays as a damping sine-cosine wave.

## 4.2.3 Autoregressive Model

The general (finite-order) autoregressive model of order , , is:

 (4.13)

Let us begin with the simplest process, the autoregressive process of first order, , that is defined as:

 (4.14)

Figure 4.7 shows two simulated time series generated from processes with zero mean and parameters and -0.7, respectively. The autoregressive parameter measures the persistence of past events into the current values. For example, if , a positive (or negative) shock affects positively (or negatively) for a period of time which is longer the larger the value of . When , the series moves more roughly around the mean due to the alternation in the direction of the effect of , that is, a shock that affects positively in moment , has negative effects on , positive in , ...

The process is always invertible and it is stationary when the parameter of the model is constrained to lie in the region . To prove the stationary condition, first we write the in the moving average form by recursive substitution of in (4.14):

 (4.15)

That is, is a weighted sum of past innovations. The weights depend on the value of the parameter : when , (or ), the influence of a given innovation increases (or decreases) through time. Taking expectations to (4.15) in order to compute the mean of the process, we get:

Given that , the result is a sum of infinite terms that converges for all value of only if , in which case . A similar problem appears when we compute the second moment. The proof can be simplified assuming that , that is, . Then, variance is:

Again, the variance goes to infinity except for , in which case . It is easy to verify that both the mean and the variance explode when that condition doesn't hold.

The autocovariance function of a stationary process is

Therefore, the autocorrelation function for the stationary model is:

That is, the correlogram shows an exponential decay with positive values always if is positive and with negative-positive oscillations if is negative (see figure 4.8). Furthermore, the rate of decay decreases as increases, so the greater the value of the stronger the dynamic correlation in the process. Finally, there is a cutoff in the partial autocorrelation function at the first lag.

It can be shown that the general process (Box and Jenkins; 1976):

• Is stationary only if the roots of the characteristic equation of the polynomial lie outside the unit circle. The mean of a stationary model is .

• Is always invertible for any values of the parameters .

• Its ACF goes to zero exponentially when the roots of are real or with sine-cosine wave fluctuations when they are complex.

• Its PACF has a cutoff at the lag, that is, .

Some examples of correlograms for more complex models, such as the , can be seen in figure 4.9. They are very similar to the patterns when the processes have real roots, but take a very different shape when the roots are complex (see the first pair of graphics of figure 4.9).

## 4.2.4 Autoregressive Moving Average Model

The general (finite-order) autoregressive moving average model of orders , , is:

It can be shown that the general process (Box and Jenkins; 1976):

• Is stationary if the component is stationary, that is, the roots of the characteristic equation lie outside the unit circle. The mean of a stationary model is

A necessary condition to hold stationarity is the following that ensures a finite mean process:

 (4.16)

• Is invertible if the component is invertible, that is, the roots of the characteristic equation lie outside the unit circle.

• Its ACF approaches to zero as lag tends to infinity, exponentially when has real roots or with sine-cosine wave fluctuations when these roots are complex.

• Its PACF decays to zero, exponentially when has real roots or with sine-cosine wave fluctuations when these roots are complex.

For example, the process is defined as:

This model is stationary if and is invertible if . The mean of the stationary process can be derived as follows:

by stationarity , and so .

The autovariance function for an stationary process (assuming ) is as follows:

The autocorrelation function for the stationary model is:

Figure 4.10 shows typical profiles of the ACF and PACF for stationary e invertible processes.