9.2 Principal Components in Practice

In practice the PC transformation has to be replaced by the respective estimators: $\mu$ becomes $\overline x$, $\Sigma$ is replaced by $\data{S}$, etc. If $g_1$ denotes the first eigenvector of $\data{S}$, the first principal component is given by $y_1=(\data{X}-\undertilde 1_n \overline x^{\top})g_1$. More generally if $\data{S} = \data{G}\data{L}\data{G}^{\top}$ is the spectral decomposition of $\data{S}$, then the PCs are obtained by

\begin{displaymath} \data{Y} = (\data{X}-\undertilde 1_{n}\overline x^{\top})\data{G}. \end{displaymath} (9.10)

Note that with the centering matrix $\data{H} = \data{I} - (n^{-1}\undertilde 1_{n}\undertilde 1_{n}^{\top})$ and $ \data{H} 1_n \overline x^{\top} = 0$ we can write
$\displaystyle \data{S}_{\data{Y}}$ $\textstyle =$ $\displaystyle n^{-1}\data{Y}^{\top}\data{H}\data{Y} = n^{-1}\data{G}^{\top}(\da... ...x^{\top})^{\top}\data{H} (\data{X}-\undertilde 1_{n}\overline x^{\top})\data{G}$  
  $\textstyle =$ $\displaystyle n^{-1}\data{G}^{\top}\data{X}^{\top}\data{H}\data{X}\data{G} = \data{G}^{\top}\data{S}\data{G}=\data{L}$ (9.11)

where $\data{L} = \mathop{\hbox{diag}}(\ell _1,\ldots ,\ell _p)$ is the matrix of eigenvalues of $\data{S}$. Hence the variance of $y_i$ equals the eigenvalue $\ell _i$!

The PC technique is sensitive to scale changes. If we multiply one variable by a scalar we obtain different eigenvalues and eigenvectors. This is due to the fact that an eigenvalue decomposition is performed on of the covariance matrix and not on the correlation matrix (see Section 9.5). The following warning is therefore important:

1mm
\begin{picture}(2.00,2.00) \par\linethickness{1.0pt}\put(0.00,0.00){\line(1,0){1... ...\line(1,-2){5.00}} \put(5.00,4.00){\makebox(0,0)[cc]{\LARGE\bf !}} \end{picture}
The PC transformation should be applied to data that have approximately the same scale in each variable.

EXAMPLE 9.2   Let us apply this technique to the bank data set. In this example we do not standardize the data. Figure 9.3 shows some PC plots of the bank data set. The genuine and counterfeit bank notes are marked by ``o'' and ``+'' respectively.

Figure 9.3: Principal components of the bank data. 31645 MVApcabank.xpl
\includegraphics[width=1\defpicwidth]{pcabank.ps}

Recall that the mean vector of $\data{X}$ is

\begin{displaymath}\overline x =\left( 214.9, 130.1, 129.9, 9.4, 10.6, 140.5 \right)^{\top}.\end{displaymath}

The vector of eigenvalues of $\data{S}$ is

\begin{displaymath}\ell =\left( 2.985, 0.931, 0.242, 0.194, 0.085, 0.035 \right)^{\top}.\end{displaymath}

The eigenvectors $g_j$ are given by the columns of the matrix

\begin{displaymath} \data{G}=\left( \begin{array}{rrrrrr} -0.044 & 0.011 & 0.32... ...-0.489 & 0.592 &-0.258 & 0.085 &-0.046\\ \end{array} \right).\end{displaymath}

The first column of $\data{G}$ is the first eigenvector and gives the weights used in the linear combination of the original data in the first PC.

EXAMPLE 9.3   To see how sensitive the PCs are to a change in the scale of the variables, assume that $X_1, X_2, X_3$ and $X_6$ are measured in $cm$ and that $X_4$ and $X_5$ remain in $mm$ in the bank data set. This leads to:

\begin{displaymath}\bar{x}=(21.49,\ 13.01,\ 12.99,\ 9.41,\ 10.65,\ 14.05)^{\top}.\end{displaymath}

The covariance matrix can be obtained from $S$ in (3.4) by dividing rows 1, 2, 3, 6 and columns 1, 2, 3, 6 by 10. We obtain:

\begin{displaymath}\ell = (2.101,\ 0.623,\ 0.005,\ 0.002,\ 0.001,\ 0.0004)^{\top}\end{displaymath}

which clearly differs from Example 9.2. Only the first two eigenvectors are given:

\begin{displaymath}g_1=(-0.005,\ 0.011,\ 0.014,\ 0.992,\ 0.113,\ -0.052)^{\top}\end{displaymath}


\begin{displaymath}g_2=(-0.001,\ 0.013,\ 0.016,\ -0.117,\ 0.991,\ -0.069)^{\top}.\end{displaymath}

Comparing these results to the first two columns of $\data{G}$ from Example 9.2, a completely different story is revealed. Here the first component is dominated by $X_4$ (lower margin) and the second by $X_5$ (upper margin), while all of the other variables have much less weight. The results are shown in Figure 9.4. Section 9.5 will show how to select a reasonable standardization of the variables when the scales are too different.

Figure 9.4: Principal components of the rescaled bank data. 31652 MVApcabankr.xpl
\includegraphics[width=1\defpicwidth]{rescale.ps}

Summary
$\ast$
The scale of the variables should be roughly the same for PC transformations.
$\ast$
For the practical implementation of principal components analysis (PCA) we replace $\mu$ by the mean $\overline{x}$ and $\Sigma$ by the empirical covariance $\data{S}$. Then we compute the eigenvalues $\ell_{1},\ldots,\ell_{p}$ and the eigenvectors $g_{1},\ldots,g_{p}$ of $\data{S}$. The graphical representation of the PCs is obtained by plotting the first PC vs. the second (and eventually vs. the third).
$\ast$
The components of the eigenvectors $g_{i}$ are the weights of the original variables in the PCs.