2.3 Quadratic Forms

A quadratic form $Q(x)$ is built from a symmetric matrix ${\data{A}}(p\times p)$ and a vector $x\in \mathbb{R}^p$:

$$Q(x) = x^{\top}\ {\data{A}}\ x = \sum ^p_{i=1}\sum ^p_{j=1} a_{ij}x_ix_j.$$ (2.21)

Definiteness of Quadratic Forms and Matrices

$$\begin{array}{ll} Q(x) > 0 \textrm{ for all } x\not= 0 & \qu... ...sitive semidefinite\index{positive semidefinite}} \end{array}$$

A matrix ${\data{A}}$ is called positive definite (semidefinite) if the corresponding quadratic form $Q(.)$ is positive definite (semidefinite). We write $\data{A}> 0 \ (\ge 0)$.

Quadratic forms can always be diagonalized, as the following result shows.

THEOREM 2.3   If ${\data{A}}$ is symmetric and $Q({\undertilde{x}})={\undertilde{x}}^{\top} {\data{A}} \undertilde{x}$ is the corresponding quadratic form, then there exists a transformation $\undertilde{x} \mapsto\Gamma^{\top}x=y$ such that

$$x^{\top}\ {\data{A}}\ x = \sum^p_{i=1} \lambda _iy^2_i,$$

where $\lambda_i$ are the eigenvalues of ${\data{A}}$.

PROOF:
${\data{A}}=\Gamma\ \Lambda \ \Gamma^{\top}$. By Theorem 2.1 and $y=\Gamma^{\top}\alpha$ we have that $x^{\top}{\data{A}} x= x^{\top} \Gamma \Lambda \Gamma^{\top} x = y^{\top} \Lambda y = \sum^p_{i=1} \lambda _i y^2_i.$ ${\Box}$

Positive definiteness of quadratic forms can be deduced from positive eigenvalues.

THEOREM 2.4   ${\data{A}}>0$ if and only if all $\lambda _i>0$, $i = 1,\ldots ,p$.

PROOF:
$0 < \lambda_1y^2_1 + \cdots + \lambda_py^2_p = x^{\top} {\data{A}} x$ for all $x\neq 0$ by Theorem 2.3. ${\Box}$

COROLLARY 2.1   If ${\data{A}}>0$, then ${\data{A}}^{-1}$ exists and $\vert{\data{A}}\vert>0$.

EXAMPLE 2.6   The quadratic form $Q(x)=x^2_1+x^2_2$ corresponds to the matrix ${\data{A}} =\left ({1\atop 0} {0\atop 1}\right )$ with eigenvalues $\lambda_1 = \lambda_2 = 1$ and is thus positive definite. The quadratic form $Q(x) = (x_1-x_2)^2$ corresponds to the matrix ${\data{A}} =\left ({\phantom{-}1\atop {-1}} {{-1}\atop \phantom{-}1}\right )$ with eigenvalues $\lambda_1 = 2 , \lambda_2 = 0$ and is positive semidefinite. The quadratic form $Q(x) = x^2_1-x^2_2$ with eigenvalues $\lambda_1 = 1 , \lambda_2 = -1$ is indefinite.

In the statistical analysis of multivariate data, we are interested in maximizing quadratic forms given some constraints.

THEOREM 2.5   If ${\data{A}}$ and ${\data{B}}$ are symmetric and ${\data{B}}>0$, then the maximum of $x^{\top} {\data{A}} x$ under the constraints $x^{\top}{\data{B}}x = 1$ is given by the largest eigenvalue of ${\data{B}}^{-1}{\data{A}}$. More generally,

$$\max_{\{x:x^{\top}{\data{B}}x=1\}}x^{\top} {\data{A}}x = \la... ...a_p = \min_{\{x:x^{\top}{\data{B}}x=1\}}x^{\top}\ {\data{A}}x,$$

where $\lambda_1,\ldots,\lambda_p$ denote the eigenvalues of ${\data{B}}^{-1}{\data{A}}$. The vector which maximizes (minimizes) $x^{\top} \data{A}x$ under the constraint $x^{\top} \data{B}x = 1$ is the eigenvector of $\data{B}^{-1}\data{A}$ which corresponds to the largest (smallest) eigenvalue of ${\data{B}}^{-1}{\data{A}}$.

PROOF:
By definition, ${\data{B}}^{1/2} = \Gamma_{{\data{B}}}\ \Lambda_{{\data{B}}}^{1/2}\ \Gamma^{\top}_{{\data{B}}}$. Set $y = {\data{B}}^{1/2}x$, then

$$\max_{\{x:x^{\top}{\data{B}}x=1\}}x^{\top}\ {\data{A}}x = \... ...1\}}y^{\top}{\data{B}}^{-1/2}\ {\data{A}}{\data{B}}^{-1/2}y.$$ (2.22)

From Theorem 2.1, let

$${\data{B}}^{-1/2}\ {\data{A}}\ {\data{B}}^{-1/2}=\Gamma\ \Lambda \ \Gamma^{\top}$$

be the spectral decomposition of ${\data{B}}^{-1/2}\ {\data{A}}\ {\data{B}}^{-1/2}$. Set

$$z = \Gamma^{\top}y \ \Rightarrow \ z^{\top}z=y^{\top}\Gamma\ \Gamma^{\top}\ y=y^{\top}y.$$

Thus (2.22) is equivalent to

$$\max_{\{z:z^{\top}z=1\}}z^{\top}\ \Lambda \ z=\max_{\{z:z^{\top}z=1\}} \sum^p_{i=1} \lambda_iz^2_i.$$

But

$$\max_z\sum \lambda _iz^2_i\le \lambda _1\underbrace{\max_z\sum z^2_i}_{=1}=\lambda_1.$$

The maximum is thus obtained by $z=(1,0,\ldots ,0)^{\top}$, i.e.,

$$y = \gamma_{\col{1}}\Rightarrow x = {\data{B}}^{-1/2}\gamma_{\col{1}}.$$

Since ${\data{B}}^{-1}{\data{A}}$ and ${\data{B}}^{-1/2}\ {\data{A}}\ {\data{B}}^{-1/2}$ have the same eigenvalues, the proof is complete. ${\Box}$

EXAMPLE 2.7   Consider the following matrices

$${\data A}=\left(\begin{array}{cc}1&2\\ 2&3\end{array}\right) ... ...d {\data B}=\left(\begin{array}{cc}1&0\\ 0&1\end{array}\right).$$

We calculate

$${\data B}^{-1}{\data A}=\left(\begin{array}{cc}1&2\\ 2&3\end{array}\right).$$

The biggest eigenvalue of the matrix ${\data B}^{-1}{\data A}$ is $2+\sqrt{5}$. This means that the maximum of $x^{\top} \data{A}x$ under the constraint $x^{\top} \data{B}x = 1$ is $2+\sqrt{5}$.

Notice that the constraint $x^{\top} \data{B}x = 1$ corresponds, with our choice of $\data{B}$, to the points which lie on the unit circle $x_1^2+x_2^2=1$.

Summary

$\ast$

A quadratic form can be described by a symmetric matrix $\data{A}$.

$\ast$

Quadratic forms can always be diagonalized.

$\ast$

Positive definiteness of a quadratic form is equivalent to positiveness of the eigenvalues of the matrix $\data{A}$.

$\ast$

The maximum and minimum of a quadratic form given some constraints can be expressed in terms of eigenvalues.