2.3 Quadratic Forms
A quadratic form
is built from a symmetric matrix
and a vector
:
![\begin{displaymath}
Q(x) = x^{\top}\ {\data{A}}\ x = \sum ^p_{i=1}\sum ^p_{j=1} a_{ij}x_ix_j.
\end{displaymath}](mvahtmlimg428.gif) |
(2.21) |
Definiteness of Quadratic Forms and Matrices
A matrix
is called positive definite (semidefinite)
if the corresponding quadratic form
is positive definite
(semidefinite). We write
.
Quadratic forms can always be diagonalized, as the following result shows.
THEOREM 2.3
If
![${\data{A}}
$](mvahtmlimg241.gif)
is symmetric and
![$Q({\undertilde{x}})={\undertilde{x}}^{\top}
{\data{A}} \undertilde{x}$](mvahtmlimg432.gif)
is the corresponding quadratic form,
then there exists a transformation
![$\undertilde{x}
\mapsto\Gamma^{\top}x=y$](mvahtmlimg433.gif)
such that
where
![$\lambda_i$](mvahtmlimg435.gif)
are the eigenvalues of
![${\data{A}}
$](mvahtmlimg241.gif)
.
PROOF:
.
By Theorem 2.1 and
we have that
Positive definiteness of quadratic forms can be deduced from positive
eigenvalues.
PROOF:
for all
by Theorem 2.3.
COROLLARY 2.1
If
![${\data{A}}>0$](mvahtmlimg440.gif)
, then
![${\data{A}}^{-1}$](mvahtmlimg304.gif)
exists and
![$\vert{\data{A}}\vert>0$](mvahtmlimg445.gif)
.
EXAMPLE 2.6
The quadratic form
![$Q(x)=x^2_1+x^2_2$](mvahtmlimg446.gif)
corresponds to the matrix
![${\data{A}} =\left ({1\atop 0} {0\atop 1}\right )$](mvahtmlimg447.gif)
with eigenvalues
![$\lambda_1 = \lambda_2 = 1$](mvahtmlimg448.gif)
and is thus positive definite.
The quadratic form
![$Q(x) = (x_1-x_2)^2$](mvahtmlimg449.gif)
corresponds to the matrix
![${\data{A}} =\left ({\phantom{-}1\atop {-1}} {{-1}\atop \phantom{-}1}\right )$](mvahtmlimg450.gif)
with eigenvalues
![$\lambda_1 = 2 , \lambda_2 = 0$](mvahtmlimg451.gif)
and is
positive semidefinite.
The quadratic form
![$Q(x) = x^2_1-x^2_2$](mvahtmlimg452.gif)
with eigenvalues
![$\lambda_1 = 1 , \lambda_2 = -1$](mvahtmlimg453.gif)
is indefinite.
In the statistical analysis of multivariate data, we are interested in
maximizing quadratic forms given some constraints.
THEOREM 2.5
If
![${\data{A}}
$](mvahtmlimg241.gif)
and
![${\data{B}}$](mvahtmlimg454.gif)
are symmetric and
![${\data{B}}>0$](mvahtmlimg455.gif)
,
then the maximum of
![$x^{\top} {\data{A}} x$](mvahtmlimg456.gif)
under
the constraints
![$x^{\top}{\data{B}}x = 1$](mvahtmlimg457.gif)
is given by the largest eigenvalue
of
![${\data{B}}^{-1}{\data{A}}$](mvahtmlimg458.gif)
. More generally,
where
![$\lambda_1,\ldots,\lambda_p$](mvahtmlimg328.gif)
denote the eigenvalues of
![${\data{B}}^{-1}{\data{A}}$](mvahtmlimg458.gif)
. The vector which maximizes (minimizes)
![$x^{\top} \data{A}x$](mvahtmlimg460.gif)
under the constraint
![$x^{\top} \data{B}x = 1$](mvahtmlimg461.gif)
is the eigenvector of
![$\data{B}^{-1}\data{A}$](mvahtmlimg462.gif)
which corresponds to the
largest (smallest) eigenvalue of
![${\data{B}}^{-1}{\data{A}}$](mvahtmlimg458.gif)
.
PROOF:
By definition,
.
Set
, then
![\begin{displaymath}
\max_{\{x:x^{\top}{\data{B}}x=1\}}x^{\top}\ {\data{A}}x
= \...
...1\}}y^{\top}{\data{B}}^{-1/2}\ {\data{A}}{\data{B}}^{-1/2}y.
\end{displaymath}](mvahtmlimg465.gif) |
(2.22) |
From Theorem 2.1, let
be the spectral decomposition of
. Set
Thus (2.22) is equivalent to
But
The maximum is thus obtained by
, i.e.,
Since
and
have the same eigenvalues, the
proof is complete.
EXAMPLE 2.7
Consider the following matrices
We calculate
The biggest eigenvalue of the matrix
![${\data B}^{-1}{\data A}$](mvahtmlimg475.gif)
is
![$2+\sqrt{5}$](mvahtmlimg476.gif)
.
This means that the maximum of
![$x^{\top} \data{A}x$](mvahtmlimg460.gif)
under the constraint
![$x^{\top} \data{B}x = 1$](mvahtmlimg461.gif)
is
![$2+\sqrt{5}$](mvahtmlimg476.gif)
.
Notice that the constraint
corresponds, with our choice of
, to the points which lie on the unit circle
.
Summary
![$\ast$](mvahtmlimg108.gif)
- A quadratic form can be described by a symmetric matrix
.
![$\ast$](mvahtmlimg108.gif)
- Quadratic forms can always be diagonalized.
![$\ast$](mvahtmlimg108.gif)
- Positive definiteness of a quadratic form is equivalent to positiveness
of the eigenvalues of the matrix
.
![$\ast$](mvahtmlimg108.gif)
- The maximum and minimum of a quadratic form given some constraints can be
expressed in terms of eigenvalues.