Several numerical methods for the multidimensional minimization will be described in this section. Many of such algorithms except the Nelder & Mead's method (see Section 7.5.1) use a one-dimensional minimization method in their individual iterations.
The simplex method attempts to enclose the minimum inside a simplex (i.e., an -dimensional convex volume
defined by
linearly independent points--vertices). Starting from an initial simplex, the algorithm inspects
the function values in vertices and constructs a new simplex using operations of reflection, expansion or contraction,
so that the final simplex is small enough to contain the minimum with the desired accuracy.
The quantlet
|
finds a minimum of a given function using Nelder and Mead's simplex method. The method can be started from more initial points at the same time; input all the initial points as columns of input matrix x0. The string parameter f contains a name of the minimized function. It is necessary to specify a maximal number of iterations maxiter. The optional parameter eps sets the termination criterium of the iterative process (it is compared with the variance of function values at vortices, hence a smaller value should be set to get the same precision as by e.g., conjugate gradients described in Section 7.5.3). The parameter step sets the length of an initial simplex. The output parameter x has three components: columns of x.minimum contain the minima of f found in the search started from the respective initial points, x.iter is the number of executed iterations and x.converged is equal to 1 if the process converged for all initial points, otherwise it is 0.
Example
XEGnum22.xpl
implements minimization of
using the downhill simplex method.
Starting from an initial estimate
, amoeba needs 410 iterations to find the following approximation
of the minimum in
:
Contents of minim.minimum [1,] 8.6837e-19 [2,] 8.9511e-19 [3,] 1.6666e-18 [4,] 2.0878e-18 Contents of minim.iter [1,] 410 Contents of minim.converged [1,] 1
Start at an initial point . In each iteration, compute
as a point minimizing the function
along a new direction, derived in some way from the local gradient. The way of choosing a new direction distinguishes
the various conjugate gradients methods.
For example, very simple method of steepest descent searches along the line from
in the direction of the local (downhill) gradient
, i.e., computes the new iteration as
, where
minimizes the restricted function
. However, this method is not very efficient (each step has to
go in perpendicular direction to the previous one, which is usually not a direction leading to the minimum).
Hence, we would like to choose a new direction based on the negative local gradient direction but at the same time
conjugated, i.e.,
-orthogonal to the previous direction(s).
In the original Fletcher-Reeves version of conjugate gradient algorithm, the new direction in all steps
is taken as a linear combination of the current gradient and the previous direction,
, with the factor
calculated from the ratio of the magnitudes
of the current and previous gradients:
Polak and Ribiere proposed to use the factor in the form
Fig. 7.8 shows a principle of conjugate gradient method. It was produced by example
XEGnum08.xpl
that
minimizes the function
starting from point
. The exact solution
is approximated
in 10 iterations as follows:
Contents of minim.xmin [1,] 7.0446e-12 [2,] 1 Contents of minim.fmin [1,] 4.9822e-23 Contents of minim.iter [1,] 10
The example
XEGnum08.xpl
uses the quantlet
|
implementing the Polak and Ribiere version of conjugate gradient method. One can call this function with only two parameters--string fname containing the name of a function to be minimized and a vector x0 with the initial estimate of the minimum location:
min = nmcongrad(fname,x0)In this case, the gradient of fname will be computed numerically using the quantlet
min = nmcongrad(fname,x0,h).If a function computing the derivatives (gradient) of fname is available, one can input its name as a string fder and call the quantlet
min = nmcongrad(fname,x0,fder).
Another example illustrating the usage of
nmcongrad
is implemented in
XEGnum09.xpl
.
The function to be minimized is defined as
. Starting from an initial estimate
,
nmcongrad
needs four iterations
to find a following approximation of the minimum in
:
Contents of minim.xmin [1,] -3.1788e-18 [2,] -4.426e-18 [3,] -4.1159e-18 [4,] 7.2989e-19 Contents of minim.fmin [1,] 4.7167e-35 Contents of minim.iter [1,] 4
The conjugate gradient method involves a line minimization (see Section 7.5.7 for more details);
the quantlet
nmlinminder
is used by default.
One can specify the name of another line minimization function in the parameter linmin. Line minimum
should be computed as precisely as possible, otherwise the convergence of the conjugate gradient method is slower;
hence, the quantlet
nmlinminappr
is not suitable for line minimization in context of
nmcongrad
.
The termination of the iterative process can be influenced by the parameters ftol and maxiter, setting the tolerance limit for the function values and maximal number of iterations, respectively.
Together with conjugate gradient methods, the family of quasi-Newton methods (called also variable metric methods)
belongs to the class of conjugate directions methods. The search direction in -th step is computed according to
the rule
Fig. 7.9 shows a principle of BFGS method. It was produced by example
XEGnum10.xpl
that
minimizes function
starting from point
. The exact solution
is approximated
in 25 iterations as follows:
Contents of minim.xmin [1,] 2.1573e-21 [2,] 0.99967 Contents of minim.fmin [1,] 3.6672e-14 Contents of minim.iter [1,] 25
The following quantlet
|
is used to find a minimum of a given function using Broyden-Fletcher-Goldfarb-Shanno method.
Similarly to
nmcongrad
, this quantlet can be called with only two parameters--string fname
containing the name of a function to be minimized and a vector x0 with the initial
estimate of the minimum location:
min = nmBFGS(fname,x0)The gradient of fname will be computed numerically using the quantlet
min = nmBFGS(fname,x0,h).If a function computing the derivatives (gradient) of fname is available, one can call the quantlet
min = nmcongrad(fname,x0,fder).
An example
XEGnum11.xpl
calls the quantlet
nmBFGS
to minimize
(see Section 7.5.3 for minimization of the same function by conjugate gradient method).
Starting from an initial estimate
,
nmBFGS
finds the following approximation
of the minimum
in two iterations:
Contents of minim.xmin [1,] 1.0756e-08 [2,] 1.4977e-08 [3,] 1.3926e-08 [4,] -2.47e-09 Contents of minim.fmin [1,] 5.4004e-16 Contents of minim.iter [1,] 2
The BFGS method also involves a line minimization; the quantlet
nmlinminappr
is used by default, because it gives
a result quicker then
nmlinmin
or
nmlinminder
and its precision is sufficient in context of
nmBFGS
. The name of another line minimization function can be specified in the parameter linmin.
The termination of the iterative process can be influenced by the parameters ftol, gtol and maxiter, setting the tolerance limit for the convergence of function values, function gradients and maximal number of iterations, respectively.
In other words, minimizing on the line
is equivalent to the one-dimensional minimization of a newly defined
(one-dimensional, of course) function
using any one-dimensional minimization method.
![]() |
Contents of grad [1,] 2 [2,] 3 Contents of fdert [1,] -1
|
serves to find a minimum of a given function along the direction direc
from the point x0 not using derivatives of fname.
The input string fname contains a name of a function to be minimized; both x0 and direc
are vectors. The second parameter fder is a dummy necessary for the compatibility with
nmlinminder
;
it can be set to any value.
Example
XEGnum12.xpl
illustrates a line minimization using the quantlet
nmlinmin
. A function
is minimized on a line
and a graph of the function is produced
(see Fig. 7.10).
Contents of minim.xlmin [1,] -0.33928 [2,] 1.5536 Contents of minim.flmin [1,] 0.39683 Contents of minim.moved [1,] -1.3393 [2,] -0.44643
The output parameter lmin consists of the abscissa lmin.xlmin of the minimum of fname
on a line
, lmin.flmin is a minimum function value
and
lmin.moved gives the vector displacement during the minimization:
.
The following quantlet is similar to
nmlinmin
but the implemented method involves the evaluation
of the derivatives of fname:
|
Hence, the string fder should contain a name of function computing
the derivatives (gradient) of fname; if left empty, the quantlet
nmgraddiff
will be used for
computing the gradient.
Example
XEGnum13.xpl
is equivalent to the example
XEGnum12.xpl
above
except of using the quantlet
nmlinminder
for line minimization. It produces the same output and Fig. 7.10.
The quantlet
nmlinminappr
finds an approximated minimum of a function on a given line:
|
In contrast to the quantlets
nmlinmin
and
nmlinminder
described above,
it finds a minimum with lower precision; on the other hand, the computation is quicker.
Hence, it is used as a default line minimization routine for
nmBFGS
but it is not convenient for
nmcongrad
.
![]() |
Example
XEGnum14.xpl
is another modification of the example
XEGnum12.xpl
, using the line-minimizing quantlet
nmlinminappr
. Fig. 7.11 and the following output show that the approximation of the line minimum found by
nmlinminappr
is less precise then the approximation found by
nmlinmin
or
nmlinminder
:
Contents of minim.xlmin [1,] -0.5 [2,] 1.5 Contents of minim.flmin [1,] 0.4375 Contents of minim.moved [1,] -1.5 [2,] -0.5 Contents of minim.check [1,] 0
Please note that
nmlinminappr
searches for a line minimum only in the positive
direction of a given direction vector direc. An additional input parameter stepmax can be used to
prevent evaluation of the function fname outside its domain. Except of output parameters lmin.xlmin,
lmin.flmin and lmin.moved described above,
nmlinminappr
returns also lmin.check.
lmin.check is equal to zero in case of numerical convergence of line minimization and equals one for
lmin.xlmin too close to x0;
means usually convergence when used in
a minimization algorithm, but a calling method should check the convergence in case of a root-finding problem.