7.4 Minimization of a Function: One-dimensional Case

Some numerical methods for finding a local minimum of a given one-dimensional function $ f(x)$ will be described in this section.

The same methods can be used for finding a maximum as well because the function $ f(x)$ has its (local, global) maximum at $ x_0$ if and only if $ -f(x)$ has its (local, global) minimum at $ x_0$.


7.4.1 Minimum Bracketing

Before starting any optimum-searching procedure, it is advisable to bracket the minimum (or maximum); otherwise one cannot be sure that there exists any solution to the given optimization problem.

A minimum (or maximum) is bracketed by three points $ (a,b,c)$, if $ a<b<c$ and $ f(b)$ is less (or greater in case of maximum) than both $ f(a)$ and $ f(c)$. If this condition holds and the function $ f(x)$ is continuous in the interval $ (a,c)$, then $ f(x)$ has a minimum for some $ x$, $ a<x<c$. A very simple iterative minimum bracketing procedure follows:

Start from any initial point. Moving a step in the downhill direction, one gets a new point in each iteration. From the third iteration on, store the last three points. New one in each iteration substitutes the oldest one in the triplet. Take larger and larger steps (the new step being computed by simple multiplying the previous one by a constant factor or obtained as a result of a parabolic interpolation applied on a previous triplet of points, see Section 7.4.3) until the downhill trend stops and one gets the last point, where the function value is greater than in the previous one.

An analogous procedure can be used for maximum bracketing; alternatively, one can use the minimum bracketing procedure for $ -f(x)$.


7.4.2 Example

The following quantlet can be used for bracketing of a minimum of a scalar function:


{a,b,c,fa,fb,fc} = 28572 nmbracket (fname {,a0,b0})

fname has to be a string with a name of the function whose minimum should be bracketed. If one has some idea about the location of the minimum, the input parameters a0 and b0 can be set to start the bracketing from a0 in the direction to b0 or vice versa, depending on the downhill direction. Otherwise one can use the simplest form

{a,b,c,fa,fb,fc} = nmbracket(fname)
starting the bracketing with the defaults values $ \texttt{a0} = 0$ and $ \texttt{b0} = 1$.

An example 28575 XEGnum05.xpl gives the following brackets for function $ f(x) = x^2 \cdot \cos \left( \frac{x + 3\pi}{4} \right)$:

Contents of bracket.a
[1,]    2.618
Contents of bracket.b
[1,]   5.2361
Contents of bracket.c
[1,]   9.4721

Contents of bracket.fa
[1,]  -6.7955
Contents of bracket.fb
[1,]  -23.743
Contents of bracket.fc
[1,]   1.0622
28579 XEGnum05.xpl

Fig. 7.3 shows a graph of $ f(x)$ and the bracketing points shown as stars.

Figure: Graph of $ f(x) = x^2 \cdot \cos \left( \frac{x + 3\pi}{4} \right)$ with stars representing the bracketing points, 28585 XEGnum05.xpl
\includegraphics[width=1.0\defepswidth]{XEGnum05.ps}


7.4.3 Parabolic Interpolation

The bracketing procedure described in Section 7.4.1 involves a parabolic interpolation. Given a triplet of points and the respective functional values, parabolic interpolation fits a parabola through these points and finds its extremum (i.e., its minimum or maximum). Using parabolic interpolation can speed up the optimizing procedure if the function is approximately parabolic near the minimum or maximum.

Let us denote the three points $ a,b,c$ and the respective function values $ f(a),f(b),f(c)$. We are looking for point $ x$ minimizing (or maximizing) $ f(x)$. Providing the parabola can be described as

$\displaystyle f(x) = \alpha_2 x^2 + \alpha_1 x + \alpha_0,
$

we can find the values of $ \alpha_i$, $ i=0,1,2$ by solving the following system:
$\displaystyle f(a)$ $\displaystyle =$ $\displaystyle \alpha_2 a^2 + \alpha_1 a + \alpha_0,$  
$\displaystyle f(b)$ $\displaystyle =$ $\displaystyle \alpha_2 b^2 + \alpha_1 b + \alpha_0,$  
$\displaystyle f(c)$ $\displaystyle =$ $\displaystyle \alpha_2 c^2 + \alpha_1 c + \alpha_0.$  

Now we want to find the point $ x$, at which the parabola has its extremum. Hence, the condition $ f'(x) = 0$ should be valid, i.e., $ 2\alpha_2 x + \alpha_1 = 0$. From here we have

$\displaystyle x=\frac{-\alpha_1}{2\alpha_2}
$

and after substituting for alphas

$\displaystyle x = \frac{1}{2}~\frac{(b^2-a^2)\{f(b)-f(c)\} - (b^2-c^2)\{f(b)-f(a)\}}{(b-a)\{f(b)-f(c)\} - (b-c)\{f(b)-f(a)\}}
$

or, alternatively

$\displaystyle x = b-\frac{1}{2}~\frac{(b-a)^2 \{f(b)-f(c)\} - (b-c)^2 \{f(b)-f(a)\}}{(b-a)\{f(b)-f(c)\} - (b-c)\{f(b)-f(a)\}}.
$

REMARK 7.5    
Minimum or maximum? As shown by the formulas above, the parabolic interpolation finds the only extremum of the parabola, not distinguishing between a minimum and a maximum. If one needs to know whether a minimum or a maximum was found, the signum of $ \alpha_2$ can be used to determine: if it is positive, the extremum is a minimum; if it is negative, one has a maximum.

Can parabolic interpolation fail? The parabolic interpolation fails if the denumerator of the formula for $ x$ is equal to zero. This happens if and only if the three given points $ [a,f(a)],[b,f(b)],[c,f(c)]$ are linear dependent, i.e., they are on the same line. In this case, the information for parabolic interpolation is insufficient.


7.4.4 Example

One can use the following quantlet to find the extremum of a parabola determined by three points.


x = 28718 nmparabint (a,b,c,fa,fb,fc)

The input parameters include the abscissas a,b and c as well as the respective function values fa, fb and fc. Using 28721 nmparabint , one can execute the inverse parabolic interpolation for more parabolas at the same time. In this case all the input parameters will be vectors whose $ i$-th components refer to the $ i$-th parabola. 28724 nmparabint returns INF if the three points are linear dependent (i.e., on the same line).

An example 28727 XEGnum06.xpl uses the parabolic interpolation through the points $ (-1,4)$, $ (0,2)$ and $ (3,8)$ to find the parabola extremum at $ x = 0.5$:

Contents of res
[1,]      0.5
28731 XEGnum06.xpl

Fig. 7.4 shows the parabola $ f(x) = x^2 - x + 2$ from which the input points (represented by stars) were taken. The minimum found by 28736 nmparabint.xpl is represented by a triangle.

Figure: Graph of $ f(x) = x^2 - x + 2$ with an interpolated minimum shown as a triangle, 28740 XEGnum06.xpl
\includegraphics[width=1.0\defepswidth]{XEGnum06.ps}


7.4.5 Golden Section Search

The principle of the golden section search method is very simple and can be used even for functions without continuous second (or even first) derivative-in such case the parabolic interpolation used in Brent's method (see Sections 7.4.7 and 7.4.9) cannot provide any additional information.

For golden section search, one has to bracket the minimum by some triplet $ a<b<c$ first (see Section 7.4.1 for details on minimum bracketing). Now, let us choose a new point $ x$ in the bracketing interval, for example between $ a$ and $ b$ ($ a<x<b<c$), and evaluate $ f(x)$. The new point $ x$ substitutes one of the original points in bracketing triplet: if $ f(x) > f(b)$, then $ (x,b,c)$ is the new triplet (with $ f(b)$ less then both $ f(x)$ and $ f(c)$); if $ f(x) < f(b)$, then $ (a,x,b)$ is the new triplet (with $ f(x)<f(b)<f(a)$). The minimum function value we have found until now is at the middle point of the new triplet in both cases. Continue this procedure until the size of the bracketing interval (i.e., the distance of its outer points) is small enough.

It can be shown divides the interval $ (a,c)$ in the ratio $ g:(1-g)$ or, vice versa, $ (1-g):g$, with $ g=\frac{3-\sqrt{5}}{2} \approx 0.38$. This $ g$ is the so called golden section.


7.4.6 Example

The following quantlet implements the golden section search method described above:


min = 28879 nmgolden (fname{,a,b,c,xtol})

If used in the very basic form

min = nmgolden(fname),
fname being a string with a name of function to be minimized, 28882 nmgolden calls the quantlet 28885 nmbracket to bracket a minimum of fname. Use
min = nmgolden(fname,a,b,c)
if a bracketing triplet $ a,b,c$ is known, i.e., the values $ a<b<c$ whose function values satisfy the conditions $ f(b)<f(a)$ and $ f(b)<f(c)$. The optional parameter xtol sets the fractional presicion of the minimum; the iterative process stops after achieving this value.

An example 28888 XEGnum07.xpl uses a golden section search with a function $ \sin(x)$ given a bracketing triplet $ \{\pi,1.3\pi,2\pi\}$ and finds its minimum $ \sin(3/2~\pi) = -1$:

Contents of res.xmin
[1,]   4.7124

Contents of res.fmin
[1,]       -1
28892 XEGnum07.xpl

Fig. 7.5 depicts a graph of $ \sin(x)$ with bracketing triplet represented by stars and the minimum shown as a triangle.

Figure: Graph of $ f(x) = \sin(x)$ with a minimum shown as a triangle, 28898 XEGnum07.xpl
\includegraphics[width=1.0\defepswidth]{XEGnum07.ps}


7.4.7 Brent's Method

The golden section search method described in Section 7.4.5 can be easily used with any continuous function; however, its convergence is only linear. If the function has at least a continuous second derivative, Brent's method finds the minimum quicker (its convergence is superlinear).

Before starting the procedure, one should bracket the minimum (see section 7.4.1 for more details). In each iteration, Brent's method reduces the bracketing interval and updates six stored points: outer points of bracketing interval $ a, b$, three points ( $ x_{{min}_i}$, $ i=1,2,3$) with the three least values found so far and the most recently evaluated point $ x_{new}$. The method combines parabolic interpolation through $ x_{{min}_i}$ (as long as the process is convergent and does not leave the bracketing interval) with golden section search.


7.4.8 Example

The following quantlet implements Brent's method for minimization of a scalar function:


min = 28998 nmbrent (fname{,a,b,c,xtol})

It is used the same way as 29001 nmgolden described in Section 7.4.6.

An example 29004 XEGnum07b.xpl uses the Brent's method for a function $ \sin(x)$ given a bracketing triplet $ \{0,1.1\pi,2\pi\}$ and finds its minimum $ \sin(3/2~\pi) = -1$:

Contents of res.xmin
[1,]   4.7124

Contents of res.fmin
[1,]       -1
29008 XEGnum07b.xpl

Fig. 7.6 depicts a graph of $ \sin(x)$ with bracketing triplet represented by stars and the minimum shown as a triangle.

Figure: Graph of $ f(x) = \sin(x)$ with a minimum shown as a triangle, 29014 XEGnum07b.xpl
\includegraphics[width=1.0\defepswidth]{XEGnum07b.ps}


7.4.9 Brent's Method Using First Derivative of a Function

This method is based on Brent's method from the Section 7.4.7. If it is possible to calculate the first derivative of the function $ f$ to be minimized, one can utilize this additional information for minimization. However, it is not very practical to simply search for roots of $ f'(x)$: first, one cannot easily distinguish maxima and minima; second, the downhill search in the root-finding method can lead out of the bracketing interval! Hence, the derivative only helps us to choose a new point within the bracket in each iteration: the sign of the derivative in the middle point $ b$ of the bracketing triplet indicates whether the new trial point $ x_{new}$ should be taken from the interval $ (a,b)$ or $ (b,c).$ The point $ x_{new}$ is computed by extrapolating $ f'(b)$ and $ f'(x_{{min}_2})$ to zero (using secant method, i.e., linear interpolation) or by bisection, if extrapolation takes us out of the bracketing interval.


7.4.10 Example

The quantlet


min = 29121 nmbrentder (fname{,fder,a,b,c,xtol})

finds a minimum of a given scalar function using Brent's method with first derivative. Its usage is very similar to 29124 nmgolden ; see Section 7.4.6 for description of the input parameters a,b,c and xtol. In the simplest form

min = nmbrentder(fname)
without the derivative of fname the quantlet calls 29127 nmgraddiff to compute the derivative numerically. One can influence the precision of the numerical approximation of the derivative by setting the step $ h$ for quantlet 29130 nmgraddiff , see Section 7.6.1:
min = nmbrentder(fname,h)

If a function computing the first derivative of fname is available, one can input its name as a string fder:

min = nmbrentder(fname,fder),
as in example 29133 XEGnum07c.xpl searching for a local minimum of $ f(x) = \sin(x)$. As a result, one gets Fig. 7.7 and the following approximation of the minimum at $ x=-\pi/2$:
Contents of res.xmin
[1,]  -1.5708

Contents of res.fmin
[1,]       -1
29137 XEGnum07c.xpl

Figure: Graph of $ f(x) = \sin(x)$ with a minimum shown as a triangle, 29143 XEGnum07c.xpl
\includegraphics[width=1.0\defepswidth]{XEGnum07c.ps}