7.3 Solving a System of Nonlinear Equations

Here we deal with numerical methods for approximation of real solutions of a system of nonlinear equations $ f_j(x) = 0$, $ j = 1,\dots,n$, i.e., finding the roots of a vector function $ F = (f_1,\ldots,f_n)$. Compared with the one-dimensional case, root-finding in the multidimensional case is much more complicated. For example, in one-dimensional case one can relatively easily bracket the roots of a given function (i.e., determine the intervals, in which at least one root of the function lies) but there are no methods for bracketing of roots of general functions in the multidimensional case! Usually we even do not know whether a root (solution of the system) exists and whether it is unique.


7.3.1 Newton-Raphson Method for Nonlinear Systems of Equations

The Newton-Raphson method is very popular also in the multidimensional case (here we have far less methods to choose from). As well as in the one-dimensional case, it is very efficient if one has a good initial approximation.

The formula for the multidimensional Newton-Raphson method can be derived similarly as in Section 7.2.2. Start again with the Taylor series of $ f_j$ centered in (vector!) $ x_i$

$\displaystyle f_j(x_i+h) = f_j(x_i) + \sum_{k=1}^n \frac{\partial f_j(x_i)}{\partial x_k} h_k + O(h^2) \textrm{ for }j=1,\dots,n.
$

The same can be written in matrix notation as

$\displaystyle F(x_i+h) = F(x_i) + JF(x_i) h + O(h^2)
$

where $ JF(x_i) = (\partial_k f_j)_{j,k=1,\dots,n}$ is the Jacobian of the vector function $ F = (f_1,\ldots,f_n)$. Neglecting the term $ O(h^2)$ and setting $ f(x_i+h) = 0$ implies

$\displaystyle JF(x_i) h = -F(x_i).
$

This system of linear equations can be solved for the vector $ h$. The new iteration is computed as $ x_{i+1} = x_i + h$.

REMARK 7.3    
Vector $ h$ points in the descent direction of the function $ \vert F\vert^2$ from $ x_i$ as shown in (Press, Teukolsky, Vetterling, and Flannery; 1992).

Computing the derivative $ f'(x)$: As well as $ f'(x)$ in one-dimensional case, $ JF(x_i)$ can be also approximated by differences (see 7.6.3).

Termination: For usual termination criteria, see Section 7.2.1.


7.3.2 Example

We can solve a nonlinear system of equations by Newton-Raphson method using the following quantlet:


z =
28273 nmnewton (fname, fder, x0{, epsf, epsx,
maxiter, checkcond})

Its simplest usage is in form

z = nmnewton(fname,"",x0),
as shown in

example 28276 XEGnum02.xpl , searching for a solution of the following system of two nonlinear equations:

$\displaystyle x^2 - 2 y - 2$ $\displaystyle =$ 0  
$\displaystyle x - y^2 - 1$ $\displaystyle =$ 0  

Starting from initial estimate $ x=0$, $ y = 0$, we get an approximation of the solution $ x=0$, $ y = -1$:
Contents of sol.x
[1,] -1.7807e-15
[2,]       -1

Contents of sol.fval
[1,]  1.5987e-14
[2,]  2.2204e-14

Contents of sol.fderval
[1,]        0       -2
[2,]        1       -3
28280 XEGnum02.xpl

If functions computing the derivatives of fname are not available and the precision of the numerical approximation of the derivatives should be influenced, one can input a step h for 28285 nmjacobian (see Section 7.6.3):

z = nmnewton(fname,h,x0).

Having the functions for computation of the derivatives of fname available, one may input their name(s) as a parameter fder:

z = nmnewton(fname,fder,x0);
fder should be either a string with a name of the function computing the Jacobian of fname or a vector of strings with names of functions that compute gradients of the first, second, ..., $ n$-th component of the vector function fname.

This possibility is shown in example 28288 XEGnum03.xpl that solves the following system:

$\displaystyle \sin(x+y)$ $\displaystyle =$ 0  
$\displaystyle \cos(x-y)$ $\displaystyle =$ 0  

The Jacobian is computed using the formula

$\displaystyle J = \left ( \begin{array}{rr}
\cos(x+y) & \cos(x+y)\\
-\sin(x-y) & \sin(x-y)
\end{array} \right )
$

Setting an initial estimate to $ x = 1$, $ y = -1$, we get a solution $ x = 0.7854$, $ y = -0.7854$. This is an approximation of $ (\pi/4,-\pi/4)$:
Contents of _tmp.x
[1,]   0.7854
[2,]  -0.7854

Contents of _tmp.fval
[1,]        0
[2,]  6.1257e-17

Contents of _tmp.fderval
[1,]        1        1
[2,]       -1        1
28292 XEGnum03.xpl

The parameters epsf, epsx and maxiter influence the termination of an iterative process of finding the solution (see Section 7.2.1): the procedure ends if sum of absolute values of fname or corrections to z are less or equal to epsf or epsx, respectively; the process is also terminated if the number of iterations exceeds maxiter.

In each iteration we multiply by $ JF(x_i)^{-1}$, the inverse matrix to the Jacobian of fname; the high condition number of the Jacobian can cause a numerical instability of the iterative process. Set the last parameter checkcond for checking the stability; A warning message will be produced if the condition number exceeds checkcond.

Unfortunately, the Newton-Raphson method can fail if the initial approximation is not close enough to the root. Failing of the method (i.e., convergence is not achieved after any reasonable number of iterations) means either that $ F$ has no roots or that the Newton-Raphson steps $ h$ were too long in some iterations (as mentioned above, each step of the Newton-Raphson method goes in the descent direction of the function $ \vert F\vert^2$, having its minimum $ \vert F\vert^2 = 0$, if a root exists). The latter possible difficulty can be solved using a modification of the Newton-Raphson method described in the Section 7.3.3

REMARK 7.4   The sensitivity of the Newton-Raphson method applied to problems with oscillating functions is increased by using numerically computed derivatives.


7.3.3 Modified Newton-Raphson Method for Systems

The Newton-Raphson method can be modified following way to achieve higher numerical stability: In each iteration, compute the Newton-Raphson step $ h$ and check whether $ \vert F(x_i + h)\vert < \vert F(x_i)\vert$. If this condition is not valid, we have to reduce step size until having an acceptable $ h$. More or less ingenious ways of reducing step size were introduced (see, e.g., (Press, Teukolsky, Vetterling, and Flannery; 1992)); however, reducing it too much can substantially decrease the convergence rate.


7.3.4 Example

The following quantlet solves a nonlinear system of equations using the modified Newton-Raphson method:


z =
28417 nmnewtonmod (fname, fder, x0{, epsf, epsx,
maxiter, checkcond})

Its usage is the same as for the quantlet 28420 nmnewton , see Section 7.3.2.

An example 28423 XEGnum04.xpl shows a problem where using the modified Newton-Raphson method instead of the original one is desirable. The following equation is to be solved:

$\displaystyle x \cdot \sin \left( \frac{1}{x^2} \right) = 0.
$

Its left-side function $ f(x) = x \sin(1/x^2)$ is highly oscillating (see Fig. 7.2).

Figure: Graph of $ f(x) = x \sin(1/x^2)$ for $ x\in [0.01, 0.01002]$. The filled circle shows the solution found by the modified Newton-Raphson method, 28427 XEGnum04.xpl
\includegraphics[width=1.0\defepswidth]{XEGnum04.ps}

Setting an initial estimate to $ x = 0.01$, the modified Newton-Raphson method gives a solution $ x = 0.010002$ in 5 iterations (shown as a filled circle at Fig. 7.2). On the other hand, the original Newton-Raphson method needs 88 iterations to find a much more distant root $ x = 0.010702$.

Contents of _tmp
[1,] "Newton-Raphson method:"

Contents of x
[1,]  0.010702

Contents of fval
[1,]  4.7088e-05

Contents of _tmp
[1,] "Modified Newton-Raphson method:"

Contents of x
[1,]  0.010002

Contents of fval
[1,] -1.9808e-05
28431 XEGnum04.xpl