# Rate of convergence

(Redirected from Linear convergence)

In numerical analysis, the speed at which a convergent sequence approaches its limit is called the rate of convergence. Although strictly speaking, a limit does not give information about any finite first part of the sequence, the concept of rate of convergence is of practical importance when working with a sequence of successive approximations for an iterative method, as then typically fewer iterations are needed to yield a useful approximation if the rate of convergence is higher. This may even make the difference between needing ten or a million iterations.

Similar concepts are used for discretization methods. The solution of the discretized problem converges to the solution of the continuous problem as the grid size goes to zero, and the speed of convergence is one of the factors of the efficiency of the method. However, the terminology in this case is different from the terminology for iterative methods.

Series acceleration is a collection of techniques for improving the rate of convergence of a series discretization. Such acceleration is commonly accomplished with sequence transformations.

## Convergence speed for iterative methods

### Basic definition

Suppose that the sequence ${\displaystyle (x_{k})_{k}}$  converges to the number ${\displaystyle L}$ .

This sequence is said to converge linearly to ${\displaystyle L}$ , if there exists a number ${\displaystyle \mu \in (0,1)}$  such that

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|}}=\mu .}$

The number ${\displaystyle \mu }$  is called the rate of convergence.

If the sequence converges, and

• ${\displaystyle \mu _{k}}$  varies from step to step with ${\displaystyle \mu _{k}\to 0}$  for ${\displaystyle k\to \infty }$ , then the sequence is said to converge superlinearly.
• ${\displaystyle \mu _{k}}$  varies from step to step with ${\displaystyle \mu _{k}\to 1}$  for ${\displaystyle k\to \infty }$ , then the sequence is said to converge sublinearly.

If the sequence converges sublinearly and additionally

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+2}-x_{k+1}|}{|x_{k+1}-x_{k}|}}=1,}$

then it is said that the sequence ${\displaystyle (x_{k})_{k}}$  converges logarithmically to ${\displaystyle L}$ .

The next definition is used to distinguish superlinear rates of convergence. The sequence converges with order ${\displaystyle q}$  to ${\displaystyle L}$  for ${\displaystyle q>1}$ [1] if

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|^{q}}}

for some ${\displaystyle M\in (0,1)}$ . In particular, convergence with order

• ${\displaystyle q=2}$  is called quadratic convergence,
• ${\displaystyle q=3}$  is called cubic convergence,
• etc.

This is sometimes called Q-linear convergence, Q-quadratic convergence, etc., to distinguish it from the definition below. The Q stands for "quotient", because the definition uses the quotient between two successive terms. A sequence that has a quadratic convergence implies that it has a superlinear rates of convergence.

A practical method to calculate the order of convergence for a sequence is to calculate the following sequence, which is converging to ${\displaystyle q}$

${\displaystyle q\approx {\frac {\log \left|{\frac {x_{n+1}-x_{n}}{x_{n}-x_{n-1}}}\right|}{\log \left|{\frac {x_{n}-x_{n-1}}{x_{n-1}-x_{n-2}}}\right|}}.}$

### Extended definition

The drawback of the above definitions is that these do not catch some sequences which still converge reasonably fast, but whose rate is variable, such as the sequence ${\displaystyle \left(b_{k}\right)_{k}}$  below. Therefore, the definition of rate of convergence is sometimes extended as follows.

Under the new definition, the sequence ${\displaystyle (x_{k})_{k}}$  converges with at least order ${\displaystyle q}$  if there exists a sequence ${\displaystyle (\varepsilon _{k})_{k}}$  such that

${\displaystyle |x_{k}-L|\leq \varepsilon _{k}\quad {\text{for all }}k\,,}$

and the sequence ${\displaystyle (\varepsilon _{k})_{k}}$  converges to zero with order ${\displaystyle q}$  according to the above "simple" definition. To distinguish it from that definition, this is sometimes called R-linear convergence, R-quadratic convergence, etc. (with the R standing for "root").

### Examples

Consider the following sequences:

{\displaystyle {\begin{aligned}a_{0}&=1,\,&&a_{1}={\frac {1}{2}},\,&&a_{2}={\frac {1}{4}},\,&&a_{3}={\frac {1}{8}},\,&&a_{4}={\frac {1}{16}},\,&&a_{5}={\frac {1}{32}},\,&&\ldots ,\,&&a_{k}={\frac {1}{2^{k}}},\,&&\ldots \\b_{0}&=1,\,&&b_{1}=1,\,&&b_{2}={\frac {1}{4}},\,&&b_{3}={\frac {1}{4}},\,&&b_{4}={\frac {1}{16}},\,&&b_{5}={\frac {1}{16}},\,&&\ldots ,\,&&b_{k}={\frac {1}{4^{\left\lfloor {\frac {k}{2}}\right\rfloor }}},\,&&\ldots \\c_{0}&={\frac {1}{2}},\,&&c_{1}={\frac {1}{4}},\,&&c_{2}={\frac {1}{16}},\,&&c_{3}={\frac {1}{256}},\,&&c_{4}={\frac {1}{65\,536}},\,&&&&\ldots ,\,&&c_{k}={\frac {1}{2^{2^{k}}}},\,&&\ldots \\d_{0}&=1,\,&&d_{1}={\frac {1}{2}},\,&&d_{2}={\frac {1}{3}},\,&&d_{3}={\frac {1}{4}},\,&&d_{4}={\frac {1}{5}},\,&&d_{5}={\frac {1}{6}},\,&&\ldots ,\,&&d_{k}={\frac {1}{k+1}},\,&&\ldots \end{aligned}}}

The sequence {ak} converges linearly to 0 with rate 1/2. More generally, the sequence k converges linearly with rate μ if |μ| < 1. The sequence {bk} also converges linearly to 0 with rate 1/2 under the extended definition, but not under the simple definition. The sequence {ck} converges superlinearly. In fact, it is quadratically convergent. Finally, the sequence {dk} converges sublinearly and logarithmically.

Linear, linear, superlinear (quadratic), and sublinear rates of convergence

## Convergence speed for discretization methods

A similar situation exists for discretization methods. The important parameter here for the convergence speed is not the iteration number k, but the number of grid points and grid spacing. In this case, the number of grid points n in a discretization process is inversely proportional to the grid spacing.

In this case, a sequence ${\displaystyle x_{n}}$  is said to converge to L with order p if there exists a constant C such that

${\displaystyle |x_{n}-L|

This is written as ${\displaystyle |x_{n}-L|={\mathcal {O}}(n^{-p})}$  using big O notation.

This is the relevant definition when discussing methods for numerical quadrature or the solution of ordinary differential equations. A practical method to calculate the rate of convergence for a discretization method is to implement the following formula:

${\displaystyle p\approx {\frac {\log(e_{\text{new}}/e_{\text{old}})}{\log(h_{\text{new}}/h_{\text{old}})}},}$

where ${\displaystyle e_{\text{new}}}$  and ${\displaystyle e_{\text{old}}}$ ) denote the errors w.r.t. the new and old step sizes ${\displaystyle h_{\text{new}}}$  and ${\displaystyle h_{\text{old}}}$  respectively.

### Examples

The sequence {dk} with dk = 1 / (k + 1) was introduced above. This sequence converges with order 1 according to the convention for discretization methods.

The sequence {ak} with ak = 2k, which was also introduced above, converges with order p for every number p. It is said to converge exponentially using the convention for discretization methods. However, it only converges linearly (that is, with order 1) using the convention for iterative methods.

The order of convergence of a discretization method is related to its global truncation error (GTE).

## Acceleration of convergence

Many methods exist to increase the rate of convergence of a given sequence, i.e. to transform a given sequence into one converging faster to the same limit. Such techniques are in general known as "series acceleration". The goal of the transformed sequence is to reduce the computational cost of the calculation. One example of series acceleration is Aitken's delta-squared process.

## References

1. ^ ${\displaystyle q}$  may be non-integer. For example, the secant method has, in the case of convergence to a regular, simple root, convergence order φ ≈ 1.618.

## Literature

The simple definition is used in

• Michelle Schatzman (2002), Numerical analysis: a mathematical introduction, Clarendon Press, Oxford. ISBN 0-19-850279-6.

The extended definition is used in

Logarithmic convergence is used in

The Big O definition is used in

• Richard L. Burden and J. Douglas Faires (2001), Numerical Analysis (7th ed.), Brooks/Cole. ISBN 0-534-38216-9

The terms Q-linear and R-linear are used in; The Big O definition when using Taylor series is used in

One may also study the following paper for Q-linear and R-linear:

• Potra, F. A. (1989). "On Q-order and R-order of convergence". J. Optim. Th. Appl. 63 (3): 415–431. doi:10.1007/BF00939805.