Open main menu

Smooth maximum

In mathematics, a smooth maximum of an indexed family x1, ..., xn of numbers is a differentiable approximation to the maximum function

and the concept of smooth minimum is similarly defined.

Contents

ExamplesEdit

For large positive values of the parameter  , the following formulation is one smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

 

  has the following properties:

  1.   as  
  2.   is the arithmetic mean of its inputs
  3.   as  

The gradient of   is closely related to softmax and is given by

 

This makes the softmax function useful for optimization techniques that use gradient descent.

LogSumExpEdit

Another smooth maximum is LogSumExp:

 

This can also be normalized if the   are all non-negative, yielding a function with domain   and range  :

 

The   term corrects for the fact that   by canceling out all but one zero exponential, and   if all   are zero.

Use in numerical methodsEdit

Other choices of smoothing functionEdit

See alsoEdit

ReferencesEdit

M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," in Proc. ESANN, Apr. 2014, pp. 271-276. (https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)