This article does not cite any sources. (March 2017) (Learn how and when to remove this template message)
In mathematics, a smooth maximum of an indexed family x1, ..., xn of numbers is a differentiable approximation to the maximum function
and the concept of smooth minimum is similarly defined.
For large positive values of the parameter , the following formulation is one smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.
has the following properties:
- is the arithmetic mean of its inputs
The gradient of is closely related to softmax and is given by
This makes the softmax function useful for optimization techniques that use gradient descent.
Another smooth maximum is LogSumExp:
This can also be normalized if the are all non-negative, yielding a function with domain and range :
The term corrects for the fact that by canceling out all but one zero exponential, and if all are zero.
Use in numerical methodsEdit
This section is empty. You can help by adding to it. (February 2015)
Other choices of smoothing functionEdit
This section is empty. You can help by adding to it. (March 2015)
M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," in Proc. ESANN, Apr. 2014, pp. 271-276. (https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)