Global optimization

Global optimization is a branch of applied mathematics and numerical analysis that attempts to find the global minima or maxima of a function or a set of functions on a given set. It is usually described as a minimization problem because the maximization of the real-valued function is obviously equivalent to the minimization of the function .

Given a possibly nonlinear and non-convex continuous function with the global minima and the set of all global minimizers in , the standard minimization problem can be given as

that is, finding and a global minimizer in ; where is a (not necessarily convex) compact set defined by inequalities .

Global optimization is distinguished from local optimization by its focus on finding the minima or maxima over the given set, as opposed to finding local minima or maxima. Finding an arbitrary local minima is relatively straightforward by using classical local optimization methods. Finding the global minima of a function is far more difficult: analytical methods are frequently not applicable, and the use of numerical solution strategies often leads to very hard challenges.

General theoryEdit

A recent approach to the global optimization problem is via minima distribution [1]. In this work, a relationship between any continuous function   on a compact set   and its global minima   has been strictly established. As a typical case, it follows that




where   is the  -dimensional Lebesgue measure of the set of minimizers  . And if   is not a constant on  , the monotonic relationship


holds for all   and  , which implies a series of monotonous containment relationships, and one of them is, for example,


And we define a minima distribution to be a weak limit   such that the identity


holds for every smooth function   with compact support in  . Here are two immediate properties of  :

(1)   satisfies the identity  .
(2) If   is continuous on  , then  .

As a comparison, the well-known relationship between any differentiable convex function and its minima is strictly established by the gradient. If   is differentiable on a convex set  , then   is convex if and only if


thus,   implies that   holds for all  , i.e.,   is a global minimizer of   on  .


Typical examples of global optimization applications include:

Deterministic methodsEdit

The most successful general exact strategies are:

Inner and outer approximationEdit

In both of these strategies, the set over which a function is to be optimized is approximated by polyhedra. In inner approximation, the polyhedra are contained in the set, while in outer approximation, the polyhedra contain the set.

Cutting-plane methodsEdit

The cutting-plane method is an umbrella term for optimization methods which iteratively refine a feasible set or objective function by means of linear inequalities, termed cuts. Such procedures are popularly used to find integer solutions to mixed integer linear programming (MILP) problems, as well as to solve general, not necessarily differentiable convex optimization problems. The use of cutting planes to solve MILP was introduced by Ralph E. Gomory and Václav Chvátal.

Branch and bound methodsEdit

Branch and bound (BB or B&B) is an algorithm design paradigm for discrete and combinatorial optimization problems. A branch-and-bound algorithm consists of a systematic enumeration of candidate solutions by means of state space search: the set of candidate solutions is thought of as forming a rooted tree with the full set at the root. The algorithm explores branches of this tree, which represent subsets of the solution set. Before enumerating the candidate solutions of a branch, the branch is checked against upper and lower estimated bounds on the optimal solution, and is discarded if it cannot produce a better solution than the best one found so far by the algorithm.

Interval methodsEdit

Interval arithmetic, interval mathematics, interval analysis, or interval computation, is a method developed by mathematicians since the 1950s and 1960s as an approach to putting bounds on rounding errors and measurement errors in mathematical computation and thus developing numerical methods that yield reliable results. Interval arithmetic helps find reliable and guaranteed solutions to equations and optimization problems.

Methods based on real algebraic geometryEdit

Real algebra is the part of algebra which is relevant to real algebraic (and semialgebraic) geometry. It is mostly concerned with the study of ordered fields and ordered rings (in particular real closed fields) and their applications to the study of positive polynomials and sums-of-squares of polynomials. It can be used in convex optimization

Stochastic methodsEdit

Several exact or inexact Monte-Carlo-based algorithms exist:

Direct Monte-Carlo samplingEdit

In this method, random simulations are used to find an approximate solution.

Example: The traveling salesman problem is what is called a conventional optimization problem. That is, all the facts (distances between each destination point) needed to determine the optimal path to follow are known with certainty and the goal is to run through the possible travel choices to come up with the one with the lowest total distance. However, let's assume that instead of wanting to minimize the total distance traveled to visit each desired destination, we wanted to minimize the total time needed to reach each destination. This goes beyond conventional optimization since travel time is inherently uncertain (traffic jams, time of day, etc.). As a result, to determine our optimal path we would want to use simulation - optimization to first understand the range of potential times it could take to go from one point to another (represented by a probability distribution in this case rather than a specific distance) and then optimize our travel decisions to identify the best path to follow taking that uncertainty into account.

Stochastic tunnelingEdit

Stochastic tunneling (STUN) is an approach to global optimization based on the Monte Carlo method-sampling of the function to be objectively minimized in which the function is nonlinearly transformed to allow for easier tunneling among regions containing function minima. Easier tunneling allows for faster exploration of sample space and faster convergence to a good solution.

Parallel temperingEdit

Parallel tempering, also known as replica exchange MCMC sampling, is a simulation method aimed at improving the dynamic properties of Monte Carlo method simulations of physical systems, and of Markov chain Monte Carlo (MCMC) sampling methods more generally. The replica exchange method was originally devised by Swendsen,[2] then extended by Geyer[3] and later developed, among others, by Giorgio Parisi.,[4][5] Sugita and Okamoto formulated a molecular dynamics version of parallel tempering:[6] this is usually known as replica-exchange molecular dynamics or REMD.

Essentially, one runs N copies of the system, randomly initialized, at different temperatures. Then, based on the Metropolis criterion one exchanges configurations at different temperatures. The idea of this method is to make configurations at high temperatures available to the simulations at low temperatures and vice versa. This results in a very robust ensemble which is able to sample both low and high energy configurations. In this way, thermodynamical properties such as the specific heat, which is in general not well computed in the canonical ensemble, can be computed with great precision.

Heuristics and metaheuristicsEdit

Main page: Metaheuristic

Other approaches include heuristic strategies to search the search space in a more or less intelligent way, including:

Response surface methodology-based approachesEdit

See alsoEdit


  1. ^ Xiaopeng Luo (2018). "Minima distribution for global optimization". arXiv:1812.03457. Cite journal requires |journal= (help)
  2. ^ Swendsen RH and Wang JS (1986) Replica Monte Carlo simulation of spin glasses Physical Review Letters 57 : 2607–2609
  3. ^ C. J. Geyer, (1991) in Computing Science and Statistics, Proceedings of the 23rd Symposium on the Interface, American Statistical Association, New York, p. 156.
  4. ^ Marco Falcioni and Michael W. Deem (1999). "A Biased Monte Carlo Scheme for Zeolite Structure Solution". J. Chem. Phys. 110 (3): 1754–1766. arXiv:cond-mat/9809085. Bibcode:1999JChPh.110.1754F. doi:10.1063/1.477812.
  5. ^ David J. Earl and Michael W. Deem (2005) "Parallel tempering: Theory, applications, and new perspectives", Phys. Chem. Chem. Phys., 7, 3910
  6. ^ Y. Sugita and Y. Okamoto (1999). "Replica-exchange molecular dynamics method for protein folding". Chemical Physics Letters. 314 (1–2): 141–151. Bibcode:1999CPL...314..141S. doi:10.1016/S0009-2614(99)01123-9.
  7. ^ Thacker, Neil; Cootes, Tim (1996). "Graduated Non-Convexity and Multi-Resolution Optimization Methods". Vision Through Optimization.
  8. ^ Blake, Andrew; Zisserman, Andrew (1987). Visual Reconstruction. MIT Press. ISBN 0-262-02271-0.[page needed]
  9. ^ Hossein Mobahi, John W. Fisher III. On the Link Between Gaussian Homotopy Continuation and Convex Envelopes, In Lecture Notes in Computer Science (EMMCVPR 2015), Springer, 2015.
  10. ^ Jonas Mockus (2013). Bayesian approach to global optimization: theory and applications. Kluwer Academic.


Deterministic global optimization:

For simulated annealing:

  • Kirkpatrick, S.; Gelatt, C. D.; Vecchi, M. P. (1983-05-13). "Optimization by Simulated Annealing". Science. American Association for the Advancement of Science (AAAS). 220 (4598): 671–680. doi:10.1126/science.220.4598.671. ISSN 0036-8075.

For reactive search optimization:

  • Roberto Battiti, M. Brunato and F. Mascia, Reactive Search and Intelligent Optimization, Operations Research/Computer Science Interfaces Series, Vol. 45, Springer, November 2008. ISBN 978-0-387-09623-0

For stochastic methods:

  • A. Zhigljavsky. Theory of Global Random Search. Mathematics and its applications. Kluwer Academic Publishers. 1991.
  • Hamacher, K (2006). "Adaptation in stochastic tunneling global optimization of complex potential energy landscapes". Europhysics Letters (EPL). IOP Publishing. 74 (6): 944–950. doi:10.1209/epl/i2006-10058-0. ISSN 0295-5075.
  • Hamacher, K.; Wenzel, W. (1999-01-01). "Scaling behavior of stochastic minimization algorithms in a perfect funnel landscape". Physical Review E. American Physical Society (APS). 59 (1): 938–941. arXiv:physics/9810035. doi:10.1103/physreve.59.938. ISSN 1063-651X.
  • Wenzel, W.; Hamacher, K. (1999-04-12). "Stochastic Tunneling Approach for Global Minimization of Complex Potential Energy Landscapes". Physical Review Letters. American Physical Society (APS). 82 (15): 3003–3007. arXiv:physics/9903008. doi:10.1103/physrevlett.82.3003. ISSN 0031-9007.

For parallel tempering:

  • Hansmann, Ulrich H.E. (1997). "Parallel tempering algorithm for conformational studies of biological molecules". Chemical Physics Letters. Elsevier BV. 281 (1–3): 140–150. doi:10.1016/s0009-2614(97)01198-6. ISSN 0009-2614.

For continuation methods:

For general considerations on the dimensionality of the domain of definition of the objective function:

  • Hamacher, Kay (2005). "On stochastic global optimization of one-dimensional functions". Physica A: Statistical Mechanics and its Applications. Elsevier BV. 354: 547–557. doi:10.1016/j.physa.2005.02.028. ISSN 0378-4371.

For strategies allowing one to compare deterministic and stochastic global optimization methods

  • Sergeyev, Ya. D.; Kvasov, D. E.; Mukhametzhanov, M. S. (2018-01-11). "On the efficiency of nature-inspired metaheuristics in expensive global optimization with limited budget". Scientific Reports. Springer Science and Business Media LLC. 8 (1): 453. doi:10.1038/s41598-017-18940-4. ISSN 2045-2322.

External linksEdit