Hamiltonian (control theory)

The Hamiltonian is a function used to solve a problem of optimal control for a dynamical system. It can be understood as an instantaneous increment of the Lagrangian expression of the problem that is to be optimized over a certain time period.[1] Inspired by, but distinct from, the Hamiltonian of classical mechanics, the Hamiltonian of optimal control theory was developed by Lev Pontryagin as part of his maximum principle.[2] Pontryagin proved that a necessary condition for solving the optimal control problem is that the control should be chosen so as to optimize the Hamiltonian.[3]

Problem statement and definition of the HamiltonianEdit

Consider a dynamical system of   first-order differential equations


where   denotes a vector of state variables, and   a vector of control variables. Once initial conditions   and controls   are specified, a solution to the differential equations, called a trajectory  , can be found. The problem of optimal control is to choose   (from some set  ) so that   maximizes or minimizes a certain objective function between an initial time   and a terminal time   (where   may be infinity). Specifically, the goal is to optimize a performance index   at each point in time,


subject to the above equations of motion of the state variables. The solution method involves defining an ancillary function known as the control Hamiltonian


which combines the objective function and the state equations much like a Lagrangian in a static optimization problem, only that the multipliers  , referred to as costate variables, are functions of time rather than constants.

The goal is to find an optimal control policy function   and, with it, an optimal trajectory of the state variable  , which by Pontryagin's maximum principle are the arguments that maximize the Hamiltonian,

  for all  

The first-order necessary conditions for a maximum are given by

  which is the maximum principle,
  which generates the state transition function  ,
  which generates  

the latter of which are referred to as the costate equations. Together, the state and costate equations describe the Hamiltonian dynamical system (again analogous to but distinct from the Hamiltonian system in physics), the solution of which involves a two-point boundary value problem, given that there are   boundary conditions involving two different points in time, the initial time (the   differential equations for the state variables), and the terminal time (the   differential equations for the costate variables; unless a final function is specified, the boundary conditions are  , or   for infinite time horizons).[4]

A sufficient condition for a maximum is the concavity of the Hamiltonian evaluated at the solution, i.e.


where   is the optimal control, and   is resulting optimal trajectory for the state variable.[5] Alternatively, by a result due to Olvi L. Mangasarian, the necessary conditions are sufficient if the functions   and   are both concave in   and  .[6]

Derivation from the LagrangianEdit

A constrained optimization problem as the one stated above usually suggests a Lagrangian expression, specifically


where   compares to the Lagrange multiplier in a static optimization problem but is now, as noted above, a function of time. In order to eliminate  , the last term on the right-hand side can be rewritten using integration by parts, such that


which can be substituted back into the Lagrangian expression to give


To derive the first-order conditions for an optimum, assume that the solution has been found and the Lagrangian is maximized. Then any perturbation to   or   must cause the value of the Lagrangian to decline. Specifically, the total derivative of   obeys


For this expression to equal zero necessitates the following optimality conditions:


If both the initial value   and terminal value   are fixed, i.e.  , no conditions on   and   are needed. If the terminal value is free, as is often the case, the additional condition   is necessary for optimality. The latter is called a transversality condition for a fixed horizon problem.[7]

It can be seen that the necessary conditions are identical to the ones stated above for the Hamiltonian. Thus the Hamiltonian can be understood as a device to generate the first-order necessary conditions.[8]

The Hamiltonian in discrete timeEdit

When the problem is formulated in discrete time, the Hamiltonian is defined as:


and the costate equations are


(Note that the discrete time Hamiltonian at time   involves the costate variable at time  [9] This small detail is essential so that when we differentiate with respect to   we get a term involving   on the right hand side of the costate equations. Using a wrong convention here can lead to incorrect results, i.e. a costate equation which is not a backwards difference equation).

Behavior of the Hamiltonian over timeEdit

From Pontryagin's maximum principle, special conditions for the Hamiltonian can be derived.[10] When the final time   is fixed and the Hamiltonian does not depend explicitly on time  , then:[11]


or if the terminal time is free, then:


Further, if the terminal time tends to infinity, a transversality condition on the Hamiltonian applies.[12]


The Hamiltonian of control compared to the Hamiltonian of mechanicsEdit

William Rowan Hamilton defined the Hamiltonian for describing the mechanics of a system. It is a function of three variables:


where   is the Lagrangian, the extremizing of which determines the dynamics (not the Lagrangian defined above),   is the state variable and   is its time derivative.

  is the so-called "conjugate momentum", defined by


Hamilton then formulated his equations to describe the dynamics of the system as


The Hamiltonian of control theory describes not the dynamics of a system but conditions for extremizing some scalar function thereof (the Lagrangian) with respect to a control variable  . As normally defined, it is a function of 4 variables


where   is the state variable and   is the control variable with respect to that which we are extremizing.

The associated conditions for a maximum are


This definition agrees with that given by the article by Sussmann and Willems.[13] (see p. 39, equation 14). Sussmann and Willems show how the control Hamiltonian can be used in dynamics e.g. for the brachistochrone problem, but do not mention the prior work of Carathéodory on this approach.[14]

Current value and present value HamiltonianEdit

In economics, the objective function in dynamic optimization problems often depends directly on time only through exponential discounting, such that it takes the form


where   is referred to as the instantaneous utility function, or felicity function.[15] This allows a redefinition of the Hamiltonian as   where


which is referred to as the current value Hamiltonian, in contrast to the present value Hamiltonian   defined in the first section. Most notably the costate variables are redefined as  , which leads to modified first-order conditions.


which follows immediately from the product rule. Economically,   represent current-valued shadow prices for the capital goods  .

Example: Ramsey–Cass–Koopmans modelEdit

In economics, the Ramsey–Cass–Koopmans model is used to determine an optimal savings behavior for an economy. The objective function   is the social welfare function,


to be maximized by choice of an optimal consumption path  . The function   indicates the utility the representative agent of consuming   at any given point in time. The factor   represents discounting. The maximization problem is subject to the following differential equation for capital intensity, describing the time evolution of capital per effective worker:


where   is period t consumption,   is period t capital per worker (with  ),   is period t production,   is the population growth rate,   is the capital depreciation rate, the agent discounts future utility at rate  , with   and  .

Here,   is the state variable which evolves according to the above equation, and   is the control variable. The Hamiltonian becomes


The optimality conditions are


in addition to the transversality condition  . If we let  , then log-differentiating the first optimality condition with respect to   yields


Inserting this equation into the second optimality condition yields


which is known as the Keynes–Ramsey rule, which gives a condition for consumption in every period which, if followed, ensures maximum lifetime utility.


  1. ^ Ferguson, Brian S.; Lim, G. C. (1998). Introduction to Dynamic Economic Problems. Manchester: Manchester University Press. pp. 166–167. ISBN 0-7190-4996-2.
  2. ^ Dixit, Avinash K. (1990). Optimization in Economic Theory. New York: Oxford University Press. pp. 145–161. ISBN 978-0-19-877210-1.
  3. ^ Kirk, Donald E. (1970). Optimal Control Theory : An Introduction. Englewood Cliffs: Prentice Hall. p. 232. ISBN 0-13-638098-0.
  4. ^ Gandolfo, Giancarlo (1996). Economic Dynamics (Third ed.). Berlin: Springer. pp. 375–376. ISBN 3-540-60988-1.
  5. ^ Seierstad, Atle; Sydsæter, Knut (1987). Optimal Control Theory with Economic Applications. Amsterdam: North-Holland. pp. 107–110. ISBN 0-444-87923-4.
  6. ^ Mangasarian, O. L. (1966). "Sufficient Conditions for the Optimal Control of Nonlinear Systems". SIAM Journal on Control. 4 (1): 139–152. doi:10.1137/0304013.
  7. ^ Léonard, Daniel; Long, Ngo Van (1992). "Endpoint Constraints and Transversality Conditions". Optimal Control Theory and Static Optimization in Economics. New York: Cambridge University Press. p. 222 [Theorem 7.1.1]. ISBN 0-521-33158-7.
  8. ^ Kamien, Morton I.; Schwartz, Nancy L. (1991). Dynamic Optimization : The Calculus of Variances and Optimal Control in Economics and Management (Second ed.). Amsterdam: North-Holland. pp. 126–127. ISBN 0-444-01609-0.
  9. ^ Jönsson, U. (2005). "A DISCRETE VERSION OF PMP" (PDF). p. 25. Archived from the original (PDF) on January 22, 2023.
  10. ^ Naidu, Desineni S. (2003). Optimal Control Systems. Boca Raton: CRC Press. pp. 259–260. ISBN 0-8493-0892-5.
  11. ^ Torres, Delfim F. M. (2002). "A Remarkable Property of the Dynamic Optimization Extremals". Investigacao Operacional. 22 (2): 253–263. arXiv:math/0212102.
  12. ^ Michel, Philippe (1982). "On the Transversality Condition in Infinite Horizon Optimal Problems". Econometrica. 50 (4): 975–985. doi:10.2307/1912772. JSTOR 1912772. S2CID 16503488.
  13. ^ Sussmann; Willems (June 1997). "300 Years of Optimal Control" (PDF). IEEE Control Systems Magazine. doi:10.1109/37.588098. Archived from the original (PDF) on July 30, 2010.
  14. ^ See Pesch, H. J.; Bulirsch, R. (1994). "The maximum principle, Bellman's equation, and Carathéodory's work". Journal of Optimization Theory and Applications. 80 (2): 199–225. doi:10.1007/BF02192933. S2CID 121749702.
  15. ^ Bævre, Kåre (Spring 2005). "Econ 4350: Growth and Investment: Lecture Note 7" (PDF). Department of Economics, University of Oslo.

Further readingEdit