Open main menu

In linear algebra, a symmetric real matrix is said to be positive definite if the scalar is strictly positive for every non-zero column vector of real numbers. Here denotes the transpose of .[1] When interpreting as the output of an operator, , that is acting on an input, , the property of positive definiteness implies that the output always has a positive inner product with the input, as often observed in physical processes.

More generally, a complex Hermitian matrix is said to be positive definite if the scalar is strictly positive for every non-zero column vector of complex numbers. Here denotes the conjugate transpose of . Note that is automatically real since is Hermitian.

Positive semi-definite matrices are defined similarly, except that the above scalars or must be positive or zero (i.e. non-negative). Negative definite and negative semi-definite matrices are defined analogously. A matrix that is not positive semi-definite and not negative semi-definite is called indefinite.

The matrix is positive definite if and only if the bilinear form is positive definite (and similarly for a positive definite sesquilinear form in the complex case). This is a coordinate realization of an inner product on a vector space.[2]

Some authors use more general definitions of definiteness, including some non-symmetric real matrices, or non-Hermitian complex ones.

DefinitionsEdit

In the following definitions,   is the transpose of  ,   is the conjugate transpose of   and   denotes the n-dimensional zero-vector.

Definitions for real matricesEdit

A   symmetric real matrix   is said to be positive definite if   for all non-zero   in  . Formally,

 

A   symmetric real matrix   is said to be positive semidefinite or non-negative definite if   for all   in  . Formally,

 

A   symmetric real matrix   is said to be negative definite if   for all non-zero   in  . Formally,

 

A   symmetric real matrix   is said to be negative semidefinite or non-positive definite if   for all   in  . Formally,

 

A   symmetric real matrix which is neither positive semidefinite nor negative semidefinite is called indefinite.

Definitions for complex matricesEdit

The following definitions all involve the term  . Notice that this is always a real number for any Hermitian square matrix  .

A   Hermitian complex matrix   is said to be positive definite if   for all non-zero   in  . Formally,

 

A   Hermitian complex matrix   is said to be positive semi-definite or non-negative definite if   for all   in  . Formally,

 

A   Hermitian complex matrix   is said to be negative definite if   for all non-zero   in  . Formally,

 

A   Hermitian complex matrix   is said to be negative semi-definite or non-positive definite if   for all   in  . Formally,

 

A   Hermitian complex matrix which is neither positive semidefinite nor negative semidefinite is called indefinite.

Consistency between real and complex definitionsEdit

Since every real matrix is also a complex matrix, the definitions of "definiteness" for the two classes must agree.

For complex matrices, the most common definition says that "  is positive definite if and only if   is real and positive for all non-zero complex column vectors  ". This condition implies that   is Hermitian (i.e. its transpose is equal to its conjugate). To see this, consider the matrices   and  , so that   and  . The matrices   and   are Hermitian, therefore   and   are individually real. If   is real, then   must be zero for all  . Then   is the zero matrix and  , proving that   is Hermitian.

By this definition, a positive definite real matrix   is Hermitian, hence symmetric; and   is positive for all non-zero real column vectors  . However the last condition alone is not sufficient for   to be positive definite. For example, if

 

then for any real vector   with entries   and   we have  , which is always positive if   is not zero. However, if   is the complex vector with entries   and  , one gets

 

which is not real. Therefore,   is not positive definite.

On the other hand, for a symmetric real matrix  , the condition "  for all nonzero real vectors  " does imply that   is positive definite in the complex sense.

NotationEdit

If a Hermitian matrix   is positive semi-definite, one sometimes writes   and if   is positive definite one writes  . To denote that   is negative semi-definite one writes   and to denote that   is negative definite one writes  .

The notion comes from functional analysis where positive semidefinite matrices define positive operators.

A common alternative notation is  ,  ,   and   for positive semi-definite and positive definite, negative semi-definite and negative definite matrices, respectively. This may be confusing, as sometimes nonnegative matrices respectively nonpositive matrices are also denoted in this way.

ExamplesEdit

  • The identity matrix   is positive definite (and as such also positive semi-definite). It is a real symmetric matrix, and, for any non-zero column vector z with real entries a and b, one has
     .

    Seen as a complex matrix, for any non-zero column vector z with complex entries a and b one has

     .
    Either way, the result is positive since   is not the zero vector (that is, at least one of   and   is not zero).
  • The real symmetric matrix
     
    is positive definite since for any non-zero column vector z with entries a, b and c, we have
     
    This result is a sum of squares, and therefore non-negative; and is zero only if  , that is, when z is the zero vector.
  • For any real invertible matrix  , the product   is a positive definite matrix. A simple proof is that for any non-zero vector  , the condition   since the invertibility of matrix   means that  
  • The example   above shows that a matrix in which some elements are negative may still be positive definite. Conversely, a matrix whose entries are all positive is not necessarily positive definite, as for example
     
    for which  

EigenvaluesEdit

Let   be an   Hermitian matrix.

  •   is positive definite if and only if all of its eigenvalues are positive.
  •   is positive semi-definite if and only if all of its eigenvalues are non-negative.
  •   is negative definite if and only if all of its eigenvalues are negative
  •   is negative semi-definite if and only if all of its eigenvalues are non-positive.
  •   is indefinite if and only if it has both positive and negative eigenvalues.

Let   be an eigendecomposition of  , where   is a unitary complex matrix whose rows comprise an orthonormal basis of eigenvectors of  , and   is a real diagonal matrix whose main diagonal contains the corresponding eigenvalues. The matrix   may be regarded as a diagonal matrix   that has been re-expressed in coordinates of the basis  . In particular, the one-to-one change of variable   shows that   is real and positive for any complex vector   if and only if   is real and positive for any  ; in other words, if   is positive definite. For a diagonal matrix, this is true only if each element of the main diagonal—that is, every eigenvalue of  —is positive. Since the spectral theorem guarantees all eigenvalues of a Hermitian matrix to be real, the positivity of eigenvalues can be checked using Descartes' rule of alternating signs when the characteristic polynomial of a real, symmetric matrix   is available.

ConnectionsEdit

A general purely quadratic real function   on   real variables   can always be written as   where   is the column vector with those variables, and   is a symmetric real matrix. Therefore, the matrix being positive definite means that   has a unique minimum (zero) when   is zero, and is strictly positive for any other  .

More generally, a twice-differentiable real function   on   real variables has local minimum at arguments   if its gradient is zero and its Hessian (the matrix of all second derivatives) is positive semi-definite at that point. Similar statements can be made for negative definite and semi-definite matrices.

In statistics, the covariance matrix of a multivariate probability distribution is always positive semi-definite; and it is positive definite unless one variable is an exact linear function of the others. Conversely, every positive semi-definite matrix is the covariance matrix of some multivariate distribution.

CharacterizationsEdit

Let   be an   Hermitian matrix. The following properties are equivalent to   being positive definite:

The associated sesquilinear form is an inner product
The sesquilinear form defined by   is the function   from   to   such that   for all   and   in  , where   is the conjugate transpose of  . For any complex matrix  , this form is linear in   and semilinear in  . Therefore, the form is an inner product on   if and only if   is real and positive for all nonzero  ; that is if and only if   is positive definite. (In fact, every inner product on   arises in this fashion from a Hermitian positive definite matrix.)
It is the Gram matrix of a set of linearly independent vectors
Let   be a list of   linearly independent vectors of some complex vector space with an inner product  . It can be verified that the Gram matrix   of those vectors, defined by  , is always positive definite. Conversely, if   is positive definite, it has an eigendecomposition   where   is unitary,   diagonal, and all diagonal elements   of   are real and positive. Let   be the real diagonal matrix with entries   so  ; then  . Now we let   be the columns of  . These vectors are linearly independent, and by the above   is their Gram matrix, under the standard inner product of  , namely  .
Its leading principal minors are all positive
The kth leading principal minor of a matrix   is the determinant of its upper-left   sub-matrix. It turns out that a matrix is positive definite if and only if all these determinants are positive. This condition is known as Sylvester's criterion, and provides an efficient test of positive definiteness of a symmetric real matrix. Namely, the matrix is reduced to an upper triangular matrix by using elementary row operations, as in the first part of the Gaussian elimination method, taking care to preserve the sign of its determinant during pivoting process. Since the kth leading principal minor of a triangular matrix is the product of its diagonal elements up to row  , Sylvester's criterion is equivalent to checking whether its diagonal elements are all positive. This condition can be checked each time a new row   of the triangular matrix is obtained.

Quadratic forms, convexity, optimizationEdit

The (purely) quadratic form associated with a real   matrix   is the function   such that   for all  .   can be assumed symmetric by replacing it with  .

A symmetric matrix   is positive definite if and only if its quadratic form is a strictly convex function.

More generally, any quadratic function from   to   can be written as   where   is a symmetric   matrix,   is a real  -vector, and   a real constant. This quadratic function is strictly convex, and hence has a unique finite global minimum, if and only if   is positive definite. For this reason, positive definite matrices play an important role in optimization problems.

Simultaneous diagonalizationEdit

A symmetric matrix and another symmetric and positive definite matrix can be simultaneously diagonalized, although not necessarily via a similarity transformation. This result does not extend to the case of three or more matrices. In this section we write for the real case. Extension to the complex case is immediate.

Let   be a symmetric and   a symmetric and positive definite matrix. Write the generalized eigenvalue equation as   where we impose that   be normalized, i.e.  . Now we use Cholesky decomposition to write the inverse of   as  . Multiplying by   and letting  , we get  , which can be rewritten as   where  . Manipulation now yields   where   is a matrix having as columns the generalized eigenvectors and   is a diagonal matrix of the generalized eigenvalues. Now premultiplication with   gives the final result:   and  , but note that this is no longer an orthogonal diagonalization with respect to the inner product where  . In fact, we diagonalized   with respect to the inner product induced by  .

Note that this result does not contradict what is said on simultaneous diagonalization in the article Diagonalizable matrix, which refers to simultaneous diagonalization by a similarity transformation. Our result here is more akin to a simultaneous diagonalization of two quadratic forms, and is useful for optimization of one form under conditions on the other.[3]

PropertiesEdit

Induced partial orderingEdit

For arbitrary square matrices  ,   we write   if   i.e.,   is positive semi-definite. This defines a partial ordering on the set of all square matrices. One can similarly define a strict partial ordering  . The ordering is called the Loewner order.

Inverse of positive definite matrixEdit

Every positive definite matrix is invertible and its inverse is also positive definite.[4] If   then  .[5] Moreover, by the min-max theorem, the kth largest eigenvalue of   is greater than the kth largest eigenvalue of  .

ScalingEdit

If   is positive definite and   is a real number, then   is positive definite.[6]

AdditionEdit

If   and   are positive definite, then the sum   is also positive definite.[6]

MultiplicationEdit

  • If   and   are positive definite, then the products   and   are also positive definite. If  , then   is also positive definite.
  • If   is positive semidefinite, then   is positive semidefinite. If   is positive definite and   has full rank, then   is positive definite.[7]

Cholesky decompositionEdit

For any matrix  , the matrix   is positive semidefinite, and  . Conversely, any Hermitian positive semi-definite matrix   can be written as  , where   is lower triangular; this is the Cholesky decomposition. If   is not positive definite, then some of the diagonal elements of   may be zero.

A hermitian matrix   is positive definite if and only if it has a unique Cholesky decomposition, i.e. the matrix   is positive definite if and only if there exists a unique lower triangular matrix  , with real and strictly positive diagonal elements, such that  .

Square rootEdit

A matrix   is positive semi-definite if and only if there is a positive semi-definite matrix   with  . This matrix   is unique,[8] is called the square root of  , and is denoted with   (the square root   is not to be confused with the matrix   in the Cholesky factorization  , which is also sometimes called the square root of  ).

If   then  .

SubmatricesEdit

Every principal submatrix of a positive definite matrix is positive definite.

TraceEdit

The diagonal entries   of a positive definite matrix are real and non-negative. As a consequence the trace,  . Furthermore,[9] since every principal sub-matrix (in particular, 2-by-2) is positive definite,

 

and thus

 

Hadamard productEdit

If  , although   is not necessary positive semidefinite, the Hadamard product   (this result is often called the Schur product theorem).[10]

Regarding the Hadamard product of two positive semidefinite matrices  ,  , there are two notable inequalities:

  • Oppenheim's inequality:  [11]
  •  .[12]

Kronecker productEdit

If  , although   is not necessary positive semidefinite, the Kronecker product  .

Frobenius productEdit

If  , although   is not necessary positive semidefinite, the Frobenius product   (Lancaster–Tismenetsky, The Theory of Matrices, p. 218).

ConvexityEdit

The set of positive semidefinite symmetric matrices is convex. That is, if   and   are positive semidefinite, then for any   between 0 and 1,   is also positive semidefinite. For any vector  :

 

This property guarantees that semidefinite programming problems converge to a globally optimal solution.

Relation with cosineEdit

The positive-definiteness of a matrix   expresses that the angle   between any vector   and its image   is always  :

 

Further propertiesEdit

  1. If   is a symmetric Toeplitz matrix, i.e. the entries   are given as a function of their absolute index differences:  , and the strict inequality

     

    holds, then   is strictly positive definite.
  2. Let   and   Hermitian. If   (resp.,  ) then   (resp.,  ).[13]
  3. If   is real, then there is a   such that  , where   is the identity matrix.
  4. If   denotes the leading   minor,   is the kth pivot during LU decomposition.
  5. A matrix is negative definite if its k-th order leading principal minor is negative when   is odd, and positive when   is even.
  6. A matrix   is positive semidefinite if and only if it arises as the Gram matrix of some set of vectors. In contrast to the positive definite case, these vectors need not be linearly independent.

A Hermitian matrix is positive semidefinite if and only if all of its principal minors are nonnegative. It is however not enough to consider the leading principal minors only, as is checked on the diagonal matrix with entries 0 and −1.

Block matricesEdit

A positive   matrix may also be defined by blocks:

 

where each block is  . By applying the positivity condition, it immediately follows that   and   are hermitian, and  .

We have that   for all complex  , and in particular for  . Then

 

A similar argument can be applied to  , and thus we conclude that both   and   must be positive definite matrices, as well.

Converse results can be proved with stronger conditions on the blocks, for instance using the Schur complement.

Extension for non-Hermitian square matricesEdit

The definition of positive definite can be generalized by designating any complex matrix   (e.g. real non-symmetric) as positive definite if   for all non-zero complex vectors  , where   denotes the real part of a complex number  .[14] Only the Hermitian part   determines whether the matrix is positive definite, and is assessed in the narrower sense above. Similarly, If   and   are real, we have   for all real nonzero vectors   if and only if the symmetric part   is positive definite in the narrower sense. It is immediately clear that  is insensitive to transposition of M.

Consequently, a non-symmetric real matrix with only positive eigenvalues does not need to be positive definite. For example, the matrix   has positive eigenvalues yet is not positive definite; in particular a negative value of   is obtained with the choice   (which is the eigenvector associated with the negative eigenvalue of the symmetric part of  ).

In summary, the distinguishing feature between the real and complex case is that, a bounded positive operator on a complex Hilbert space is necessarily Hermitian, or self adjoint. The general claim can be argued using the polarization identity. That is no longer true in the real case.

ApplicationsEdit

Heat conductivity matrixEdit

Fourier's law of heat conduction, giving heat flux   in terms of the temperature gradient   is written for anisotropic media as  , in which   is the symmetric thermal conductivity matrix. The negative is inserted in Fourier's law to reflect the expectation that heat will always flow from hot to cold. In other words, since the temperature gradient   always points from cold to hot, the heat flux   is expected to have a negative inner product with   so that  . Substituting Fourier's law then gives this expectation as  , implying that the conductivity matrix should be positive definite.

See alsoEdit

NotesEdit

  1. ^ "Appendix C: Positive Semidefinite and Positive Definite Matrices". Parameter Estimation for Scientists and Engineers: 259–263. doi:10.1002/9780470173862.app3.
  2. ^ Stewart, J. (1976). "Positive definite functions and generalizations, an historical survey". Rocky Mountain J. Math. 6 (3): 409–434. doi:10.1216/RMJ-1976-6-3-409.
  3. ^ Horn & Johnson (1985), p. 218 ff.
  4. ^ Horn & Johnson (1985), p. 397
  5. ^ Horn & Johnson (1985), Corollary 7.7.4(a)
  6. ^ a b Horn & Johnson (1985), Observation 7.1.3
  7. ^ Horn, Roger A.; Johnson, Charles R. (2013). "7.1 Definitions and Properties". Matrix Analysis (2nd ed.). Cambridge University Press. p. 431. ISBN 978-0-521-83940-2. "Observation 7.1.8 Let   be Hermitian and let  :
    • Suppose that A is positive semidefinite. Then   is positive semidefinite,  , and  
    • Suppose that A is positive definite. Then  , and   is positive definite if and only if rank(C) = m"
  8. ^ Horn & Johnson (1985), Theorem 7.2.6 with  
  9. ^ Horn & Johnson (1985), p. 398
  10. ^ Horn & Johnson (1985), Theorem 7.5.3
  11. ^ Horn & Johnson (1985), Theorem 7.8.6
  12. ^ Styan (1973)[full citation needed]
  13. ^ Bhatia, Rajendra (2007). Positive Definite Matrices. Princeton, New Jersey: Princeton University Press. p. 8. ISBN 978-0-691-12918-1.
  14. ^ Weisstein, Eric W. Positive Definite Matrix. From MathWorld--A Wolfram Web Resource. Accessed on 2012-07-26

ReferencesEdit

External linksEdit