In mathematics, a matrix norm is a vector norm in a vector space whose elements (vectors) are matrices (of given dimensions).

PreliminariesEdit

Given a field   of either real or complex numbers, let   be the K-vector space of matrices with   rows and   columns and entries in the field  . A matrix norm is a norm on  

This article will always write such norms with double vertical bars (like so:  ). Thus, the matrix norm is a function   that must satisfy the following properties:[1][2]

For all scalars   and matrices  ,
  •   (positive-valued)
  •   (definite)
  •   (absolutely homogeneous)
  •   (sub-additive or satisfying the triangle inequality)

The only feature distinguishing matrices from rearranged vectors is multiplication. Matrix norms are particularly useful if they are also sub-multiplicative:[1][2][3]

  •  [Note 1]

Every norm on Kn×n can be rescaled to be sub-multiplicative; in some books, the terminology matrix norm is reserved for sub-multiplicative norms.[4]

Matrix norms induced by vector normsEdit

Suppose a vector norm   on   and a vector norm   on   is given. Any   matrix A induces a linear operator from   to   with respect to the standard basis, and one defines the corresponding induced norm or operator norm or subordinate norm on the space   of all   matrices as follows:

 

where   denotes the supremum. This norm measures how much the mapping induced by   can stretch vectors. Depending on the vector norms  ,   used, notation other than   can be used for the operator norm.

Norms induced by p-normsEdit

If the p-norm for vectors (1 ≤ p ≤ ∞) is used for both spaces   and  , then the corresponding operator norm is:[2]

 

These induced norms are different from the "entry-wise" p-norms and the Schatten p-norms for matrices treated below, which are also usually denoted by  

In the special cases of   the induced matrix norms can be computed or estimated by

 

which is simply the maximum absolute column sum of the matrix;

 

which is simply the maximum absolute row sum of the matrix.

In the special case of   (the Euclidean norm or  -norm for vectors), the induced matrix norm is the spectral norm. The spectral norm of a matrix   is the largest singular value of   (i.e., the square root of the largest eigenvalue of the matrix  , where   denotes the conjugate transpose of  ):[5]

 

where   represents the largest singular value of matrix  . Also,

 

since   and similarly   by singular value decomposition (SVD). There is another important inequality:

 

where   is the Frobenius norm. Equality holds if and only if the matrix   is a rank-one matrix or a zero matrix. This inequality can be derived from the fact that the trace of a matrix is equal to the sum of its eigenvalues.

When   we have an equivalent definition for   as  . It can be shown to be equivalent to the above definitions using the Cauchy–Schwarz inequality.

For example, for

 

we have that

 
 

PropertiesEdit

Any operator norm is consistent with the vector norms that induce them, giving

 

Suppose  ;  ; and   are operator norms induced by the respective pairs of vector norms  ;  ; and  . Then,

 

this follows from

 

and

 

Square matricesEdit

Suppose   is an operator norm on the space of square matrices   induced by vector norms   and  . Then, the operator norm is a sub-multiplicative matrix norm:

 

Moreover, any such norm satisfies the inequality

  (1)

for all positive integers r, where ρ(A) is the spectral radius of A. For symmetric or hermitian A, we have equality in (1) for the 2-norm, since in this case the 2-norm is precisely the spectral radius of A. For an arbitrary matrix, we may not have equality for any norm; a counterexample would be

 

which has vanishing spectral radius. In any case, for any matrix norm, we have the spectral radius formula:

 

Consistent and compatible normsEdit

A matrix norm   on   is called consistent with a vector norm   on   and a vector norm   on  , if:

 

for all   and all  . In the special case of m = n and  ,   is also called compatible with  .

All induced norms are consistent by definition. Also, any sub-multiplicative matrix norm on   induces a compatible vector norm on   by defining  .

"Entry-wise" matrix normsEdit

These norms treat an   matrix as a vector of size  , and use one of the familiar vector norms. For example, using the p-norm for vectors, p ≥ 1, we get:

 

This is a different norm from the induced p-norm (see above) and the Schatten p-norm (see below), but the notation is the same.

The special case p = 2 is the Frobenius norm, and p = ∞ yields the maximum norm.

L2,1 and Lp,q normsEdit

Let   be the columns of matrix  . The   norm[6] is the sum of the Euclidean norms of the columns of the matrix:

 

The   norm as an error function is more robust, since the error for each data point (a column) is not squared. It is used in robust data analysis and sparse coding.

For p, q ≥ 1, the   norm can be generalized to the   norm as follows:

 

Frobenius normEdit

When p = q = 2 for the   norm, it is called the Frobenius norm or the Hilbert–Schmidt norm, though the latter term is used more frequently in the context of operators on (possibly infinite-dimensional) Hilbert space. This norm can be defined in various ways:

 

where   are the singular values of  . Recall that the trace function returns the sum of diagonal entries of a square matrix.

The Frobenius norm is an extension of the Euclidean norm to   and comes from the Frobenius inner product on the space of all matrices.

The Frobenius norm is sub-multiplicative and is very useful for numerical linear algebra. The sub-multiplicativity of Frobenius norm can be proved using Cauchy–Schwarz inequality.

Frobenius norm is often easier to compute than induced norms, and has the useful property of being invariant under rotations (and unitary operations in general). That is,   for any unitary matrix  . This property follows from the cyclic nature of the trace ( ):

 

and analogously:

 

where we have used the unitary nature of   (that is,  ).

It also satisfies

 

and

 

where   is the Frobenius inner product.

Max normEdit

The max norm is the elementwise norm with p = q = ∞:

 

This norm is not sub-multiplicative.

Note that in some literature (such as Communication complexity), an alternative definition of max-norm, also called the  -norm, refers to the factorization norm:

 

Schatten normsEdit

The Schatten p-norms arise when applying the p-norm to the vector of singular values of a matrix.[2] If the singular values of the   matrix   are denoted by σi, then the Schatten p-norm is defined by

 

These norms again share the notation with the induced and entry-wise p-norms, but they are different.

All Schatten norms are sub-multiplicative. They are also unitarily invariant, which means that   for all matrices   and all unitary matrices   and  .

The most familiar cases are p = 1, 2, ∞. The case p = 2 yields the Frobenius norm, introduced before. The case p = ∞ yields the spectral norm, which is the operator norm induced by the vector 2-norm (see above). Finally, p = 1 yields the nuclear norm (also known as the trace norm, or the Ky Fan 'n'-norm[7]), defined as

 

where   denotes a positive semidefinite matrix   such that  . More precisely, since   is a positive semidefinite matrix, its square root is well-defined. The nuclear norm   is a convex envelope of the rank function  , so it is often used in mathematical optimization to search for low rank matrices.

Monotone normsEdit

A matrix norm   is called monotone if it is monotonic with respect to the Loewner order. Thus, a matrix norm is increasing if

 

The Frobenius norm and spectral norm are examples of monotone norms.[8]

Cut normsEdit

Another source of inspiration for matrix norms arises from considering a matrix as the adjacency matrix of a weighted, directed graph.[9] The so-called "cut norm" measures how close the associated graph is to being bipartite:

 

where AKm×n.[9][10][11] Equivalent definitions (up to a constant factor) impose the conditions 2|S|>n & 2|T|>m; S=T; or ST=∅.[10]

The cut-norm is equivalent to the induced operator norm ‖·‖∞→1, which is itself equivalent to the another norm, called the Grothendieck norm.[11]

To define the Grothendieck norm, first note that a linear operator K1K1 is just a scalar, and thus extends to a linear operator on any KkKk. Moreover, given any choice of basis for Kn and Km, any linear operator KnKm extends to a linear operator (Kk)n→(Kk)m, by letting each matrix element on elements of Kk via scalar multiplication. The Grothendieck norm is the norm of that extended operator; in symbols:[11]

 

The Grothendieck norm depends on choice of basis (usually taken to be the standard basis) and k.

Equivalence of normsEdit

For any two matrix norms   and  , we have that:

 

for some positive numbers r and s, for all matrices  . In other words, all norms on   are equivalent; they induce the same topology on  . This is true because the vector space   has the finite dimension  .

Moreover, for every vector norm   on  , there exists a unique positive real number   such that   is a sub-multiplicative matrix norm for every  .

A sub-multiplicative matrix norm   is said to be minimal, if there exists no other sub-multiplicative matrix norm   satisfying  .

Examples of norm equivalenceEdit

Let   once again refer to the norm induced by the vector p-norm (as above in the Induced Norm section).

For matrix   of rank  , the following inequalities hold:[12][13]

  •  
  •  
  •  
  •  
  •  

Another useful inequality between matrix norms is

 

which is a special case of Hölder's inequality.

See alsoEdit

NotesEdit

  1. ^ The condition only applies when the product is defined, such as the case of square matrices (m = n).

ReferencesEdit

  1. ^ a b Weisstein, Eric W. "Matrix Norm". mathworld.wolfram.com. Retrieved 2020-08-24.
  2. ^ a b c d "Matrix norms". fourier.eng.hmc.edu. Retrieved 2020-08-24.
  3. ^ Malek-Shahmirzadi, Massoud (1983). "A characterization of certain classes of matrix norms". Linear and Multilinear Algebra. 13 (2): 97–99. doi:10.1080/03081088308817508. ISSN 0308-1087.
  4. ^ Horn, Roger A. (2012). Matrix analysis. Johnson, Charles R. (2nd ed.). Cambridge: Cambridge University Press. pp. 340–341. ISBN 978-1-139-77600-4. OCLC 817236655.
  5. ^ Carl D. Meyer, Matrix Analysis and Applied Linear Algebra, §5.2, p.281, Society for Industrial & Applied Mathematics, June 2000.
  6. ^ Ding, Chris; Zhou, Ding; He, Xiaofeng; Zha, Hongyuan (June 2006). "R1-PCA: Rotational Invariant L1-norm Principal Component Analysis for Robust Subspace Factorization". Proceedings of the 23rd International Conference on Machine Learning. ICML '06. Pittsburgh, Pennsylvania, USA: ACM. pp. 281–288. doi:10.1145/1143844.1143880. ISBN 1-59593-383-2.
  7. ^ Fan, Ky. (1951). "Maximum properties and inequalities for the eigenvalues of completely continuous operators". Proceedings of the National Academy of Sciences of the United States of America. 37 (11): 760–766. Bibcode:1951PNAS...37..760F. doi:10.1073/pnas.37.11.760. PMC 1063464. PMID 16578416.
  8. ^ Ciarlet, Philippe G. (1989). Introduction to numerical linear algebra and optimisation. Cambridge, England: Cambridge University Press. p. 57. ISBN 0521327881.
  9. ^ a b Frieze, Alan; Kannan, Ravi (1999-02-01). "Quick Approximation to Matrices and Applications". Combinatorica. 19 (2): 175–220. doi:10.1007/s004930050052. ISSN 1439-6912. S2CID 15231198.
  10. ^ a b Lovász László (2012). "The cut distance". Large Networks and Graph Limits. AMS Colloquium Publications. Vol. 60. Providence, RI: American Mathematical Society. pp. 127–131. ISBN 978-0-8218-9085-1. Note that Lovász rescales A to lie in [0, 1].
  11. ^ a b c Alon, Noga; Naor, Assaf (2004-06-13). "Approximating the cut-norm via Grothendieck's inequality". Proceedings of the Thirty-sixth Annual ACM Symposium on Theory of Computing. STOC '04. Chicago, IL, USA: Association for Computing Machinery: 72–80. doi:10.1145/1007352.1007371. ISBN 978-1-58113-852-8. S2CID 1667427.
  12. ^ Golub, Gene; Charles F. Van Loan (1996). Matrix Computations – Third Edition. Baltimore: The Johns Hopkins University Press, 56–57. ISBN 0-8018-5413-X.
  13. ^ Roger Horn and Charles Johnson. Matrix Analysis, Chapter 5, Cambridge University Press, 1985. ISBN 0-521-38632-2.

BibliographyEdit

  • James W. Demmel, Applied Numerical Linear Algebra, section 1.7, published by SIAM, 1997.
  • Carl D. Meyer, Matrix Analysis and Applied Linear Algebra, published by SIAM, 2000. [1]
  • John Watrous, Theory of Quantum Information, 2.3 Norms of operators, lecture notes, University of Waterloo, 2011.
  • Kendall Atkinson, An Introduction to Numerical Analysis, published by John Wiley & Sons, Inc 1989