Differential entropy

Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy, a measure of average surprisal of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not.[1]: 181–218  The actual continuous version of discrete entropy is the limiting density of discrete points (LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete entropy.

DefinitionEdit

Let   be a random variable with a probability density function   whose support is a set  . The differential entropy   or   is defined as[2]: 243 

 

For probability distributions which don't have an explicit density function expression, but have an explicit quantile function expression,  , then   can be defined in terms of the derivative of   i.e. the quantile density function   as [3]: 54–59 

 .

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure  .[4]: 183–184  For example, the differential entropy of a quantity measured in millimeters will be log(1000) more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of log(1000) more than the same quantity divided by 1000.

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution   has negative differential entropy

 

being less than that of   which has zero differential entropy. Thus, differential entropy does not share all properties of discrete entropy.

Note that the continuous mutual information   has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of   and   as these partitions become finer and finer. Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps), [5] including linear [6] transformations of   and  , and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

For the direct analogue of discrete entropy extended to the continuous space, see limiting density of discrete points.

Properties of differential entropyEdit

  • For probability densities   and  , the Kullback–Leibler divergence   is greater than or equal to 0 with equality only if   almost everywhere. Similarly, for two random variables   and  ,   and   with equality if and only if   and   are independent.
  • The chain rule for differential entropy holds as in the discrete case[2]: 253 
 .
  • Differential entropy is translation invariant, i.e. for a constant  .[2]: 253 
 
  • Differential entropy is in general not invariant under arbitrary invertible maps.
In particular, for a constant  
 
For a vector valued random variable   and an invertible (square) matrix  
 [2]: 253 
  • In general, for a transformation from a random vector to another random vector with same dimension  , the corresponding entropies are related via
 
where   is the Jacobian of the transformation  .[7] The above inequality becomes an equality if the transform is a bijection. Furthermore, when   is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and  .
  • If a random vector   has mean zero and covariance matrix  ,   with equality if and only if   is jointly gaussian (see below).[2]: 254 

However, differential entropy does not have other desirable properties:

  • It is not invariant under change of variables, and is therefore most useful with dimensionless variables.
  • It can be negative.

A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

Maximization in the normal distributionEdit

TheoremEdit

With a normal distribution, differential entropy is maximized for a given variance. A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.[2]: 255 

ProofEdit

Let   be a Gaussian PDF with mean μ and variance   and   an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that   has the same mean of   as  .

Consider the Kullback–Leibler divergence between the two distributions

 

Now note that

 

because the result does not depend on   other than through the variance. Combining the two results yields

 

with equality when   following from the properties of Kullback–Leibler divergence.

Alternative proofEdit

This result may also be demonstrated using the variational calculus. A Lagrangian function with two Lagrangian multipliers may be defined as:

 

where g(x) is some function with mean μ. When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition   and the requirement of fixed variance  , are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero:

 

Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields:

 

Using the constraint equations to solve for λ0 and λ yields the normal distribution:

 

Example: Exponential distributionEdit

Let   be an exponentially distributed random variable with parameter  , that is, with probability density function

 

Its differential entropy is then

   
 
 
 

Here,   was used rather than   to make it explicit that the logarithm was taken to base e, to simplify the calculation.

Relation to estimator errorEdit

The differential entropy yields a lower bound on the expected squared error of an estimator. For any random variable   and estimator   the following holds:[2]

 

with equality if and only if   is a Gaussian random variable and   is the mean of  .

Differential entropies for various distributionsEdit

In the table below   is the gamma function,   is the digamma function,   is the beta function, and γE is Euler's constant.[8]: 219–230 

Table of differential entropies
Distribution Name Probability density function (pdf) Entropy in nats Support
Uniform      
Normal      
Exponential      
Rayleigh      
Beta   for    
 
 
Cauchy      
Chi      
Chi-squared      
Erlang      
F    
 
 
Gamma      
Laplace      
Logistic      
Lognormal      
Maxwell–Boltzmann      
Generalized normal      
Pareto      
Student's t      
Triangular      
Weibull      
Multivariate normal  
 
   

Many of the differential entropies are from.[9]: 120–122 

VariantsEdit

As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities.[10]: 181–218 

A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points). If   is further constrained to be a probability density, the resulting notion is called relative entropy in information theory:

 

The definition of differential entropy above can be obtained by partitioning the range of   into bins of length   with associated sample points   within the bins, for   Riemann integrable. This gives a quantized version of  , defined by   if  . Then the entropy of   is[2]

 

The first term on the right approximates the differential entropy, while the second term is approximately  . Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be  .

See alsoEdit

ReferencesEdit

  1. ^ Jaynes, E.T. (1963). "Information Theory And Statistical Mechanics" (PDF). Brandeis University Summer Institute Lectures in Theoretical Physics. 3 (sect. 4b).
  2. ^ a b c d e f g h Cover, Thomas M.; Thomas, Joy A. (1991). Elements of Information Theory. New York: Wiley. ISBN 0-471-06259-6.
  3. ^ Vasicek, Oldrich (1976), "A Test for Normality Based on Sample Entropy", Journal of the Royal Statistical Society, Series B, 38 (1): 54–59, JSTOR 2984828.
  4. ^ Gibbs, Josiah Willard (1902). Elementary Principles in Statistical Mechanics, developed with especial reference to the rational foundation of thermodynamics. New York: Charles Scribner's Sons.
  5. ^ Kraskov, Alexander; Stögbauer, Grassberger (2004). "Estimating mutual information". Physical Review E. 60 (6): 066138. arXiv:cond-mat/0305641. Bibcode:2004PhRvE..69f6138K. doi:10.1103/PhysRevE.69.066138. PMID 15244698. S2CID 1269438.
  6. ^ Fazlollah M. Reza (1994) [1961]. An Introduction to Information Theory. Dover Publications, Inc., New York. ISBN 0-486-68210-2.
  7. ^ "proof of upper bound on differential entropy of f(X)". Stack Exchange. April 16, 2016.
  8. ^ Park, Sung Y.; Bera, Anil K. (2009). "Maximum entropy autoregressive conditional heteroskedasticity model" (PDF). Journal of Econometrics. Elsevier. 150 (2): 219–230. doi:10.1016/j.jeconom.2008.12.014. Archived from the original (PDF) on 2016-03-07. Retrieved 2011-06-02.
  9. ^ Lazo, A. and P. Rathie (1978). "On the entropy of continuous probability distributions". IEEE Transactions on Information Theory. 24 (1): 120–122. doi:10.1109/TIT.1978.1055832.
  10. ^ Jaynes, E.T. (1963). "Information Theory And Statistical Mechanics" (PDF). Brandeis University Summer Institute Lectures in Theoretical Physics. 3 (sect. 4b).

External linksEdit