Talk:Invariant estimator

Equivariant?

Can someone say where the term "equivariant" has been used ... it is not in any of my dictionaries of maths or stats. Melcombe (talk) 09:32, 12 May 2008 (UTC)

It appears, for example, in Lehmann and Casella, Theory of Point Estimation. --Zvika (talk) 18:17, 12 May 2008 (UTC)

non-Bayesian?

Should such prominence be given to "non-Bayesian"? After all ideas of invariance can be applied to Bayesian estimation just as well. Consider for example HPD (highest posterior density) estimation (either point or interval estimates), which is not invariant to transformation of the parameters. Melcombe (talk) 09:14, 19 May 2008 (UTC)

Can you provide a source dealing with Bayesian equivariant estimators? I haven't encountered one. --Zvika (talk) 18:33, 19 May 2008 (UTC)

I am not sure that it is important to actually find examples of invariant Bayesian estimators, just that the idea of invariance can be applied to estimation in a Bayesian setting, where this would go beyond the impractical definition given in Bayesian estimation since contexts with prescribed loss functions are rare. Thus things like Maximum a posteriori need to be included as Bayesian estimates, and also the expectation of the posterior distribution, both in contexts of no-loss-function. Obviously HPD estimates are not invariant to transformations of the parameter space, and nor are expected values of the posterior, and it is important to be able to say this. Melcombe (talk) 09:41, 30 May 2008 (UTC)

Hi again Melcombe, nice to hear from you. You raise two separate points.

1. As I understand it, there is a difference between Bayesian estimation and Bayes estimator. The fact that currently one redirects to the other is simply a result of the fact that we don't yet have an article on Bayesian estimation. Bayesian estimation refers to the general approach of using of prior information (as opposed to frequentist or classical point estimation); this includes MAP, MMSE, etc. By contrast, a Bayes estimator is an estimator which is optimal under the "minimum Bayes risk" criterion (see, e.g., Lehmann and Casella, p.225).
   So, as far as this issue is concerned, I think the best solution is to create a new article Bayesian estimation which would discuss the various Bayesian approaches to estimation and provide links to Bayes estimator, minimum mean square error, maximum a posteriori, etc. I would be happy to work with you on such an article. I do not think MAP should be included in Bayes estimator because it is not a Bayes estimator under this definition.
2. Concerning the relation between invariance/equivariance and Bayes methods: As I said above, the critical issue is to find a reliable source dealing with this question. Bayes estimators might or might not be invariant, but unless you can find a source saying so, including such statements in Wikipedia constitutes original research. I have not myself looked specifically for such a source, so it may well be present in Bayesian textbooks like Berger's. I urge you to go ahead and find such a source; it will definitely improve the article if you do.

Zvika (talk) 17:52, 30 May 2008 (UTC)

I had forgotten to set a watch on the page. A reference is Bernardo & Smith's "Bayesian Theory" (Wiley, 1994/2000). They cover (briefly): invariance for model formulation, invariance for prior distributions and non-informative priors and invariance in estimation. The last is most directly relevant here: they have about 2/3rds of a page (p454) for example :"From a Bayesian point of view, for invariance to be a relevant notion it must be true that the transformation involved also applies to the prior distribution". So at least it has been considered in Bayesian contexts. It may have been found to be non-useful, but this is perhaps no more than in general statistics, where ideas of "unbiasedess" pervade even though this leads to non-invariant procedures. Thus ideas of "invariance" may be best seen as possibly providing a guide to finding reasonable procedures, but not as being a definitive requirement. Melcombe (talk) 09:03, 2 June 2008 (UTC)

Excellent! In this case, I would definitely support a discussion of the Bayesian aspects of invariance in the article. --Zvika (talk) 14:24, 2 June 2008 (UTC)

Reading through the passage you quoted, I noticed that it appears in Appendix B, "Non-Bayesian Approaches". Furthermore, as you said, the basic conclusion of this passage is that invariance is a rather limited technique from a Bayesian point of view. On the other hand, frequentist books give quite a lot of space to invariance (an 80-page chapter in Lehmann and Casella), including a more general formulation than that which appears in Bernardo and Smith. So perhaps the discussion of Bayesian aspects should be a passing statement saying that invariance is not usually applied in a Bayesian setting, with a short explanation why as explained by B&S. What do you think? --Zvika (talk) 07:06, 3 June 2008 (UTC)

Broadly speaking I agree, at least when dealing with questions of finding "optimal" estimators. However, it may be that (presently) this has too much emphasis in the article and that there needs to be more initial discussion of at least the two special cases of: (i) invariance to permutations of observations in the iid case; (ii) invariance to non-linear transformations of parameters. For case (i), all Bayesian procedures (and all likelihood-based procedures) would be invariant to this type of transformation. There would then remain the question of what to do about the other types of invariance that B&S mention and which are both: not directly about estimation and not referred to in the article Invariant (mathematics). Melcombe (talk) 09:38, 3 June 2008 (UTC)

Concerning (i), I find it difficult to think of any reasonable estimator which is not invariant for such a transformation, so I don't know what would be the goal of mentioning this (except perhaps as a trivial example). I didn't quite understand what you're referring to in (ii) - what other types of invariance do B&S mention? I only see a mention of location invariance. In any case, if you've got a concrete idea of what you think should be changed, go ahead and do it, so we have something tangible to discuss. --Zvika (talk) 12:23, 3 June 2008 (UTC)

As mentioned above B&S discuss invariance for model formulation and invariance for prior distributions and non-informative priors: these are listed in their index. Further, I have found in "Essentials of Statistical Inference" by Young & Smith(Cambridge UP, 2005, p86-87) treatment of an "invariant statistic", with a separate (distinct) definition of "equivariant" which then leads on to "equivariant estimator". They also have "maximal invariant". Thus there may be a need to clarify these (different?) levels of invariance for estimators. Melcombe (talk) 13:31, 3 June 2008 (UTC)

As I say, go ahead and edit the article as you see fit, and then we can discuss concrete issues. It could definitely use a discussion of maximal invariance. Iirc, this is analogous to the concept of maximal sufficiency. --Zvika (talk) 17:21, 3 June 2008 (UTC)

I have made some major changes, including replacing the previous context-setting stuff and putting in a revised section structure which would separate-off the more mathematical bits. I stopped at the section-heading "optimal invariant estimators" ... possibly the present content should be moved into the mathematical bit and be replaced with some more context. More generally, I think it would be good to have an article "Invariance (statistics)" to give a outline of uses of invariance ideas in statistics and one or more articles paralleling this one to cover things like "invariant statistic" to allow the emphasis in this one to be on estimation and optimal estimators. I am not clear what you expect the coverage to be here... I was thinking of "optimal" as being optimal within a given class of estimators, rather than optimal over all possible estimtors. For example I would think a reference at least to estimates of the type "optimal linear combination of order statistics" for location-scale problems would be in order.

Melcombe (talk) 13:06, 13 June 2008 (UTC)

Overall I like your changes. My only comment is that perhaps the background section on Bayesian vs. deterministic estimation would be more appropriate in a general article on estimation, as this background is really required for understanding just about any estimation-related idea. Perhaps there should be a whole article on this, with a link to it from the lead of any article describing either a deterministic or a Bayesian concept. --Zvika (talk) 06:39, 14 June 2008 (UTC)

Invariant Test

I suggest to start a new article, Invariant Test which should include the definition of Invariant Statistic Model and Invariant hypothesis, Invariant Function (maximal invariant function). Jackzhp (talk) 01:05, 3 January 2011 (UTC)