Bayes not Bust! Why Simplicity is no Problem for Bayesians |
| |
| Authors: | Dowe David L; Gardner Steve; Oppy Graham |
| |
| Institution: | Clayton School of Information Technology, Monash University, Clayton VIC. Australia 3800, http://www.csse.monash.edu.au/~dld |
| |
| Abstract: | The advent of formal definitions of the simplicity of a theoryhas important implications for model selection. But what isthe best way to define simplicity? Forster and Sober (1994])advocate the use of Akaike's Information Criterion (AIC), anon-Bayesian formalisation of the notion of simplicity. Thisforms an important part of their wider attack on Bayesianismin the philosophy of science. We defend a Bayesian alternative:the simplicity of a theory is to be characterised in terms ofWallace's Minimum Message Length (MML). We show that AIC isinadequate for many statistical problems where MML performswell. Whereas MML is always defined, AIC can be undefined. WhereasMML is not known ever to be statistically inconsistent, AICcan be. Even when defined and consistent, AIC performs worsethan MML on small sample sizes. MML is statistically invariantunder 1-to-1 re-parametrisation, thus avoiding a common criticismof Bayesian approaches. We also show that MML provides answersto many of Forster's objections to Bayesianism. Hence an importantpart of the attack on Bayesianism fails. - Introduction
- TheCurve Fitting Problem
- 2.1 Curves and families of curves
- 2.2 Noise
- 2.3 Themethod of Maximum Likelihood
- 2.4 ML and over-fitting
- Akaike's Information Criterion(AIC)
- The Predictive Accuracy Framework
- The Minimum MessageLength (MML) Principle
- 5.1 The Strict MML estimator
- 5.2 Anexample: Thebinomial distribution
- 5.3 Properties ofthe SMML estimator
- 5.3.1 Bayesianism
- 5.3.2 Languageinvariance
- 5.3.3Generality
- 5.3.4 Consistencyand efficiency
- 5.4 Similarity to false oracles
- 5.5 Approximationsto SMML
- Criticisms of AIC
- 6.1 Problems with ML
- 6.1.1 Smallsample biasin a Gaussian distribution
- 6.1.2 Thevon Misescircular and von Mises—Fisherspherical distributions
- 6.1.3 The Neyman–Scottproblem
- 6.1.4 Neyman–Scott,predictive accuracyandminimum expected KL distance
- 6.2 Otherproblems with AIC
- 6.2.1 Univariate polynomial regression
- 6.2.2 Autoregressiveeconometric time series
- 6.2.3 Multivariatesecond-orderpolynomial modelselection
- 6.2.4 Gapor no gap:a clustering-like problem forAIC
- 6.3 Conclusionsfrom the comparison of MML and AIC
- Meeting Forster's objectionsto Bayesianism
- 7.1 The sub-family problem
- 7.2 Theproblem of approximation,or, which framework forstatistics?
- Conclusion
- Details of the derivation of the Strict MMLestimator
- MML, AIC and the Gap vs. No Gap Problem
- B.1 Expectedsize of the largest gap
- B.2 Performanceof AIC on thegap vs. no gap problem
- B.3 Performanceof MML in thegap vs. no gap problem
|
| |
| Keywords: | |
| 本文献已被 Oxford 等数据库收录！ |
|
