Bill Jefferys wrote:
>
> At 10:59 AM +0200 4/3/02, Juergen Schmidhuber wrote:
> >The theory of inductive inference is Bayesian, of course.
> >But Bayes' rule by itself does not yield Occam's razor.
>
> "By itself?" No one said it did. Of course assumptions must be made.
> At minimum one always has to choose priors in Bayesian inference.
>
> Our paper shows that there is a Bayesian interpretation that yields
> something very suggestive of Ockham's razor. It is appealing in that
> if one has a "simple" versus a "complex" hypothesis, "simple" meaning
> that the prior probability is concentrated and "complex" meaning that
> it is vague and spread out, "simple" meaning that you don't have many
> knobs to tweak, "complex" meaning the opposite, then the "simple"
> hypothesis will be favored over the "complex" one unless the data lie
> well away from where the "simple" hypothesis has placed its bets.
> Bayesians distinguish this from Ockham's formulation by calling it
> the "Bayesian Ockham's razor", recognizing that it is not what
> William of Ockham wrote, "Entia non sunt multiplicanda sine
> necessitate" (or one of his other genuine formulations).
>
> Please don't read more into our article than is there.
>
> "By itself." First you said that the AP "by itself" has no predictive
> power. I missed the "by itself" so misunderstood you, but when I
> understood what you were saying I agreed. Now you say that Bayes'
> rule "by itself" does not yield Ockham's razor. Jim and I never said
> that it did. I am hard pressed to see how anything nontrivial
> relating to the real world can be gotten from any principle "by
> itself," so I don't regard these comments as very profound, or very
> useful.
>
> [Remainder of article snipped]
>
> Bill
One has to choose priors. Exactly. To repeat the nontrivial point:
The only choice you need to obtain Occam's razor is to restrict
yourself to priors computable in the limit (this is not much of a
restriction in the sense that other priors are beyond constructive
description anyway). Then you'll automatically prefer few knobs to
tweak, or, more generally, short descriptions of the
observations:
http://www.idsia.ch/~juergen/toesv2/
Juergen (will be out of town until April 16)
Received on Thu Apr 04 2002 - 01:43:18 PST