Variational Bayes and a problem of reliable communication: II. Infinite systems

We consider a family of estimation problems not admitting conventional analysis because of singularity and measurability issues. We define posterior distributions for the family by a variational technique analogous to that used to define Gibbs measures in statistical mechanics. The family of estimation problems, which arise in the asymptotic analysis of error-control codes, is parametrized by a code rate, R∈(0,∞); this is shown to be analogous to the absolute temperature of statistical mechanics. The family undergoes an (Ehrenfest) first-order phase transition at a critical code rate C (the channel capacity), where there is a convex set of posterior distributions. At all other code rates, there is only one posterior distribution; if R < C, this is the Dirac measure located at the source sequence, whereas if R > C it has infinite support. In a result reflecting the Dobrushin construction, we show that these posterior distributions are asymptotically consistent with those of families of finite-sequence error-control codes. © 2012 IOP Publishing Ltd and SISSA Medialab srl.