CSM-349 - Benford's Law: An Empirical Investigation and a Novel Explanation

This report describes an investigation into Benford?s Law for the distribution of leading digits in real data sets. A large number of such data sets have been examined and it was found that only a small fraction of them conform to the law. Three classes of mathematical model of processes that might account for such a leading digit distribution have also been investigated. We found that based on the notion of taking the product of many random factors the most credible. This led to the identification of a class of lognormal distributions, those whose shape parameter exceeds 1, which satisfy Benford?s Law. This in turn led us to a novel explanation for the law: that it is fundamentally a consequence of the fact that many physical quantities cannot meaningfully take negative values. This enabled us to develop a simple set of rules for determining whether a given data set is likely to conform to Benford?s Law. Our explanation has an important advantage over previous attempts to account for the law: it also explains which data sets will not have logarithmically distributed leading digits. Some techniques for generating data that satisfy Benford?s law are described and the report concludes with a summary and a discussion of the practical implications.