The Significance of Differences Interval: Assessing the Statistical or Substantive Difference between Two Quantities of Interest

The standard practice in discussing results has shifted from coefficients to substantive quantities of interest. Hypothesis testing nowadays entails computing substantive effects with the 95% CI for alternative scenarios and comparing them. Absent an evaluation of the difference in estimates, current practice often cannot provide a definitive answer. When CIs overlap, the estimates \emph{may or may not} be statistically different. This ambiguity invites mistakes, as analysts turn to ill-advised conjectures to infer whether estimates are distinct. My literature survey indicates this is a widespread problem, with more than half of the articles not providing the evidence required to assess significance of differences. One practical solution is to report instead significance of differences intervals (SDIs), which can be used for direct comparisons. I expand the SDI method to accommodate unpaired sample data, asymmetric distributions, and, for substantive significance, differences larger than zero. I also provide an easy-to-use software to compute SDIs.