Type I error for smoke detector: probability of alarm given no fire=0.05
Bayesian: probability of fire given current air data
Frequentist smoke alarm designed as most research is done:
Set the alarm trigger so as to have a 0.8 chance of detecting an inferno
Advantage of actionable evidence quantification:
Set the alarm to trigger when the posterior probability of a fire exceeds 0.02 while at home and at 0.
In my opinion, null hypothesis testing and p-values have done significant harm to science. The purpose of this note is to catalog the many problems caused by p-values. As readers post new problems in their comments, more will be incorporated into the list, so this is a work in progress.
The American Statistical Association has done a great service by issuing its Statement on Statistical Significance and P-values. Now it’s time to act.
In trying to guard against false conclusions, researchers often attempt to minimize the risk of a “false positive” conclusion. In the field of assessing the efficacy of medical and behavioral treatments for improving subjects’ outcomes, falsely concluding that a treatment is effective when it is not is an important consideration. Nowhere is this more important than in the drug and medical device regulatory environments, because a treatment thought not to work can be given a second chance as better data arrive, but a treatment judged to be effective may be approved for marketing, and if later data show that the treatment was actually not effective (or was only trivially effective) it is difficult to remove the treatment from the market if it is safe.