Statistical significance

There has been a long-term debate about statistical significance testing and the use of p-values. In 2019, The American Statistician published a special issue with an editorial: ‘Moving to a World Beyond “p < 0.05”‘.  Hydrologists are generally dilettantes in the statistical world but there are some important lessons for us.

The single key message from the editorial (Wasserstein et al., 2019) is:

“Statistically significant” – don’t say it and don’t use it.

Some don’ts:

  • Don’t base your conclusion on whether an association or effect was found to be statistically significant
  • Don’t believe that an association or effect exists just because it was statistically significant
  • Don’t believe an association or effect is absent just because it was not statistically significant
  • Don’t conclude anything about the scientific or practical importance of a finding based solely on its statistical significance (or lack therefore).

Instead, the recommendation is: “Accept uncertainty. Be Thoughtful, Open and Modest” (ATOM).

Accepting uncertainty involves accompanying every point estimate by a measure of its uncertainty.  If we recognise and publicise uncertainty it will provide the motivation to do things better.  Embracing uncertainty will highlight the areas where more work is required.  It also helps us to be modest.  Just think of the huge uncertainty in many of the estimates we use in hydrology: losses, flows from extrapolated ratings, 1% flood levels.

Being thoughtful is about carefully planning studies, designing them well and executing them carefully. It’s also understanding the practical use and application of results and considering multiple approaches for solving problems.  It’s being clear about the level of confidence users should have in results.

Openness in reporting is important.  There is a minimum amount of information that should be provided so that others can understand what was done (see the previous post for further discussion on this).  In the scientific community there is a move to providing sufficient information so that others can undertake an alternative analysis of data.

Being modest means understanding and clearly expressing the limitations of your work and not using unwarranted certainty.

Accepting uncertainty, being thoughtful, open and modest is clearly a worthwhile goal, not just for statisticians, but also for hydrologists.

Reference

Wasserstein, R. L., Schirm, A. L. and Lazer, N. A. (2019) Moving to a World Beyond “p < 0.05”.  The American Statistician, 73: sup1, 1-19, DOI:10.1080/00031305.2019.1583913

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s