Statistical Significance and Practical Significance

When taking decisions, it is important to clearly understand the distinction between statistical significance and practical or market significance.

For instance, let us consider the findings of a product validation test (such as BASES) that indicate, with statistical significance, that a new formulation has the potential to increase a brand’s sales by a million dollars. However, if the anticipated sales gain is insufficient to outweigh the costs associated with introducing the new variant, then the increase in sales lacks the necessary significance to warrant the launch of the variant. It lacks practical significance.

Statistical significance indicates the presence of an effect, whereas practical significance is a measure of how meaningful or important the effect is in the real world.

In another example, a retail bank implemented several initiatives aimed at high-value customers, which led to an increase in their customer satisfaction rating from 3.0 to 3.5 on a 5-point scale. This increase suggests that the initiatives had an impact on customer satisfaction. However, if the p-value for the data is 0.1, it implies that the result is not statistically significant at the standard significance level (α=0.05). In such a case, there is a 10% probability that the observed difference is merely due to sampling error.

To strengthen the confidence in the findings and justify the implementation of these new initiatives, one possible approach would be to increase the sample size. By doing so, if the results become statistically significant, it would raise the level of confidence that the observed difference is indeed “real” and not due to chance alone. Increasing the sample size can help provide stronger evidence of the effectiveness of the initiatives and support the decision to move forward with their implementation.