Statistical Significance

What is Statistical Significance?

Statistical significance is the claim that the results or observations from an experiment are due to an underlying cause, rather than chance. Researchers conduct hypothesis testing to determine statistical significance.

Financial analysts often analyze their models to determine if a change in actions will make a statistically significant difference. For example, a business might examine the number of sales leads generated from two different advertisements to see if a change in their advertising provides a statistically significant number of additional positive responses from potential customers.

Hypothesis Testing

Statistical significance is commonly determined through hypothesis testing. The hypothesis is a researcher’s theory or belief about something before testing their theory. It is also referred to as the alternative hypothesis.

The alternative hypothesis contrasts with the null hypothesis. The null hypothesis is that the researcher’s theory is not true, and there is no underlying cause present during an experiment. If testing shows that the researcher’s theory is true, we reject the null hypothesis, and the alternative hypothesis is validated.

Integral to hypothesis testing is the concept of a “p-value.” The p-value is the probability that the observations in testing a hypothesis result from random chance instead of an underlying cause. A higher p-value indicates the higher likelihood the observations were due to change. A lower p-value indicates the higher likelihood the observations were due to a cause theorized in the hypothesis.

The statistically significant range of possible p-values is determined by the researcher. Typically, a p-value of 5% or less indicates statistical significance, rejecting the null hypothesis. In practical English, such a p-value indicates that the observed data demonstrates only a 5% (or lower) probability of being due solely to chance.

Practical Example

Consider a company that wants to test the theory that its stock being mentioned on a certain business television show attracts a statistically significant number of new investors. It may construct an experiment where they arrange to get their stock mentioned on the show every other Tuesday for three months. The company can then compare the number of new investors from Tuesdays on the show with the number of new investors from Tuesdays not on the show.

The number of new investors on Tuesdays when their stock is not mentioned would be taken as the baseline average of new investors. If the average number of new investors on Tuesdays when their stock is mentioned is substantially higher than the baseline average, the company can conclude that the result is statistically significant and that it is to the company’s advantage to arrange for it to be mentioned.

Statistical Significance in Business

All types of businesses commonly conduct experiments to determine statistical significance. Another instance is the action of pharmaceutical companies in drug testing. All drugs are tested to confirm whether someone taking the drug possesses a statistically significant effect in terms of how it affects the medical condition for which the drug is to be prescribed.

Companies that conduct business online often test for statistically significant results from variations in advertising. The variations tested might include the content of ads, where online the ads are run, or what time of day the ads are run. Such testing enables a company to construct and deliver its advertising in a manner that is most effective and efficient in generating sales.

Price variations are another area where businesses conduct research to see whether there is a significant impact on their sales revenues due to changing the price of their product. This research helps the business determine the optimal price level that generates the maximum revenue or profit.

Overall, testing for statistical significance helps the executives of a company make wise decisions concerning how they operate their business and market their products or services.

Keep Learning

In order to help you become a world-class analyst and advance your career to your fullest potential, these additional resources will be very helpful:

• Basic Statistics Concepts in Finance
• Empirical Evidence
• P-Value
• Total Probability Rule
• See all data science resources