I DO!

Ever wondered, why we don't say "data is telling the truth"? Rather we say, "data doesn't lie". Because that's how Hypothesis testing is performed!

But I believe in another known hypothesis, that "Data doesn't lie, People with data lie". I have been working on the framework to Reject my null hypothesis, H0 = People with data don't lie; it will need a lot of data of previous tests to test the hypothesis, or rather perform meta-analysis.

In this blog, I want to discuss why is it important to understand and identify error types in hypothesis testing; not able to identify (intentionally or unintentionally)  correct error types in a given business scenario leads to "lying" with data.

To start with, first it is important to understand how a hypothesis testing should be formulated. Three things which are critical in Hypothesis testing: Null Hypothesis (H0) - status quo, Alternate Hypothesis (H1) - Desired result, and significance level (alpha) - when do you want to reject the null (and hence end up accepting alternate).

Several decisions we make in personal or professional life are based on probability, even if we don't work out the math behind it (stats to be specific!). Let's say two data scientists working together realise their mutual feelings for each other and start dating. Eventually, they come across the situation to decide to get married or not. Social norms suggest that the terms to get married are based on mutual emotions, feelings, understanding and of course - love!

However, getting married being such an important decision in life, which requires: relationship update announcement to friends and family, commitment to each other, future planning, and expenses towards getting married. In this scenario, since getting married 'changes' the current relationship, and requires significant efforts, commitment and money, the couple wants to ensure that likelihood of this decision going wrong is minimised. The status quo in this case would be: there is no significant change in their mutual feelings for each other and the couple continues with their relationship as it is; this is Null hypothesis. And hence, the alternate hypothesis or desired result is that there is a significant change in their mutual feelings and their feelings can't be justified anymore with the boyfriend-girlfriend relationship.

Let's understand the importance of deciding the Null and Alternate hypothesis here. We will assume that 'liking' each other can justify being in a relationship, while certain degree of 'love' is required to form a marriage. Both 'like' and 'love' are defined by level of mutual feelings; 'love' requiring higher level of positive feelings. Just to be able to convert this into a statistical problem, we will consider 'like' and 'love' as mutually exclusive events, i.e., a couple shifts from being 'liking' each other to being in 'love', then it will not be counted as a 'like' event, and will be counted only as 'love' event. Although people in 'love', or married still 'like' each other (or do they??), for further explanation of concepts this will not be considered.

Feeling levels for 'like' and 'love' are different for different couples, i.e., some couples finding each other good looking may be ready to commit for marriage, while others might want to maintain just a casual relationship even after their first child together (Ross and Rachel)! If we collect data for thousands of couples about their feeling-level for 'like' and 'love' we will get distribution of feeling-level for the two events. The distribution of feeling-level for 'like' and 'love' can be plotted as normal distributions. Feeling-level (on x-scale) can start with: 0 - I don't hate you, 1 - I find you good looking, 2 - I enjoy talking to you .......... 8 - It is a great experience to raise a kid with you, 9 - I want to have more kids with you and can live on a deserted island with our little family, 10 - I love your parents and they love me back! Y-axis will show probability of 'liking' or 'loving' each other. Now, note here theoretically there can be couple which is at feeling level '8' but still defines their relationship as 'casually seeing each other'. However the probability of such a couple existing is so very low that it may not even exists in the given data set. Most couples who are 'in relationship' would have mutual feelings around 3-5, while most married couples would have mutual feelings around 6-8.

Since I am not a psychologist, the scale defined above is less likely to be correct. But whatever the levels be, there will be an overlap between the two normal distribution curves, that is to say, there are certain levels (around 4-6) which may fall under 'like' curve for certain couples and 'love' curve for other couples. This overlap is of great importance, since this overlap gives rise to all types of error, and bad decisions. Coming back to our data scientists, the guy wants to evaluate if they have sufficient mutual feelings for each other to qualify for being in 'love' and hence get married. So he gets down on his knee and pops the question, "I have felt increase in our mutual feelings and need to evaluate if we should get married. I propose the null hypothesis that the increased mutual feeling-level in not significant and is caused by chance, and hence the status-quo of just being 'in-relationship' should continue."

Here, the girl can't simply accept the proposal, rather in order for her to 'accept' the marriage proposal, (or the alternate hypothesis), she needs to ensure that the measured increase in feeling-level has very low probability of lying under 'like' distribution curve of feelings. This 'very-low' probability of measured feeling-level in 'like' curve (null hypothesis) is called 'significance level' or alpha. So the girl starts evaluating and realises that they enjoy talking to each other, spending time with each other, enjoy vacations with each other, have no issues living under the same roof, have same taste, get along with each other's friends, have similar plans for future in terms of work, health and family, and many more similarities. She goes to her friends and says, "He proposed me the null hypothesis!", and the friends ask "what did you say?", and she says, "I rejected, I rejected the null with 95% confidence!!"

The above scenario ensures that decision of moving into a new status, which requires significant effort and money, is not based on observations which occurred by chance, but are caused by a significant change in feeling-levels (or any other measured attribute or independent variable). The final outcome of 'like' or 'love' is dependent variable or target variable. In this case, if the girl wants to be 'really sure' that she is in 'love' [distribution], she will keep the significance level (alpha) to be very low, that means, she will say, "I will reject the null hypothesis of being in 'like' distribution only if the feelings level is more than 7". There is still a chance (probability) that the mutual feelings of 8-10 represent 'liking' to each other and not necessarily 'love', but the probability is very small, and that is the 'risk' the girl is ready to take.

If the measurements suggest that the feeling level is 8, and the couple ends up getting married, and if they live 'happily ever after', this would be classified as True Positive. However, if the measured value 8 actually belonged to 'like' distribution (which the couple would realise in few years after marriage!), this would be classified as 'False positive' or Type-I error.

On the other hand, if the girl wants to further minimise risk of Type-I error, she would keep the threshold at level 8 rather than 7. In this case, measured value of 8 would be considered in the 'like' distribution, and the girl will not have sufficient reason to reject the null hypothesis. If the measured value of 8 was actually by chance, and the real 'love' did not exist between two, not getting married would be a True-negative. However, as in case of Ross/Rachel an observation of 8 (living together, raising a kid) being considered casual, and can still 'see other people', not realising how much they actually 'love' each other is a "missed opportunity" of getting married, or False Negative or Type-II error.

If this sounds confusing, don't worry, they call it 'confusion matrix' on purpose.

Depending on business scenario, cost of making Type-I error (following false positive or false alarm) vs. Type-II error (cost of missed opportunity), the alpha level is selected. But making an error is unavoidable, and this has to be decided before the hypothesis testing is conducted.

Taking a decision with a given risk of making Type-I or Type-II error is acceptable, as long as the hypothesis is formulated correctly. But when I mentioned, 'people with data lie', this relates to how people use statistics or hypothesis testing to manipulate the results to (mis)guide decisions.

What if the data scientist in 'love' formulated the proposal as, "I have sufficient evidence that we are in 'love', and hence propose to get married." In this scenario, the measured feeling level between 4-10 would be in 'love' distribution with probability of >5% (alpha level 0.05 or confidence interval 0.95). Any measured value >= 4 will lead to decision of getting married. Even with such small values of feeling-level, the girl will not be able to 'reject the hypothesis' of being in 'love'. The girl would reason, "I like spending weekends with the guy, there is 6% chance that this means 'love', so I can't completely reject that I love this guy".

This time she goes to her friends, and says, "He proposed for marriage, and I failed to reject!!". And we get Ross/Carole!

People might want to use such hypothesis to prove certain 'changes', and say the test was run with 95% confidence! But as business leaders (when it is proposed by managers) or common people (when it is proposed by media) we need to understand the formulation of hypothesis, both null and alternate, before feeling great about the 'confidence level'.

Assuming the hypothesis are formulated correctly, the next step would be to understand the different performance measures from the confusion matrix, and using the right measure. But let's keep that for the next blog.