I QUIT!

Recently I analysed the famous IBM's (fictitious) HR Attrition data, and joined other 250,000 fellow analysts who have done this previously! In case you missed this analysis along with other things such as watching Titanic, you can find it here, here, here or here.

I would like to discuss about two things in this blog:

1. Use of right performance measure for model evaluation
2. Analysis of human behaviour

1. Right Performance Matrix

I observed that several solutions talk about model accuracy, AUC-ROC etc. as performance measure for the given problem statement. It is well known that Model accuracy is not an apt performance measure parameter for unbalanced data. Model accuracy doesn't help much more than 'pure chance'
or prior probability of the model. That is to say, if the interested target class (attrite employees in current scenario, ~16%) is very small as compared to the other class (~84%), then a simple model with overall accuracy of 85% is just as good as picking a random observation and assigning it to the major class (non-attrite in this case).

Which performance parameter should be used in this case? There is no single answer to this question, and right performance measure depends on the business scenario. Referring back to my previous blog, quality of business decision greatly depends on making a right hypothesis and apt significance level to test. In the given scenario, each individual employee will be evaluated for determining probability of him/her leaving the company, which depends on measured values of several independent variables (salary, age, overtime.....). If the independent variables can be combined in a single function (such as using linear regression or NN), then the function output and probability of employee leaving can be plotted for people who left (target = 1), and people who did not leave (target = 0). In order to predict if a current employee will leave the company or not, the null hypothesis (or status quo) would be "employee will stay with the company", while alternate becomes "employee is looking for change and will resign". Again, significant overlap can be expected between probability distribution curves of the employees who have stayed and who have left the company.

If the model predicts an employee to be leaving the company, but the employee stays, it would be a false positive or false alarm or Type-I error. However, if the model fails to predict about an employee who actually left, this would be a missed opportunity (to retain the employee) or False negative or Type-II error. A correctly identified employee who was planning to leave would be True positive. Now, depending on the cost of making Type-I or Type-II error, a suitable model performance parameter should be used for optimising the model. Let's say, if it takes 2 Hrs of HR time to discuss and understand from a predicted employee reason for his / her possible resignation, vs. it takes 80 Hrs of total time for backfill. In this case, if model predicts several False positive cases (false alarms or high Type-I error) who were not planning to leave the company, then HR would end up spending few extra days in discussions with these false positive employees. However, if the model fails to predict actual positive resignation cases, then the backfill efforts may take weeks or months! Not to mention losing company's IP to competitors.

You may want the model to predict maximum number of True positive cases, even if it produces false positives in doing so. Not only model accuracy, but also AUC-ROC may not give desired results in this case. The most important parameter in this case is sensitivity (or recall or True Positive Rate), i.e. out of total actual positive cases, how many positive cases did the model predict? Other secondary important parameter would be Precision (or positive predicted value), i.e. out of total predicted positive cases, how many are actual positive? While sensitivity (or recall) dictates "how complete the predicted results are?", precision dictates "how relevant the predicted results are?". Details on precision and recall on Wiki are very helpful in understanding.

So which measure encompasses these two measures? F-Score! Given that Sensitivity or recall is more important in given scenario, beta > 1 should be used to calculate F-score, and to compare model performance. If the cost of dealing with False positive increases, then beta value can be lowered and precision to be improved. For e.g. if HR resources are limited, and want to discuss with limited  employees who have very high probability of resigning. Anyway, this filtering and prioritisation can be done even after model prediction, and some of the employees might fall under 'desired attrition' group.

In other business scenario when cost of making Type-I error (False positives, false alarm) is very high, False positive rate (FPR) or False discovery rate (FDR) should be looked into, or further measurements should be done before accepting the positive cases. For e.g., if a False positive results into sending a patient in amputation surgery, or leads to a marriage, more data gathering / tests should be conducted before taking the decision.


2. Analysis of Human Behaviour

After I completed the above analysis, optimising the Precision and Recall of various models, I realised different people-analytics models have something in common. All of these models are trying to predict people's 'behaviour' given environmental conditions, recent & past experiences, demographics, their background (education, family etc.). In a way, these models are trying to predict people's behaviour based on their attitude, which is based on their values, which are based on their beliefs:

source: New Zealand Immigration Advisers Authority

Each of these underlying variables (attitude, values, beliefs) are result of several independent variables as shown above, and these three variables become independent variables along with few others to predict final dependent variable - behaviour. The targeted 'customer behaviour' in different business scenario can have different units: attrition (HR problem), customer churn (services - banking, telecom, healthcare etc.), ad-responders (marketing).

The given problem of employee attrition provides some detail about demographic, and relationship with current & past companies. Similar data when available in other business scenario can help in customer profiling (clustering) as well as predictions (classification). However, what is missing from the given fictitious data is recent behaviour of employees during couple of months before resigning. Such data has shown to be very helpful in churn analysis for service industry. Customers show significant change in service-usage before making a switch to other service provider, and this sudden change in usage pattern can help predict and hopefully prevent a losing customer. If similar data is monitored about employee's engagement each month, a sudden change in employee's behaviour would help predict an unsatisfied employee, who might be looking for external options. This is to say, rather than just focusing on the customer/employee profile, companies should also monitor how the customer/employee is recently behaving to understand if the customer/employee is going to leave your services/company.

For e.g. any organisation or telecom company or a bank will have employees / customers with varied backgrounds (age, salary, marital status etc.) and it may become very difficult to classify  the employees / customers based on such static variables. However, employees or customers who are planning to leave the organisation / service provider might start acting differently than they used to act previously, or act similarly to others who had left earlier. And hence, a comprehensive HR analytics would require not only collecting static (during joining) or quasi-static (yearly) data about employees, but dynamic data (weekly or monthly) for improved prediction. Sometimes even couple of weeks of heads-up is good enough for HR to intervene and take corrective actions to retain high value employees.

I DO!

Ever wondered, why we don't say "data is telling the truth"? Rather we say, "data doesn't lie". Because that's how Hypothesis testing is performed!

But I believe in another known hypothesis, that "Data doesn't lie, People with data lie". I have been working on the framework to Reject my null hypothesis, H0 = People with data don't lie; it will need a lot of data of previous tests to test the hypothesis, or rather perform meta-analysis.

In this blog, I want to discuss why is it important to understand and identify error types in hypothesis testing; not able to identify (intentionally or unintentionally)  correct error types in a given business scenario leads to "lying" with data.

To start with, first it is important to understand how a hypothesis testing should be formulated. Three things which are critical in Hypothesis testing: Null Hypothesis (H0) - status quo, Alternate Hypothesis (H1) - Desired result, and significance level (alpha) - when do you want to reject the null (and hence end up accepting alternate).

Several decisions we make in personal or professional life are based on probability, even if we don't work out the math behind it (stats to be specific!). Let's say two data scientists working together realise their mutual feelings for each other and start dating. Eventually, they come across the situation to decide to get married or not. Social norms suggest that the terms to get married are based on mutual emotions, feelings, understanding and of course - love!

However, getting married being such an important decision in life, which requires: relationship update announcement to friends and family, commitment to each other, future planning, and expenses towards getting married. In this scenario, since getting married 'changes' the current relationship, and requires significant efforts, commitment and money, the couple wants to ensure that likelihood of this decision going wrong is minimised. The status quo in this case would be: there is no significant change in their mutual feelings for each other and the couple continues with their relationship as it is; this is Null hypothesis. And hence, the alternate hypothesis or desired result is that there is a significant change in their mutual feelings and their feelings can't be justified anymore with the boyfriend-girlfriend relationship.

Let's understand the importance of deciding the Null and Alternate hypothesis here. We will assume that 'liking' each other can justify being in a relationship, while certain degree of 'love' is required to form a marriage. Both 'like' and 'love' are defined by level of mutual feelings; 'love' requiring higher level of positive feelings. Just to be able to convert this into a statistical problem, we will consider 'like' and 'love' as mutually exclusive events, i.e., a couple shifts from being 'liking' each other to being in 'love', then it will not be counted as a 'like' event, and will be counted only as 'love' event. Although people in 'love', or married still 'like' each other (or do they??), for further explanation of concepts this will not be considered.

Feeling levels for 'like' and 'love' are different for different couples, i.e., some couples finding each other good looking may be ready to commit for marriage, while others might want to maintain just a casual relationship even after their first child together (Ross and Rachel)! If we collect data for thousands of couples about their feeling-level for 'like' and 'love' we will get distribution of feeling-level for the two events. The distribution of feeling-level for 'like' and 'love' can be plotted as normal distributions. Feeling-level (on x-scale) can start with: 0 - I don't hate you, 1 - I find you good looking, 2 - I enjoy talking to you .......... 8 - It is a great experience to raise a kid with you, 9 - I want to have more kids with you and can live on a deserted island with our little family, 10 - I love your parents and they love me back! Y-axis will show probability of 'liking' or 'loving' each other. Now, note here theoretically there can be couple which is at feeling level '8' but still defines their relationship as 'casually seeing each other'. However the probability of such a couple existing is so very low that it may not even exists in the given data set. Most couples who are 'in relationship' would have mutual feelings around 3-5, while most married couples would have mutual feelings around 6-8.

Since I am not a psychologist, the scale defined above is less likely to be correct. But whatever the levels be, there will be an overlap between the two normal distribution curves, that is to say, there are certain levels (around 4-6) which may fall under 'like' curve for certain couples and 'love' curve for other couples. This overlap is of great importance, since this overlap gives rise to all types of error, and bad decisions. Coming back to our data scientists, the guy wants to evaluate if they have sufficient mutual feelings for each other to qualify for being in 'love' and hence get married. So he gets down on his knee and pops the question, "I have felt increase in our mutual feelings and need to evaluate if we should get married. I propose the null hypothesis that the increased mutual feeling-level in not significant and is caused by chance, and hence the status-quo of just being 'in-relationship' should continue."

Here, the girl can't simply accept the proposal, rather in order for her to 'accept' the marriage proposal, (or the alternate hypothesis), she needs to ensure that the measured increase in feeling-level has very low probability of lying under 'like' distribution curve of feelings. This 'very-low' probability of measured feeling-level in 'like' curve (null hypothesis) is called 'significance level' or alpha. So the girl starts evaluating and realises that they enjoy talking to each other, spending time with each other, enjoy vacations with each other, have no issues living under the same roof, have same taste, get along with each other's friends, have similar plans for future in terms of work, health and family, and many more similarities. She goes to her friends and says, "He proposed me the null hypothesis!", and the friends ask "what did you say?", and she says, "I rejected, I rejected the null with 95% confidence!!"

The above scenario ensures that decision of moving into a new status, which requires significant effort and money, is not based on observations which occurred by chance, but are caused by a significant change in feeling-levels (or any other measured attribute or independent variable). The final outcome of 'like' or 'love' is dependent variable or target variable. In this case, if the girl wants to be 'really sure' that she is in 'love' [distribution], she will keep the significance level (alpha) to be very low, that means, she will say, "I will reject the null hypothesis of being in 'like' distribution only if the feelings level is more than 7". There is still a chance (probability) that the mutual feelings of 8-10 represent 'liking' to each other and not necessarily 'love', but the probability is very small, and that is the 'risk' the girl is ready to take.

If the measurements suggest that the feeling level is 8, and the couple ends up getting married, and if they live 'happily ever after', this would be classified as True Positive. However, if the measured value 8 actually belonged to 'like' distribution (which the couple would realise in few years after marriage!), this would be classified as 'False positive' or Type-I error.

On the other hand, if the girl wants to further minimise risk of Type-I error, she would keep the threshold at level 8 rather than 7. In this case, measured value of 8 would be considered in the 'like' distribution, and the girl will not have sufficient reason to reject the null hypothesis. If the measured value of 8 was actually by chance, and the real 'love' did not exist between two, not getting married would be a True-negative. However, as in case of Ross/Rachel an observation of 8 (living together, raising a kid) being considered casual, and can still 'see other people', not realising how much they actually 'love' each other is a "missed opportunity" of getting married, or False Negative or Type-II error.

If this sounds confusing, don't worry, they call it 'confusion matrix' on purpose.

Depending on business scenario, cost of making Type-I error (following false positive or false alarm) vs. Type-II error (cost of missed opportunity), the alpha level is selected. But making an error is unavoidable, and this has to be decided before the hypothesis testing is conducted.

Taking a decision with a given risk of making Type-I or Type-II error is acceptable, as long as the hypothesis is formulated correctly. But when I mentioned, 'people with data lie', this relates to how people use statistics or hypothesis testing to manipulate the results to (mis)guide decisions.

What if the data scientist in 'love' formulated the proposal as, "I have sufficient evidence that we are in 'love', and hence propose to get married." In this scenario, the measured feeling level between 4-10 would be in 'love' distribution with probability of >5% (alpha level 0.05 or confidence interval 0.95). Any measured value >= 4 will lead to decision of getting married. Even with such small values of feeling-level, the girl will not be able to 'reject the hypothesis' of being in 'love'. The girl would reason, "I like spending weekends with the guy, there is 6% chance that this means 'love', so I can't completely reject that I love this guy".

This time she goes to her friends, and says, "He proposed for marriage, and I failed to reject!!". And we get Ross/Carole!

People might want to use such hypothesis to prove certain 'changes', and say the test was run with 95% confidence! But as business leaders (when it is proposed by managers) or common people (when it is proposed by media) we need to understand the formulation of hypothesis, both null and alternate, before feeling great about the 'confidence level'.

Assuming the hypothesis are formulated correctly, the next step would be to understand the different performance measures from the confusion matrix, and using the right measure. But let's keep that for the next blog.