'The future depends on what you do in the present'

 – Mahatma Gandhi

There was a saying a couple of decades ago, “Everyone wants to follow [Mahatma] Gandhi, but no one wants to be [Mahatma] Gandhi”.  Although the saying may not be relevant today, given so many people, including Entrepreneurs, business leaders, and politicians are working towards creating a Personal Brand, a different interpretation of the saying can still help the society.  If the ‘saying’ was scalable, it may drive a complete nation (such as India) to become a ‘leader’, rather than following those which are decades ahead.

Several people in India plan to travel and long to settle in developed countries, which not so long ago, were at a similar economic development level as India.  It is not entirely a bad thing for our country, but an opportunity to understand our shortcomings and make an improvement plan for the nation.

During our recent vacation in Singapore my wife lost her phone in public places with hundreds of people around, twice in a single day; both times we got it back within 30 minutes! Very small sample size to conclude anything, but the probability of that [safe return of phone] happening in India would not be significant. There are several other societies which are developed on the principles of honesty and/or orderliness, with minimum variance in processes, and chaos being confined to a manageable level.  So, what resources these societies harnessed to achieve this development, and do we have these resources in India to follow the same path?

One may argue that some nations have/had resources which India may/does not have: Oil, sea-ports, gold, minerals, and now rare earth magnets! However, I am biased towards one particular resource which could play a significant role in a nation’s development: people’s emotions. Every society has this resource, but needs to harness it suitably: either in favor of few people or in favor of the nation.  So what are ‘Emotions’? As per Elisabeth Kübler-Ross, a Swiss-American psychiatrist human emotions consist of either ‘Love’ or ‘Fear’.  People do what they do for either of these two emotions. Even Maslov’s Hierarchy of Needs can be related to these emotions as:

• Base two layers ‘Physiological’ and ‘Safety’ Needs guided by the emotion of Fear [of extinction]
• Top three layers of ‘Love / Belonging’, ‘Esteem’ and ‘Self Actualization’ guided by Love [for friends/family/society, society’s perception of someone, own perception of oneself and finally for the spiritual self]

As per Maslov, unless the base two layers (guided by Fear) are not satisfied, one has the least motivation to work towards the top three layers.

Several mythological (or real) epics such as Mahabharata and Ramayana discuss social structures that were well developed, stable and satisfied basic needs of most participants. In such societies perhaps people would have followed their hearts and try to achieve what they ‘Love’ – leading to the efficient growth of society. Not debating if these epics were real or myths, but such social structures were rare to be observed for the documented part of the last millennia. Most societies lived in the fear of frequent wars, posing a challenge to our existence. Out of the need for existence (or fear of extinction), several leaders worked towards social developments to satisfy basic needs and retain the strength to survive. Some might also be guided by the Love towards Humanity and desire for a Better Life.

Back to the present, most of the developed nations, be it communist, monarchies or city-sized democracy have harnessed the emotion of Fear among citizens to some level and leveraged it to bring law & order in their society. This law & order is based on unambiguous rules, surveillance, detailed record-keeping, strict punishments and sometimes even controlled cash-flows. Once the basic needs and safety of the subjects is established, the government provides for facilities which the citizens ‘Love’; and the people in the developing nations such as India, long for. People born and brought up within these societies may not perceive how the governments have harnessed the power of Fear to retain social order, since such policies are typically introduced slowly leading to a quasi-static change. However, for an outsider coming from a developing nation, these changes are sudden or rather lead to a [cultural] shock. The question is how this can be implemented in India?

Creating Fear to Establish Law & Order in India

The above-discussed concepts are indeed well known to most leaders and have been applied in every society from time to time; India is no different. It’s just that when such changes are proposed by the government, several other social groups/leaders portray these changes as unnecessary, harsh, illogical, and even non-constitutional. Their actions are simply driven by the ‘Fear’ of losing their power over the public. Let us analyze a few of the recent actions by the government, which might be guided by the need to create orderliness in the system.

Demonetization:

The demonetization was carried out, with the ‘hope’ (or at least the government wanted us to believe) that the black money would be ‘caught’. The intentions and consequences always remained debatable. Not commenting on the efficiency of the process or the success of this project, two things which were achieved through this were: 

1. Fear that the government is capable of such drastic measures
2. Wide acceptance of UPI and electronic payments

At this point, it is worth reiterating “The future depends on what you do in the present”. This can also be interpreted as “What you do in the present, affects how the future will be shaped”. Hence, your Present actions can not change the Past, but can only help shape the Future. Was catching black-money ever an intention of demonetization? Or was it done to create the Fear, leading to the easy implementation of technological advancement? This not only will help the public in the future but will also enable the government to track cash flow in society.

Will people stop generating/using black money? Of course not. But large population shifting to electronic/traceable/legitimate transaction methods, the efforts will be much less for the government to track anomalies in cash-flow and hence, a higher probability of catching illegitimate transactions.

CAA / NRC:

This being a very sensitive topic, rather than getting into the details or discussing the moral implications, let's analyze it only at the surface. Government’s [said] intention of catching the illegal immigrants or people’s perception that a particular class of the society will suffer, may not be the actual intention or outcome of this move.

It is not a secret that every society faces infiltration from neighboring societies with relatively fewer opportunities. If ‘Fear’ is a form of energy driving people’s behavior, people will flee from a society with high ‘Fear’ to one with low ‘Fear’ – as per the second law of thermodynamics. Please note, the legal migration of wealthy & worthy not necessarily follow this analogy; the wealthy & worthy would have crossed the two base layers of Maslov’s pyramid and are in pursuit of happiness.  To curb the illegal infiltration, some countries want to build walls, while ours wants to create a system. With this system, will the government catch all or even most illegal immigrants currently in the system? Perhaps not. If influential people could figure out ways to legitimize each & every currency bill during demonetization, the [illegal] immigrants are perhaps worth much more effort.

However, this should create Fear in the society, that once again the government is capable of drastic measures. When the current population complies with the new system, creating a traceable database for the government, and it will be much easier in the future to track and contain illegal infiltration. Once again, the efforts here are not intended to change what has happened in the past, but to help shape a better future.

Yet, similar to demonetization, this step of government has seen a lot of opposition and criticism, by those who ‘Fear’ to lose their power.

FASTag

It is very surprising that government’s moves such as demonetization and CAA have been widely and openly opposed, but no one has come out on the roads to oppose FASTag – on the contrary, they have complied with FASTag to come out on the road! If this doesn’t sound serious, read here how FASTag helped to catch a stolen vehicle. In developed countries typically all vehicles have some sort of RF identifier, which is useful for toll collection, pulling service records, vehicle identification in case of emergency, etc.; the WIIFM for the end-user. However, in places like Singapore, where the whole city is full of such RF receivers, the government could (IF they already don’t) use it to track any given vehicle, and hence the user. India is currently implementing FASTag for toll collection on the highways, but soon it will be implemented in the cities to track traffic violations.

So, when FASTag will be used to track non-stolen cars, and the cab booking apps will be required to share user data with the government for security purposes (such as a cab booking platform used to track user location even AFTER the cab ride), most of us could be tracked. People thought that demonetization and CAA are bad, because the government was trying to catch your past sins and where you came from, but with FASTag the government will catch where you are going! Imagine yourself at the immigration counter of Suvarnabhumi Airport, which of the following questions will make you most uncomfortable:

1. Where are you coming from?
2. How much Baht are you carrying?
3. What places are you going to visit?

Information on where you came from, hence knowing your sanctuary, what are you spending on, and where are you traveling, the power of this data is only limited by your imagination! All developed countries, even those who promote immigration, ensure you live in the country to generate sufficient digital trail, and analyze it before saying ‘we accept her one of us’.

What are our options?

Firstly, one should realize that the government is seeking only a fraction of data that you have already shared with FAAMG. With all its efforts, the government can not know as many details about individuals as these tech companies already know. How the data will be used (end result) may justify the means.

Singapore police proudly say that they can track Anyone in Singapore – it’s a routine task. And when Indian police track a fleeing criminal, we make an episode of Crime Patrol. No offense to Indian police or other armed forces. I have very high regard for them, that they carry out such tough tasks with minimal information, risking their own lives, while their counterparts in developed countries have all the data at their disposal.

Travel and immigration on Indian passport require so much scrutiny because India does not have strict laws to check illegal infiltration in the country. If our own systems were air-tight, our citizenship would be valued more internationally.

Money laundering and cash transaction have led to tax evasion. While 2% of the total population pays tax, many others reap the benefits of the system. If people are afraid of hoarding money and are open to paying taxes for keeping their money in the banks, it not only reduces the burden on the taxpayers but also provides more money in the system for circulation. On being accused of involvement in demonetization policy by an [not-so] Aam-Aadmi, Mr. Vijay Shekhar Sharma aptly replied, “Dear Sir, the biggest beneficiary is our country.”

The choices we have are: slow march on the roads or use the FAST-lane to zoom through the barriers. We will be paying the price for the development of the nation in terms of our data, but aren’t we paying it anyway in one way or another for the past 70 years?

I QUIT!

Recently I analysed the famous IBM's (fictitious) HR Attrition data, and joined other 250,000 fellow analysts who have done this previously! In case you missed this analysis along with other things such as watching Titanic, you can find it here, here, here or here.

I would like to discuss about two things in this blog:

1. Use of right performance measure for model evaluation
2. Analysis of human behaviour

1. Right Performance Matrix

I observed that several solutions talk about model accuracy, AUC-ROC etc. as performance measure for the given problem statement. It is well known that Model accuracy is not an apt performance measure parameter for unbalanced data. Model accuracy doesn't help much more than 'pure chance'
or prior probability of the model. That is to say, if the interested target class (attrite employees in current scenario, ~16%) is very small as compared to the other class (~84%), then a simple model with overall accuracy of 85% is just as good as picking a random observation and assigning it to the major class (non-attrite in this case).

Which performance parameter should be used in this case? There is no single answer to this question, and right performance measure depends on the business scenario. Referring back to my previous blog, quality of business decision greatly depends on making a right hypothesis and apt significance level to test. In the given scenario, each individual employee will be evaluated for determining probability of him/her leaving the company, which depends on measured values of several independent variables (salary, age, overtime.....). If the independent variables can be combined in a single function (such as using linear regression or NN), then the function output and probability of employee leaving can be plotted for people who left (target = 1), and people who did not leave (target = 0). In order to predict if a current employee will leave the company or not, the null hypothesis (or status quo) would be "employee will stay with the company", while alternate becomes "employee is looking for change and will resign". Again, significant overlap can be expected between probability distribution curves of the employees who have stayed and who have left the company.

If the model predicts an employee to be leaving the company, but the employee stays, it would be a false positive or false alarm or Type-I error. However, if the model fails to predict about an employee who actually left, this would be a missed opportunity (to retain the employee) or False negative or Type-II error. A correctly identified employee who was planning to leave would be True positive. Now, depending on the cost of making Type-I or Type-II error, a suitable model performance parameter should be used for optimising the model. Let's say, if it takes 2 Hrs of HR time to discuss and understand from a predicted employee reason for his / her possible resignation, vs. it takes 80 Hrs of total time for backfill. In this case, if model predicts several False positive cases (false alarms or high Type-I error) who were not planning to leave the company, then HR would end up spending few extra days in discussions with these false positive employees. However, if the model fails to predict actual positive resignation cases, then the backfill efforts may take weeks or months! Not to mention losing company's IP to competitors.

You may want the model to predict maximum number of True positive cases, even if it produces false positives in doing so. Not only model accuracy, but also AUC-ROC may not give desired results in this case. The most important parameter in this case is sensitivity (or recall or True Positive Rate), i.e. out of total actual positive cases, how many positive cases did the model predict? Other secondary important parameter would be Precision (or positive predicted value), i.e. out of total predicted positive cases, how many are actual positive? While sensitivity (or recall) dictates "how complete the predicted results are?", precision dictates "how relevant the predicted results are?". Details on precision and recall on Wiki are very helpful in understanding.

So which measure encompasses these two measures? F-Score! Given that Sensitivity or recall is more important in given scenario, beta > 1 should be used to calculate F-score, and to compare model performance. If the cost of dealing with False positive increases, then beta value can be lowered and precision to be improved. For e.g. if HR resources are limited, and want to discuss with limited  employees who have very high probability of resigning. Anyway, this filtering and prioritisation can be done even after model prediction, and some of the employees might fall under 'desired attrition' group.

In other business scenario when cost of making Type-I error (False positives, false alarm) is very high, False positive rate (FPR) or False discovery rate (FDR) should be looked into, or further measurements should be done before accepting the positive cases. For e.g., if a False positive results into sending a patient in amputation surgery, or leads to a marriage, more data gathering / tests should be conducted before taking the decision.


2. Analysis of Human Behaviour

After I completed the above analysis, optimising the Precision and Recall of various models, I realised different people-analytics models have something in common. All of these models are trying to predict people's 'behaviour' given environmental conditions, recent & past experiences, demographics, their background (education, family etc.). In a way, these models are trying to predict people's behaviour based on their attitude, which is based on their values, which are based on their beliefs:

source: New Zealand Immigration Advisers Authority

Each of these underlying variables (attitude, values, beliefs) are result of several independent variables as shown above, and these three variables become independent variables along with few others to predict final dependent variable - behaviour. The targeted 'customer behaviour' in different business scenario can have different units: attrition (HR problem), customer churn (services - banking, telecom, healthcare etc.), ad-responders (marketing).

The given problem of employee attrition provides some detail about demographic, and relationship with current & past companies. Similar data when available in other business scenario can help in customer profiling (clustering) as well as predictions (classification). However, what is missing from the given fictitious data is recent behaviour of employees during couple of months before resigning. Such data has shown to be very helpful in churn analysis for service industry. Customers show significant change in service-usage before making a switch to other service provider, and this sudden change in usage pattern can help predict and hopefully prevent a losing customer. If similar data is monitored about employee's engagement each month, a sudden change in employee's behaviour would help predict an unsatisfied employee, who might be looking for external options. This is to say, rather than just focusing on the customer/employee profile, companies should also monitor how the customer/employee is recently behaving to understand if the customer/employee is going to leave your services/company.

For e.g. any organisation or telecom company or a bank will have employees / customers with varied backgrounds (age, salary, marital status etc.) and it may become very difficult to classify  the employees / customers based on such static variables. However, employees or customers who are planning to leave the organisation / service provider might start acting differently than they used to act previously, or act similarly to others who had left earlier. And hence, a comprehensive HR analytics would require not only collecting static (during joining) or quasi-static (yearly) data about employees, but dynamic data (weekly or monthly) for improved prediction. Sometimes even couple of weeks of heads-up is good enough for HR to intervene and take corrective actions to retain high value employees.

I DO!

Ever wondered, why we don't say "data is telling the truth"? Rather we say, "data doesn't lie". Because that's how Hypothesis testing is performed!

But I believe in another known hypothesis, that "Data doesn't lie, People with data lie". I have been working on the framework to Reject my null hypothesis, H0 = People with data don't lie; it will need a lot of data of previous tests to test the hypothesis, or rather perform meta-analysis.

In this blog, I want to discuss why is it important to understand and identify error types in hypothesis testing; not able to identify (intentionally or unintentionally)  correct error types in a given business scenario leads to "lying" with data.

To start with, first it is important to understand how a hypothesis testing should be formulated. Three things which are critical in Hypothesis testing: Null Hypothesis (H0) - status quo, Alternate Hypothesis (H1) - Desired result, and significance level (alpha) - when do you want to reject the null (and hence end up accepting alternate).

Several decisions we make in personal or professional life are based on probability, even if we don't work out the math behind it (stats to be specific!). Let's say two data scientists working together realise their mutual feelings for each other and start dating. Eventually, they come across the situation to decide to get married or not. Social norms suggest that the terms to get married are based on mutual emotions, feelings, understanding and of course - love!

However, getting married being such an important decision in life, which requires: relationship update announcement to friends and family, commitment to each other, future planning, and expenses towards getting married. In this scenario, since getting married 'changes' the current relationship, and requires significant efforts, commitment and money, the couple wants to ensure that likelihood of this decision going wrong is minimised. The status quo in this case would be: there is no significant change in their mutual feelings for each other and the couple continues with their relationship as it is; this is Null hypothesis. And hence, the alternate hypothesis or desired result is that there is a significant change in their mutual feelings and their feelings can't be justified anymore with the boyfriend-girlfriend relationship.

Let's understand the importance of deciding the Null and Alternate hypothesis here. We will assume that 'liking' each other can justify being in a relationship, while certain degree of 'love' is required to form a marriage. Both 'like' and 'love' are defined by level of mutual feelings; 'love' requiring higher level of positive feelings. Just to be able to convert this into a statistical problem, we will consider 'like' and 'love' as mutually exclusive events, i.e., a couple shifts from being 'liking' each other to being in 'love', then it will not be counted as a 'like' event, and will be counted only as 'love' event. Although people in 'love', or married still 'like' each other (or do they??), for further explanation of concepts this will not be considered.

Feeling levels for 'like' and 'love' are different for different couples, i.e., some couples finding each other good looking may be ready to commit for marriage, while others might want to maintain just a casual relationship even after their first child together (Ross and Rachel)! If we collect data for thousands of couples about their feeling-level for 'like' and 'love' we will get distribution of feeling-level for the two events. The distribution of feeling-level for 'like' and 'love' can be plotted as normal distributions. Feeling-level (on x-scale) can start with: 0 - I don't hate you, 1 - I find you good looking, 2 - I enjoy talking to you .......... 8 - It is a great experience to raise a kid with you, 9 - I want to have more kids with you and can live on a deserted island with our little family, 10 - I love your parents and they love me back! Y-axis will show probability of 'liking' or 'loving' each other. Now, note here theoretically there can be couple which is at feeling level '8' but still defines their relationship as 'casually seeing each other'. However the probability of such a couple existing is so very low that it may not even exists in the given data set. Most couples who are 'in relationship' would have mutual feelings around 3-5, while most married couples would have mutual feelings around 6-8.

Since I am not a psychologist, the scale defined above is less likely to be correct. But whatever the levels be, there will be an overlap between the two normal distribution curves, that is to say, there are certain levels (around 4-6) which may fall under 'like' curve for certain couples and 'love' curve for other couples. This overlap is of great importance, since this overlap gives rise to all types of error, and bad decisions. Coming back to our data scientists, the guy wants to evaluate if they have sufficient mutual feelings for each other to qualify for being in 'love' and hence get married. So he gets down on his knee and pops the question, "I have felt increase in our mutual feelings and need to evaluate if we should get married. I propose the null hypothesis that the increased mutual feeling-level in not significant and is caused by chance, and hence the status-quo of just being 'in-relationship' should continue."

Here, the girl can't simply accept the proposal, rather in order for her to 'accept' the marriage proposal, (or the alternate hypothesis), she needs to ensure that the measured increase in feeling-level has very low probability of lying under 'like' distribution curve of feelings. This 'very-low' probability of measured feeling-level in 'like' curve (null hypothesis) is called 'significance level' or alpha. So the girl starts evaluating and realises that they enjoy talking to each other, spending time with each other, enjoy vacations with each other, have no issues living under the same roof, have same taste, get along with each other's friends, have similar plans for future in terms of work, health and family, and many more similarities. She goes to her friends and says, "He proposed me the null hypothesis!", and the friends ask "what did you say?", and she says, "I rejected, I rejected the null with 95% confidence!!"

The above scenario ensures that decision of moving into a new status, which requires significant effort and money, is not based on observations which occurred by chance, but are caused by a significant change in feeling-levels (or any other measured attribute or independent variable). The final outcome of 'like' or 'love' is dependent variable or target variable. In this case, if the girl wants to be 'really sure' that she is in 'love' [distribution], she will keep the significance level (alpha) to be very low, that means, she will say, "I will reject the null hypothesis of being in 'like' distribution only if the feelings level is more than 7". There is still a chance (probability) that the mutual feelings of 8-10 represent 'liking' to each other and not necessarily 'love', but the probability is very small, and that is the 'risk' the girl is ready to take.

If the measurements suggest that the feeling level is 8, and the couple ends up getting married, and if they live 'happily ever after', this would be classified as True Positive. However, if the measured value 8 actually belonged to 'like' distribution (which the couple would realise in few years after marriage!), this would be classified as 'False positive' or Type-I error.

On the other hand, if the girl wants to further minimise risk of Type-I error, she would keep the threshold at level 8 rather than 7. In this case, measured value of 8 would be considered in the 'like' distribution, and the girl will not have sufficient reason to reject the null hypothesis. If the measured value of 8 was actually by chance, and the real 'love' did not exist between two, not getting married would be a True-negative. However, as in case of Ross/Rachel an observation of 8 (living together, raising a kid) being considered casual, and can still 'see other people', not realising how much they actually 'love' each other is a "missed opportunity" of getting married, or False Negative or Type-II error.

If this sounds confusing, don't worry, they call it 'confusion matrix' on purpose.

Depending on business scenario, cost of making Type-I error (following false positive or false alarm) vs. Type-II error (cost of missed opportunity), the alpha level is selected. But making an error is unavoidable, and this has to be decided before the hypothesis testing is conducted.

Taking a decision with a given risk of making Type-I or Type-II error is acceptable, as long as the hypothesis is formulated correctly. But when I mentioned, 'people with data lie', this relates to how people use statistics or hypothesis testing to manipulate the results to (mis)guide decisions.

What if the data scientist in 'love' formulated the proposal as, "I have sufficient evidence that we are in 'love', and hence propose to get married." In this scenario, the measured feeling level between 4-10 would be in 'love' distribution with probability of >5% (alpha level 0.05 or confidence interval 0.95). Any measured value >= 4 will lead to decision of getting married. Even with such small values of feeling-level, the girl will not be able to 'reject the hypothesis' of being in 'love'. The girl would reason, "I like spending weekends with the guy, there is 6% chance that this means 'love', so I can't completely reject that I love this guy".

This time she goes to her friends, and says, "He proposed for marriage, and I failed to reject!!". And we get Ross/Carole!

People might want to use such hypothesis to prove certain 'changes', and say the test was run with 95% confidence! But as business leaders (when it is proposed by managers) or common people (when it is proposed by media) we need to understand the formulation of hypothesis, both null and alternate, before feeling great about the 'confidence level'.

Assuming the hypothesis are formulated correctly, the next step would be to understand the different performance measures from the confusion matrix, and using the right measure. But let's keep that for the next blog.