PSM : So What Next??
So far you have been exposed to the fundamental questions of what is propensity matching, why is it a good choice for causal effect evaluation, how is it used in day to day applications and what are the different techniques and challenges? Now for the NEXT BIG QUESTION? WHAT NEXT? We are hoping we address this in detail in this blog so that you can get a (right) direction to proceed while employing this technique.
Image Source: http://1.bp.blogspot.com/–QhyAvmz1ow/UT–34XJFrI/AAAAAAAAV20/gtJ0OcRXQrI/s1600/j.mp:14PR7bT.jpg
Propensity Matching is a fairly young technique just here around 20 years or so (we are counting in turtle years by the way). The scope of this technique is therefore growing with time. Also, everyday(figuratively) new applications of PSM techniques are being discovered. Let’s explore these applications together.
In one of the earlier blog posts, we analyzed two case studies which employed PSM, one dealing with analyzing the effect of coaching/tuitions on SAT scores, the other dealing with analyzing the outcome evaluation of medical therapy management at Retail Pharmacy. Although, origin of PSM is in the healthcare and medical industry, the applications of PSM are branching out to educational and even economic industries. This primarily comes from the fact that PSM is an easy and convenient technique to use. Easy because it is fairly simple to understand and employ. Convenient because it adheres to moral and ethical principles. And lastly, it is the one which you run to, when you need to compare apples with oranges (Back to our fruit salad).
Applications of PSM, nowadays, range widely from dealing with economic issues, such as comparing performance of firms which received government assistance[1], to making a difference in the social sector by analyzing whether a child being victim of bullying result in future delinquency[2]. Since, the factors that decide whether to provide assistance to a firm or not are not under researcher’s control, PSM provides a solid platform to compare the two samples by looking into the after-effects of the treatment. Similarly, whether a person is bullied or not is an unfortunate event which can’t be predicted. However, the adverse effects of such an event can be analyzed more objectively through the use of PSM. It provides a basis for comparison between a victim and a normal individual so as to understand the extent to which such events alter a person’s life
PSM : Pros and Cons!!
However, PSM is not a magic trick which can be employed to each and every A/B testing scenario. Just there exists Mr. Hyde which is a counterpart to Dr. Jekyll, PSM also has its own dark side. This dark side lies within the assumptions of PSM. It is important to understand these assumptions and verify whether they are applicable to your individual scenario. One of these assumptions is the aspect of hidden bias.
Image source:http://www.clipartbest.com/cliparts/RTd/egL/RTdegLRT9.png
No matter how many attributes we measure or the amount of data that we collect, there are few factors that remain unmeasured but still do have the power to influence the outcome. PSM technique assumes that the effect of these factors on the outcome is not significant enough. This assumption might be untrue in many cases where these factors actually have a significant effect. Consider a case in the healthcare industry. We judge the effectiveness of a drug based on observable vital stats of a person. This makes sense enough to employ PSM and PSM works well in such scenarios since most of the factors affecting a person’s survival(drug’s positive effect) are dependent on these vital stats which can be measured.
Now consider a scenario, where we compare the effect of coaching on SAT scores of various factors. Confounding variables are demographic attributes, aptitude, IQ etc which are measurable. However, the hidden bias exist in the form of attention span of a student. It is not possible to measure the number of times a student pays attention during tuitions or while studying. This unmeasured confounder however, does have a significant effect on the outcome. PSM doesn’t handle this type of bias well and hence might give out inaccurate comparisons.
Fortunately, these days, sentiment analysis is a growing trend where you measure the possible effect of such unmeasurable confounders and determine whether this effect is significant enough to manipulate the output.
In a nutshell:
- PSM is a handy technique to employ you need to compare the outcomes of a non-randomized experiment.
- PSM is finding its applications from healthcare industry to the economic sectors, social sectors and even housing industry.
- It is important to understand the underlying assumptions made while employing this technique since these assumptions are not always applicable to business scenarios
- If biases exist, it is important to gauge the effect of unmeasured confounders on the outcome and then determine whether PSM results can be safely utilized.
Key Terminology
Unmeasurable confounders: Factors which exist but which can’t be accurately measured and which have an effect on the outcome of the experiment
Hidden Bias: Effect that unmeasurable confounders have on the outcome
References
[1] Cristian Rotaru, Sezim Dzhumasheva,and Franklin Soriano(2012),”Propensity Score Matching: An Application using the ABS Business Characteristics Survey”
[2] Dr. Jennifer Wong(2013), “Does Bully Victimization Predict Future Delinquency?” url:http://cjb.sagepub.com/content/40/11/1184.abstract?rss=1
Posted by malvikar | Filed under 10_Propensity score matching, Uncategorized