Baig Academy

Here's a very surprising way in which correlation fails. This phenomenon is also called Simpson's Paradox.

Within segments of data, there might be one pattern - a strong negative association. But on the whole, all segments combined, the pattern might completely reverse.

A metric like correlation that only looks at the aggregate doesn't capture the true pattern. This is a great reason why you shouldn't just go with numerical summaries, but visualize the data too!

Correlation is used everywhere, but it is not perfect. Let's see one major drawback of correlation that every data science professional should be aware of.

High sensitivity to outliers! One single data point can deceive you into believing there is a strong correlation, even though there is none. To understand why, check out the formulation for calculating correlation.

Correlation is an essential tool in the kit of any data professional. The Pearson correlation measures the linear relationship between two variables.

It varies between -1 and +1. -1 means a perfect negative correlation i.e., one variable decreases as the other increases. And +1 means both increase together. Around 0 means no correlation or no linear pattern.

Some key uses -
- Exploratory data analysis - correlation matrices and heatmaps allow us to quickly identify hidden patterns
- Dimensionality reduction - it helps identify redundant features
- Feature selection - improves predictive models' performance by limiting to valuable features
- Data validation - surprising/unexpected correlations can help identify data quality issues, or anomalies and helps in data cleaning.
- Business Insights - identify key drivers, e.g. ad spend is positively correlated with sales, helping better investment decisions