13/12/2020
Exploratory Data Analysis (EDA)
1. Introduction
Exploratory Data Analysis (EDA) refers to - performing initial critical investigations into any given data to understand the data and deriving maximum insights from it. With EDA, we can discover patterns in the data, identify anomalies and fix it, do hypothesis testing for correlation or goodness of fit, find summary statistics of the variables and graphical representation of the data as bird's eye view before advanced analysis.
Exploratory Data Analysis can be divided into 2 forms – Quantitative and Graphical. Also, in each form, we can analyze either one variable or multiple variables simultaneously, namely Univariate and Multivariate Analysis. So, EDA can be divided into 4 categories – Quantitative-Univariate, Quantitative-Multivariate, Graphical-Univariate and Graphical-Multivariate Analysis.
2. Marketing Campaign Case
The data of a marketing campaign – mails or communications to the customers for persuading them for a particular type of response – either feedback or buying the product. We will perform EDA on it using python. Following 6 steps need to be followed:
Step #1: Import Necessary Libraries for analysis.
Step #2: Import Data and Check if it is the correct data.
Step #3: Find summary statistics of all numeric variables; Types of variables; Missing values in the data; Filling in missing Data – Mean for numeric variables and Mode for categorical variables.
Step #4: Univariate analysis: 1. Target variable i.e., Customer response – yes or no? 2. Any other variable e.g., Profession – distribution of the professions of the customers to whom the communication was made during the campaign. We can do a similar analysis for all the categorical variables just to get insight. 3. Outlier Analysis of numeric variables e.g., Customer Age. To find outlier customers i.e., very young or very old - laying beyond 3 times the Interquartile range above or below. Interquartile range = IQR = Middle 50 percentile customers by age.
Step #5: Bivariate analysis: We can analyze, dependent and one independent variable to find a relation between the two e.g., Response Rate vs Customer with or without loan; or Response in a particular month; Response vs level of education; Response vs on which day communication was made and so on. Insights => Response is better when contacted on Monday; During the month of May; When people are married; When they do not have a loan; Profession is admin; Education is university degree; Contacted through mobile; They are new customers; the previous outcome was non-existent.
Step #6: Multivariate Analysis: Simultaneous correlations between all the variables. And later, categorical variables can be encoded for further analysis.
3. Conclusion
With EDA, we can find out, a summary statistic for all the variables, identify missing values and replace it, perform univariate analysis on individual variables to get a trend in the data, perform bivariate analysis to identify the relationship between dependent and other individual variables, do correlation analysis for all the variables simultaneously and finally, encode the data to make it ready for further analysis.
Article Credit: Amit Hiremath, MBA 1st Year, Batch of 2020-2022
