Logistic Regression in R

Introduction Logistic Regression in R

Logistic Regression in R also known as binary classification problems. They are used to predict an outcome as (1 or 0 either yes/ no) for an independent variable. To understand logistic regression in R, it is important to know the basic linear regression, works with the continuous outcome variable. More specifically we can say it is an extension of linear regression. In this article, we shall discuss various methods used to calculate model and evaluation. Logistic regression is used for solving classification problems in machine learning.

How does Logistic Regression in R works?

Logistic regression is a technique used in the field of statistics measuring the difference between a dependent and independent variable with the guide of logistic function by estimating the different occurrence of probabilities. They can be either binomial (has yes or No outcome) or multinomial (Fair vs. poor very poor). The probability values lie between 0 and 1 and the variable should be positive (<1). It targets on the dependent variable and has the following steps to follow:

1. n- no.of fixed trials on a taken dataset
2. with two outcomes trial
3. the outcome of the probability should be independent of each other
4. The probability of success and failures must be the same at each trial.

In this, we are considering an example by taking the ISLR package, which provides various datasets for training. To fit the model, generalized linear model function (glm) is used here. To build a logistic regression glm function is preferred and gets the details of them using a summary for analysis task.

Working Steps:

The working steps on logistic regression follow certain term elements like

• Modeling the probability or doing probability estimation
• prediction
• Initializing threshold value (High or Low specificity)
• Confusion matrix
• The plotting area under the curve(AUC)

Examples

Below are some example of Logistic Regression in R:

Data Loading:

Installing the ISLR package.

require(ISLR)

Loading required package: ISLR

For this article, we are going to use a dataset ‘Weekly’ in RStudio. The dataset implies the summary details of the weekly stock from 1990 to 2010.

require(ISLR)

names(OJ)

Output:

[1] “Purchase”   “WeekofPurchase” “StoreID”        “PriceCH”

[5] “PriceMM”       “DiscCH”         “DiscMM”         “SpecialCH”

[9] “SpecialMM”   “LoyalCH”      “SalePriceMM”    “SalePriceCH”

[13] “PriceDiff”      “Store7”         “PctDiscMM”      “PctDiscCH”

[17] ” ListPriceDiff”  “STORE”

str(OJ)

Shows 1070 Observations of 18 variables.

Our dataset has 1070 observations and 18 different variables. here we have Special MM And special CH has a dependent outcome. Let’s take a Special MM attribute to have a correct observation and an accuracy of 84 %.

table(OJ$SpecialMM)

0  1

897 173

Next to find the probability

897/1070

[1] 0.8383178

In the next step for a better sample Splitting the data set into training and testing data set is a goo

library(caTools)

set.seed(88)

split=sample.split(OJ$SpecialMM, SplitRatio = 0.84)

Considering qt has a training set and qs has test set sample data.

qt=subset(OJ,split==TRUE)

qs=subset(OJ,split==FALSE)

nrow(qt)

[1] 898

nrow(qs)

[1] 172

Therefore we have 898 Training set and 172 testing samples.

Next using Summary () gives the details of deviance and co-efficient tables for regression analysis.

QualityLog=glm(SpecialMM~SalePriceMM+WeekofPurchase ,data=qt,family=binomial)

summary(QualityLog)

Output:

Call: glm(formula = SpecialMM ~ SalePriceMM + WeekofPurchase, family = binomial, data = qt) Deviance Residuals: Min        1Q        Median      3Q        Max -1.2790  -0.4182  -0.3687  -0.2640  2.4284 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept)  2.910774  1.616328 1.801  0.07173 . SalePriceMM -4.538464  0.405808 -11.184  < 2e-16 *** WeekofPurchase  0.015546  0.005831  2.666  0.00767 ** — Null deviance:794.01 on 897 degrees of freedom Residual deviance: 636.13  on 895  degrees of freedom AIC: 642.13 Number of Fisher Scoring iterations: 5

From the above analysis, it is said that the coefficients table gives positive values for WeekofPurchase and they have at least two stars which imply they are the significant codes to the model.

Prediction Technique:

Here we shall use predict Train function in this R package and provide probabilities we are using argument named type=response. Let’s see the prediction applied to the training set (qt). The R predicts the outcome in the form of P(y=1|X) with the boundary probability of 0.5.

predictTrain = predict(QualityLog, type=”response”)

Summary results in a median, mean and min, max values.

summary(predictTrain) Execution gives

Min. 1st Qu.Median Mean 3rd Qu.Max.

0.02192 0.03342 0.07799 0.16147 0.25395 0.89038

tapply (predictTrain,qt$SpecialMM)

To compute the average for the true probabilities tapply() function is used.

tapply(predictTrain,qt$SpecialMM,mean)

0        1

0.1224444 0.3641334

Therefore, we find in the above statement that the possibility of true SpecialMM means value is0.34 and for true poor value is 0.12.

Calculating threshold value:

if P is > T– prediction is poor Special MM

if P is <T– Prediction is good

Classification Matrix:

table(qt$SpecialMM,predictTrain >0.5)

FALSE TRUE

0  746   7

1   105   40

To compute Sensitivity and Specificity

40/145

[1] 0.2758621

746/753

[1] 0.9907039

Testing set Prediction

predictTest = predict(QualityLog, type = “response”, newdata = qs)

table(qs$SpecialMM,predictTest >= 0.3)

FALSE TRUE

0   130 14

1     10  18

table(qs$SpecialMM,predictTest >= 0.5)

FALSE TRUE

0   140   4

1    18    10

Calculating accuracy

150/172

[1] 0.872093

There are 172 cases from which 144 are good and 28 are poor.

Plotting ROC curve:

This is the last step by plotting the ROC curve for performance measurements. A good AUC value should be nearer to 1 not to 0.5. Checking with the probabilities 0.5, 0.7, 0.2 to predict how the threshold value increases and decreases. It is done by plotting threshold values simultaneously in the ROC curve. A good choice is picking considering higher sensitivity.

Logistic Regression techniques

Let’s see an implementation of logistic using R, as it makes very easy to fit the model. There are two types of techniques:

• Multinomial Logistic Regression
• Ordinal Logistic Regression

Former works with response variables when they have more than or equal to two classes. later works when the order is significant.

Conclusion

Hence, we have learned the basic logic behind regression alongside we have implemented Logistic Regression on a particular dataset of R. A binomial or binary regression measures categorical values of binary responses and predictor variables. They play a vital role in analytics wherein industry experts are expecting to know the linear and logistic regression. They have their own challenges and in the practical example, we have done with the steps on data cleaning, pre-processing. Altogether we have seen how logistic regression solves a problem of categorical outcome in a simple and easy way.

Recommended Articles