During this practical, two different classification methods will be covered: K-nearest neighbours and logistic regression.
One of the packages we are going to use is class. For this, you will probably need to
install.packages("class") before running the
library(MASS) library(class) library(ISLR) library(tidyverse)
This practical will be mainly based around the
default dataset which contains credit card loan data for 10 000 people. With the goal being to classify credit card cases as
no based on whether they will default on their loan.
balanceis mapped to the x position,
incomeis mapped to the y position, and
defaultis mapped to the colour. Can you see any interesting patterns already?
facet_grid(cols = vars(student))to the plot. What do you see?
ifelse()(0 = not a student, 1 = student). Then, randomly split the Default dataset into a training set
default_train(80%) and a validation set
If you haven’t used the function
ifelse() before, please feel free to review it in Chapter 5 Control Flow (particular section 5.2.2) in Hadley Wickham’s Book Advanced R, this provides a concise overview of choice functions (
if()) and vectorised if (
Now that we have explored the dataset, we can start on the task of classification. We can imagine a credit card company wanting to predict whether a customer will default on the loan so they can take steps to prevent this from happening.
The first method we will be using is k-nearest neighbours (KNN). It classifies datapoints based on a majority vote of the k points closest to it. In
class package contains a
knn() function to perform knn.
income(but no basis functions of those variables) in the
default_traindataset. Set k to 5. Store the predictions in a variable called
Remember: make sure to review the
knn() function through the help panel on the GUI or through typing “?knn” into the console. For further guidance on the
knn() function, please see Section 4.6.5 in An introduction to Statistical Learning
default) mapped to the colour aesthetic, and one with the predicted class (
knn_5_pred) mapped to the colour aesthetic. Hint: Add the predicted class
default_validdataset before starting your
ggplot()call of the second plot. What do you see?
knn_2_predvector generated from a 2-nearest neighbours algorithm. Are there any differences?
During this we have manually tested two different values for K, this although useful in exploring your data. To know the optimal value for K, you should use cross validation.
The confusion matrix is an insightful summary of the plots we have made and the correct and incorrect classifications therein. A confusion matrix can be made in
R with the
table() function by entering two
conf_2NN <- table(predicted = knn_2_pred, true = default_valid$default) conf_2NN
## true ## predicted No Yes ## No 1899 55 ## Yes 31 15
To learn more these, please see Section 4.4.3 in An Introduction to Statistical Learning, where it discusses Confusion Matrices in the context of another classification method Linear Discriminant Analysis (LDA).
KNN directly predicts the class of a new observation using a majority vote of the existing observations closest to it. In contrast to this, logistic regression predicts the
log-odds of belonging to category 1. These log-odds can then be transformed to probabilities by performing an inverse logit transform:
p = 1⁄(1 + ℇ -α)
where α indicates log-odds for being in class 1 and p is the probability.
Therefore, logistic regression is a
probabilistic classifier as opposed to a
direct classifier such as KNN: indirectly, it outputs a probability which can then be used in conjunction with a cutoff (usually 0.5) to classify new observations.
Logistic regression in
R happens with the
glm() function, which stands for generalized linear model. Here we have to indicate that the residuals are modeled not as a Gaussian (normal distribution), but as a
family = binomialto fit a logistic regression model
Now we have generated a model, we can use the
predict() method to output the estimated probabilities for each point in the training dataset. By default
predict outputs the log-odds, but we can transform it back using the inverse logit function of before or setting the argument
type = "response" within the predict function.
lr_mod. You can choose for yourself which type of visualisation you would like to make. Write down your interpretations along with your plot.
Another advantage of logistic regression is that we get coefficients we can interpret.
lr_modmodel and interpret the coefficient for
balance. What would the probability of default be for a person who is not a student, has an income of 40000, and a balance of 3000 dollars at the end of each month? Is this what you expect based on the plots we’ve made before?
Let’s visualise the effect
balance has on the predicted default probability.
balance_dfwith 3 columns and 500 rows:
balanceranging from 0 to 3000, and
incomealways the mean income in the
lr_modto output the predicted probabilities for different values of
balance. Then create a plot with the
balance_df$balancevariable mapped to x and the predicted probabilities mapped to y. Is this in line with what you expect?
Now let’s do another - slightly less guided - round of KNN and/or logistic regression on a new dataset in order to predict the outcome for a specific case. We will use the Titanic dataset also discussed in the lecture. The data can be found in the
/data folder of your project. Before creating a model, explore the data, for example by using