Machine Learning 12 Slides

Logistic Regression It's Not What You Think.

Slide 1 / 12

Use ← and → to navigate

Swipe left / right on mobile

Guide Notes & Explanation

Accompanying breakdown for this slide deck

Logistic regression has a confusing name. It contains the word "regression," which typically describes predicting a continuous number. But this algorithm is actually used for classification tasks.
It predicts which category an observation belongs to. For example, it can classify an email as spam or not spam. It can predict if a transaction is fraudulent or legitimate.
The name comes from its historical development. It uses a linear regression-like approach but then transforms its output. This transformation allows it to predict probabilities for categories.

At its heart, logistic regression estimates probabilities. It calculates the probability that a given input belongs to a default category. If the probability is above 0.5, it predicts one class; below, it predicts the other.
It does this by using the logistic function. This function, also known as the sigmoid function, takes any real number. It then squashes that number into a value between 0 and 1.
This output is interpreted as a probability. This simple yet powerful mechanism is the key to its success.

The magic happens with the sigmoid function. It is the mathematical engine that enables classification. Its formula is straightforward.
The equation for the sigmoid function is: P(Y=1) = 1 / (1 + e^(-z))
Here, 'z' is the linear combination of inputs and coefficients. 'e' is the base of the natural logarithm. The output P(Y=1) is the probability of the event occurring.
This S-shaped curve is perfect for modeling probabilities. It ensures outputs are always sensible, bounded between 0 and 1.

A probability alone is not a classification. We need a clear rule to make a decision. This is where the threshold comes in.
By default, the threshold is set at 0.5. If the predicted probability is >= 0.5, the outcome is predicted as class 1. If it is < 0.5, the outcome is predicted as class 0.
This threshold can be adjusted based on business needs. For instance, in medical testing, we might lower the threshold to catch more sick patients.

Despite being a classifier, it has a linear core. The 'z' in the sigmoid equation is a linear function. It looks just like a linear regression equation.
z = b0 + b1*x1 + b2*x2 + ... + bn*xn
The coefficients (b1, b2,...) are learned from the data. They represent the log odds of the outcome. This linear combination is then passed to the nonlinear sigmoid.
This blend of linear and nonlinear is why it's so effective.

Logistic regression is a fundamental algorithm in machine learning. Its popularity stems from several key advantages. It is fast to train and easy to implement.
It provides not just a class label, but a probability score. This probabilistic interpretation is incredibly valuable for risk assessment. It also requires less computational power than complex models.
Furthermore, the coefficients are interpretable. We can understand the impact of each feature on the outcome.

Logistic regression isn't limited to two classes. It can be extended for multiclass classification. This version is called Multinomial Logistic Regression.
It uses the softmax function instead of the sigmoid function. The softmax function outputs probabilities for each class. The probabilities for all classes sum to one.
This makes it a versatile tool for various classification challenges. It is a go-to method for many real-world problems.

You encounter logistic regression daily, often without knowing it. Banks use it to assess credit risk and flag fraudulent transactions. It helps determine the likelihood a loan will default.
Marketing teams use it for customer churn prediction. It identifies users who are likely to stop using a service. Email providers rely on it to filter spam from your inbox.
Its applications are vast across finance, healthcare, and tech.

When starting a classification project, begin with logistic regression. It provides a strong and interpretable baseline performance. You can compare more complex models against it.
If a complicated model cannot outperform logistic regression, it may be unnecessary. It helps validate the predictive power of your features. It is a benchmark for model performance.
This makes it an essential first step in any data scientist's workflow.

Remember, logistic regression is for classification, not regression. The name is a historical artifact that can be misleading. Its output is a probability that is then used for categorization.
It is a simple, powerful, and highly interpretable algorithm. It should be a core part of every machine learning toolkit. Do not overlook it in favor of more complex models.
Understand it, use it, and appreciate its elegance and utility.