Logistic Regression Notes & Questions | Class 12 Future Skills & AI | Drishti OS

What you'll learn

How logistic regression solves binary classification problems
The sigmoid function and why it's used
Log-loss (cross-entropy) as the cost function
How the decision boundary is formed
Gradient descent for optimising logistic regression
Evaluating a classifier with precision and recall

Key concepts

Binary Classification

Logistic regression predicts the probability that an input belongs to one of two classes (0 or 1).

Examples: spam vs not-spam, pass vs fail, disease vs healthy.

Despite its name, logistic regression is a classification algorithm, not a regression one.

Sigmoid Function

The sigmoid function maps any real number to a value between 0 and 1, making it ideal for probability output.

σ(z) = 1 / (1 + e⁻ᶻ)

where z = w·x + b (the linear combination of weights and input features)

z value	σ(z) output
Very large positive	≈ 1.0
0	0.5
Very large negative	≈ 0.0

The model outputs P(y = 1 | x). If P > 0.5, predict class 1; otherwise predict class 0.

Decision Boundary

The decision boundary is where P = 0.5, i.e., where z = 0.

w·x + b = 0

This is a linear boundary in the feature space. Logistic regression is a linear classifier.

For non-linear boundaries, features can be engineered (e.g., x², x₁x₂) before applying logistic regression.

Log-Loss (Cross-Entropy Cost)

Logistic regression uses log-loss instead of mean squared error.

L(ŷ, y) = −[y log(ŷ) + (1 − y) log(1 − ŷ)]

Situation	Loss
y = 1, ŷ ≈ 1	Low loss (correct, confident)
y = 1, ŷ ≈ 0	High loss (wrong, confident)
y = 0, ŷ ≈ 0	Low loss (correct, confident)
y = 0, ŷ ≈ 1	High loss (wrong, confident)

Log-loss penalises confident wrong predictions heavily.

Gradient Descent for Logistic Regression

The goal is to find weights w and bias b that minimise the total log-loss.

Update rule:

w := w − α × ∂L/∂w b := b − α × ∂L/∂b

Symbol	Meaning
α (alpha)	Learning rate — controls step size
∂L/∂w	Gradient of loss with respect to weights

Repeat until convergence (loss stops decreasing significantly).

Evaluation Metrics: Precision and Recall

Using a confusion matrix:

	Predicted Positive	Predicted Negative
Actual Positive	TP (True Positive)	FN (False Negative)
Actual Negative	FP (False Positive)	TN (True Negative)

Precision = TP / (TP + FP) — of all predicted positives, how many were correct?

Recall = TP / (TP + FN) — of all actual positives, how many did we catch?

F1 Score = 2 × (Precision × Recall) / (Precision + Recall) — harmonic mean of both

Priority	Use
Minimise false positives	Maximise Precision (e.g., spam filter)
Minimise false negatives	Maximise Recall (e.g., cancer detection)

Quick check

Why is logistic regression preferred over linear regression for binary classification? What problem does the sigmoid function solve?
A logistic regression model outputs 0.72 for a test sample. What class does it predict, and what does the number 0.72 represent?
Explain why log-loss penalises a confident wrong prediction more severely than a close wrong prediction.
A medical test for a rare disease must catch every positive case, even at the cost of false alarms. Should the team optimise for precision or recall? Explain.
A model is trained with a very large learning rate (α). What problem might this cause during gradient descent, and how would you fix it?

Logistic Regression