Support Vector Machines Notes & Questions | Class 12 Future Skills & AI | Drishti OS

What you'll learn

What classification is and how SVMs solve it
The concepts of hyperplane, margin, and support vectors
How the kernel trick extends SVMs to non-linear problems
The difference between hard margin and soft margin SVMs
When to choose SVM over other classifiers

Key concepts

The Classification Problem

Given labelled data points (e.g., spam/not-spam, cat/dog), a classifier learns a decision boundary that separates the classes. At prediction time, new points are classified based on which side of the boundary they fall on.

Hyperplane

In 2D: a hyperplane is a line. In 3D: a plane. In higher dimensions: a general flat boundary.

An SVM finds the optimal hyperplane — the one that best separates the two classes.

For a linear SVM in 2D: w·x + b = 0

Points where w·x + b > 0 → Class +1

Points where w·x + b < 0 → Class −1

Margin and Support Vectors

Term	Definition
Margin	The distance between the hyperplane and the nearest data points from each class
Support Vectors	The data points closest to the hyperplane (they "support" its position)
Maximum Margin	SVM selects the hyperplane that maximises the margin

Maximising the margin minimises generalisation error — the model is less likely to misclassify unseen data.

Class A (○)          Class B (×)
  ○   ○   ○ | × × ×
        ○ --|-- ×
  ○   ○   ○ | × × ×
         ↑margin↑
      hyperplane

Kernel Trick

When data is not linearly separable in its original space, the kernel trick maps data into a higher-dimensional space where a linear hyperplane can separate it — without explicitly computing the transformation.

Kernel	Best for
Linear	Linearly separable data, high-dimensional text data
RBF (Radial Basis Function)	Non-linear, complex boundaries (most common default)
Polynomial	Moderate non-linearity
Sigmoid	Neural-network-like behaviour

RBF kernel: K(x, x') = exp(−γ ||x − x'||²)

γ controls how tightly the kernel fits training points. High γ → risk of overfitting.

Soft Margin SVM (C parameter)

Real data is rarely perfectly separable. Soft-margin SVM allows some misclassification using a slack variable ξ.

The C parameter controls the trade-off:

C value	Effect
Large C	Small tolerance for misclassification; tighter fit (risk of overfitting)
Small C	More tolerance; wider margin, simpler boundary (risk of underfitting)

When to Use SVM

Use SVM when...	Avoid SVM when...
Data has clear margin of separation	Dataset is very large (SVMs are slow on big data)
High-dimensional feature space (e.g., text, genomics)	Need probability estimates
Small to medium datasets	Features need heavy interpretation
Non-linear boundaries (with RBF kernel)	Data has heavy noise and many overlapping classes

Quick check

What are support vectors, and why does the SVM algorithm focus specifically on these points rather than all training data?
A dataset has two classes that form concentric circles — inner circle is class A, outer ring is class B. Can a linear SVM classify this correctly? What would you do instead?
Explain the effect of setting C to a very large value in a soft-margin SVM. What problem does this risk?
Why is the kernel trick computationally efficient compared to explicitly computing a high-dimensional feature map?
A student says "SVMs always outperform logistic regression." Describe one scenario where logistic regression would be preferred over SVM.

Support Vector Machines