Random Forest & Ensembles

• What ensemble learning is and why it outperforms single models
• How bagging reduces variance in decision trees
• The random feature selection that makes Random Forest unique
• How voting (classification) and averaging (regression) combine predictions
• Feature importance scores from Random Forest
• The bias-variance trade-off and where Random Forest sits

Ensemble Methods

An ensemble combines multiple models to produce better predictions than any individual model.

"The wisdom of the crowd" — diverse, imperfect models together are more reliable than one strong model alone.

Decision Trees (the building block)

A decision tree splits data at each node using the feature and threshold that best separates classes.

Weakness of single trees: They are prone to overfitting — a deep tree memorises training data but performs poorly on new data.

Bagging (Bootstrap Aggregating)

1. Draw B random samples from the training data with replacement (bootstrap samples).
2. Train a separate decision tree on each sample.
3. Combine predictions:
   • Classification → majority vote
   • Regression → average

Because each tree sees different data, the ensemble is more robust than any single tree.

Random Forest = Bagging + Random Feature Selection

At each split in each tree, Random Forest considers only a random subset of features (not all features). This forces the trees to be diverse — they can't all rely on the same dominant feature.

Typical subset size: √p features for classification, p/3 for regression (where p = total features)

Voting (Classification)

Each tree casts a vote for a class. The class with the most votes wins.

Tree 1: Dog     Tree 2: Cat     Tree 3: Dog
Tree 4: Dog     Tree 5: Cat

Vote: Dog = 3, Cat = 2  →  Prediction: Dog

Feature Importance

Random Forest can rank which features matter most by measuring how much each feature reduces impurity (e.g., Gini impurity) across all trees.

This is useful for feature selection and understanding the model.

Bias-Variance Trade-off

Random Forest reduces variance by averaging many high-variance trees. Each individual tree overfits slightly, but the ensemble averages out the errors.

Random Forest vs Single Decision Tree

1. Explain why combining 100 slightly inaccurate decision trees in a Random Forest often gives better results than using one highly tuned tree.

2. What is the role of random feature selection at each split in a Random Forest? Why not just use all features?

3. A Random Forest is trained on a dataset with 16 features. How many features would typically be considered at each split for a classification task?

4. Define bias and variance in the context of machine learning. Does a deep decision tree have high bias or high variance? Explain.

5. Describe a real-world application where knowing the feature importance scores from a Random Forest would be genuinely useful.