Decision Trees and Random Forest
Decision trees are the most interpretable model: they ask yes/no questions like “is age > 30?” and split. A single tree overfits easily, memorizing noise. Random forest builds many trees on different samples and votes, which smooths the noise.
Two knobs matter most: max_depth and min_samples_leaf. Deep trees memorize, shallow trees underfit. Forests reduce variance without much tuning, which is why they are a strong baseline for tabular data.
You can read feature importances from a forest to see which columns matter, but be careful: correlated features split importance.
Overfitting with depth
Limit max_depth and require min_samples_leaf to force the tree to generalize. Use validation to pick them.
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=200, max_depth=8, min_samples_leaf=5, random_state=42)
forest.fit(X_train, y_train)Feature importance
Forest gives importances that sum to 1. Use them to prune useless columns, but check permutation importance for correlated features.
importances = forest.feature_importances_
print(sorted(zip(feature_names, importances), key=lambda x: -x[1])[:5])Syntax
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(X_train, y_train)
forest.predict(X_test[:5])max_depth=None memorizes training data
Default RandomForest has max_depth=None, which grows until pure leaves and memorizes noise. On small tabular data this gives 100% train but 70% test. Set max_depth 6-12 and check validation. This micro-blog thread is the top “why my forest overfits” post.
RandomForestClassifier(max_depth=8, min_samples_leaf=5) # not NoneGood to know
- Single trees overfit, forests vote to reduce variance.
- Tune max_depth and min_samples_leaf.
- Feature importances help select columns.