Dimensionality Reduction with PCA
Dimensionality reduction compresses many columns into fewer while keeping the important variance. Principal Component Analysis finds new axes that capture the most spread and projects your data onto them.
It helps with visualization, speed, and noise reduction. If you have 100 features, PCA can keep 95% of variance with 10 components. It is not magic, it just rotates the space.
Always scale before PCA, otherwise large-valued features dominate the components.
Keep variance, drop noise
Pick number of components by explained variance. 95% is a common target.
from sklearn.decomposition import PCA
pca = PCA(n_components=0.95, random_state=42)
X_pca = pca.fit_transform(X_scaled)
print(pca.explained_variance_ratio_.sum(), X_pca.shape)Fit only on train
Learn PCA from train only, then transform test. Fitting on whole data leaks test structure.
pca.fit(X_train_scaled)
X_test_pca = pca.transform(X_test_scaled)Syntax
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_scaled)PCA leaks if fit on whole data
Fitting PCA on all data before split leaks test variance into train and inflates scores. Fit on train only, transform test. This GitHub issue is the top PCA leakage post.
pca.fit(X_train_scaled) # not X
X_test_pca = pca.transform(X_test_scaled)Good to know
- PCA keeps variance, drops dimensions.
- Scale before PCA.
- Fit on train, transform test.