Train/Test Split and Cross-Validation

Splitting is not just 80/20 and done. How you split decides whether your score is honest. Doing any preprocessing before splitting leaks test information into training and inflates your score.

Cross-validation splits multiple times and averages the scores, so one lucky split does not fool you. Pipelines bundle preprocessing and modeling so leakage is impossible.

If you tune hyperparameters, you need a third split: validation. Train on train, pick best params on validation, finally test on test once. Reusing test for tuning is the most common leakage in Kaggle.

Pipeline prevents leakage

A pipeline learns imputation and scaling only from train, then applies to test. Doing fit_transform on whole data before split is the classic leak.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = Pipeline([("imp", SimpleImputer()), ("scale", StandardScaler()), ("clf", LogisticRegression())])
pipe.fit(X_train, y_train)
pipe.score(X_test, y_test)

Cross-validation

Use cross_val_score to get 5 scores and their mean. This tells you how stable your model is.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipe, X, y, cv=5)
print(scores.mean(), scores.std())

Syntax

from sklearn.model_selection import train_test_split, cross_val_score
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
pipe.fit(X_train, y_train)

# Bad: SimpleImputer().fit_transform(X) before split
# Good: Pipeline([("imp", SimpleImputer()), ("model", clf)])