Datasets, CSVs and Missing Values
Good data beats a fancy algorithm. Most beginners start with CSV files from Kaggle or UCI, then use pandas to inspect them. The first thing you will see is missing values: blank cells where data was not collected.
How you fill those blanks matters. Filling with the overall mean before splitting leaks test information into training. You must learn the mean from train only, then apply it to both train and test.
Kaggle is the best place to practice because you can download clean CSVs and see how others cleaned them. Always read the dataset description to know what each column means and whether the target is balanced.
Reading CSV correctly
Use pandas read_csv, check head(), info(), and describe(). Look for object columns that should be numbers and date columns that need parsing.
import pandas as pd
df = pd.read_csv("housing.csv")
print(df.info())
print(df.isnull().sum())Missing values without leakage
Learn imputation from train only. Modern sklearn does this with pipelines so test data never leaks.
from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy="mean")
X_train_imp = imp.fit_transform(X_train)
X_test_imp = imp.transform(X_test) # not fit_transformSyntax
df = pd.read_csv("housing.csv")
df.fillna(df.mean(numeric_only=True), inplace=True)
# better: use SimpleImputer in a pipelineKaggle leakage story that cost a competition
In a famous Kaggle competition, a team filled missing values using the whole dataset mean before splitting and got 98% accuracy — on test it dropped to 70%. GitHub discussion shows this is the top leakage trap. Use pipelines to prevent it.
from sklearn.pipeline import Pipeline
pipe = Pipeline([("imp", SimpleImputer()), ("model", LinearRegression())])Good to know
- Kaggle and UCI are best for starter datasets.
- Always check head(), info(), isnull().sum().
- Impute from train only to avoid leakage.