SVM and k-Nearest Neighbors

Support Vector Machines and k-Nearest Neighbors are both distance-based. They decide by measuring how far points are. If one feature is in dollars and another in bedrooms, dollars dominate distance, so the model mostly ignores bedrooms.

Scaling fixes this. StandardScaler makes all features mean 0 and variance 1 so they contribute equally. Without scaling, both models perform poorly and behave unpredictably.

Both are slow on large data because they compare to many points. Use them for small to medium tabular sets, not for millions of rows without approximation.

Scaling is mandatory

Always scale before distance models. Use a pipeline so scaling is learned from train only.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
pipe = Pipeline([("scale", StandardScaler()), ("svm", SVC(C=1.0))])
pipe.fit(X_train, y_train)

Choosing k and C

For k-NN, small k is noisy, large k is smooth. For SVM, C trades margin for errors. Use GridSearchCV to pick them.

from sklearn.model_selection import GridSearchCV
params = {"svm__C":[0.1,1,10], "svm__gamma":["scale","auto"]}
GridSearchCV(pipe, params, cv=3).fit(X_train, y_train)

Syntax

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)
knn.predict(X_test_scaled)

Pipeline([("scale", StandardScaler()), ("knn", KNeighborsClassifier())])