NLP Basics: Text to Vectors
Natural Language Processing turns text into numbers so models can read it. The simplest way is TF-IDF, which weights rare words higher. Modern ways use embeddings that capture meaning.
You will build a sentiment classifier: positive vs negative reviews. Start with TF-IDF + Naive Bayes, then try linear SVM. Both are strong baselines before transformers.
Text needs cleaning: lowercasing, removing punctuation, and handling missing values. But do not over-clean and remove meaning like “not”.
TF-IDF vs embeddings
TF-IDF counts words weighted by rarity. Embeddings like Word2Vec or BERT capture context. Start with TF-IDF, then try embeddings if needed.
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer(stop_words="english", max_features=5000)
X = vec.fit_transform(texts)Sentiment pipeline
Vectorize, then classify. Use a pipeline so vocabulary is learned from train only.
from sklearn.pipeline import Pipeline
pipe = Pipeline([("tfidf", TfidfVectorizer()), ("clf", MultinomialNB())])
pipe.fit(train_texts, y_train)Syntax
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.fit_transform(["hello world", "hello ML"])
print(X.toarray())TF-IDF vs CountVectorizer matters
CountVectorizer counts, TF-IDF downweights “the”. On sentiment data TF-IDF often gains 5-10% F1. This GitHub issue is the top “why my text model is bad” fix.
TfidfVectorizer(max_features=5000) # try vs CountVectorizer()Good to know
- TF-IDF is a strong baseline for text.
- Embeddings capture meaning better but need more data.
- Always pipeline text vectorization to avoid leakage.