Four seconds against twenty two minutes. That is the honest gap between a TF-IDF model scoring 0.8542 macro F1 on a four class text problem and a fine tuned DistilBERT scoring 0.9013 on exactly the same split. Both are defensible. Which one belongs in your serving path depends on three questions tutorials almost never ask: how often does this thing retrain, what latency budget does the caller have, and does 0.047 of macro F1 change a single decision anybody downstream actually makes. Most of the time the answer to that last one is no, and the four second model wins.

Text as a matrix of counts

Where our project stands: last part we rebuilt the churn network in PyTorch and landed a test average precision of 0.5445 on the 20,000 row churn matrix standing in for the public Telco Customer Churn file. This part steps sideways on purpose, because a churn table has no free text worth modelling until you attach support tickets to it. Two datasets run through what follows. Public 20 newsgroups corpus that ships inside scikit-learn does the teaching, because it is four classes, small enough to iterate on, and arrives in one function call. Near the end I fold the technique back into churn by turning ticket text into features and measuring whether average precision moves at all.

Every model you have trained so far expects a rectangular matrix of numbers. A document is not that. Bag of words is the oldest bridge between the two, and it is brutally simple: build a vocabulary of every term in the corpus, then represent each document as a vector counting how many times each vocabulary term appears in it. Word order vanishes. Grammar vanishes. What survives is which words showed up and how often, which turns out to be enough for a surprising number of problems.

# tested against: Python 3.10.12, scikit-learn 1.7.2, numpy 2.2.6,
# sentence-transformers 5.5.1, transformers 5.14.1
import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer

cats = ['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian']
strip = ('headers', 'footers', 'quotes')

train = fetch_20newsgroups(subset='train', categories=cats, remove=strip)
test = fetch_20newsgroups(subset='test', categories=cats, remove=strip)

vec = CountVectorizer(min_df=2)
X = vec.fit_transform(train.data)
density = X.nnz / (X.shape[0] * X.shape[1])
print('documents', X.shape[0], 'vocabulary', X.shape[1])
print('stored values', X.nnz, 'density', round(density, 5))

documents 2257 vocabulary 12057
stored values 208413 density 0.00766

Density of 0.00766 means fewer than eight cells in every thousand hold a number. Stored as a dense float64 array that matrix would eat 218 MB for 2,257 short documents; sparse it costs about 2.5 MB. Scale that to a million support tickets and the difference decides whether the job runs on a laptop or needs a cluster. That is why fit_transform hands back a sparse matrix rather than an array, and why most linear models in scikit-learn accept sparse input directly without you converting anything.

Setting min_df=2 did quiet work there. Terms appearing in exactly one document across the whole corpus are usually typos, product IDs or one person’s unusual spelling, and they cannot generalise by construction. Dropping them cut the vocabulary from 30,241 to 12,057 without moving accuracy at all. My default for any new corpus is min_df=2 on the first pass, then raise it if the vocabulary is still unwieldy.

Weighting terms with TF-IDF

Raw counts have an obvious defect. A word appearing in every document tells you nothing about which class a document belongs to, yet it gets the largest count and therefore the loudest voice. TF-IDF fixes this by multiplying each term frequency by an inverse document frequency factor, roughly the log of how many documents there are divided by how many contain that term. Common terms get shrunk toward zero. Terms concentrated in a handful of documents get amplified. It is a one line change to the vectoriser and it bought me 0.088 of macro F1 on this corpus.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
from sklearn.metrics import f1_score

pipe = make_pipeline(
    TfidfVectorizer(min_df=2, ngram_range=(1, 2), sublinear_tf=True),
    LinearSVC(C=1.0),
)
pipe.fit(train.data, train.target)
pred = pipe.predict(test.data)

print('features', len(pipe[0].get_feature_names_out()))
print('macro f1', round(f1_score(test.target, pred, average='macro'), 4))

features 57340
macro f1 0.8542

Two arguments there earn their place. Setting sublinear_tf=True replaces the term frequency with one plus its log, which stops a word repeated forty times in a rant from dominating the vector. Adding ngram_range=(1, 2) keeps adjacent word pairs alongside single words, so not working becomes its own feature instead of dissolving into two useless tokens. Bigrams took the vocabulary from 12,057 to 57,340 and lifted macro F1 by 0.019 here. On short, sentiment heavy text such as support tickets the lift is usually larger, because negation is exactly what unigrams destroy.

Word embeddings and what they fixed

Bag of words has a second defect, deeper than word order. Every term is its own dimension, orthogonal to all others, so refund and reimbursement are as unrelated as refund and giraffe. Any document using one word where your training set used the synonym scores zero similarity. Embeddings attack that directly by learning a short dense vector for each word from the company it keeps, so words appearing in similar contexts end up near each other in the space. Where TF-IDF gave us 57,340 sparse dimensions of pure surface form, an embedding gives a few hundred dense dimensions that carry something closer to meaning.

Averaging word vectors to represent a whole document works badly. Sentence embedding models are trained specifically so the vector for an entire passage is useful on its own, and that is what you should reach for. MiniLM is my default: 384 dimensions, small enough to run on CPU, and good enough that I rarely reach past it before measuring.

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

enc = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
print('max seq length', enc.max_seq_length)

Etr = enc.encode(train.data, batch_size=64, show_progress_bar=False)
Ete = enc.encode(test.data, batch_size=64, show_progress_bar=False)
print('embedding shape', Etr.shape)

clf = LogisticRegression(max_iter=2000).fit(Etr, train.target)
print('macro f1', round(f1_score(test.target, clf.predict(Ete), average='macro'), 4))

max seq length 256
embedding shape (2257, 384)
macro f1 0.8697

Notice that first printed line, because it caused me grief once and it will cause you grief too. MiniLM truncates at 256 tokens, silently. No warning, no exception, just a vector computed from the opening paragraph while the rest of a long document is discarded. On this corpus the median document is 118 tokens so it barely matters. On a corpus of long email threads it matters enormously, and the fix is to chunk each document, encode the chunks and pool the results rather than pretending one vector represents four pages.

Attention and contextual vectors

Classic embeddings assign one fixed vector per word, which means charge has a single representation whether the sentence is about a credit card or a battery. Transformer models, introduced in the 2017 Vaswani paper, remove that limitation. Each token starts with a base vector, then a self attention layer lets every token look at every other token in the sequence and update itself accordingly. Stack several such layers and the vector for charge at position nine now depends on the whole sentence around it. That single idea is what separates a transformer from everything above it in this post.

Attention costs something, and it costs it quadratically. Every token attends to every other token, so doubling sequence length roughly quadruples the compute. That is why these models carry a hard maximum length, and why the first thing worth measuring on any new corpus is how many documents exceed it.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('distilbert-base-uncased')
lengths = [len(tok(d).input_ids) for d in train.data]

print('median tokens', int(np.median(lengths)))
print('max tokens', max(lengths))
print('over 512', sum(1 for n in lengths if n > 512))

Token indices sequence length is longer than the specified maximum
sequence length for this model (7947 > 512). Running this sequence
through the model will result in indexing errors.
median tokens 118
max tokens 7947
over 512 121

That message is worth reading carefully, because it is a warning at tokenisation time and an IndexError at forward pass time. Passing truncation=True, max_length=512 to the tokeniser makes it go away, at the cost of throwing away everything past token 512 in 121 of our 2,257 training documents. Median length of 118 tells you truncation is cheap here. Median length of 900 would tell you a completely different story, and would push me toward chunking or a long context model instead.

Fine tuning, latency and cost

Read that table as a cost curve rather than a leaderboard. Moving from counts to TF-IDF bought 0.088 macro F1 for three extra seconds of training and no extra serving cost whatsoever. Moving from TF-IDF to frozen embeddings bought 0.016 and multiplied per document latency by sixty. Moving from frozen embeddings to a fine tuned transformer bought a further 0.032 and turned a four second job into a twenty two minute GPU job that now needs a GPU in the serving path as well.

Here is my verdict, and it is not a neutral one. Reach for TF-IDF with a linear model first, every time. It trains in seconds, retrains nightly on commodity hardware, produces coefficients you can read off and explain to a stakeholder, and on most business text lands within a few points of anything heavier. Avoid fine tuning a transformer as your opening move. Not because it performs badly, but because you inherit GPU dependency, a checkpoint to version, tokeniser drift and a serving path forty times slower, all before anybody has established that the extra accuracy changes a decision. Frozen sentence embeddings sit in a comfortable middle: cheap to compute once, easy to cache, and they slot into the tabular models you already know how to run.

Text features inside a churn model

Back to our churn project. Subscription customers who are about to leave often say so first, in a support ticket, weeks before the cancellation lands in the billing table. That makes ticket text a genuine leading indicator and exactly the sort of signal a tabular model cannot see. Approach that works and stays cheap: encode the most recent ticket per customer once, compress those 384 dimensions down to something a gradient boosting model can absorb, and paste the result onto the feature matrix from Part 6.

from sklearn.decomposition import PCA

# tickets_train aligns row for row with X_train from Part 6
Etr = enc.encode(tickets_train, batch_size=64, show_progress_bar=False)
Ete = enc.encode(tickets_test, batch_size=64, show_progress_bar=False)

pca = PCA(n_components=16, random_state=0).fit(Etr)
print('variance kept', round(pca.explained_variance_ratio_.sum(), 4))

X_train_txt = np.hstack([X_train, pca.transform(Etr)])
X_test_txt = np.hstack([X_test, pca.transform(Ete)])
print('columns before', X_train.shape[1], 'after', X_train_txt.shape[1])

variance kept 0.6118
columns before 20 after 36

Result worth pausing on: dense embeddings beat sparse TF-IDF badly here, and the reason is structural rather than linguistic. Tree models split on one feature at a time, so handing them 300 sparse columns that are almost always zero gives them very little to split on. Sixteen dense components carry the same information in a shape trees can actually use, and they cost a third of the fit time. Whenever text features meet a tree model, compress first. That trade off does not apply to linear models, which handle wide sparse input perfectly well, and it is the sort of interaction exploratory analysis will not warn you about in advance.

One operational detail decides whether any of this survives contact with production. Encoding text is slow per document and fast per batch, so batch size drives your entire cost model. Measuring it takes ten minutes and saves arguments later.

Read that curve as a serving design decision, not a benchmark. If you score churn nightly you batch at 64 and encode a million tickets in about fifteen minutes on ordinary CPUs. If you score on request, one document at a time, you get 91 per second and a per call cost roughly thirteen times higher for identical output. Anywhere the text is available before the prediction is needed, encode ahead of time and cache the vector against the customer. Part 23 comes back to precisely this pattern under its proper name.

Recommended text stack for support ticket features

If you are adding text to an existing model this quarter, build it in this order. Start with a TF-IDF vectoriser at min_df=2 and ngram_range=(1, 2), inside the pipeline, feeding a linear model, and record that score as your baseline. Then encode the same text with all-MiniLM-L6-v2, compress to sixteen or thirty two components with PCA, and attach those columns to your tabular features. On our churn project that second step moved average precision from 0.5341 to 0.5771, the largest single jump since feature engineering back in Part 6, and it added nine seconds to a nightly job. Only if that still leaves a gap worth money should you fine tune a transformer, and when you do, budget for the GPU in serving rather than just in training.

Run the four approaches from the table on your own text this week, on one split, with one metric, and write the numbers down. Almost nobody does this, which is why so many teams carry a transformer they cannot justify. Part 19 turns to time series forecasting, where the split itself is the thing most people get wrong, and it will make the leakage lesson from earlier look mild by comparison.

References

• TfidfVectorizer API reference, scikit-learn
• fetch_20newsgroups loader and the remove argument, scikit-learn
• Sentence Transformers 5.5.1 on PyPI, quickstart and model list
• Vaswani et al, Attention Is All You Need, 2017
• Telco Customer Churn dataset, IBM repository

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts