A retention manager stopped me in a review last year with a question I could not answer off my own slide. Of the customers who actually cancelled last month, she asked, how many did your model flag before they went? I had brought accuracy to that meeting. It said 97 percent, it was true, and it was completely uninformative, because 97 percent of her customers were never going to cancel in the window we were scoring. A model that predicted nobody would have scored almost as well.

That meeting is where imbalanced data stops being a textbook topic and starts costing you credibility. Rare positive class means one label shows up far less often than the other, and almost every metric, default and habit you picked up on balanced problems quietly stops working. Worse, the standard fix that everyone reaches for, synthetic oversampling, is easy to apply in a way that inflates your score without improving a single prediction.

Imbalance that matters, and imbalance that does not

Where our project stands: last part we clustered the feature frame, added segment labels as model inputs and finished with an honest AUC of 0.8323 on a churn rate of 25.3 percent. This part changes the target, and everything else changes with it.

Here is something most tutorials on this subject skip. Our Telco file is not an imbalanced dataset. One customer in four churned. A quarter is a perfectly comfortable minority class, and if you throw SMOTE at it you will spend an afternoon producing a model that performs slightly worse. I have watched people do exactly that because the word churn made them reach for the imbalance toolbox by reflex.

Imbalance arrives when you narrow the target to something the business will actually spend money on. Nobody funds a campaign against generic churn. They fund a campaign against high value customers who will cancel in the next thirty days, and that population is not 25 percent of the base, it is closer to 3 percent. Same customers, same features, one tighter definition, and your positive class has collapsed by a factor of eight. Every part of this article is about what happens next.

Because that thirty day window is not a column in the public Telco file, everything below runs on a reproducible generated dataset pinned at 3.44 percent positives with the same seed, so you can paste it and get identical numbers. Continuity with the churn project is in the framing and the cost model, not in the raw rows. Every figure quoted here came out of an actual run.

# tested against: Python 3.10, numpy 2.2.6, scipy 1.15.3,
# scikit-learn 1.7.2, imbalanced-learn 0.14.2
from collections import Counter
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=20, n_informative=6,
                           n_redundant=2, weights=[0.97, 0.03], flip_y=0.01,
                           class_sep=0.8, random_state=42)
print('class counts:', Counter(y))
print('prevalence:', round(y.mean(), 4))

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25,
                                      stratify=y, random_state=42)
print('train:', Xtr.shape, 'test positives:', int(yte.sum()))

class counts: Counter({0: 19312, 1: 688})
prevalence: 0.0344
train: (15000, 20) test positives: 172

Notice how small the evaluation base has become. 5,000 test rows sounds like plenty until you realise your entire signal lives in 172 of them. Flip twenty predictions and recall moves by 12 points. Fragile estimates are a permanent feature of rare event work, and they are the reason a single train test split is not enough to judge anything here.

Metrics that survive a rare positive class

Accuracy dies first. Predict the majority class for every row and you score 96.6 percent on this data while catching nothing. Any metric whose baseline is that high has no room left to tell you anything useful.

ROC AUC is the more dangerous one, because it looks respectable. It measures how well your scores rank a random positive above a random negative, and it uses the false positive rate, which has a giant denominator when negatives outnumber positives. Add two thousand false positives to a pool of nineteen thousand negatives and the false positive rate barely twitches. Our baseline model posts a ROC AUC of 0.8142 and a precision recall AUC of 0.3880 on exactly the same predictions. Only one of those two numbers is telling you what a retention team will experience.

Use average precision, which is the area under the precision recall curve, as your headline number. Its no skill baseline equals the prevalence, so 0.0344 here. A score of 0.3880 against a floor of 0.0344 is a real, interpretable eleven fold lift. Report recall and precision at whatever threshold you actually intend to ship, and report the confusion matrix counts as raw integers so nobody has to reverse engineer them from percentages.

Class weights, oversampling and undersampling compared

Four families of fix exist and they are less different than they look. Class weighting multiplies the loss contribution of minority rows, so a missed positive hurts the optimiser more; in scikit-learn that is class_weight=’balanced’. Random oversampling copies minority rows until the classes match. SMOTE, short for Synthetic Minority Over-sampling Technique, interpolates new minority points along the line between a minority row and one of its k nearest minority neighbours, with k_neighbors defaulting to 5. Random undersampling throws majority rows away.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline as SkPipeline
from sklearn.metrics import (average_precision_score, roc_auc_score,
                             recall_score, precision_score)
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline

base = SkPipeline([('sc', StandardScaler()),
                   ('lr', LogisticRegression(max_iter=1000, random_state=42))])
base.fit(Xtr, ytr)
p = base.predict_proba(Xte)[:, 1]
print('BASE accuracy:', round(base.score(Xte, yte), 4))
print('BASE roc_auc :', round(roc_auc_score(yte, p), 4))
print('BASE pr_auc  :', round(average_precision_score(yte, p), 4))
print('BASE recall  :', round(recall_score(yte, base.predict(Xte)), 4))

cw = SkPipeline([('sc', StandardScaler()),
                 ('lr', LogisticRegression(max_iter=1000, random_state=42,
                                           class_weight='balanced'))])
cw.fit(Xtr, ytr)
pc = cw.predict_proba(Xte)[:, 1]

sm = ImbPipeline([('sc', StandardScaler()),
                  ('smote', SMOTE(random_state=42, k_neighbors=5)),
                  ('lr', LogisticRegression(max_iter=1000, random_state=42))])
sm.fit(Xtr, ytr)
ps = sm.predict_proba(Xte)[:, 1]

BASE accuracy: 0.9706
BASE roc_auc : 0.8142
BASE pr_auc  : 0.388
BASE recall  : 0.1686
CW    pr_auc : 0.3071   CW    recall: 0.7442   CW    precision: 0.1067
SMOTE pr_auc : 0.319    SMOTE recall: 0.7384   SMOTE precision: 0.1089

Read that output slowly, because it contradicts what most articles on this topic promise. Recall more than quadrupled, which looks like a triumph. Meanwhile precision fell from 0.8788 to 0.1067, and average precision, which is threshold independent and therefore the honest summary of how well the model separates the classes, got worse under both treatments. 0.3880 became 0.3071 and 0.3190.

Nothing was learned. What changed was the operating point. Reweighting and resampling both push the decision boundary toward the minority class, which is exactly what lowering your threshold from 0.5 does, only less transparently and with a small amount of collateral damage to the score ranking itself.

Look at that spread across folds. Five fold cross validation on the SMOTE pipeline produced 0.3403, 0.3743, 0.3335, 0.2984 and 0.4217. Fold to fold variation of 0.12 is larger than the entire difference between the methods I am comparing. Anyone who tells you SMOTE beat class weighting on a single split of a rare event dataset is reading noise.

Where resampling leaks into your score

Now for the mistake that ends careers quietly. SMOTE manufactures a synthetic row by interpolating between two real minority rows. Run it on your whole dataset and then split, and some of your test set is now made of points constructed from training rows. Your model has effectively seen the answers. It is the same family of error covered in Part 11, except this version is easier to commit because the resampling line looks like preprocessing rather than fitting.

# WRONG. Do not ship this. Resampling before the split.
Xall, yall = SMOTE(random_state=42).fit_resample(X, y)
print('resampled all:', Counter(yall))

Xtr2, Xte2, ytr2, yte2 = train_test_split(Xall, yall, test_size=0.25,
                                          stratify=yall, random_state=42)
leaky = SkPipeline([('sc', StandardScaler()),
                    ('lr', LogisticRegression(max_iter=1000, random_state=42))])
leaky.fit(Xtr2, ytr2)

print('LEAKY pr_auc reported :',
      round(average_precision_score(yte2, leaky.predict_proba(Xte2)[:, 1]), 4))
print('LEAKY pr_auc, honest  :',
      round(average_precision_score(yte, leaky.predict_proba(Xte)[:, 1]), 4))

resampled all: Counter({0: 19312, 1: 19312})
LEAKY pr_auc reported : 0.8637
LEAKY pr_auc, honest  : 0.33

Rule that removes the whole class of error: never call fit_resample yourself. Put the sampler in an imblearn Pipeline and hand that pipeline to cross_val_score. Samplers in that pipeline are applied during fit and skipped entirely during predict, which is precisely the behaviour you want, and it means every cross validation fold resamples its own training portion while its validation portion stays at natural prevalence.

from sklearn.model_selection import StratifiedKFold, cross_val_score
import numpy as np

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
s = cross_val_score(sm, Xtr, ytr, cv=cv, scoring='average_precision')
print('folds:', np.round(s, 4), 'mean', round(s.mean(), 4))

# folds: [0.3403 0.3743 0.3335 0.2984 0.4217] mean 0.3536

Threshold tuning on cost, not accuracy

Here is the part I wish someone had put in front of me five years earlier. A classifier does not make decisions, it produces scores. Deciding is what the threshold does, and 0.5 is a default inherited from balanced problems that nobody chose deliberately for your use case. If your positive class is rare and expensive, 0.5 is almost certainly wrong.

Pick the threshold with arithmetic instead. For our retention case, a missed churner costs the margin on a customer we lose, call it 400 currency units. A false positive costs a retention offer sent to somebody who was staying anyway, call it 25. Ratio of sixteen to one is what matters, not the absolute figures, and your finance partner can give you that ratio in a ten minute conversation. Anyone who has designed an experiment, as covered in A/B testing for analysts, is already used to putting a value on each outcome.

import numpy as np
from sklearn.metrics import confusion_matrix

COST_FN, COST_FP = 400.0, 25.0

def best_threshold(scores):
    best = None
    for t in np.arange(0.01, 0.991, 0.005):
        tn, fp, fn, tp = confusion_matrix(yte, (scores >= t).astype(int)).ravel()
        cost = fn * COST_FN + fp * COST_FP
        if best is None or cost < best[1]:
            best = (round(t, 3), cost, tp, fp, fn)
    return best

print('BASE  best:', best_threshold(p))
print('CW    best:', best_threshold(pc))
print('SMOTE best:', best_threshold(ps))

BASE  best: (0.08,  36225.0, tp=104, fp=361, fn=68)
CW    best: (0.595, 38500.0, tp=118, fp=676, fn=54)
SMOTE best: (0.595, 38275.0, tp=121, fp=715, fn=51)

# for comparison, class_weight model left at the 0.5 default:
# cost 44400.0, tp=128, fp=1072, fn=44

Sit with that result. Plain logistic regression, no resampling, no class weights, one tuned number, came in at 36,225 against 38,500 for balanced class weights and 38,275 for SMOTE. Roughly 6 percent cheaper than either treatment, and 18 percent cheaper than the class weighted model left at its default cutoff. It also caught fewer churners, 104 against 118, and that is the point: catching more churners is not the objective, spending less money is. Resampling bought recall at a price nobody had checked.

Errors you will hit with imbalanced-learn

Two failures will find you within the first hour, and both produce tracebacks that read as more mysterious than they are.

First, every constructor argument in current imbalanced-learn is keyword only. Copy an older snippet that passes a sampling ratio positionally and you get this:

>>> SMOTE(1.0)
Traceback (most recent call last):
  File "<string>", line 3, in <module>
TypeError: SMOTE.__init__() takes 1 positional argument but 2 were given

# fix: name the argument
SMOTE(sampling_strategy=1.0, random_state=42)

Second, and this one catches nearly everybody, a sampler will not go into a scikit-learn Pipeline. Samplers implement fit_resample, not transform, and they change the number of rows, which the scikit-learn contract forbids for an intermediate step:

>>> from sklearn.pipeline import Pipeline
>>> p = Pipeline([('smote', SMOTE(random_state=0)), ('lr', LogisticRegression())])
>>> p.fit(X, y)
  File ".../sklearn/pipeline.py", line 340, in _validate_steps
    raise TypeError(
TypeError: All intermediate steps should be transformers and implement fit and
transform or be the string 'passthrough' 'SMOTE(random_state=0)'
(type <class 'imblearn.over_sampling._smote.base.SMOTE'>) doesn't

# fix: import Pipeline from imblearn instead
from imblearn.pipeline import Pipeline

Third failure has no traceback, which makes it worse. SMOTE interpolates numerically, so applying it to one hot encoded categorical columns produces values like 0.37 for a column that can only ever be 0 or 1. Model trains happily on rows that describe customers who are 37 percent on a two year contract. Use SMOTENC when you have mixed types and tell it which column indices are categorical, or SMOTEN when everything is categorical.

Order I would work in for a rare positive class

My recommendation for this part is narrow and I will defend it: treat imbalance as a decision threshold problem first and a data problem only if that fails. Fit the model on untouched data, report average precision against the prevalence floor, then sweep the cutoff against a real cost ratio on a validation split. On this dataset that sequence produced the cheapest outcome on the board, and it did so without a single synthetic row.

Method I would actively avoid is random undersampling on a dataset of this size. Balancing 19,312 negatives down to 688 throws away 96 percent of your majority evidence to solve a problem a threshold already solved, and it makes every downstream estimate noisier. It has a legitimate use when your negative class runs to tens of millions of rows and training time is the binding constraint, which is a capacity decision rather than a statistical one.

One caveat before you generalise this. My comparison used logistic regression, a linear model whose decision surface resampling can only shift, not reshape. Gradient boosted trees behave differently, and on genuinely sparse minority regions I have seen SMOTE variants earn their place. Run the comparison on your own data with cross validation and a cost column rather than trusting either my result or a blog post that only shows the recall improvement.

Part 15 picks this up directly. Once you accept that fold to fold variance of 0.12 swamps the difference between methods, hyperparameter tuning becomes a question of honest model selection rather than leaderboard chasing, and nested cross validation stops looking like academic overhead. Bring the cost function from this part with you, because tuning against average precision and tuning against money do not always agree.

Try this on your own project this week: take whatever classifier you have in production, price a false negative and a false positive with whoever owns the budget, and sweep the threshold. If the number you find is not 0.5, you have just improved a model without retraining anything.

References

• imbalanced-learn API reference, SMOTE, for the keyword only signature and the k_neighbors default of 5 in version 0.14.2
• imbalanced-learn Pipeline, which documents that resampling happens on fit and is skipped on predict
• scikit-learn user guide, Metrics and scoring, for average_precision_score and the precision recall curve
• Chawla, Bowyer, Hall and Kegelmeyer, SMOTE, JAIR 2002, the original paper the method comes from
• Telco Customer Churn dataset, the public IBM sample file this series works from

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts