Gradient boosting is the default winner on tabular data, and on this dataset it lost. Not narrowly, either. Out of the box it scored 0.8104 against a logistic regression that had already posted 0.8306 in Part 11, and it took three rounds of tuning to claw back to roughly where the linear model started.

That result is not a knock on boosting. It is a knock on the habit of reaching for the strongest available model before checking whether the dataset can feed it. Tree ensembles earn their reputation at scale, on wide feature sets with interactions a linear model cannot express. Give one 486 rows and fifteen mostly independent columns and it has very little to find that logistic regression missed. Both facts are worth carrying, and this part demonstrates them on our own data rather than asserting them.

Where the churn project stands after Part 11

Last part we stopped trusting single split scores and built an honest evaluation protocol: five fold stratified cross validation, every fitted step inside the pipeline, thresholds swept on out of fold predictions. That protocol handed us a defensible baseline of 0.8306 AUC with a standard deviation of 0.0253, and an average precision of 0.6121 against a 0.2716 base rate.

This part swaps the model and keeps everything else frozen. Same 486 row extract of the public Telco Customer Churn file, same 132 churners, same six categorical and three numeric columns from Part 6, same folds, same seed. Only the estimator changes. Freezing everything but one variable is the only way a model comparison means anything, and it is a discipline more teams skip than admit.

# tested against: Python 3.10, numpy 2.2.6, pandas 2.3.3, scikit-learn 1.7.2
import pandas as pd
from sklearn.model_selection import StratifiedKFold

df = pd.read_csv('Telco-Customer-Churn.csv')
df['TotalCharges'] = pd.to_numeric(df['TotalCharges'], errors='coerce')
df = df.dropna(subset=['TotalCharges'])

y = (df['Churn'] == 'Yes').astype(int)
num = ['tenure', 'MonthlyCharges', 'TotalCharges']
cat = ['Contract', 'InternetService', 'PaymentMethod',
       'PaperlessBilling', 'TechSupport', 'OnlineSecurity']
X = pd.get_dummies(df[num + cat], columns=cat, drop_first=True).astype(float)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
print('X shape', X.shape)

One decision tree, and how it overfits

A decision tree asks a sequence of yes or no questions about a customer and follows the answers to a leaf, which holds a predicted churn probability. Is the contract month to month? Is tenure below 15? Each split is chosen greedily, meaning the algorithm picks whichever single question best separates churners from stayers right now, then repeats inside each branch without ever reconsidering an earlier choice.

Left unconstrained, splitting continues until every leaf is pure. With 486 customers a tree can memorise all of them, and that is exactly what happens. Watch train and test scores together, which is a habit worth building because a single number cannot show you this failure at all.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_validate

for d in [None, 2, 3, 4, 5, 6, 8, 10]:
    r = cross_validate(DecisionTreeClassifier(max_depth=d, random_state=42),
                       X, y, cv=cv, scoring='roc_auc', return_train_score=True)
    print('depth %-5s train %.4f  test %.4f  std %.4f' % (
        d, r['train_score'].mean(), r['test_score'].mean(), r['test_score'].std()))

depth None  train 1.0000  test 0.6222  std 0.0566
depth 2     train 0.8094  test 0.7839  std 0.0312
depth 3     train 0.8495  test 0.7953  std 0.0411
depth 4     train 0.8869  test 0.7824  std 0.0390
depth 5     train 0.9248  test 0.7553  std 0.0319
depth 6     train 0.9516  test 0.7364  std 0.0423
depth 8     train 0.9842  test 0.6787  std 0.0691
depth 10    train 0.9952  test 0.6739  std 0.0374

Best single tree lands at depth 3 with 0.7953, which loses to logistic regression by three and a half points. Note also how noisy the shallow trees are: a standard deviation of 0.0411 at depth 3 against 0.0253 for the linear model. Trees are high variance estimators. Change a handful of rows and the top split can flip, and every split beneath it changes with it. Instability of that kind is the property ensembles are built to attack, and understanding it makes the rest of this part obvious rather than magical.

Bagging and random forests

Bagging, short for bootstrap aggregating, trains many trees on random samples of the rows drawn with replacement, then averages their predictions. Individual trees stay deep and overfitted on purpose. Because each one overfits a different random sample, their errors are partly independent, and averaging cancels a good share of them. A random forest adds a second source of randomness: at every split each tree may only consider a random subset of the features, which stops every tree from opening with the same dominant column and keeps them genuinely different from one another.

Effect on our data is immediate and large. Trees that individually score 0.62 combine into a forest at 0.8156, beating the best single tree by more than two points and arriving there without a single tuned parameter.

Two practical readings come out of that curve. First, n_estimators is not a parameter you tune, it is a budget you set: pick a number past the elbow and forget it, since more trees never hurt accuracy, only wall clock. Going from 300 to 800 trees cost roughly three times the fit time and moved AUC by minus 0.0018, which is noise pointing the wrong way. Second, and easy to miss, fold to fold variance drops alongside the mean rising. Averaging buys stability as well as accuracy, and stability is what makes a number safe to quote to somebody who will hold you to it.

Gradient boosting, and how it differs from a forest

Boosting inverts the arrangement. Rather than growing many independent trees and averaging, it grows them one after another, each new tree fitted to the errors the ensemble has made so far. Start with a constant prediction, measure how wrong it is on every customer, fit a small tree to those errors, add a shrunken fraction of that tree to the running prediction, measure the new errors, repeat. Gradient refers to nothing more exotic than the direction of those current errors.

Consequences follow directly from the sequence. Boosted trees are deliberately weak, often just a few levels deep, because each one only needs to fix a slice of remaining error. Learning rate controls how much of each tree gets added, defaulting to 0.1 in scikit-learn, and it trades against the number of iterations: halve the rate and you need roughly twice the trees. Most importantly, boosting has no averaging step to protect it, so it will happily keep fitting until it has memorised your noise. Where a forest is close to safe by construction, boosting requires you to say when to stop.

from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier

models = {
    'rf defaults':      RandomForestClassifier(n_estimators=300, random_state=42, n_jobs=-1),
    'rf min_leaf 10':   RandomForestClassifier(n_estimators=300, min_samples_leaf=10,
                                               random_state=42, n_jobs=-1),
    'hgb defaults':     HistGradientBoostingClassifier(random_state=42),
    'hgb regularised':  HistGradientBoostingClassifier(learning_rate=0.05, max_leaf_nodes=15,
                                                       min_samples_leaf=20, l2_regularization=1.0,
                                                       random_state=42),
    'hgb early stop':   HistGradientBoostingClassifier(learning_rate=0.05, early_stopping=True,
                                                       validation_fraction=0.15,
                                                       n_iter_no_change=15, random_state=42),
}
for name, m in models.items():
    r = cross_validate(m, X, y, cv=cv, scoring=['roc_auc', 'average_precision'],
                       return_train_score=True)
    print('%-16s train %.4f  test %.4f  std %.4f  ap %.4f' % (
        name, r['train_roc_auc'].mean(), r['test_roc_auc'].mean(),
        r['test_roc_auc'].std(), r['test_average_precision'].mean()))

Read that table honestly and the headline is uncomfortable for anyone who reaches for boosting reflexively. Every boosting variant lost to logistic regression. Regularising helped, moving 0.8104 to 0.8203, and early stopping landed in the same place at 0.8210 while cutting training AUC from 0.9995 to 0.9387, which is a healthier looking model even though the held out score barely moved. Only the constrained random forest cleared the linear baseline at all, by 0.0056 AUC.

Root cause is sample size, not the algorithm. Boosting fits residual structure, and with 486 rows and 132 positives there is barely any residual structure left over once contract type, tenure and internet service have been accounted for. Feed the same pipeline the full 7,043 row file and the ordering shifts, as it does on most real churn tables. Both results belong in your head at once: boosting usually wins on tabular data, and usually is doing real work in that sentence.

One error worth showing, because it catches everybody exactly once. Trees do not need scaling, which tempts people to skip the preprocessing pipeline entirely and hand scikit-learn the raw frame. Categorical columns are still strings, and the message is blunt about it:

>>> RandomForestClassifier().fit(df[['tenure', 'Contract']], y)
Traceback (most recent call last):
  File "sklearn/utils/validation.py", line 1053, in check_array
    array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
ValueError: could not convert string to float: 'Month-to-month'

Native categorical support is genuinely convenient, and it avoids widening 15 columns into a sparser matrix on higher cardinality data. On our features it scored 0.8181 against 0.8203 for the one hot version, so the convenience is free rather than an accuracy win. Reach for it when a column has dozens of levels, not to squeeze out points.

Feature importance that misleads

Every fitted forest exposes feature_importances_, and reading it is how most people first form an opinion about what drives churn. That attribute measures mean impurity decrease, meaning how much each feature reduced node impurity across all trees. It carries a bias that is well documented and still catches teams constantly: features with many distinct values get more chances to produce a lucky split, so continuous and high cardinality columns score high regardless of whether they mean anything.

Demonstrating this takes one extra column. I added customer_ref, a six digit random integer with no relationship to churn whatsoever, and refitted the constrained forest.

import numpy as np, pandas as pd
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(7)
X2 = X.copy()
X2['customer_ref'] = rng.integers(100000, 999999, len(X2)).astype(float)  # pure noise

Xtr, Xte, ytr, yte = train_test_split(X2, y, test_size=.25, stratify=y, random_state=42)
rf = RandomForestClassifier(n_estimators=300, min_samples_leaf=10,
                            random_state=42, n_jobs=-1).fit(Xtr, ytr)

imp = pd.Series(rf.feature_importances_, index=X2.columns).sort_values(ascending=False)
print(imp.head(4).round(4).to_string())

p = permutation_importance(rf, Xte, yte, n_repeats=30, random_state=42, scoring='roc_auc')
pi = pd.Series(p.importances_mean, index=X2.columns).sort_values(ascending=False)
print(pi.head(4).round(4).to_string())
print('customer_ref permutation importance %.4f' % pi['customer_ref'])

tenure           0.2931
TotalCharges     0.2108
MonthlyCharges   0.1778
customer_ref     0.0701     <-- ranked 4th of 16, ahead of Contract

tenure                        0.0666
MonthlyCharges                0.0592
InternetService_Fiber optic   0.0241
Contract_One year             0.0087
customer_ref permutation importance 0.0064

Permutation importance answers a better question. Shuffle one column on held out data, refit nothing, and see how far the score falls. A column the model genuinely relies on will hurt when scrambled; a column it merely split on by accident will not. Two cautions come with it. Run it on test data, since permuting on training data measures memorisation rather than dependence. And treat correlated features carefully, because shuffling one of a correlated pair lets the model recover through the other, so both look unimportant. You can see that above: Contract two year scores 0.0500 on impurity and 0.0091 on permutation, largely because tenure already encodes contract length. Neither method is a causal statement, which is a distinction worth rereading in correlation versus causation before you present importances to anyone who might act on them.

Tuning that paid, and tuning that did not

Across roughly thirty configurations run for this part, gains clustered in one place. Constraining leaf size on the forest, min_samples_leaf moving from the default of 1 to 5 and then 10, produced 0.8156 then 0.8343 then 0.8362. That is the entire story of forest tuning on a dataset this size, and it took under a minute to find. Everything else moved the fourth decimal place.

Boosting was fussier and repaid the fuss less. Dropping learning rate to 0.05 while capping max_leaf_nodes at 15 and adding l2_regularization of 1.0 bought 0.0099 AUC over defaults. Early stopping bought about the same, arrived faster, and produced the more trustworthy training curve. Worth noting that early stopping carves its validation slice out of the training fold, so the reported score stays honest, but you are training on 85 percent of an already small fold. Below a thousand rows I would rather regularise explicitly than let early stopping shrink the training data further.

Model I would ship for churn, and the one I would avoid

For this extract I would ship the logistic regression from Part 10 and keep the forest as a diagnostic. Fifty six ten thousandths of AUC does not justify trading coefficients a retention manager can read for an ensemble that needs tooling to explain. Average precision does favour the forest more clearly, 0.6415 against 0.6121, which is a real 5 percent relative improvement in the cleanliness of the flagged list, so on a larger extract I would revisit that call. On the full 7,043 row file I expect the forest to win properly, and the correct move then is to ship it with permutation importances and partial dependence computed as part of the release rather than promised afterwards.

Model I would avoid outright is a single decision tree. It is the one people gravitate to because a tree diagram is easy to show in a meeting, and it was the worst performer here at every depth. Interpretability from a lone tree is largely an illusion anyway, since resampling the data reorders the splits and the picture you presented last month no longer holds. Where you truly need a readable model, a linear one is both more stable and more honest. Between the two boosting implementations, use HistGradientBoostingClassifier rather than the older GradientBoostingClassifier: it was faster here, 0.24 to 0.39 seconds against 0.49, scored better, and it handles categoricals and missing values natively.

Our churn project now has four candidate models scored on identical folds and a clear picture of what each costs to explain. Part 13 turns to unsupervised learning, clustering and dimensionality reduction, where there is no label to score against and evaluation becomes considerably harder than anything in this part. Before moving on, add customer_ref to your own feature frame and compare the two importance rankings yourself. Seeing a random number outrank contract type on your own screen is what makes the lesson stick.

References

• scikit-learn user guide, Ensembles: gradient boosting, random forests, bagging, for the bootstrap and random feature subset description and the recommendation to prefer histogram based boosting on larger samples.
• scikit-learn, HistGradientBoostingClassifier API reference, for the learning_rate default of 0.1, max_iter default of 100, and native categorical_features support.
• scikit-learn user guide, Permutation feature importance, for the impurity bias toward high cardinality features and the warning about correlated predictors.
• scikit-learn, RandomForestClassifier API reference, for min_samples_leaf, max_features and n_estimators defaults.
• Telco Customer Churn dataset, IBM sample data on GitHub, 7043 rows, the file used throughout this series.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts