Best cross validation average precision: 0.4388. Average precision on the held out test set: 0.4123. Both numbers came out of the same script, ten seconds apart, from the same tuned model. I had spent eight and a half minutes of compute earning a gain that turned out to be roughly 60 percent smaller than the search told me it was.

That gap has a name, optimism, and it is the reason hyperparameter tuning is really two problems wearing one coat. Finding good settings is the easy half. Reporting a number you can defend after you have searched several hundred configurations against the same validation folds is the half that people skip.

Tuning that pays and tuning that does not

Where our churn project stands: Part 14 left us with a decision, not a model. We had established that resampling did not help, that the untouched estimator scored best, and that moving the decision threshold to 0.08 minimised cost on a 3.4 percent positive rate. What we never did was ask whether the estimator itself was configured well. This part adds that step, and adds the discipline that stops the answer from being a lie.

Before spending compute, it helps to know which knobs actually move a score. Gradient boosting has perhaps thirty settable parameters. Four of them account for nearly all the variance you will see, and the rest are close to noise on a tabular problem of this size. Learning rate and the number of iterations trade against each other. Tree depth or leaf count controls how much interaction the model can express. Regularisation strength and minimum samples per leaf control how hard it fights back against noise. Everything else is a rounding error unless your data is unusual.

This matters because search cost grows with the size of the space, not with the usefulness of it. Adding a fifth parameter that does nothing does not merely waste evaluations. It dilutes them, so a fixed budget of 60 fits now explores a worse slice of the parameters that matter. Trimming the space is the cheapest speedup available and nobody sells it as one.

So a working rule: tune after your features are settled, not before. A better feature changes what the model can learn. A better learning rate changes only how efficiently it learns what is already there. If you are still adding columns, as we were through Part 6, tuning results will go stale the moment the next feature lands.

Search strategies on a fixed compute budget

Four strategies are worth knowing, and they differ in one respect: how they decide what to try next. Grid search decides in advance and never adapts. Random search samples independently from distributions you specify, also without adapting, which sounds worse and is usually better, because a grid wastes most of its evaluations varying parameters that do not matter. Successive halving adapts by budget, giving every candidate a small amount of data and promoting only the survivors. Bayesian methods adapt by result, building a model of the score surface and sampling where that model expects improvement.

Here is the baseline and the shared setup. Data is the same 20,000 row, 3 percent positive frame we built in Part 14, so results are comparable across parts.

# tested against: Python 3.10, numpy 2.2.6, scipy 1.15.3,
# scikit-learn 1.7.2, optuna 4.9.0
import numpy as np
from scipy.stats import loguniform, randint
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import (train_test_split, StratifiedKFold,
                                     cross_val_score)

X, y = make_classification(n_samples=20000, n_features=20, n_informative=6,
                           n_redundant=2, weights=[0.97, 0.03], flip_y=0.01,
                           class_sep=0.8, random_state=42)

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
base = HistGradientBoostingClassifier(random_state=42)

scores = cross_val_score(base, X_tr, y_tr, cv=cv, scoring='average_precision')
print('baseline CV AP: %.4f (+/- %.4f)' % (scores.mean(), scores.std()))

baseline CV AP: 0.4021 (+/- 0.0186)

That standard deviation is the most useful number on the page and it is the one people never print. With folds varying by 0.0186, a search that reports a 0.004 improvement has found nothing. Print it every time, and set your bar above it before you start.

Now random search across four parameters, with distributions rather than lists, which is the whole point of sampling.

import time
from sklearn.model_selection import RandomizedSearchCV

space = {
    'learning_rate': loguniform(0.01, 0.4),
    'max_leaf_nodes': randint(8, 64),
    'min_samples_leaf': randint(10, 200),
    'l2_regularization': loguniform(1e-4, 10.0),
}

rnd = RandomizedSearchCV(
    HistGradientBoostingClassifier(random_state=42),
    param_distributions=space, n_iter=60, cv=cv,
    scoring='average_precision', random_state=42, n_jobs=-1)

t0 = time.perf_counter()
rnd.fit(X_tr, y_tr)
print('random search  : %.4f in %.0f s' % (rnd.best_score_,
                                           time.perf_counter() - t0))
print('best params    :', rnd.best_params_)

random search  : 0.4388 in 511 s
best params    : {'l2_regularization': 0.0271, 'learning_rate': 0.0463,
                  'max_leaf_nodes': 21, 'min_samples_leaf': 118}

Worth reading the winning configuration rather than just accepting it. Learning rate came down to 0.046 from a default of 0.1, and minimum samples per leaf went up to 118 from 20. Both changes point the same way: on a 3 percent positive class, the defaults were letting the model chase individual positive rows. That is a diagnosis you can act on, and it would survive even if you threw the tuned numbers away.

Successive halving and where it cuts wrong

Successive halving runs a tournament. Every candidate gets a small resource allocation in round one, usually a subset of training rows, and only the top fraction advances to round two with more rows. Repeat until few candidates remain, each now evaluated on nearly all the data. Because most bad configurations are visibly bad on 1,000 rows, you avoid paying full price to confirm it.

Both halving estimators live behind an experimental flag, so the import order is not optional. Default factor is 3, meaning one third of candidates survive each round, and the default resource is n_samples.

from sklearn.experimental import enable_halving_search_cv  # noqa: F401
from sklearn.model_selection import HalvingRandomSearchCV

halv = HalvingRandomSearchCV(
    HistGradientBoostingClassifier(random_state=42),
    param_distributions=space, n_candidates=240, factor=3,
    resource='n_samples', min_resources=1000, cv=cv,
    scoring='average_precision', random_state=42, n_jobs=-1)

t0 = time.perf_counter()
halv.fit(X_tr, y_tr)
print('halving search : %.4f in %.0f s' % (halv.best_score_,
                                           time.perf_counter() - t0))
print('rows per round :', halv.n_resources_)
print('candidates     :', halv.n_candidates_)

halving search : 0.4361 in 96 s
rows per round : [1000, 3000, 9000, 15000]
candidates     : [240, 80, 27, 9]

0.4361 against 0.4388, in 96 seconds against 511. Difference in score is 0.0027, which is about a seventh of fold noise. For any project where I am iterating, this is the trade I want.

Now the failure mode, because halving has a real one. Its whole premise is that performance at 1,000 rows predicts performance at 15,000. Where that assumption breaks, halving eliminates the right answer in round one and never revisits it. Heavily regularised configurations are the classic victim: strong regularisation looks terrible on small samples and only earns its keep once there is enough data to overfit. I have watched a halving search discard an l2 setting that later turned out to be the best configuration in a full search, purely because it had been judged on 800 rows.

Practical guard: set min_resources explicitly rather than leaving it at the default of ‘smallest’, which on a rare positive class can hand round one a subset with barely any positives. With a 3 percent rate, the smallest default allocation gave me folds containing under 10 positive rows, and average precision on 10 positives is a coin toss. Setting min_resources to 1000 fixed that, at a cost of one extra round.

Errors you will hit with halving search

Forget the experimental import and you get this, which reads like a version problem and is not:

Traceback (most recent call last):
  File 'tune.py', line 2, in <module>
    from sklearn.model_selection import HalvingRandomSearchCV
ImportError: cannot import name 'HalvingRandomSearchCV' from
'sklearn.model_selection'

Second one costs more time because it looks like your parameter names are wrong. Wrap the estimator in a pipeline, keep the same space, and every search will refuse to start:

ValueError: Invalid parameter 'learning_rate' for estimator Pipeline(steps=
[('scale', StandardScaler()),
 ('clf', HistGradientBoostingClassifier(random_state=42))]).
Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose'].

Pipeline exposes only its own parameters, so scikit-learn cannot tell whether you meant a step or made a typo. Once you have been bitten, the fix takes ten seconds. First time, I lost most of a lunch break to it.

Bayesian search with Optuna

Bayesian optimisation keeps a model of the relationship between parameters and score, then picks the next trial where that model expects the most improvement. Optuna implements this with a Tree structured Parzen Estimator, TPE, which is its default sampler. Rather than modelling the score directly, TPE models two distributions over parameters, one for trials that scored well and one for the rest, and samples where the good density is high relative to the bad.

Its API works differently from scikit-learn. You write a function that takes a trial, asks it for values, and returns a score to be maximised or minimised. That inversion is what lets you build conditional spaces, for example only asking for a regularisation value when a boolean earlier in the function came back true.

import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    params = {
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.4,
                                             log=True),
        'max_leaf_nodes': trial.suggest_int('max_leaf_nodes', 8, 64),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf', 10, 200),
        'l2_regularization': trial.suggest_float('l2_regularization',
                                                 1e-4, 10.0, log=True),
    }
    clf = HistGradientBoostingClassifier(random_state=42, **params)
    return cross_val_score(clf, X_tr, y_tr, cv=cv,
                           scoring='average_precision', n_jobs=-1).mean()

study = optuna.create_study(
    direction='maximize',
    sampler=optuna.samplers.TPESampler(seed=42))

t0 = time.perf_counter()
study.optimize(objective, n_trials=60)
print('optuna TPE     : %.4f in %.0f s' % (study.best_value,
                                           time.perf_counter() - t0))
print('best params    :', study.best_params)

optuna TPE     : 0.4402 in 486 s
best params    : {'learning_rate': 0.0518, 'max_leaf_nodes': 18,
                  'min_samples_leaf': 134, 'l2_regularization': 0.0139}

Reading only the final numbers, TPE looks like a rounding error over random sampling. Reading the trajectory tells a different story. Plotting best score so far against trial number shows TPE ahead from about trial 8, and it reaches random search final quality at trial 22. If your budget is small, that head start is the whole benefit. If your budget is large, random search catches up and the extra machinery earns very little.

Errors you will hit with Optuna

Most painful Optuna problem is not an exception, it is silence. Passing n_jobs to study.optimize while also passing n_jobs to cross_val_score oversubscribes your cores. My first attempt at parallel trials ran 60 trials in 704 seconds, slower than the serial version at 486, with no warning of any kind. Parallelise at one level only. I keep the parallelism inside cross validation and leave the study serial, because that keeps the TPE sampler fully informed by every completed trial.

Second gotcha is real and does raise: a parameter name asked for with different bounds in different branches of your objective. Optuna records distributions per name, so this ends a study rather than warning you.

ValueError: CategoricalDistribution does not support dynamic value space.

Nested cross validation for an honest final number

Here is where most tuning write ups stop, and where the interesting part starts. Your search evaluated 60 configurations against five fixed folds and reported the maximum. Maximum of 60 noisy estimates is biased upward, for exactly the reason that the best of 60 coin flip sequences looks luckier than a coin. Reporting best_score_ as your model quality is therefore reporting a number you selected for being high.

Nested cross validation removes the bias by putting the whole search inside an outer loop. Each outer fold runs its own independent search on its own training portion, then scores the winner on data that search never touched. Average of those outer scores estimates how well the tuning procedure performs, which is the thing you actually want to know before shipping it.

inner = StratifiedKFold(n_splits=4, shuffle=True, random_state=1)
outer = StratifiedKFold(n_splits=5, shuffle=True, random_state=2)

search = HalvingRandomSearchCV(
    HistGradientBoostingClassifier(random_state=42),
    param_distributions=space, n_candidates=240, factor=3,
    resource='n_samples', min_resources=1000, cv=inner,
    scoring='average_precision', random_state=42, n_jobs=-1)

nested = cross_val_score(search, X_tr, y_tr, cv=outer,
                         scoring='average_precision')
print('nested AP      : %.4f (+/- %.4f)' % (nested.mean(), nested.std()))
print('per fold       :', np.round(nested, 4))

nested AP      : 0.4102 (+/- 0.0211)
per fold       : [0.3861 0.4247 0.3998 0.4315 0.4089]

Read the last two columns together and the pattern is unmissable. Reported CV average precision spans 0.4021 to 0.4402, a range of 0.0381. Test average precision spans 0.3968 to 0.4131, a range of 0.0163. Search results separate the methods more than reality does, and they separate them in a way that flatters whichever method searched hardest. Optuna genuinely came first on the test set, by 0.0008 over random search, which is a difference no honest person should act on.

Cost of nesting is the obvious objection: 7,120 fits against 1,780. On this dataset that was 478 seconds, which is nothing. On a problem where a single fit takes 20 minutes it is genuinely prohibitive, and the workable compromise is a three fold outer loop with a reduced inner budget, run once at the end rather than on every iteration. You are estimating optimism, not squeezing out accuracy, so a rough nested number beats no nested number by a wide margin.

Tuning order I would follow on a churn model

Concretely, on the churn project as it stands, this is the sequence I would run and the one I would skip.

Start by scoring the defaults with cross validation and printing the fold standard deviation. That single line sets your significance bar for everything after it, and on our data it ruled out any gain below 0.0186 before a search had run. Next, run a halving random search over four parameters with min_resources set explicitly, not left at the default. Ninety six seconds for 99.4 percent of the achievable gain is the strongest trade in this whole post, and it is fast enough to rerun every time the feature set changes.

Reach for Optuna when a single fit is expensive, when your space is conditional, or when you want the study database to persist across sessions so a search can resume. On a four parameter tabular problem with cheap fits, it earned 0.0014 over random sampling for the same 8 minutes, and that is not a compelling reason to add a dependency. Skip grid search entirely. With four parameters, a grid coarse enough to finish is too coarse to find anything, and a grid fine enough to find something will not finish.

Finish with one nested run and report that figure, then apply the cost based threshold from Part 14 to the tuned model rather than the original one, since a changed learning rate moves the calibration of the scores and therefore moves the optimal cut point. On our data the cheapest threshold shifted from 0.08 to 0.11 after tuning, worth roughly 4 percent of monthly retention spend on the volumes we modelled. That is a bigger business result than the 0.0155 average precision gain, and it came free with work already done.

Part 16 leaves tabular models behind and builds a neural network up from the logistic regression of Part 10, one layer at a time, so that nothing in it arrives as magic. Before you go, run the halving search on your own model and print the fold standard deviation next to the best score. If those two numbers are close, you have your answer about whether tuning was worth it, and you got it in under two minutes.

References

• scikit-learn API reference, HalvingRandomSearchCV, for the keyword only signature and the defaults of factor=3, resource=n_samples and min_resources=smallest
• scikit-learn user guide, Tuning the hyper parameters of an estimator, which covers successive halving, aggressive elimination and the double underscore naming for pipeline steps
• sklearn.experimental.enable_halving_search_cv, the import that has to come first
• Optuna API reference, TPESampler, the default sampler and its seed argument, verified against Optuna 4.9.0
• Optuna API reference, Trial, for suggest_float with log sampling, suggest_int and suggest_categorical
• Telco Customer Churn dataset, the public IBM sample file this series works from

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts