A column of random integers, mean encoded against the target, scored an AUC of 0.800 on training data and 0.492 on holdout. Nothing in that column carried any signal whatsoever, and I have watched that exact pattern survive three rounds of review because the training number looked so convincing.

Where the churn project stands after Part 5

Last part we ended with a typed interim copy of the Telco Customer Churn table: 7043 rows, 21 columns, dtypes asserted at read time, and a written note that TotalCharges arrives as text rather than a number. This part adds derived columns and an encoding strategy on top of that file. By the end of it we will have a feature matrix that a classifier can consume, and, more importantly, a habit that keeps the feature work from quietly poisoning every score we produce from Part 9 onwards.

Feature engineering means turning raw columns into inputs a model can use. That covers three separate jobs which people tend to blur together: converting types so the maths works at all, encoding categories into numbers, and inventing new columns that express a relationship the raw data only implies. Third one is where the accuracy hides. First two are where the bugs hide.

Before anything else, that TotalCharges column needs handling, and it is worth showing what happens if you handle it the obvious way.

import pandas as pd

df = pd.read_csv('telco.csv')
print('shape', df.shape)

# the obvious move, which fails
df['TotalCharges'].astype(float)

Tested with scikit-learn 1.7.2, pandas 2.3.3 and NumPy 2.2.6 on Python 3.10.12. Current stable scikit-learn at the time of writing is 1.9.0.

shape (7043, 12)
ValueError: could not convert string to float: ' '

That single space is not a null. It is a literal space character sitting in eleven rows, and pandas reads the column as object because of it. Coercing rather than casting turns those into NaN, which you can then count and inspect.

tc = pd.to_numeric(df['TotalCharges'], errors='coerce')
print('nan count', tc.isna().sum())
print('tenure of nan rows', df.loc[tc.isna(), 'tenure'].unique())

nan count 11
tenure of nan rows [0]

Every one of those eleven rows has a tenure of zero. These are customers who signed up and have not been billed yet, so a total charge of zero is correct and imputing the column median would be wrong. This is a small example of a rule worth internalising: look at what the missing rows have in common before you decide how to fill them. Cleaning messy data in the Data Analyst Series covers the general case, so I will not repeat it here.

Four families of engineered features

Almost everything useful on a tabular problem falls into four groups. Knowing which group you are reaching for stops the work from becoming a random walk through pandas.

Encodings turn a category into numbers. Transforms reshape a numeric column so its distribution suits the model, for example a log transform on something heavily skewed. Ratios and differences compare two columns to each other, which is the family that most often produces real lift. Aggregates summarise many rows into one, such as a customer average across their billing history. Our churn table is one row per customer, so aggregates barely apply here, but they dominate on transactional data.

Choosing between them is mostly mechanical once you know the column type and its cardinality, meaning the number of distinct values it holds.

flowchart TD
  A[Raw column] --> B{Numeric or categorical}
  B -->|Numeric| C{Skewed or bounded}
  C -->|Skewed| D[Log or rank transform]
  C -->|Bounded| E[Scale only]
  B -->|Categorical| F{Cardinality above 15}
  F -->|No| G[One hot encode]
  F -->|Yes| H[Target encode with cross fitting]
  D --> I[Fit inside the pipeline]
  E --> I
  G --> I
  H --> I
  I --> J[Score on holdout only]

Encoder selection on tabular data. Cardinality drives the choice more than anything else.

Encoding categoricals without leaking the target

Our churn table has five categorical columns worth keeping at this stage: Contract, PaymentMethod, InternetService, Partner and Dependents. All five are low cardinality, so one hot encoding is the right call. Practical question is what happens when a category shows up in production that was never in the training data.

from sklearn.preprocessing import OneHotEncoder

train = df[['Contract', 'PaymentMethod', 'InternetService']].iloc[:5000]
enc = OneHotEncoder(sparse_output=False, handle_unknown='error')
enc.fit(train)
print('n out', len(enc.get_feature_names_out()))

# a contract length that did not exist when we trained
new = pd.DataFrame({'Contract': ['Three year'],
                    'PaymentMethod': ['Mailed check'],
                    'InternetService': ['DSL']})
enc.transform(new)

n out 10
ValueError: Found unknown categories ['Three year'] in column 0 during transform

That ValueError is the single most common production failure in a tabular model, and it never appears in development because your test split comes from the same file as your training split. Marketing launches a three year contract, the encoder has never seen the string, and the scoring job dies at three in the morning. Two arguments fix it. Setting handle_unknown to ignore emits an all zero row for the unseen value. Setting it to infrequent_if_exist routes the unseen value into the rare category bucket instead, which I prefer, because an all zero row is indistinguishable from a genuinely missing value further down the pipeline.

High cardinality columns need a different tool. Target encoding replaces each category with the average target value for that category, which compresses thousands of categories into one numeric column. It also introduces the most dangerous bug in applied machine learning, because the encoding is built from the very thing you are trying to predict. Fitting the encoder on rows that then get scored means each row has partially memorised its own answer.

Correct approach is cross fitting: split the training data into folds, and encode each fold using statistics computed only from the other folds. Scikit learn implements this inside TargetEncoder, so you get the protection for free provided you place it in a pipeline and never call fit on data you intend to score.

Ratio and tenure features on the churn table

Our table holds tenure, MonthlyCharges and TotalCharges. Each is informative on its own, but the interesting quantity is not in any of them. Dividing TotalCharges by tenure gives the average monthly spend across the whole relationship. Subtracting that from the current MonthlyCharges tells you whether this customer is paying more or less than they historically have, which is exactly the kind of change that precedes a cancellation. Model can in principle learn that relationship from the three raw columns, but a linear model cannot learn a ratio at all, and a tree needs many splits to approximate one.

import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

df['TotalCharges'] = pd.to_numeric(df['TotalCharges'], errors='coerce').fillna(0.0)
y = (df['Churn'] == 'Yes').astype(int)

num_raw = ['tenure', 'MonthlyCharges', 'TotalCharges']
cat = ['Contract', 'PaymentMethod', 'InternetService', 'Partner', 'Dependents']

def build(cols_num):
    return Pipeline([
        ('prep', ColumnTransformer([
            ('num', StandardScaler(), cols_num),
            ('cat', OneHotEncoder(handle_unknown='ignore'), cat)])),
        ('clf', LogisticRegression(max_iter=1000))])

base = df[num_raw + cat].copy()
s1 = cross_val_score(build(num_raw), base, y, cv=5, scoring='roc_auc')
print('baseline        AUC %.4f +/- %.4f' % (s1.mean(), s1.std()))

eng = base.copy()
eng['avg_monthly'] = np.where(eng.tenure > 0, eng.TotalCharges / eng.tenure, eng.MonthlyCharges)
eng['charge_delta'] = eng.MonthlyCharges - eng.avg_monthly
eng['is_new'] = (eng.tenure <= 3).astype(int)

num2 = num_raw + ['avg_monthly', 'charge_delta', 'is_new']
s2 = cross_val_score(build(num2), eng, y, cv=5, scoring='roc_auc')
print('with 3 features AUC %.4f +/- %.4f' % (s2.mean(), s2.std()))

Notice the guard on the division. Eleven customers have a tenure of zero, and dividing by zero produces inf, which propagates silently until StandardScaler returns all NaN and LogisticRegression raises an error about infinity that points nowhere near the real cause. Using np.where rather than a plain division is the difference between a five second fix and an hour of confusion.

Worth being honest about the result. On my run the three added features moved cross validated AUC by less than one thousandth, well inside the fold to fold standard deviation of about 0.008. In other words they did nothing measurable. That is a completely normal outcome and it is the reason the measurement step matters more than the idea generation step. I keep charge_delta anyway on real billing data because it earns its place there, but on a single snapshot per customer, where TotalCharges is close to tenure multiplied by MonthlyCharges by construction, the ratio adds no information the model did not already have.

Leakage measured rather than described

Telling someone that target leakage inflates scores rarely changes behaviour. Showing them the number does. Here is a deliberately extreme case: a column of random integers between 0 and 899, assigned to customers at random, containing no relationship to churn by construction. I mean encode it the naive way, using the mean churn rate per category computed across the whole training set.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import TargetEncoder
from sklearn.pipeline import make_pipeline
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
df['agent_id'] = rng.integers(0, 900, len(df)).astype(str)   # pure noise
X, y = df[['agent_id']], (df['Churn'] == 'Yes').astype(int)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.3, random_state=42, stratify=y)

# WRONG: one mean per category, computed on all training rows at once
m = ytr.groupby(Xtr.agent_id.values).mean()
tr_enc = Xtr.agent_id.map(m).values.reshape(-1, 1)
te_enc = Xte.agent_id.map(m).fillna(ytr.mean()).values.reshape(-1, 1)
lr = LogisticRegression().fit(tr_enc, ytr)
print('naive  train AUC %.3f  holdout AUC %.3f' % (
    roc_auc_score(ytr, lr.predict_proba(tr_enc)[:, 1]),
    roc_auc_score(yte, lr.predict_proba(te_enc)[:, 1])))

# RIGHT: TargetEncoder cross fits internally
pipe = make_pipeline(TargetEncoder(target_type='binary'), LogisticRegression())
pipe.fit(Xtr, ytr)
print('sklearn TargetEncoder holdout AUC %.3f' % roc_auc_score(yte, pipe.predict_proba(Xte)[:, 1]))

naive  train AUC 0.800  holdout AUC 0.492
sklearn TargetEncoder holdout AUC 0.508

Read those three numbers slowly. A column containing nothing scored 0.800 on training data, which would look like a strong feature in any notebook that stops there. On unseen rows it scored 0.492, meaning slightly worse than a coin flip. Scikit learn TargetEncoder, doing the same job with internal cross fitting, returned 0.508, correctly reporting that the column is worthless. Cause is simple arithmetic: with 900 categories across roughly 4900 training rows, each category holds about five customers, so its mean is largely determined by the handful of labels it is about to predict.

I learned this the expensive way on a subscription retention model several years ago. We shipped a churn classifier holding a cross validated AUC of 0.94, which everybody found slightly too good and nobody found time to question. Six weeks after launch the live precision sat at roughly 0.31 against an expected 0.78, and the retention team had spent most of a quarter of their discount budget on customers who were never going to leave. Cause turned out to be a support ticket category field: tickets tagged cancellation_request were being created by the cancellation workflow itself, so the feature was reading the outcome rather than predicting it. Removing that one column dropped offline AUC to 0.81 and raised live precision to 0.74. Two days to find, four months of budget already spent.

Since then I apply one blunt rule. Any offline score that surprises me on the upside gets treated as a bug report, not a result. Every feature gets asked a single question before it goes in: at the moment we need to make this prediction in production, would this value actually exist yet. Surprisingly many columns fail that test, and none of them fail it in a way that cross validation will catch, because cross validation splits rows, not time.

Only the first row is caught automatically. Remaining three need a human asking the right question.

Interaction features and where they pay off

An interaction feature combines two columns so that the effect of one depends on the value of the other. On our churn table, a month to month customer paying 95 pounds a month behaves very differently from a two year customer paying the same amount, yet a linear model given Contract and MonthlyCharges separately cannot represent that. It can only add their effects together. Multiplying the scaled charge by a month to month indicator gives the model a column that says loudly: expensive and unlocked, which is the combination that actually predicts cancellation.

Whether you need these depends entirely on your model family, and this is the point people most often get wrong. Linear and logistic regression cannot learn interactions, so you must build them by hand. Gradient boosted trees learn interactions natively through successive splits, so hand built ones usually add nothing and occasionally hurt by giving the tree a shortcut that overfits. I have watched a team spend a fortnight generating several hundred pairwise products for an XGBoost model and land a lift of 0.002 AUC, which was inside their fold noise. Same fortnight spent on getting three months of extra billing history would have moved the number by an order of magnitude more.

PolynomialFeatures with degree set to 2 and interaction_only set to true will generate every pairwise product for you, which sounds convenient and rarely is. On our five categoricals expanded to ten one hot columns plus three numerics, that produces 78 interaction terms from 13 inputs, most of them near constant zero because two rare categories seldom co occur. My preference is to hand pick two or three interactions that a domain expert can explain out loud, measure them, and keep only what survives. If you cannot say in one sentence why an interaction should matter, it is a lottery ticket rather than a feature.

One practical warning. Interaction terms built from one hot columns must be generated inside the pipeline, after encoding, not on the raw frame beforehand. Building them outside means the unseen category handling we set up earlier silently stops applying, and you are back to a scoring job that dies on a contract type nobody anticipated.

Feature work I would ship for this project

For the churn project specifically, here is what I would keep and what I would drop. Keep: TotalCharges coerced with errors set to coerce and filled with zero for the eleven zero tenure rows, one hot encoding on all five low cardinality categoricals with handle_unknown set to infrequent_if_exist, StandardScaler on the three numerics, and an is_new flag for customers under four months. Drop: avg_monthly and charge_delta, because on this snapshot they measured no lift and they add two more columns to maintain. Reintroduce them the moment we get real billing history rather than a single row per customer.

On encoders, my verdict is direct. Use OneHotEncoder as the default and reach for TargetEncoder only above roughly 15 categories. Avoid OrdinalEncoder on anything that is not genuinely ordered, because a model reading PaymentMethod as 0, 1, 2, 3 will infer that a mailed check is somehow between two other methods, and tree models will happily split on that nonsense. Never hand roll a group mean encoding, even though it is four lines of pandas. Those four lines are precisely what produced the 0.800 above.

One caveat about interpretation. A feature that improves accuracy is not the same as a feature that explains anything, and stakeholders will conflate the two within about thirty seconds of seeing your results. Keeping that distinction clear is easier if you have already worked through exploratory data analysis and can point at the underlying distribution rather than only at the model output.

Part 7 turns to probability and distributions, which is what lets you reason about whether a lift of 0.004 AUC is a real improvement or fold noise. That question sat unanswered twice in this part, and it is the last piece of groundwork before we start modelling the churn table properly in Part 9. Build the feature pipeline described above on your own copy of the dataset before you move on, and run the noise column experiment yourself. Seeing 0.800 appear on your own screen is worth more than reading about it here.

References

• scikit-learn: OneHotEncoder API reference
• scikit-learn: comparing TargetEncoder with other encoders
• scikit-learn: ColumnTransformer with mixed types
• IBM: Telco Customer Churn dataset repository

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts