Three hundred and twelve. That was every scoring request our churn endpoint handled in its busiest week, and I had sized it to survive two hundred a second. Nobody had lied to me. I had simply never asked the retention team when they actually read a churn score, and their answer, when I finally asked, was once a day at six in the morning.

Cost of a wrong serving choice

Serving means getting a number out of a trained artifact and into whatever consumes it. That sentence hides an enormous amount of engineering, because a prediction sitting in a Python process is worth nothing until a campaign tool, a screen or a downstream job can read it. Every choice in this part is really a choice about that last step.

My own version of that lesson was expensive in a quiet way. I stood up a real time churn endpoint on a managed container service: three replicas, one vCPU and two gigabytes each, sitting behind a load balancer, because three replicas felt like the responsible number. Monthly bill came to roughly forty seven dollars once the balancer was counted. Eight months passed before anyone opened the cost report line by line, so about three hundred and seventy six dollars had gone on serving predictions that a nightly job produced in one and a half seconds. We deleted the endpoint on a Thursday and replaced it with a scheduled job writing to a table. Retention never noticed, because their campaign tool had always read that table at six in the morning and never once called the API in between.

Ask two questions before writing any serving code. First, when does the consumer read the prediction, and second, is the input to the model knowable before that moment. Answer both honestly and the serving shape usually picks itself. Skip them and you end up defending infrastructure that exists because it seemed professional.

Three serving shapes, side by side

Batch scoring runs on a schedule, takes a whole population of rows, and writes predictions to a table or file. Real time serving, sometimes called online or synchronous serving, answers a network request with a prediction for one row or a handful of rows while the caller waits. Streaming sits between them: a long lived consumer reads events off a queue or log as they arrive, scores each one, and writes to a sink, with nobody blocking on the answer.

Notice how often stakeholders say real time when they mean fresh. Those words point at different problems. A retention manager complaining that scores are stale usually wants yesterday evening replaced by this morning, which is a scheduling change, not an architecture change. Moving a daily job to hourly costs almost nothing and solves the actual complaint; standing up an endpoint solves a complaint nobody made.

Batch scoring for the churn model

Last part we packaged the churn project into telco-churn 0.3.0, with a churn train command that writes a versioned pipeline artifact to disk. This part takes that artifact and gives other systems a way to consume what it produces. Batch first, because batch is what churn actually needs, and because the batch path is the one you can debug by looking at a file.

Scoring code is shorter than people expect, because all the feature engineering from Part 6 lives inside the saved pipeline object rather than in this script. Saving the fitted pipeline, not the bare estimator, is what keeps the two halves honest.

# src/telco_churn/batch_score.py
# versions pinned for this example: Python 3.10.12, scikit-learn 1.7.2,
# pandas 2.3.3, joblib 1.5.3, pyarrow 21.0.0

import datetime as dt
import joblib
import pandas as pd

from telco_churn.config import ARTIFACT_DIR, DATA_DIR

def main() -> None:
    model = joblib.load(ARTIFACT_DIR / "churn_pipeline_0.3.0.joblib")
    frame = pd.read_parquet(DATA_DIR / "scoring_batch.parquet")

    ids = frame["customerID"]
    features = frame.drop(columns=["customerID"])
    proba = model.predict_proba(features)[:, 1]

    scored = pd.DataFrame(
        {
            "customerID": ids,
            "churn_probability": proba.round(4),
            "model_version": "0.3.0",
            "scored_on": dt.date.today(),
        }
    )
    stamp = dt.date.today().strftime("%Y%m%d")
    scored.to_parquet(ARTIFACT_DIR / f"scores_{stamp}.parquet", index=False)
    print(scored.shape)
    print(scored.head(3).to_string(index=False))

if __name__ == "__main__":
    main()

$ time python -m telco_churn.batch_score
(7043, 4)
customerID  churn_probability model_version  scored_on
7590-VHVEG             0.6218         0.3.0 2026-07-19
5575-GNVDE             0.0431         0.3.0 2026-07-19
3668-QPYBK             0.5107         0.3.0 2026-07-19

real    0m1.812s
user    0m2.394s
sys     0m0.271s

Real time endpoint with FastAPI

Sometimes the input genuinely is not knowable in advance. A quote form where the customer types a contract length, a fraud check on a card that has never been seen, a session that only exists once somebody clicks: those cases have no precomputed row to look up. FastAPI plus uvicorn is my default there, mostly because pydantic gives you input validation for free and because the resulting service is small enough to read in one sitting.

# src/telco_churn/serve.py
# versions pinned for this example: fastapi 0.139.2, uvicorn 0.51.0,
# pydantic 2.13.4, scikit-learn 1.7.2, pandas 2.3.3, Python 3.10.12

import joblib
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel, Field

MODEL_VERSION = "0.3.0"
model = joblib.load(f"artifacts/churn_pipeline_{MODEL_VERSION}.joblib")
FEATURE_ORDER = list(model.feature_names_in_)

class Customer(BaseModel):
    tenure: int = Field(ge=0, le=200)
    MonthlyCharges: float = Field(ge=0)
    TotalCharges: float = Field(ge=0)
    Contract: str
    InternetService: str
    PaymentMethod: str

app = FastAPI(title="churn-scoring", version=MODEL_VERSION)

@app.get("/healthz")
def healthz():
    return {"status": "ok", "model_version": MODEL_VERSION}

@app.post("/score")
def score(customer: Customer):
    row = pd.DataFrame([customer.model_dump()])
    row = row.reindex(columns=FEATURE_ORDER)
    proba = float(model.predict_proba(row)[0, 1])
    return {
        "churn_probability": round(proba, 4),
        "model_version": MODEL_VERSION,
    }

$ uvicorn telco_churn.serve:app --host 0.0.0.0 --port 8000 --workers 4
INFO:     Started parent process [8123]
INFO:     Started server process [8127]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

$ curl -s -X POST localhost:8000/score 
    -H 'content-type: application/json' 
    -d '{"tenure":2,"MonthlyCharges":70.7,"TotalCharges":151.65,
         "Contract":"Month-to-month","InternetService":"Fiber optic",
         "PaymentMethod":"Electronic check"}'
{"churn_probability":0.6218,"model_version":"0.3.0"}

Now for the line that cost me an afternoon. Early versions of that handler had no reindex call, and everything passed locally. In production a Java client serialised its JSON with fields in a different order, pandas built the DataFrame in that order, and scikit-learn refused.

Traceback (most recent call last):
  File "/app/src/telco_churn/serve.py", line 33, in score
    proba = float(model.predict_proba(row)[0, 1])
  File "/usr/lib/python3/site-packages/sklearn/pipeline.py", line 761,
    in predict_proba
    Xt = transform.transform(Xt)
ValueError: The feature names should match those that were passed during fit.
Feature names must be in the same order as they were in fit.

Worth knowing: this error is a gift compared to the alternative. Older pipelines that took a plain NumPy array would have accepted the scrambled columns silently and returned a confident, meaningless probability. Column name checking inside scikit-learn turns a wrong answer into a loud failure, which is why I always feed a named DataFrame rather than an array at serving time.

If writing a service feels like more than the problem deserves, MLflow will stand one up from a logged model with a single command, exposing a POST endpoint at /invocations that accepts a JSON payload. It is a reasonable way to get something callable in five minutes, and a poor way to run production traffic, because you inherit its request schema and its error handling instead of your own.

# mlflow 3.14.0
$ mlflow models serve -m runs:/8f3c1d94a2/model -p 8001 --env-manager local

Streaming scores and where they earn their keep

Streaming means a process that stays alive, subscribes to a topic on a broker such as Kafka, Kinesis or Pub/Sub, and scores each event shortly after it is published. Nothing blocks on the result. Output goes to another topic, a cache or a table, and some other system picks it up when it is ready. Latency measured end to end is usually a second or two, not milliseconds, because the broker itself batches.

Genuine cases exist. Card fraud has to be judged inside the authorisation window. Ad selection has to resolve before a page paints. Session level personalisation only makes sense while the session is open. What these share is that the decision is made inside the event’s own lifetime, and a score computed an hour later would have no consumer at all.

Churn is not one of those cases and I would not stream it. Somebody’s probability of leaving in the next thirty days does not meaningfully change between 09:14 and 09:15, and pretending otherwise buys you a broker to operate, consumer lag to monitor, and replay semantics to reason about. Duplicate delivery is the part teams underestimate: brokers give at least once delivery by default, so a replayed partition will score the same customer twice, and unless your sink is keyed on customer and event id you will have two rows disagreeing about the same person. Fixing that properly costs more engineering than the entire batch path did.

Verdict on this one is blunt. Choose streaming when the consuming decision expires in seconds. Avoid it when a business process reads the score on a human schedule, which covers churn, credit review, propensity scoring, lifetime value and almost every model a marketing team asks for.

Latency budget arithmetic

Suppose you do need an endpoint. Somebody will hand you a latency target, and you should immediately break it into stages, because guessing which stage dominates is almost always wrong. Here is what a single scoring request actually spent, averaged over ten thousand calls against the service above.

Read that table again, because it contains the most useful thing in this part. Roughly two thirds of a one row request is spent moving data into and through pandas, and one fifth is the actual gradient boosted model from Part 12. Teams who chase latency by shrinking the model are optimising the smallest slice. Accepting a list of rows per request and scoring them in one call moves far more, since that 1.9 milliseconds of construction is paid once for fifty customers instead of fifty times.

Look at what changed between the first two groups. Median latency went from nine milliseconds to eight, which is noise. At the ninety ninth percentile it fell from seventy four to twenty seven, because a single Python process handles one request at a time and every unlucky caller queues behind whoever arrived first. Scikit-learn releases the global interpreter lock in parts of its numerical code, but not enough of it to save you, so processes beat threads for this workload. Setting workers to roughly the number of vCPUs available, then measuring, is a better starting point than any rule someone quotes at you.

Rolling out a new model version safely

Serving is not something you do once. Every retrain produces a new artifact, and swapping it under live traffic is its own small discipline. Batch makes this pleasantly boring: write the new version to a separate dated file, run both for a week, compare score distributions, then repoint the table when you are satisfied. Rolling back means changing one line of configuration, and yesterday’s file is still sitting on disk where you left it.

An endpoint offers three common patterns. Shadow mode sends a copy of production traffic to the new version and throws its answers away after logging them. Canary routes a small fraction of real traffic to the new version and watches for damage. Blue green keeps two complete environments and flips a router between them. My preference for models, which differs from what I would do for ordinary application code, is shadow for a week and then a straight cutover, skipping canary entirely.

Reasoning behind that comes from an expensive mistake. I once canaried a churn model at ten percent of traffic and left it there for three weeks, waiting for evidence that never arrived, because the churn label itself takes thirty days to materialise. No amount of patience was going to make ten percent of a low volume stream significant. Shadow mode would have answered the question we could actually answer quickly, which is how often the two versions disagree: when we finally ran it, the versions differed by more than 0.1 in probability on twelve percent of customers, and two of those cases turned out to be a mis-mapped contract category. One day of shadow logging beat three weeks of canary.

Whichever pattern you choose, stamp the model version into the scoring response and into every batch row, as the code above does. Without it, an incident starts with somebody guessing which artifact was live at nine in the morning. Part 24 covers the registry that makes those version strings mean something beyond a filename you typed by hand.

Serving choice I would make for churn

For the churn project I would run a nightly batch job writing to a warehouse table, and nothing else, until somebody produces a screen where a human waits on a score. If that screen arrives, I would add a thin read API over the precomputed table rather than a model endpoint, because looking up a row is faster, cheaper and impossible to get subtly wrong. A live model endpoint is the third choice, not the first, and streaming does not enter the conversation for a thirty day churn horizon.

What I would avoid, having done it: standing up an endpoint because it demonstrates that the model is finished. Serving infrastructure is a running cost and an on call surface, and it should exist because a consumer needs it, not because a project needs a milestone. Should you find yourself unable to name the system that will call your endpoint, that is your answer.

One crack in all of this is already visible. Batch scoring reads features computed by one code path, an endpoint computes them in another, and the two drift apart the moment somebody edits a transformation on only one side. Part 23 covers feature stores and training serving skew, which is the machinery built to stop exactly that. Before you get there, take the churn service above, add a test that asserts the endpoint and the batch file agree on ten sampled customers, and watch how quickly that test starts failing once two people are editing features.

References

• Deployments Concepts, FastAPI documentation
• Deploy MLflow Model as a Local Inference Server, MLflow 3.14.0
• Model persistence, scikit-learn 1.7.2 documentation
• Deployment, uvicorn 0.51.0 documentation
• Models, pydantic 2.13.4 documentation

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts