A model I spent three weeks training answers one prediction in about six milliseconds. Arguing over how to put it in front of real traffic took longer than the training did, and it should not have, because both answers were already sitting in runbooks I owned. Serving is where an infrastructure background stops being an interview story and starts paying rent, so this part is deliberately concrete: two ways to serve the classifier from Part 13, working code for each, and a clear rule for which one your problem actually needs.
Two serving modes, one model
Last part mapped your service runbooks onto the model and named the two checks that are new. This part serves the model those runbooks describe. Same artifact, the incident classifier trained on a month of one cluster cpu and memory metrics, now put to work in two shapes. Batch scoring runs it over a file of new windows on a cron and writes a scored table. A real time endpoint wraps it in an API so another system can ask about one window and get an answer in milliseconds. Nothing about the model changes between them; what changes is how the rows arrive and how fast the answer must come back.
Most tutorials open a serving lesson by standing up a REST API, and for infrastructure work that default is usually wrong. A capacity forecast read once a week, an anomaly scan over yesterday metrics, a nightly risk score per cluster: none of these needs a request to return in six milliseconds. They need a reliable job that runs on time and lands its output where a dashboard or a ticketing rule can read it. Reach for a batch job first. Add a real time endpoint only when a prediction has to happen inside a request that a human or another service is waiting on. Deep mechanics of batch, real time and streaming serving live in the Data Science Series; this part is the operator cut, the parts that touch cron, ports, workers and pagers.
| Dimension | Batch scoring job | Real time endpoint |
|---|---|---|
| Latency need | minutes to hours is fine | a single request is waiting, milliseconds matter |
| What you operate | a scheduled job, like any cron you run | a long lived service with health checks and autoscaling |
| Cost shape | pay for one run, then nothing | pay to keep it warm around the clock |
| Typical infra use | capacity review, nightly risk scan, backfills | inline gating, an operator tool asking live |
| Failure blast radius | one late or wrong file, easy to rerun | every caller feels it at once |
Saving the model as a servable artifact
Before either serving mode, the trained model has to leave the notebook as a file you can version and ship. scikit-learn persists with joblib, and two habits from operations carry straight over: put the version in the filename so a rollback is obvious, and read the path from the environment so the same code runs in every stage. One caution the scikit-learn docs are blunt about, a joblib file is a pickle and loading an untrusted one runs arbitrary code, so treat model files like any other artifact, from a source you control only.
# tested with python 3.12, scikit-learn 1.6.1, joblib 1.4.2, pandas 2.2.3
import os, joblib
# clf was trained in Part 13 on a month of one cluster cpu and memory metrics
joblib.dump(clf, 'incident_clf-v3-sklearn1.6.1.joblib')
# read the path from the environment, never hardcode it in the service
os.environ.setdefault('MODEL_PATH', 'incident_clf-v3-sklearn1.6.1.joblib')
print('feature order the model expects:')
print(list(clf.feature_names_in_))
feature order the model expects: ['cpu_p95', 'mem_p95', 'cpu_slope', 'mem_slope', 'load1_mean']
That printed list is not decoration. scikit-learn 1.x records the feature names it was fit on in feature_names_in_, and it will check them at prediction time. Writing the sklearn version into the filename matters for a plain reason: there is no supported way to load a model into a different scikit-learn version than the one that saved it, so a version bump is a retrain, not a silent upgrade. Versioning the file, the pipeline and the config together is a Part 20 job, a model registry, which is an artifact repository with model metadata; for now the filename convention is enough to roll back by hand.
Batch scoring on a schedule
Where the project stands: last part we mapped runbooks, two parts back we trained the classifier. This part scores real windows with it. A batch job is thirty lines, load the model, read a file of new metric windows, predict, write the results. Here is the first cut, and it has a bug I have shipped for real more than once.
# batch_score.py tested with scikit-learn 1.6.1, pandas 2.2.3
import os, joblib, pandas as pd
model = joblib.load(os.environ['MODEL_PATH'])
new = pd.read_csv('metrics_2026_07.csv') # 100000 rows, one cluster, hourly windows
# grabbing columns by hand, in the order they felt natural
features = new[['mem_p95', 'cpu_p95', 'cpu_slope', 'mem_slope', 'load1_mean']]
new['risk'] = model.predict(features)
new[['window_start', 'risk']].to_csv('scored.csv', index=False)
print(new['risk'].value_counts())
ValueError: The feature names should match those that were passed during fit. Feature names must be in the same order as they were in fit.
I hand picked the columns and put mem_p95 before cpu_p95. Older code would have shrugged, lined the arrays up by position and returned confidently wrong risk scores, which is the worse outcome because nothing errors. scikit-learn 1.x refuses, because it kept the names and the order from fit. Do not paper over it by dropping the names; fix the cause by asking the model what order it wants and reindexing to exactly that.
# let the model define the contract features = new.reindex(columns=model.feature_names_in_) new['risk'] = model.predict(features) print(new['risk'].value_counts())
risk 0 93412 1 6588 Name: count, dtype: int64
Now the run is honest: 6,588 of 100,000 windows flagged, scored in one vectorised call, written to a file a capacity review or an alert rule can read. Put that script on the same scheduler you already trust, add the data validation gate from last part in front of it, and you have a production scorer without a single long lived service to babysit. This batch_score.py is the reference artifact for this part; it runs standalone against any CSV whose columns are a superset of the model features.
A real time endpoint with FastAPI
When a prediction has to happen inside a live request, wrap the model in an HTTP service. FastAPI is the common choice, uvicorn runs it, and pydantic validates the incoming JSON so a malformed request fails cleanly instead of reaching the model. One rule decides whether this service is fast or slow: load the model once when the process starts, not on every request. Here is the shape that gets it right.
# serve.py tested with fastapi 0.115.6, uvicorn 0.34.0, pydantic 2.9, pandas 2.2.3
import os, joblib
import pandas as pd
from contextlib import asynccontextmanager
from fastapi import FastAPI
from pydantic import BaseModel
ml = {}
@asynccontextmanager
async def lifespan(app: FastAPI):
ml['model'] = joblib.load(os.environ['MODEL_PATH']) # loaded once, at startup
ml['features'] = list(ml['model'].feature_names_in_)
yield
ml.clear()
app = FastAPI(lifespan=lifespan)
class Window(BaseModel):
cpu_p95: float
mem_p95: float
cpu_slope: float
mem_slope: float
load1_mean: float
@app.post('/predict')
def predict(w: Window):
row = pd.DataFrame([[w.cpu_p95, w.mem_p95, w.cpu_slope, w.mem_slope, w.load1_mean]],
columns=ml['features'])
return {'risk': int(ml['model'].predict(row)[0])}
Building the input as a one row DataFrame with the model own feature list keeps the same contract the batch job enforces, so a client cannot silently swap two fields. Run it and call it:
$ uvicorn serve:app --host 0.0.0.0 --port 8000 --workers 4
INFO: Started server process [12841]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000
$ curl -s -X POST http://localhost:8000/predict -H 'content-type: application/json'
-d '{"cpu_p95":0.94,"mem_p95":0.88,"cpu_slope":0.02,"mem_slope":0.05,"load1_mean":7.1}'
{"risk":1}
Here is the failure that trips people who follow older tutorials. Nearly every serving guide still loads the model in an @app.on_event(‘startup’) handler, and current FastAPI answers with a warning:
DeprecationWarning:
on_event is deprecated, use lifespan event handlers instead.
Read more about it in the FastAPI docs for Lifespan Events.
on_event has been deprecated since FastAPI 0.93 in favour of the lifespan context manager the code above uses, which keeps startup and shutdown in one place. A nastier version of the same mistake is loading the model inside the endpoint function, so every request reads a two megabyte file off disk before it can answer. That turns a six millisecond prediction into a call dominated by a load of roughly 180 milliseconds, and it scales with traffic, so the service gets slower exactly when it is busiest. Load once at startup, hold it in memory, and the request path does only the prediction. The two serving paths, both fed by the one artifact, look like this.
Latency, workers and serving capacity
Capacity planning for a model endpoint is the exercise you already run for any service, with one measurement you have to take yourself. A short benchmark settles it. This is where the batch versus per row gap in the opening chart came from.
# bench.py python 3.12, scikit-learn 1.6.1
import time, os, joblib, pandas as pd
model = joblib.load(os.environ['MODEL_PATH'])
X = pd.read_csv('metrics_2026_07.csv').reindex(columns=model.feature_names_in_)
t0 = time.perf_counter()
_ = model.predict(X) # one vectorised call, 100000 rows
print('batch 100k rows:', round((time.perf_counter() - t0) * 1000, 1), 'ms')
t0 = time.perf_counter()
for _, r in X.head(1000).iterrows(): # one row at a time
_ = model.predict([r.values])
print('loop 1k rows:', round((time.perf_counter() - t0) * 1000, 1), 'ms')
batch 100k rows: 41.3 ms loop 1k rows: 5820.0 ms
Read those two numbers. Scoring 100,000 rows in one call took 41 milliseconds; scoring 1,000 rows one at a time took 5.82 seconds, which is about 5.8 milliseconds of overhead per call and would put a full 100,000 rows near ten minutes. That per call cost is your real time latency floor, roughly six milliseconds of model plus framework, and it is why a batch job that has all the rows at once is thousands of times cheaper than the same model answering them one by one. As an aside, passing r.values as a bare array makes scikit-learn warn that the input has no valid feature names, another reason to keep a DataFrame with named columns on the request path.
From that floor, sizing is arithmetic you have done before. A synchronous scikit-learn prediction holds the worker while it runs, so throughput per process is roughly one over the latency, near 160 predictions per second per worker at six milliseconds. uvicorn with four workers on a four core box gets you into the mid hundreds per second, and past that you scale out with the same autoscaler and load balancer you already operate. Two additions are model specific. Batch predictions where you can, since one call of 500 rows is far cheaper than 500 calls, the same batching and caching logic that cuts LLM latency. And budget the second cost centre, retraining, as a scheduled job with its own bill rather than something you fire off whenever accuracy feels low.
Serve batch first, add real time when you need it
Both serving modes are shapes you already know, a cron job and a small HTTP service, wrapped around an artifact that happens to have been fit from data. Your instinct to reach for the simplest reliable thing is correct here, and for most infra predictions the simplest reliable thing is a batch job. Save real time for the requests that cannot wait, and when you build it, put the model load and the feature contract where they belong, at startup and inside a named DataFrame.
Next part watches this served model in production, because a model that is up and fast can still be quietly wrong, and drift is just observability pointed at prediction quality.
References
- scikit-learn, Model persistence, joblib and security
- FastAPI, Lifespan Events
- Uvicorn, Deployment and workers
- Data Science Series, Serving Machine Learning Models Batch and Real Time
- AI Engineering Series, LLM Caching, Batching and Latency
About The Author
DrJha