Why does it take eleven weeks to put a model in front of a customer when the model itself took four days to build. That question came from a finance director, not from an engineer, and it was the most useful sentence anyone said to me that quarter. She was not asking about algorithms. She was asking why the distance between a working notebook and a served prediction was roughly fifteen times the distance to the working notebook.

Almost every honest answer to her question was architectural. Nobody owned the path from a warehouse table to a feature that a live endpoint could read in single digit milliseconds. Nobody had decided whether training data came from a snapshot or a query. Six of those eleven weeks were spent rebuilding, in a serving language, transformations that already existed in a notebook. A platform is what removes those weeks, and this part is a design for one.

Planes of a data science platform

Where the churn project stands: Part 26 left us with a pipeline that can retrain a model, compare it against the incumbent and block a merge when quality drops. What we have never described is the machinery that pipeline runs on. Everything so far assumed a warehouse existed, a registry existed, an endpoint existed. This part names each of those and says what I would choose.

Splitting a platform into planes is not decoration. A plane is a boundary that one team can own, one budget can fund and one contract can describe. Draw the boundaries badly and you get the eleven week problem, where a change to a feature requires coordinated edits in four repositories owned by three teams. Draw them well and a data scientist can add a feature, retrain, and promote without filing a ticket.

Count the arrows that cross a boundary in that diagram. There are six. Those six are the entire surface area of your platform contract, and in my experience they account for most incidents. A schema change in the warehouse silently alters a feature. A registry promotion lands a model whose input signature no longer matches what the endpoint sends. Nothing inside a plane broke. A seam did.

Ownership of those seams is worth settling on a whiteboard before anyone writes a line of infrastructure. My preference is that each contract has exactly one owning team and one written expectation, phrased as something measurable: this table is refreshed by 04:00 with a row count within 5 percent of yesterday, this online store answers a batch of twenty entity keys under 60 milliseconds at p99, this registry alias only ever points at a version that passed the merge gate. Vague ownership produces a specific failure mode I have watched three times now. During an incident, two teams each believe the other is investigating, and the first forty minutes are spent establishing that nobody is. A one line contract per seam removes that entirely, and costs an afternoon.

Data plane decisions that lock you in

Start here, because this is the layer you will not get to change. Compute engines get swapped every few years without much drama. Storage layout and the modelling conventions on top of it outlive three generations of tooling, which is why a clean dimensional model is worth more to a machine learning platform than most people building one expect. If that phrase is unfamiliar, it means organising warehouse tables around business entities and events rather than around whatever the source system emitted, and the analyst series covers it properly in data modeling basics for analytics.

Two decisions matter more than the rest. First, whether training reads a query or a snapshot. A query is convenient and quietly non reproducible, because the underlying table keeps changing beneath you. A snapshot written to object storage with an immutable path costs storage and buys you the ability to rebuild a model exactly, eighteen months later, when a regulator asks. I take the snapshot every time. Storage is the cheapest thing in the diagram.

Second, whether point in time correctness is enforced by the platform or left to individual notebooks. Point in time correctness means a training row only ever sees feature values that existed at the moment being predicted, and it is the single most common source of leakage in a churn model. Left to individuals, it is done correctly by roughly half of them. Enforced in the feature layer, with an event timestamp on every source and an as of join at retrieval, it is done correctly by all of them. That is not a tooling preference, it is a correctness argument, and it is why the feature plane exists at all.

Feature layer, build or buy

Every platform review I have sat in eventually stalls on this one, so let me give the decision rule I actually use before giving the reasoning. Buy when three or more teams will share the component and a managed option meets your latency target at an annual floor under about fifteen percent of the platform budget. Build the thin version otherwise. The rule sounds arbitrary until you have paid an annual floor that dwarfed your usage.

Two lessons came out of that, and only one of them is about money. Keep feature definitions in your own repository as declarative objects, whatever engine executes them. Open source Feast is designed around exactly this split, with an offline store for building training sets and an online store holding only the latest value per entity key, and the definitions sit in your code either way. When the definitions are portable, changing the engine is a nine day job. When they live in a vendor console, changing the engine is a rewrite, and you will not do it, which is precisely what the pricing model assumes.

Orchestration and training layer

Orchestration attracts more architectural argument than it deserves. A scheduler runs jobs in an order, retries them, and tells you when one failed. Airflow does that, and version 3.3.0 landed in July 2026 with a coordinator layer that lets individual tasks run outside Python while the scheduling logic stays in Python, which removes the main reason teams used to reach for something else. My verdict: use the orchestrator your data engineering team already operates, and spend the argument you saved on the model registry contract instead.

Registry design deserves that attention because it is where deployment gets its meaning. MLflow models are addressed as a URI, and an alias gives a mutable pointer at a specific version, so a production service targets a name and an alias rather than a version number. Promotion then becomes a single act of moving that alias, fully audited, reversible in seconds. Serving code never learns a version number, which means a rollback is not a deployment. I would avoid any design where the endpoint hardcodes a version, because every rollback becomes a release, and releases at three in the morning are how good models get replaced by bad ones.

Serving layer sizing

Most churn models do not need a real time endpoint, and building one anyway is the most common overspend I see. If a retention team acts on a churn score once a week, a nightly batch write into the warehouse and the customer relationship system serves them perfectly, at a fraction of the cost and with none of the availability obligations. Part 22 walked through that choice in detail. Architecturally, the rule I hold to is that latency requirements come from the action, never from the model.

Where real time genuinely applies, such as scoring a churn risk at the moment a customer opens a cancellation page, the design constraint is not model inference. Inference on a gradient boosted churn model runs in low single digit milliseconds. Feature retrieval is the expensive half, and the diagram below is the contract that keeps it honest.

Sizing follows from that picture. Budget your latency by hop, not in aggregate. On our churn endpoint the split was roughly 6 milliseconds of network, 41 of feature retrieval, 3 of inference and 38 of the surrounding application, for an 88 millisecond p99. Anyone attacking that number by making the model smaller would have been optimising 3 percent of the total. Naming the hops before you tune is what stops that.

One serving decision that repays thinking about early is whether models get their own container each or share one. A container per model is cleaner to reason about and is what most managed platforms sell you. It also means every model pays for its own idle capacity, and with four models at modest traffic we were paying for roughly 3.5 idle replicas at all times. Colocating those four behind a single process, loading each from the registry at startup and routing on a path segment, took our serving line from 1,600 to 1,150 dollars a month without touching latency. What it cost was blast radius: a bad model version now takes down four endpoints instead of one. I accept that trade below about five models and reject it above ten, because past that point the isolation is worth more than the idle capacity, and a single deployment holding a dozen models becomes something nobody wants to be paged for.

Cost and operations trade off, with real numbers

Platform posts that compare architectures almost never show a bill, so here is ours. Same churn workload, same team of four, measured across two consecutive quarters. Left hand column is the fully managed stack we started on. Right hand column is the assembled stack we moved to. I have included operations hours because a cost comparison that omits them is dishonest.

Governance built in from day one

Access control is treated as a phase two item on most platform plans I review, and phase two rarely arrives. Retrofitting it is expensive for a structural reason: once feature views, training jobs and endpoints exist without any notion of an owner, adding one means editing every object and negotiating with every consumer. Feast added role based access control as a first class part of its architecture rather than as a wrapper, and the reason is exactly this. Ownership belongs in the object model.

Three things I would build in from the first sprint, and no more than three, because governance scope creep kills platforms too. Every feature view carries an owner and a data sensitivity label. Every registered model version carries the snapshot identifier it trained on, which the registry already supports through lineage back to the producing run. Every promotion writes an audit entry naming a human. That set is small enough to build in a week and sufficient to answer the three questions an auditor asks: who owns this, what did it learn from, and who approved it going live. Part 29 goes considerably further into model risk, but a platform without those three fields cannot support anything Part 29 describes.

Reference architecture I would build today

For a team of two to six data scientists shipping a handful of models, of which our churn project is a fair example, here is the concrete answer. Warehouse you already run, with immutable training snapshots in object storage. Feature definitions declared in your own repository, materialised on a schedule into Postgres, with a managed key value store only once read volume genuinely demands it. MLflow for tracking and the registry, promotion by alias so a rollback never becomes a release. Orchestration on whatever your data engineers already operate. Batch scoring by default, a real time endpoint only where the action that consumes the score happens in a session. Model metrics into the observability stack that already pages someone.

What I would avoid, stated plainly: a second orchestrator bought specifically for machine learning, a managed feature platform below roughly three consuming teams, and any serving design where a version number is hardcoded outside the registry. Each of those looks like sophistication in a design review and reads as an unfunded liability eighteen months later.

One last note on sequencing, which the finance director inadvertently taught me. Build the seams before you build the planes. A platform with a mediocre feature store and a clean contract between training and serving outperforms a platform with an excellent feature store and no contract at all, because the first one can be improved a piece at a time and the second one can only be replaced. Part 28 takes the same architecture and sizes the compute underneath it, which is where the next order of magnitude of cost sits. If you are designing a platform this quarter, write down your six boundary contracts before you write down a single product name, and see how many of them you can already describe precisely.

References

• MLflow Model Registry documentation, covering model URIs, aliases such as champion, tags and lineage back to the producing run
• Feast architecture overview, including the push model, feature transformation engines and role based access control
• Feast offline store reference, on building training datasets and materialising into the online store
• Apache Airflow release notes, current stable 3.3.0 released 6 July 2026, including the coordinator layer
• Data modeling basics for analytics, Data Analyst Series, assumed as background for the data plane
• IBM Telco Customer Churn dataset, the running project across this series

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts