A model that scores 0.86 offline and 0.71 in production almost never has a modelling problem. Two different pieces of code computed its inputs, and only one of them was ever tested.

This gap has a name, training serving skew, and it is the single most common reason a data science project quietly stops delivering value after launch. Nobody files a bug, because nothing is broken in the sense a monitoring system understands. Requests succeed, latency is fine, scores come out between zero and one. They are simply wrong, and they get wronger every time somebody edits a transformation on one side of the wall.

Skew, defined in plain terms

Training serving skew is any difference between the feature values a model saw during training and the feature values it sees when making a real prediction. Google Cloud architecture guidance defines it that way and lists a feature store as the standard remedy, which tells you how routine the failure is. Note carefully what it is not. Skew is not drift. Drift, covered in Part 25, is the world changing underneath a model that is otherwise being fed correctly. Skew is your own code disagreeing with itself while the world stays perfectly still.

Three flavours show up in practice, and they need different fixes. Schema skew is a column that is a string on one path and a float on the other, or a category the encoder never saw. Value skew is the ugly one: both paths produce a number, both numbers look plausible, and they were computed by different logic. Timing skew is subtler still, where both paths run identical code but one of them reads data that did not exist yet at prediction time, or reads data that is already stale by the time the request arrives.

Schema skew announces itself. A type error or an unseen category raises an exception and somebody gets paged, which is the best possible outcome. Value skew and timing skew are silent, and silence is what makes them expensive. My rule after several of these is uncomfortable but useful: assume value skew exists in every system where features are computed in more than one place, and design a measurement rather than an argument.

Where two code paths diverge

Picture how the churn project got here. Training features came out of a SQL query against the warehouse, written by whoever was closest to the data model. Batch scoring reuses roughly that query on a schedule. Then somebody needs a live score for a customer on a support call, and a Python function appears inside the endpoint that reproduces the same fields from whatever the application database happens to expose. Three implementations of one idea now exist, and only the first one has ever been compared against a label.

Nobody sets out to build that. Each step is locally reasonable and takes an afternoon. Divergence arrives later, usually through a small edit: an analyst changes a filter in the warehouse query to exclude cancelled trial accounts, ships it, and has no idea a Python file in a different repository needs the same change. Six weeks pass. Offline metrics still look fine, because offline metrics are computed from the warehouse path that was edited correctly.

Here is where I paid for the lesson. Our churn model held an AUC around 0.86 in cross validation and something close to 0.71 once we scored real customers, which I spent a full Tuesday blaming on distribution shift. Actual cause was a feature named avg_spend_90d. Training SQL computed it over a trailing ninety day window. Serving code computed it over the customer entire history, because whoever wrote the endpoint read the feature name as an average of spend and did not read the window. For anyone with tenure under ninety days both paths agreed exactly, which is precisely why my spot checks passed. About 63 percent of the scored population had longer tenure than that, and for them the number was systematically too low. One shared function and a redeploy brought production back to 0.84. Two days of my life went to a naming convention.

Point in time correctness

Timing skew deserves its own section because it is the one people get wrong even when there is a single shared implementation. Suppose you are predicting whether a customer churns, and you build a training row for customer 1001 as of March 12. Every feature in that row must reflect only what was knowable on March 12. Pull the current value of support_tickets_total from a table that has been updated ever since, and you have handed the model information from the future.

Feast documentation describes the correct operation as a point in time join, sometimes called a last known good value temporal join. For each entity row you supply an event_timestamp, and that timestamp acts as an inclusive upper bound: only feature values observed at or before it are joined in. Miss this and offline metrics inflate in a way that never survives contact with production, which is the leakage problem from Part 11 arriving through a different door.

Writing that join yourself is possible and painful. In plain SQL it becomes a correlated lateral join per feature group, ordered by timestamp descending, limited to one row, repeated for every feature view you want, and every one of those joins is a place to introduce an off by one on the inequality. Getting this right once, in a library, is a large part of what a feature store is selling. Note also that data cleaning decisions from cleaning messy data have to live on the shared side of the wall, otherwise your null handling becomes another divergence point.

Anatomy of a feature store

Strip away the marketing and a feature store is four things. A registry holds feature definitions as code, so a feature has one authoritative meaning. An offline store, usually your existing warehouse or object storage, holds full history and answers training and batch scoring queries with point in time correctness. An online store, typically Redis, DynamoDB or Postgres, holds only the latest value per entity and answers single row lookups in single digit milliseconds. A materialisation process copies values from offline to online on a schedule or on push.

Read that diagram carefully, because it contains the point most vendor pages bury. Two databases do not stop skew. Definitions living in one registry and being applied to both paths stop skew. Feast makes this explicit through feature services, which group the exact features a model version consumes, so a model can name what it needs rather than a human reassembling the list on each side.

Be equally clear about what a feature store does not do. Feast documentation states plainly that it is not an ETL system, not an orchestrator, not a warehouse and not a database. It will not manage your train and test splits, will not version your labels, and its data quality integrations are experimental. Teams who install one expecting a platform end up with a registry plus all their old problems, now distributed across one more component that needs upgrading.

Churn features that survive both paths

Last part we had the Telco churn model served two ways, a nightly batch job writing to a table and a FastAPI endpoint answering single requests. This part we audit those two paths feature by feature and decide which ones can safely live outside a store. Not everything needs the machinery, and pretending otherwise is how a two model team ends up maintaining infrastructure for twenty.

Pattern here generalises well beyond churn. Features read straight off a request are cheap and safe. Features aggregated over a time window against data the caller does not hold are expensive to recompute live and easy to get subtly wrong, so they are the ones worth precomputing and looking up. Splitting your feature list along that line is the highest value hour in this whole part.

If you do reach for Feast, configuration is smaller than people expect. A repository holds Python definitions plus one config file naming the two stores. Tested against Feast 0.64.0 on Python 3.10.12, a minimal local setup looks like this.

# feature_store.yaml, tested against Feast 0.64.0, Python 3.10.12
project: telco_churn
registry: data/registry.db
provider: local
online_store:
  type: sqlite
  path: data/online_store.db
entity_key_serialization_version: 3

First real failure most people hit is not in that file. Call get_historical_features with an entity frame that lacks the timestamp column and you get a blunt refusal: ValueError: Please provide an entity_df with a column named event_timestamp representing the time of events. Cause is exactly the constraint described earlier, since without a timestamp per row Feast cannot decide which observation was knowable, so it declines rather than guessing. Fix is to add an event_timestamp column holding the moment each prediction would have been made, not the moment you ran the query. Teams commonly set it to now for every row, which quietly turns training data into a snapshot of the present and reintroduces the leakage the tool exists to prevent.

Skew audit numbers from the churn project

Arguing about skew is unproductive. Measuring it takes an afternoon and settles everything. Job I now write on every project does the same thing: sample a thousand customers, run them through the batch path and the online path on the same day, and record for each feature the share of customers where the two values differ by more than one percent. Results below come from that job on the churn service before we fixed anything.

Shape of that chart is typical, and it should change how you plan the work. Skew is rarely spread evenly. Two or three windowed aggregates carry nearly all of it, which means a targeted fix beats a platform migration on both time and risk. Below is what the same audit looked like translated into what a stakeholder cares about.

Look at that first row for a moment, because it explains why skew survives so long inside competent teams. Offline AUC dropped very slightly after the fix. Every metric a retention manager notices improved sharply, and precision in the top five percent of scores, which decides who actually gets a call, went up by more than half. Optimising the number in your notebook and optimising the outcome are not the same activity, a distinction the analyst series makes early in correlation and causation.

Build, buy, or write a shared library

Three roads lead out of this problem. Managed feature stores such as Tecton, Databricks Feature Store, Vertex AI Feature Store and SageMaker Feature Store hand you a registry, both stores and materialisation, in exchange for money and lock in. Self hosted Feast gives you the same abstractions under Apache 2.0 with your existing warehouse and a Redis or DynamoDB instance underneath, in exchange for operating it. Third road, which almost nobody writes blog posts about, is a Python package containing one function per feature, imported by the training job, the batch job and the endpoint alike.

My verdict, stated plainly. Under about ten models and one team, write the shared library and a nightly reconciliation job, and put your effort into data modeling discipline instead of infrastructure. Between ten and fifty models across several teams, where the same customer aggregate is being recomputed by four different people, adopt Feast, because at that point the registry is genuinely solving a coordination problem that code review cannot. Managed offerings are worth the invoice only when low latency online serving is on your critical path and you have no platform engineer to run Redis properly.

Option I would avoid is the homegrown feature store: a Postgres table called features, a cron job filling it, and a convention that everybody reads from there. Reason is that the two hardest parts, point in time correct historical joins and consistent materialisation, are exactly the parts that get skipped in a homegrown build, so you inherit the operational burden of a feature store while keeping the leakage risk of not having one. I have watched that pattern eat six weeks of a good engineer and produce something strictly worse than the shared library it replaced.

Recommendation for a small churn team

For the churn project as it stands, one model and one team, do not install a feature store. Move the six feature computations into the telco_churn package built in Part 21, so training, batch scoring and the endpoint import identical functions. Add an event_timestamp to every training row and enforce the point in time rule by hand in the training query. Then write the reconciliation job, schedule it daily, and route its failure to the same place your other alerts go.

Revisit that decision when a second model wants avg_spend_90d. Duplication of one aggregate across two models is the honest trigger for adopting Feast, and it is a much better trigger than a conference talk. Should you get there, start with the local sqlite setup above, materialise one feature view, and only then argue about Redis.

One thread stays loose. Everything here assumes you can tell which model version produced a given score and which feature definitions it was trained against, and right now you cannot, because that information lives in somebody notebook and a filename. Part 24 covers experiment tracking, model registries and versioning, which is the record keeping that makes a skew audit interpretable six months later. Before then, run the reconciliation job once against your own project and write down the number. Most people are surprised, and being surprised now is much cheaper than being surprised in a quarterly review.

References

• Feature retrieval and point in time joins, Feast 0.64 documentation
• Introduction and what Feast is not, Feast documentation
• MLOps: continuous delivery and automation pipelines in machine learning, Google Cloud
• Feature engineering and serving, Databricks documentation

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts