A risk officer once stopped a churn model launch with a single question: who approved this, and what did they see before they approved it. Nobody in the room could answer. We had a model with an AUC of 0.86, a serving endpoint, dashboards, an on call rota, and no record of a human decision anywhere in the chain. That gap is what this part is about, and it is a far more common failure than a bad algorithm.

Model risk in plain terms

Model risk is the chance that a decision made from a model output turns out to be wrong in a way that costs you money, breaks a law, or damages someone. Note the shape of that sentence. Risk does not live in the model file. It lives in the decision the model drives. A churn score sitting in a table is inert. A churn score that automatically withholds a retention offer from ten thousand accounts a month is a business process with a statistical engine bolted into it, and that is what needs governing.

US regulators break this into three drivers, and I find the split useful well outside banking. Inherent risk comes from the model itself: how complex it is, how many assumptions it rests on, how good the input data is. Exposure is how much business rides on the output, usually measurable as portfolio size or revenue touched. Purpose is what the output is used for, with regulatory and financial uses ranking higher than exploratory ones. Purpose and exposure together give you materiality, and materiality is what should set how hard you work on any given model.

Two consequences fall out of that framing immediately. First, a technically excellent model can carry high risk purely by being misused, which is why scope creep onto a model built for something else is one of the most dangerous things a team can do quietly. Second, risk aggregates. Five models that all consume the same customer tenure feature share a single point of failure, and if that feature breaks, five business processes fail on the same morning. Governance that only looks at models one at a time will never see that coming.

Worth separating from a related idea you already know. Correlation is not causation, and a model that predicts well can still encode a relationship that reverses the moment you act on it, an issue I covered for analysts in correlation vs causation in data analysis. Governance does not fix that. What governance does is force someone to write down the assumption, so that when it fails there is a document to go back to rather than an argument about what everyone remembers.

What changed in 2026 for anyone shipping models

If you learned model governance any time in the last decade, part of what you learned is now superseded. On 17 April 2026 the Federal Reserve, the OCC and the FDIC jointly issued SR 26-2, Revised Guidance on Model Risk Management, which supersedes and replaces SR 11-7 from 2011 along with the 2021 anti money laundering model statement. SR 11-7 was the document everyone cited, including plenty of teams with no banking connection at all, because it was the clearest articulation of validation practice in print. Citing it now dates you.

Most of the substance carries forward. Effective challenge, conceptual soundness, outcomes analysis, ongoing monitoring and the model inventory all survive intact. What shifts is emphasis: the new guidance is explicitly risk based and tailored, and it says outright that practices appropriate for one organisation may be inappropriate for another. It also narrows applicability to banking organisations above thirty billion dollars in total assets, and it excludes simple spreadsheet arithmetic and deterministic rule engines from the definition of a model entirely.

On the European side the timeline moved too, and in the other direction. Under the original AI Act, obligations for Annex III high risk systems, which is the use case list covering employment, credit scoring, education and essential services, were due to apply from 2 August 2026. The Digital Omnibus simplification package changed that. Parliament endorsed it on 16 June 2026 and the Council gave final approval on 29 June 2026, deferring Annex III obligations to 2 December 2027 and obligations for AI embedded in regulated products under Annex I to 2 August 2028.

My reading of that deferral is that it is a trap for the unwary. Sixteen extra months sounds generous until you cost out what conformity assessment actually involves for a credit or hiring model: documented risk management across the lifecycle, data governance evidence, human oversight design, technical documentation, registration in an EU database, and CE marking. Teams I have watched attempt this from a standing start have taken nine to twelve months of part time effort per system. Treat December 2027 as the date the evidence has to exist, not the date you start collecting it.

Building a model inventory that an auditor can read

Where the churn project stands: Part 24 gave us a tracked experiment history and a registry, Parts 25 and 26 gave us monitoring and a deployment pipeline, and Parts 27 and 28 sized the platform underneath. Every one of those produces machine readable facts. What we still lack is the human layer: who owns this, what is it allowed to decide, and what happens when it is wrong. That layer is the model inventory, and it is the single highest leverage artefact in the whole of governance.

My strong preference, and this is a verdict not a survey, is to keep the inventory record in the model repository as a checked in file, generated into a central view by CI. I would avoid a standalone governance platform that lives outside engineering. I have seen three of those bought and all three drifted out of date inside a year, because the people with the facts had no reason to open the tool. A file next to the code gets updated in the same pull request that changes the model. That is the whole trick.

model_id: churn-propensity-v4
owner: retention-analytics
materiality: high
purpose: selects accounts for a retention discount offer
inherent_risk: medium
decision_authority: automated below 0.62, human review above
population: active subscribers, roughly 47000 scored monthly
last_validation: 2026-04-02
next_validation: 2026-10-02
known_limitations:
  - underperforms on two year contracts, see fairness report
  - assumes billing feed latency under 24 hours
monitoring: psi_daily, auc_weekly, segment_recall_monthly
fallback: tenure rule, offer to bottom tenure decile
approved_by: model-risk-committee, 2026-04-11

Two fields in there do more work than the rest. Decision authority states exactly where the model acts alone and where a person intervenes, which is the question every regulator and every incident review eventually reaches. Fallback states what the business does when the model is unavailable or untrusted, and writing it down forces a conversation that teams otherwise have for the first time during an outage. Known limitations is the third one I would never drop, because an auditor who finds a limitation you already documented reads it as maturity, whereas one who discovers it themselves reads it as a control failure.

Materiality then drives effort. Below is the tiering I use, with the review cadence and the rough validation cost per model that I have observed across teams of five to fifteen data scientists. Applying tier one rigour to every model is the classic way a governance programme becomes universally hated and quietly abandoned within eighteen months.

Effective challenge and independent validation

Effective challenge is the term of art for critical analysis by someone competent enough to find real problems, independent enough to say so, and senior enough to make the change happen. All three conditions matter and teams usually fail on the second or third. A reviewer who reports to the person whose bonus depends on the model shipping is not independent, however skilled. A reviewer with no authority to block a release produces a document, not a control.

Validation itself has three components worth naming separately. Conceptual soundness asks whether the design, the assumptions and the data selection make sense before you look at any score. Outcomes analysis compares predictions to what actually happened, through backtesting, holdout evaluation or outlier analysis. Ongoing monitoring asks whether the model still performs as products, customers and market conditions move underneath it. Skip the first and you get a well tested model answering the wrong question, which is the most expensive failure mode I know because nobody catches it for months.

Vendor and third party models get the same treatment and usually get less. You will often be refused the training data and sometimes the methodology, and the guidance is clear that this does not remove the obligation. What you can always do is outcomes analysis on your own population, which is frequently more informative than the vendor documentation anyway. If a supplier will not tell you the training population, benchmark their scores against a simple model you build yourself and see whether the lift justifies the licence. I have seen that exercise end two contracts.

Fairness testing, and where it goes wrong

Aggregate metrics hide everything interesting. Our churn model reported a respectable recall of 0.72 overall, and on that number alone it looked ready. Breaking recall down by contract type told a completely different story, and it is a story that generalises far beyond this dataset: rare positive cases inside a large segment get sacrificed to overall accuracy, silently, because the optimiser has no reason to care about them.

Read the two year bar again. Missing 61 percent of the churners in that segment means the retention team was effectively working blind on the customers with the highest lifetime value, while the reported headline metric looked healthy. Nobody in that chain did anything wrong. Aggregation did the damage on its own. This is why I insist on a per segment breakdown as a release gate rather than as an investigation you run after a complaint.

Two warnings before you go and measure fairness. First, the common definitions are mathematically incompatible: you cannot generally have equal false positive rates, equal false negative rates and equal predictive value across groups at the same time unless base rates match. Pick the one that matches the harm you care about, write down why, and expect to defend the choice. Second, measuring by protected attributes usually requires collecting them, which can conflict with data minimisation obligations. Resolve that with your privacy counsel before you build the pipeline, not after. Broader ground on bias and consent is covered in ethics, privacy and bias in data analysis from the analyst series.

Documentation an audit actually asks for

Audits ask narrower questions than teams expect. In every review I have sat through, the questions reduce to five. What does this model decide and for whom. What evidence existed at approval. Who approved it and were they independent. How do you know it still works. What happens when it does not. Documentation that answers those five crisply beats a hundred page technical report that answers none of them directly.

Practically, that means four artefacts per tier one or tier two model. A model card covering purpose, population, performance by segment and known limitations. A validation report with findings, severity and remediation status. An approval record naming the individual and the date. And a monitoring specification stating the metrics, thresholds and escalation path, which you built in Part 25 and which now just needs to be pointed at from the inventory rather than rewritten.

Continuity is the reason documentation exists, more than compliance. Data scientists move on. A colleague of mine inherited a pricing model whose author had left eighteen months earlier, with a notebook, no writeup and a comment that read do not change this threshold. Nobody could establish what the threshold protected against, so nobody touched it for two years while the market moved. A single paragraph from the original author would have saved that. Write documentation for the person who inherits your work, and it will happen to satisfy the auditor as a side effect.

Governance operating model for a data science team

Structure follows from one idea: whoever builds a model cannot be the only person who signs it off. Financial services formalised this as three lines of defence, and while the name is unnecessarily military, the separation it enforces is sound and scales down to a team of six without much ceremony.

One point regularly gets muddled. Internal audit should not revalidate models. Its job is to assess whether the governance framework is rigorous and whether policy is being followed, which is a different skill and a different question. Audit teams that start rebuilding models duplicate the second line, burn goodwill, and leave the actual control gap untested. If your auditors are recomputing AUC, someone has misread the mandate.

Where to start if you have no governance at all

Start with the inventory and nothing else. Spend one week listing every model your organisation runs, with an owner, a purpose and a materiality tier for each. That list alone will surprise you, because roughly a third of what turns up is something nobody knew was still scoring, and finding those is worth more than any policy you could write in the same week. Do it as a checked in file per model, not a spreadsheet, so it decays slower.

Then take your single highest tier model and run one honest independent validation on it. Give the reviewer authority to block, and let them use it once, publicly. That one event establishes more governance culture than a year of policy circulation, because everybody learns that the gate is real. Add per segment metrics to your release checks in the same sprint. Everything else, the committees, the templates, the annual attestations, can follow at the pace the organisation tolerates.

Governance done this way costs a data science team perhaps five percent of its capacity, and I have never seen it fail to pay for itself, usually the first time a model breaks quietly and somebody can point to exactly which assumption gave way. Part 30 closes the series with the path from here to data science architect and what to study next.

Your move this week: open a file, list every model you can name in ten minutes, and put an owner against each one. Whatever you cannot name an owner for is the first real finding of your governance programme. Reply and tell me how many you found.

References

• SR 26-2, Revised Guidance on Model Risk Management, Federal Reserve, OCC and FDIC, 17 April 2026. Source for the supersession of SR 11-7, the materiality framing, the effective challenge definition and the generative AI scope exclusion in footnote 3
• NIST AI 100-1, Artificial Intelligence Risk Management Framework 1.0, January 2023. Source for the Govern, Map, Measure and Manage structure referenced above
• EU AI Act implementation timeline, Future of Life Institute. Source for the original application dates in the table
• EU AI Act Omnibus agreement and postponed high risk deadlines, Gibson Dunn. Source for the deferral of Annex III obligations to 2 December 2027 and Annex I to 2 August 2028

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts