"Which model is in front of customers right now, and can we roll it back to last week’s?" A product manager asked me that in a standup, and I could not answer either half. Our nightly tune wrote the assistant’s checkpoint to the same S3 prefix every night, no version, so a quiet MT-Bench regression from 6.8 to 6.1 had been live for three days and the good checkpoint was already overwritten. Recovering it meant re-running the whole 220 minute tune and hoping. A model registry is what turns that question into a one line answer and a ten second rollback, and this part wires one into the assistant on OpenShift AI.

Picking up from a tuned checkpoint

Last part spread the assistant’s tune across GPUs and dropped a fresh Granite 3.1 8B checkpoint into object storage, gated on an MT-Bench score. That is where the pipeline stops today: a good model exists, and nobody can say which run made it, what it scored, or how to get back to the previous one. This part gives every checkpoint an identity. Same tune from Part 10, same synthetic data from Part 9, same evaluation gate from Part 11. What we add is a registered ModelVersion at the end of the pipeline, so a green run leaves a named, scored artifact you can find in six months.

Red Hat did not build this from scratch. Underneath sits the Kubeflow Model Registry, an upstream project whose data model is three ML Metadata types: a RegisteredModel is the named thing, a ModelVersion is one revision of it, and a ModelArtifact is the pointer to where that revision’s files actually live. Red Hat’s contribution is the operator that installs it, RBAC that ties registry access to OpenShift groups, and the support line. Model registry shipped as Technology Preview in OpenShift AI 2.16 and is a supported component in the 3.x line this series targets. If a tutorial talks about a bare MLMD server, that is the same metadata layer one level down.

What a model registry stores, and what it stays out of

Here is the single misunderstanding that causes the most grief: registering a model does not move the model. A tuned 8B model in bf16 is roughly 16 GB of safetensors, and that stays exactly where the tune wrote it, in S3 or an OCI image. What lands in the registry is a few kilobytes of metadata and a URI. Confuse the two and you will go looking for your weights inside a MySQL database that only ever held a pointer. Keep the split clear and everything else about the registry makes sense.

A registry is not an experiment tracker either, and the two get conflated. An experiment tracker records every training run, most of which you throw away, and its natural home is MLflow tracking and its registry. OpenShift AI’s registry is the curated shelf downstream of that: the handful of versions good enough to consider serving. The lifecycle below is what this part builds, from a gated tune to a promoted version a serving stack can pull.

Picture the hierarchy with the assistant in it. One RegisteredModel, support-assistant, is the shelf. Under it sit ModelVersions, 1.2.0 then 1.3.0 and so on, each an immutable revision. Every version owns exactly one ModelArtifact, the record that says the bytes live at a given S3 URI in a given format. Custom properties hang off the version, so a query like "give me every version where mt_bench is above 6.5 and stage is production" is answerable without opening a single checkpoint. That queryability is the whole difference between a registry and a folder of timestamped directories.

Keeping the bytes out of the registry is a deliberate design, not a limitation. A 16 GB checkpoint already has a home in S3 with its own lifecycle, backup and access control, and copying it into a metadata store would duplicate gigabytes and split the source of truth in two. The registry instead records the one URI everyone should agree on, and if you later package the model as an OCI image for air gapped serving, the artifact simply points at the image reference instead. Source of truth stays in one place, and the registry stays small enough to query in milliseconds.

flowchart LR
  T[tune Granite 3.1 8B] --> A[artifact in S3]
  A --> R[register ModelVersion]
  R --> E[evaluate MT Bench]
  E -->|clears bar| P[set stage production]
  E -->|below bar| K[keep as candidate]
  P --> S[KServe pulls by version]
  S -->|regression found| B[repoint to prior version]

Enabling the model registry component

Enablement is a cluster admin step done once, not per project. Set the modelregistry component to Managed in the DataScienceCluster and name the namespace its registry instances live in. Registry data persists in a MySQL database, so the instance you create afterward needs a database connection, and this is the first place a setup quietly goes wrong.

# Tested on Red Hat OpenShift AI 3.4 self managed
# Backend: Kubeflow Model Registry, MySQL 8 metadata store, model-registry Python client
# enable the component once, cluster wide
apiVersion: datasciencecluster.opendatahub.io/v1
kind: DataScienceCluster
metadata:
  name: default-dsc
spec:
  components:
    modelregistry:
      managementState: Managed
      registriesNamespace: rhoai-model-registries

$ oc apply -f dsc-modelregistry.yaml
datasciencecluster.opendatahub.io/default-dsc configured
# after you create a registry instance from the dashboard, its pods land here
$ oc get pods -n rhoai-model-registries
NAME                                          READY   STATUS
support-registry-rest-6d8f9c7b4-2x9pn         1/1     Running
support-registry-grpc-7c4b8d69f-lm4qt         1/1     Running
$ oc get service -n rhoai-model-registries | grep support-registry
support-registry-rest   ClusterIP   ...   8080/TCP

One structural point people miss on day one: a registry instance is shared, not per project. It lives in rhoai-model-registries, and any team with the right OpenShift group binding can register into it and read from it, which is what makes it a cross team shelf rather than one more silo per namespace. Grant a group access from the dashboard or with a RoleBinding, and keep write access narrower than read, so a stray notebook cannot archive a production version by accident. For the assistant I gave the platform team write and every product team read, and that split alone prevented two "who changed the live model" incidents in the first month.

Registering a tuned Granite version from Python

From a workbench the model-registry client is the clean path. You connect to the registry route, then call register_model with a name, the S3 URI of the artifact, a version string and whatever metadata you want to be able to search on later. Credentials come from the mounted service account, the same pattern the pipeline and training jobs already use, so no key is typed into the code.

# pip install --pre model-registry   (verified against the 3.x client)
from model_registry import ModelRegistry
from model_registry import utils

token = open('/var/run/secrets/kubernetes.io/serviceaccount/token').read()
registry = ModelRegistry(
    'https://support-registry-rest.apps.example.com',
    author='pranay',            # who registered it, stored on the version
    user_token=token,           # from the mounted SA, not a hardcoded key
)                               # defaults to a secure connection on port 443

model = registry.register_model(
    'support-assistant',                                    # RegisteredModel name
    uri=utils.s3_uri_from('granite-retrain/epoch-1', 'assistant-models'),
    version='1.3.0',
    model_format_name='safetensors',
    model_format_version='1',
    storage_key='assistant-s3-conn',   # the data connection with the S3 creds
    description='nightly tuned granite for the support assistant',
    metadata={
        'base_model': 'granite-3.1-8b',
        'sdg_dataset': 's3://assistant-pipelines/2026-07/',
        'training_run': 'granite-retrain-0728',
        'mt_bench': 6.8,
        'stage': 'candidate',
    },
)
print(model.id, model.name)

# a registered model plus its first version, metadata attached
17  support-assistant
$ # read it back to confirm the pointer, not the bytes, is what was stored
>>> v = registry.get_model_version('support-assistant', '1.3.0')
>>> v.custom_properties['mt_bench'], v.custom_properties['stage']
(6.8, 'candidate')
>>> registry.get_model_artifact('support-assistant', '1.3.0').uri
's3://assistant-models/granite-retrain/epoch-1'

In the pipeline from Part 14 this becomes one more component at the end of the graph, after evaluate and gated on the same score, so a registered version and a green run are the same event. Make that step idempotent on the artifact but not on the version: an identical checkpoint should register once, and a genuinely new checkpoint should always get a new version rather than reuse one. Get that backwards and you either register the same weights ten times or, worse, quietly overwrite the record of what shipped.

My first automated registration crashed the second night. The pipeline computed the version string from the base model name, so every nightly run tried to register 1.3.0 again, and the client refused a duplicate version under the same model.

Traceback (most recent call last):
  File 'register_step.py', line 21, in <module>
  File '.../model_registry/_client.py', line 312, in register_model
model_registry.exceptions.StoreError: a model version '1.3.0' already
exists for registered model 'support-assistant'

That refusal is the registry protecting you: a version is immutable once written, so two different checkpoints can never wear the same version and quietly overwrite the record. The fix is to make the version string carry something unique per run, the pipeline run id or the date, so 1.3.0 becomes 1.3.0-0728. Immutable versions are the whole point; if you find yourself wanting to overwrite one, you actually want a new version.

What you record on each version is what makes the registry worth having six months later, when the person who tuned the model has moved teams. This is the metadata schema I put on every version of the assistant, and it is worth copying wholesale into any registry you stand up.

Promotion, stages and rollback

Registering a version does not make it the one you serve, and the registry deliberately does not choose for you. Promotion is a property you set, and a serving stack reads. When a new version clears the MT-Bench bar, mark it production and demote the previous one; when it regresses, do nothing and the old one stays live. You update a version’s properties and push the change back.

# promote a version that beat the bar, demote the one it replaces
new = registry.get_model_version('support-assistant', '1.3.0-0728')
if float(new.custom_properties['mt_bench']) >= 6.5:
    old = registry.get_model_version('support-assistant', '1.2.0-0721')
    old.custom_properties['stage'] = 'archived'
    registry.update(old)
    new.custom_properties['stage'] = 'production'
    registry.update(new)
    print('promoted 1.3.0-0728, archived 1.2.0-0721')
# promoted 1.3.0-0728, archived 1.2.0-0721

Answering that standup question is now a query, not an archaeology dig. Iterating the versions and filtering on stage with registry.get_model_versions('support-assistant') returns exactly the revision customers are talking to, its MT-Bench score and the training run that made it. Before the registry, that answer took an afternoon of reading pipeline logs and guessing at S3 timestamps; after it, an on call engineer reads it off a dashboard in under a minute. One footgun to respect while iterating: the backend treats a version list as a circular buffer, so use the client’s pager rather than a raw loop or you will spin forever.

Pick a version scheme before the first register call, because a version cannot be renamed later. I use strings where major and minor track the base model and tuning recipe and a date suffix guarantees uniqueness per run, so 1.3.0-0728 reads as third recipe on the 3.1 base, tuned on the 28th. Whatever you choose, make it sortable, make it carry the run identity, and write it down so the pipeline and the humans agree on what a version number means.

Rollback is now the cheap operation the standup question wanted. Because the previous version’s artifact was never overwritten and its pointer is still in the registry, putting it back in front of users is a matter of flipping the stage and letting the serving layer pull the older URI, which the next part wires to KServe. This is the same discipline the AI Engineering Series argues for at the application layer in deployment, versioning and rollback, applied to weights instead of prompts. This chart is why it matters: recovering a known good model from the registry is seconds, against a full re-tune measured in hours.

One honest caveat on the seconds long rollback: repointing weights fixes the model, not everything around it. If the regression actually came from a change in the serving runtime, the prompt template or the retrieval corpus, an older model on a newer stack can behave differently again, which is one reason the version metadata records the training run and the dataset. Pair a rollback with the drift and regression signals from monitoring models in production, so you roll back to a version you have watched behave, not just an older number.

Where registration breaks

Keep this lookup next to the workbench. Five symptoms cover almost every registry problem I have hit on this platform, and each maps to a one line cause and fix.

Register every version before you serve one

References

• Red Hat OpenShift AI, register, version and promote models with the model registry
• Model Registry Python client, register_model, versions and S3 URIs
• Kubeflow Model Registry, upstream project and data model

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts