Halfway through a vendor review in May, our CFO asked how long it would take to move the support assistant to a different provider if the renewal went badly. Two weeks, I said. Actual answer, when we ran the exercise properly in June, was closer to six, and almost none of that time went on the API calls.

That gap between the estimate and the reality is worth understanding before you need it, because every argument about provider risk gets made in the language of API compatibility, and API compatibility is the cheap part.

Four layers of provider coupling

Ask where a switch actually hurts and you get four separate answers, and only one of them is code.

Wire format is the shape of the request and response. Message roles, parameter names, streaming events, error classes. This layer is genuinely cheap to abstract and it is the only one most teams abstract. Our adapter for it came to 41 lines per provider.

Capability is what the provider will actually guarantee. Whether a schema is enforced or merely suggested, whether prompt caching exists and how it is keyed, whether you get token level log probabilities, whether tool calls can run in parallel. Capabilities do not translate. They are present or absent, and code that assumed one has to change shape when it is gone.

Behaviour is how a specific model answers your specific prompts. Two models can both be excellent and disagree about whether an ambiguous ticket needs escalation. Our escalation prompt carried nine examples chosen because they fixed failures on one model family, and roughly half of them were solving problems the other family never had.

Operational surface is everything around the call. Rate limit shapes, retirement schedules, region availability, data residency terms, invoicing granularity, whose status page you watch at 3am. Nothing in your codebase abstracts this, and it is usually what procurement actually cares about.

Portability options compared

Six approaches show up in real codebases. They are not equivalent, and three of them are worse than doing nothing.

Row two deserves a warning louder than a table cell, so it gets its own section below. Row three is the one people over trust: a framework abstraction is written against the intersection of provider features, which means it is always slightly behind whichever provider you actually want to use well.

Latency cost of indirection, measured

Objections to a gateway are usually about latency, so we measured it rather than argued about it. Same prompt, same model, 2,000 requests per path, run from the same region as the service, on our docs assistant answer call with roughly 3,400 input tokens and 180 output tokens.

Three milliseconds for an in process adapter is not a trade off, it is a rounding error against a 2.4 second p99. Forty nine milliseconds for a co located gateway is defensible on a workload where the model itself takes 800. Two hundred and twenty eight milliseconds because somebody put the gateway in the wrong region is a self inflicted wound, and I have seen it survive for months because nobody attributed it correctly.

Writing a task interface instead of a model wrapper

Last part gave the assistant versioned prompts and pinned model snapshots. This part puts a seam behind that pin so a second provider can sit next to the first without touching business logic.

Most portability layers fail because they abstract the wrong noun. If your interface method is called complete and takes messages, temperature and max_tokens, you have rebuilt one provider API with a different import path, and you inherit its assumptions. Name your methods after what the application wants: answer_from_docs, classify_ticket, summarise_thread. Prompt and provider both become implementation detail, which is the only arrangement where a switch stays local.

# assistant/llm.py, tested on Python 3.12.3, openai 2.46.0, anthropic 0.117.0
# Keys read from OPENAI_API_KEY and ANTHROPIC_API_KEY. Never hardcode them.
import os, time
from dataclasses import dataclass

@dataclass
class Answer:
    text: str
    input_tokens: int
    output_tokens: int
    provider: str
    latency_ms: int

class OpenAIAdapter:
    name = "openai"
    supports_schema = True
    def __init__(self, model: str):
        from openai import OpenAI
        self.client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
        self.model = model
    def answer(self, system: str, question: str, max_tokens: int) -> Answer:
        t0 = time.perf_counter()
        r = self.client.chat.completions.create(
            model=self.model,
            max_tokens=max_tokens,
            messages=[{"role": "system", "content": system},
                      {"role": "user", "content": question}],
        )
        return Answer(r.choices[0].message.content,
                      r.usage.prompt_tokens, r.usage.completion_tokens,
                      self.name, int((time.perf_counter() - t0) * 1000))

class AnthropicAdapter:
    name = "anthropic"
    supports_schema = True
    def __init__(self, model: str):
        import anthropic
        self.client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
        self.model = model
    def answer(self, system: str, question: str, max_tokens: int) -> Answer:
        t0 = time.perf_counter()
        r = self.client.messages.create(
            model=self.model,
            max_tokens=max_tokens,
            system=system,                      # top level, not a message
            messages=[{"role": "user", "content": question}],
        )
        return Answer(r.content[0].text,
                      r.usage.input_tokens, r.usage.output_tokens,
                      self.name, int((time.perf_counter() - t0) * 1000))

PROMPTS = {  # prompts fork per family, this is the point
    "openai":    "You answer staff questions from product docs. Cite section ids.",
    "anthropic": "Answer staff questions using only the supplied docs. Cite section ids.",
}

def answer_from_docs(adapter, question: str, max_tokens: int = 400) -> Answer:
    return adapter.answer(PROMPTS[adapter.name], question, max_tokens)

Running the same question through both adapters gives you the shape you want in your logs:

Answer(provider='openai',    input_tokens=3412, output_tokens=171, latency_ms=806)
Answer(provider='anthropic', input_tokens=3388, output_tokens=204, latency_ms=884)

Where a compatibility endpoint quietly lies

Common advice says provider switching is solved: point your existing SDK at a different base URL and change the model string. Several providers publish an endpoint that accepts the OpenAI request shape, and it does work. What it does not do is fail loudly when it cannot honour something you asked for.

In March we shifted the ticket classifier onto a compatibility endpoint over a weekend to get out from under a rate limit ceiling. Ninety minutes of testing, all green, shipped Sunday evening. By Wednesday our downstream ticket router was creating records with missing fields, and structured output validity had fallen from 99.4 percent to 86.0 percent. Nothing had thrown. Three days went into that before somebody read the compatibility page properly and found response_format listed as ignored, alongside seed, logit_bias, presence_penalty and frequency_penalty. Anthropic documents this plainly and also states the layer is intended for testing rather than production, which is advice we had skipped past on the way to the code sample.

Here is the failure reproduced small enough to see. Structured output is requested, validation is strict, and the model answers with a perfectly reasonable sentence instead of JSON:

# compat_trap.py, openai 2.46.0, pydantic 2.11.7
import os, json
from openai import OpenAI
from pydantic import BaseModel, ValidationError

class Triage(BaseModel):
    category: str
    urgent: bool

client = OpenAI(
    api_key=os.environ["ANTHROPIC_API_KEY"],
    base_url="https://api.anthropic.com/v1/",   # compatibility endpoint
)

r = client.chat.completions.create(
    model="claude-sonnet-4-6",
    max_tokens=200,
    response_format={"type": "json_object"},   # accepted, then ignored
    messages=[{"role": "user",
               "content": "Triage: printer offline after firmware update."}],
)
print(repr(r.choices[0].message.content[:80]))
print(Triage.model_validate_json(r.choices[0].message.content))

'This looks like a hardware or driver issue following a firmware'
Traceback (most recent call last):
  File "compat_trap.py", line 24, in <module>
    print(Triage.model_validate_json(r.choices[0].message.content))
pydantic_core._pydantic_core.ValidationError: 1 validation error for Triage
  Invalid JSON: expected value at line 1 column 1 [type=json_invalid]

Cross provider failover with a router

Switching providers permanently and failing over to one temporarily are different problems that share plumbing. Failover needs cooldowns, retry budgets and per error class routing, which is enough machinery that writing it yourself is a poor use of a week. LiteLLM Router covers it, and its three fallback families map neatly onto the failure modes from Part 6.

# failover.py. litellm version not pinned here; Router API verified against
# docs.litellm.ai on 20 July 2026. [VERIFY exact version before you deploy]
import os
from litellm import Router

router = Router(
    model_list=[
        {"model_name": "answer-primary",
         "litellm_params": {"model": "openai/gpt-4o",
                            "api_key": os.environ["OPENAI_API_KEY"]}},
        {"model_name": "answer-backup",
         "litellm_params": {"model": "anthropic/claude-sonnet-4-6",
                            "api_key": os.environ["ANTHROPIC_API_KEY"]}},
        {"model_name": "answer-long",
         "litellm_params": {"model": "anthropic/claude-opus-4-8",
                            "api_key": os.environ["ANTHROPIC_API_KEY"]}},
    ],
    num_retries=2,
    fallbacks=[{"answer-primary": ["answer-backup"]}],
    context_window_fallbacks=[{"answer-primary": ["answer-long"]}],
)

resp = router.completion(
    model="answer-primary",
    messages=[{"role": "user", "content": "Summarise ticket 41822 in one line."}],
    mock_testing_fallbacks=True,      # forces the primary to fail
)
print(resp.model, resp.choices[0].message.content[:60])

claude-sonnet-4-6 Customer reports intermittent sync failures after the 4.2

Separating context window fallbacks from general fallbacks matters more than it looks. A rate limit should route to an equivalent model on another provider. An oversized prompt should route to a larger context model, which may well be the same provider. Collapsing both into one fallback list means every long document quietly gets answered by whichever model happened to be second in the list, and your cost per request climbs without a deploy to blame.

One caution on failover as an availability story. If your prompts are tuned for provider A and you fail over to provider B, you have not maintained your service, you have degraded it in a way your uptime dashboard reports as success. Run your eval set against the backup path too, accept a lower bar there, and record which bar you accepted.

Surfaces that stay stuck

Some things do not port, and pretending otherwise sets up the six week surprise. Prompt caching is the clearest example: cache keying, minimum cacheable prefix length and time to live differ enough that a prompt laid out to maximise cache hits on one provider can lose most of them on another. We watched a cache hit rate of 71 percent fall to 12 percent on a straight port, and the fix was restructuring the prompt, not a config flag.

Stateful conversation features are the other trap. Server side conversation state, where you send a reference to a prior response instead of the full history, is convenient and provider specific. Keep your own transcript as the source of truth even when a provider offers to hold it, otherwise your history lives inside the thing you might need to leave. Same reasoning applies to server side tool execution, hosted retrieval and provider specific safety filter configuration. Vendor particulars for the big four are covered in the AWS and Google Cloud series rather than repeated here.

Eval baselines fork as well, and this is the cost people forget to budget. Your regression suite from Part 22 encodes thresholds calibrated on one model. A second provider needs its own baseline run, its own thresholds and its own history before those numbers mean anything. Four weeks of our six went here, on re-tuning prompts and rebuilding baselines, not on adapters. Treat the pipeline discipline in CI/CD for machine learning pipelines as the shape to copy: a provider is a new target in the matrix, not a variable substitution.

Order to run an actual switch

Once a switch is agreed, sequencing decides whether it takes two weeks or two months. Our six week run was slow partly because we did these steps in roughly the reverse order, discovering prompt problems after the adapter work was already merged and reviewed. Here is the sequence I would use now, and the reason each step sits where it does.

1. Run your eval set against the candidate first, with no integration work at all. Use the compatibility endpoint or a throwaway script; correctness of the plumbing does not matter yet, only whether the model can do the job on your data. Two days. If the candidate cannot reach your baseline with a lightly adapted prompt, stop here and save yourself the other five weeks. We killed one candidate at this gate on a 61 percent answer accuracy against a 88 percent baseline, and killing it early was the cheapest decision in the whole project.

2. Write the capability assertion tests before the adapter. Schema conformance, stop sequences, tool call shape, token usage reporting, cache hit reporting. Half a day, and it converts every later surprise into a red test rather than a quiet regression.

3. Re-tune prompts against the new family, with the eval set as the referee. Budget the largest single block of time here; ours ran to about four weeks of part time work across two people. Resist copying the old prompt across and editing it. Start from the task description and add only the examples that fix failures this model actually shows, because most of your accumulated examples are scar tissue from a different model.

4. Build the adapter and wire it behind the task interface. About a day, and genuinely uneventful if steps one to three went well. This is the step everyone estimates and the only one that fits in the estimate.

5. Recompute unit cost against the new tokeniser, then canary on real traffic. Five percent of tenants for a week, with the new provider tagged in every trace so you can slice quality, latency and spend by provider without guessing. Establish the new baseline from that week; do not inherit the old thresholds.

6. Keep the old provider configured and warm for a month. Not as a fallback in the routing sense, but as a one config move reversal, on the evidence that quality problems in this kind of system are usually reported by a human three weeks after they start, not by a monitor within an hour.

Recommendation for the docs assistant

For a single service at this size, I would build the in process task interface and add a router library inside it, and I would not run a gateway yet. Three milliseconds of overhead, about a day per new provider, no new tier to operate. A gateway earns its keep once four or more services need central keys, budgets and spend attribution, and at that point move the same adapters behind it rather than rewriting.

Avoid the compatibility endpoint as a production migration path. It is a fine way to spend an afternoon comparing two models on your own eval set, and a bad way to serve customers, because silent parameter dropping turns a code change into a quality regression with no error to page on.

On Monday, do one thing: write down every provider capability your correctness depends on, then write a CI test that asserts each one against a live call. Schema conformance, stop sequences, tool call shape, cache reporting. Anything on that list without a test is a thing you will discover the hard way, on a weekend, three days late.

Part 30 closes the series with the path into this role and what to learn next, pulling together the thread that started with a single API call in Part 2.

References

• OpenAI SDK compatibility, Claude Platform Docs, including the field by field support table showing response_format, seed and logit_bias as ignored.
• Fallbacks and provider failover, LiteLLM docs, covering fallbacks, context_window_fallbacks and content_policy_fallbacks.
• Router load balancing and routing strategies, LiteLLM docs.
• Conversation state, OpenAI API docs, on server side response storage and chaining.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts