Roughly one line in twenty of the last AI feature I shipped was a model call. Everything around it, retrieval, schema validation, retries, caching, logging and a test set that could tell me when an answer had quietly got worse, was the actual work.
That ratio is the job description, and it is the thing nobody puts in a job advert. Vendor quickstarts open with five lines of Python that return a sentence, which makes the model call look like the product. In a system that real people depend on, the model call is the smallest and most replaceable part of what you build.
Key takeaways
An AI engineer builds applications on top of models somebody else trained. Training is not part of the job, and in most such roles you will never open a training loop.
Difficulty concentrates outside the model: getting the right context in front of it, checking what comes back, and noticing when quality drops.
Python backend skills transfer almost completely. Non determinism, absent ground truth and cost that scales with input length are what you have to learn fresh.
On version one of our documentation assistant, wiring the model call took 1.5 days out of 30. Retrieval and evaluation together took 13.
Before writing any code, write 20 questions with the answers you would accept. That set outlives every other decision you make.
Daily work of an AI engineer
Here is a week that would be unremarkable in this role. Monday, a support lead reports that the assistant is confidently citing a policy that was withdrawn in March, so you trace the request, find the withdrawn document still sitting in the index, and add a freshness filter. Tuesday, you discover that filter cut recall on three other question types, so you write ten more test questions to pin the behaviour down. Wednesday goes on a schema, because a downstream service wants structured fields and the model returns valid JSON about 94 percent of the time, which is not good enough. Thursday, someone in finance asks why last month cost 3,200 dollars, and you find that a retry loop was re sending the full document context on every timeout. Friday, a new model version is announced and you have to decide whether to move.
Not one of those days involved training anything. Four of them involved measurement, and that proportion holds. My rough split across a year of this work: about 30 percent on retrieval and context assembly, 25 percent on evaluation and regression testing, 20 percent on ordinary application engineering such as queues, caching and error handling, 15 percent on cost and latency, and maybe 10 percent on prompts themselves. Prompt writing gets the attention because it is visible and quotable. It is the part I spend least time on once a feature is past its first week.
Version one of the documentation assistant that runs through this series took thirty working days. Below is where those days actually went, measured rather than remembered, because I logged them at the time for exactly this argument.
Four roles drawn against one system
Role comparisons written in the abstract are useless, so let us put all four against a single system and see who owns what. Picture an internal assistant that answers staff questions from product documentation, a support ticket archive and a changelog. Everything in this series builds that system, so it is worth drawing the ownership lines once, properly.
Notice that the ML engineer band in that picture is optional. Plenty of production assistants contain no custom trained model at all, which is why an organisation can hire an AI engineer and ship a working feature without ever employing anyone who has trained anything. Notice too that the AI engineer band touches every other band, which is the honest reason this role is hard to hire for. It is a systems job wearing a modelling job costume.
| Role | Owns | Done looks like | Main artifact | Failure mode I see most |
|---|---|---|---|---|
| AI engineer | Behaviour of a system built on somebody else model | Feature passes an eval set at an agreed threshold, inside a cost and latency budget | A running service plus the eval suite that guards it | Shipping with no eval set, so quality changes are invisible until a user complains |
| ML engineer | A model trained on your own data, and its serving path | Model beats the incumbent on a held out set and meets a serving SLA | A versioned model in a registry | Training a model where retrieval or a prompt would have been enough |
| Data scientist | Framing the question and deciding what counts as success | A defensible answer, with the uncertainty stated | An analysis and a metric definition | Metric that nobody can compute in production |
| Backend developer | Everything deterministic around the feature | Tests pass, latency and error budgets hold | A service and its contract | Treating a model call like any other idempotent API call |
Where common advice is wrong
Job adverts for AI engineer roles still list PyTorch, deep learning theory and backpropagation. Candidates read that and spend three months on a neural network course. In the role as it is actually practised, you will not touch a training loop, and the same three months spent on retrieval quality, evaluation design and cost accounting would make you employable twice over. Which model to use is close to the most reversible decision in the whole stack, and it is the one most beginners agonise over first. Get your eval set right and swapping models becomes an afternoon.
Skills that carry over from backend work
Good news for anyone arriving from ordinary application development: most of what you know still applies, and it applies harder than usual. Model providers rate limit aggressively, time out under load and occasionally return malformed output, so the retry, backoff and circuit breaker instincts you built calling payment gateways are worth more here than any AI specific knowledge. What changes is a short list, but each item on it is genuinely awkward the first time.
| What you already do | What changes with a model in the loop |
|---|---|
| Assert on an exact response in a unit test | Same input can return different text on every call, so assertions move from equality to scored thresholds over a set of cases |
| Retry a failed request | Every retry re sends the entire input and is billed again, so a retry storm is a cost incident as well as a latency one |
| Validate a payload against a schema | Producer of the payload is a model that will occasionally ignore your schema, so validation needs a repair or reject path, not just a 400 |
| Size a request by payload bytes | Cost and latency track token count and there is a hard ceiling called the context window, so input length becomes a design constraint |
| Fix a bug by reading a stack trace | Wrong answer produces no error at all, so tracing what context was assembled matters more than any exception handler |
Row four deserves emphasis because it catches nearly everyone. A context window is the maximum number of tokens a model can consider at once, and it behaves like a budget rather than a limit you occasionally brush against. Once retrieval starts stuffing documents into a request, you are trading answer quality against both money and response time on every single call, and that trade off has no default setting. Context window explained covers the concept; Part 5 covers what to do about it in a budget you have to defend.
Documentation assistant we build across this series
One project runs the length of these thirty parts. Our imaginary employer is a mid sized software company whose support team answers the same forty questions every week from three sources that never agree with each other: product documentation, an archive of resolved tickets, and a changelog. Staff want a single place to ask. Customers eventually want the same thing, with tighter rules about what it may say.
So that you can follow along on real material rather than three toy text files, we use a public corpus with the same awkward shape: the getsentry/sentry-docs repository. It gives us several thousand documentation pages in Markdown and MDX, a public issue history that stands in nicely for a support ticket archive, and a commit and release trail that behaves like a changelog. It is openly licensed, it contains code samples, nested tables and platform specific variants, and it will break a naive chunker in at least four interesting ways. That is exactly why it was chosen over something tidy.
| Phase | Parts | What the assistant can do by the end |
|---|---|---|
| Calling a model well | 1 to 6 | Answer from a prompt alone, returning validated structured output, inside a known cost per request, without falling over on a rate limit |
| Retrieval | 7 to 13 | Answer from the actual docs corpus, with retrieval quality measured separately from answer quality |
| Agents and tools | 14 to 19 | Look up live ticket status, run multi step work, and escalate to a human at a defined gate |
| Quality and safety | 20 to 25 | Survive a model upgrade without silent regression, resist prompt injection, and be debuggable when it misbehaves |
| Production | 26 to 30 | Run cheaply, respond fast, and move to a different provider without a rewrite |
Where new AI engineers lose their first six months
Two mistakes account for most of the wasted time I have watched, and I have made both of them personally. First, reaching for training when retrieval would do. Second, shipping without any way to tell whether the thing got better or worse.
Second mistake is quieter and more expensive. Teams ship a feature, tune the prompt over a few weeks, and end up with no idea whether their current version is better than the one from three weeks ago, because the only evidence is how the last four answers felt. I now treat a written eval set as a precondition for starting, not as something to add later. Sixty questions written by two people who answer them for a living took an afternoon and became the most reused artefact in the whole project. It survived two model changes, a retrieval rewrite and a provider migration. Evaluating GenAI output gives the concepts; Part 20 builds the set.
A third pattern is worth naming even though it is less common: mistaking an agent for an architecture. An agent, loosely, is a loop where a model chooses which tool to call next rather than following a path you wrote. Agents are genuinely useful and Parts 14 to 19 build one. They are also the answer to far fewer problems than current enthusiasm suggests, and a workflow you wrote yourself is cheaper, faster and debuggable. AI agents explained is the concept; Part 15 argues the case against, which is the part most people skip.
Learning order I would follow
If you asked me to compress this series into advice for your next six weeks, it would be four items in this order. Learn to make a single model call reliable, meaning structured output, retries, timeouts and a known cost per request. Learn retrieval properly, because it is where quality actually comes from and where most of the engineering lives. Learn evaluation, because without it every later decision is a guess dressed as judgement. Only then look at agents, fine tuning and orchestration frameworks, all of which are much easier to reason about once you can measure.
What I would avoid, said plainly: starting with a framework. Picking up an orchestration library before you have written a raw model call yourself means you inherit somebody else opinions about retries, context assembly and prompt structure without knowing they are opinions. Write forty lines of plain Python first, feel where it hurts, then choose the framework that fixes your specific pain. Same argument applies to fine tuning, which Part 7 takes apart in detail, and to vector databases, which are a storage decision masquerading as a strategy.
One action for Monday, and it costs an afternoon. Sit with whoever answers questions in your organisation and write down twenty real questions alongside the answer they would accept. Do not clean them up. Keep the badly worded ones, because those are what users type. That file is your first eval set, it will tell you within a week whether any of this is working, and every part after this one assumes you have it. Part 2 makes the first model call against those questions and shows what the raw output looks like before any of the machinery exists.
References
- getsentry/sentry-docs, the public documentation corpus this series builds against
- OpenAI Python SDK releases, version 2.46.0 as of 17 July 2026, the version baseline for Part 2
- OpenAI API changelog, worth subscribing to before you depend on any model name
- AI engineer as a distinct application layer role, a useful outside view on the same argument
- What a data scientist actually does, Data Science Series Part 1, the neighbouring role in more detail
- What RAG is, GenAI Series, the concept behind Phase 2 of this series
About The Author
DrJha