Prompt to Production: Turning an AI Experiment Into a System
A prompt that works is a demonstration. A system is that prompt plus everything you do when it is wrong, and the failure path is the part almost every team leaves until after the first complaint.
On This Page
Somebody spent a weekend on a prompt and it now does something genuinely useful. It drafts the reply, classifies the enquiry, summarises the ticket. The demo lands, the room agrees it should go live, and a date goes on a board. That is the moment the work changes shape entirely, and almost nobody schedules for it.
The claim this piece defends: the prompt is the smallest part of what you are about to ship. AI productionisation is mostly the work of surrounding a probabilistic component with things that are not probabilistic — an input contract, a measurement, a defined path for the bad answer, a version history, a budget, and a way to notice. Skip any of them and the system still runs. It just stops being something you can defend.
None of this is exotic engineering. It is the ordinary discipline any other service gets, applied to a component that happens to answer in sentences. What makes it easy to skip is that the thing already appears to work, which is never true of a half-built API. So the shortcut is available, it is tempting, and the integration work that follows a skipped step always costs more than the step.
What AI productionisation actually changes
The experiment and the system share a prompt and share nothing else. Every property that made the demo pleasant was handed to it for free by having exactly one careful user, who was also its author.
| Concern | In the experiment | In the system |
|---|---|---|
| Input | Chosen by the person running it | Whatever arrives: empty, enormous, in another language, hostile |
| Output | Read by someone who can tell it is wrong | Consumed by code, or shown to a customer who cannot |
| Correctness | Judged in the room, once | Judged against a labelled set, repeatedly |
| Failure | Rerun it with different wording | A defined path, with an owner and a message |
| Change | Edit the text in a browser tab | A versioned change with a before and an after |
| Cost | Nobody looks | Per call, per user, per month, against a ceiling |
| Visibility | The author is watching | Logs, sampled review, and something that alerts |
Read the right-hand column as a work list rather than a warning. For a small feature it is roughly a fortnight of engineering, and it is the fortnight that decides whether the thing survives contact with real users. It is also what a technical discovery should have produced before anybody agreed to a launch date.
Inputs stop being the ones you chose
In testing, the input is a well-formed paragraph typed by someone who wants the system to succeed. In production it is a forwarded email thread carrying six signatures, a screenshot pasted as text, three words, forty thousand words, or a deliberate attempt to make the model say something embarrassing.
The input contract
Write down what the system accepts before you write down what it does. Maximum length, minimum length, expected language, required fields, and what counts as an input this feature is simply not for. That document takes twenty minutes, and it converts a whole class of mysterious behaviour into a rejected request with a clear message attached.
Treat everything arriving from a user, a webhook or a scraped page as untrusted, in exactly the sense the OWASP Top Ten has meant for two decades. Text that reaches a model is text that can carry instructions, and a model handed both a document and a tool has been handed a way for that document to act. The mitigations are dull and effective: bound what the model may do, never put retrieved text where a system instruction belongs, and check the output against the contract instead of trusting it.
What to do with input you cannot serve
Refuse it, visibly, with a route to a person. The failure we see most often is a system that quietly does its best with an input it should have declined, producing an answer that is fluent, confident and about the wrong thing. A visible refusal costs one support message. A confident wrong answer costs the trust of everyone who reads it, and occasionally a correction letter.
Evaluation is the entry fee, not the finish line
You cannot improve what you cannot measure, and you cannot measure a language model by looking at it. That argument has been made at length elsewhere, including by us in building an evaluation set before you trust a model, so the short version will do here: a few dozen real inputs, each with an agreed correct answer, run on every change, is the minimum that lets you tell an improvement from a rearrangement.
The part that gets skipped is running it automatically. An evaluation set kept in a spreadsheet and run when somebody remembers is a document, not a control. Wire it into the pipeline that already runs your tests, as a workflow on every change, and the score becomes something a reviewer sees before merging rather than something a customer finds after deploying.
Automated scoring has limits worth stating plainly. It catches regressions well and judges tone badly, so a sampled human read stays part of the system rather than a phase you eventually exit. Our parent company has written about what that reviewer is actually for in the reviewer’s pass, and the logic holds whether the output is prose or a decision.
The step most teams skip
It is not evaluation. Teams have heard that lecture and many now run a set. The step still going missing is the one immediately after: deciding, in writing, what the system does when the answer is wrong. Ask a team what happens on a bad answer and the reply usually describes what happens on a good one.
The output contract
Constrain the output before you handle its failures, because most handling is impossible otherwise. Ask for a defined structure, validate it on arrival, and treat a response that does not match as a failure rather than as text to be parsed hopefully. Free-form prose consumed by code is the most common source of breakage in these systems, and it is entirely self-inflicted.
The pattern is not unique to AI. A product like Nichevio assembles a site from structured, customisable widgets rather than a free-form canvas for the same underlying reason: a bounded slot can be validated, rendered and changed later, while a blob can only be pasted somewhere and hoped over.
What happens on a bad answer
There are four honest options, and each failure type needs one chosen: retry with a changed request, fall back to a deterministic path, hand the case to a person with the context attached, or decline and say so. Choosing none of them does not avoid the decision. It just means the default is to ship whatever came back.
Work an example. A system classifies inbound enquiries into six categories. The provider times out: retry once, then fall back to the catch-all queue, because a late enquiry in the right place is worse than an immediate one in the general pile. The response arrives but names a seventh category that does not exist: reject it at the validator and route to the catch-all, then count it, because a rising count means the taxonomy has drifted away from what people actually write. The response is valid but the model reports low confidence: escalate to a person with the original text attached. Three failure types, three different answers, and none of them obvious until somebody sat down and enumerated them.
A model rarely fails by erroring. It fails by returning something plausible and wrong, at the same latency, in the same shape as a correct answer. Monitoring built around exceptions will show a clean dashboard through the whole incident. Assume the quiet failure is the normal one, and design the checks around that assumption rather than around crashes.
Nobody ships a model. They ship a system with a model inside it, and the system gets judged on the days the model is wrong.
Versioning something nobody calls code
A prompt is configuration that changes behaviour, which makes it code regardless of what it is stored in. Keep it in the repository with everything else, so the version history records who changed the wording, when, and what the evaluation scored before and after. A prompt edited in a vendor console at four on a Friday, by someone with good intentions and no record, is the most common cause of a system that "was working last week".
Version the model choice too. Providers deprecate versions, retire them, and change behaviour underneath a stable name, so the model identifier belongs in configuration where it can be read, pinned and rolled back. When it changes, the evaluation set runs again. That is the entire ceremony, and it is the difference between an upgrade and a surprise.
Alongside the prompt, keep the thing the prompt is for: a short written statement of the job, the accepted inputs, the output shape and the failure behaviour. It is what a new engineer reads instead of reverse-engineering three hundred lines of instructions, and it turns the question of who writes your documentation into something checkable rather than something argued about.
Cost stops being invisible
During the experiment, inference cost is somebody’s personal allowance and nobody thinks about it. In production it scales with usage, which is the thing you were hoping would grow. Two properties make that awkward rather than merely expensive: cost per request is variable, because it moves with input length, and the most enthusiastic users are the most expensive ones.
So measure per call, per user and per feature from the first day rather than the first invoice, and set a ceiling that does something when it is reached. Retrieval that pastes an entire knowledge base into every request is the usual culprit, and it is often also why the answers got vaguer. Trim the context and both problems improve together, which is the rare case where the cheap option is the better one.
Latency deserves the same treatment, because it is a cost paid by the user rather than the account. A model call sits in the middle of a page load or a queue, and its slowest responses are considerably slower than its median ones. Decide what the interface does while it waits, decide when waiting becomes a timeout, and cache the requests that repeat. A surprising share of production traffic is the same question asked again.
Monitoring a system that fails quietly
Standard application monitoring watches errors, latency and saturation. A model-backed feature can be healthy on all three while producing nonsense, so the instrumentation has to watch different things. At minimum, record enough per request to answer this: what did it see, what did it say, and what did the person do next.
- The request and response, with the prompt version. Redacted where it must be, retained for a defined window, and searchable. Without this, no incident can be reconstructed afterwards.
- Validation failures, counted separately from errors. Output that did not match the contract is the earliest signal that something upstream has changed.
- Refusals and fallbacks. A rising refusal rate is a story about your inputs. A falling one, straight after a prompt edit, is usually a guardrail somebody removed by accident.
- The human signal after the answer. Edits, retries, rejections, escalations to support. The cheapest quality measure available, and almost nobody logs it.
- A standing sample for review. A handful of real interactions read by a person each week, on a rota, with somewhere to record what was wrong with them.
All of it decays without an owner, which is the same quiet ending most automations nobody is responsible for arrive at. Put a name against the feature, put the review in a calendar, and treat the connection between systems as infrastructure rather than a favour.
What we would build first
If the prompt works and a date exists, this is the order we would run it in, and why each step sits where it does.
- Write the job statement. One paragraph: what this does, for whom, and what it must never do. Everything downstream is a test of that paragraph.
- Define the input and output contracts. What is accepted, what comes back, and what a violation means. Do this before touching the prompt again.
- Build the evaluation set from real inputs. Thirty is a start. Include the awkward ones you would rather not think about, because those are the ones that ship.
- Choose a failure path per failure type. Retry, fall back, escalate, decline. Write it beside the contract so the decision outlives the person who made it.
- Version the prompt and pin the model. In the repository, reviewed like any other change, with the evaluation running before anything merges.
- Instrument, then launch narrow. One team, one segment, one week. A narrow launch with logging beats a wide launch with confidence.
The honest concession: this sequence is disproportionate for some things. An internal tool that drafts something a person always rewrites needs the job statement and a light review, and skipping the rest is a fair trade. The line we draw is whether an output reaches a customer or triggers an action with nobody in between. Cross it and every step becomes non-negotiable, including telling people what they are talking to, which is a published commitment for us rather than a preference, and increasingly what the detection argument comes down to in practice.
Start with the failure path. Not because it is the hardest part, but because writing it drags every other decision into the open: you cannot say what happens on a bad answer without first agreeing what a bad answer is, who owns it, and what the system owes the person on the other side. If that is where you are stuck, describe the step you are automating and what it costs on the day it goes wrong.
Common questions.
What does it take to move an AI prototype into production?
Six things beyond the prompt itself: a written contract for accepted inputs, a structured output that can be validated, an evaluation set that runs automatically, a defined path for wrong or unavailable answers, version control over the prompt and the model identifier, and monitoring that records requests, responses and what the user did next. Most of that is ordinary engineering discipline applied to an unusual component.
How do you monitor an AI feature that does not throw errors?
Log the request, the response and the prompt version on every call, then measure the signals that reveal quiet failure: output that fails validation, refusal and fallback rates, and what a person did afterwards. Edits, retries, rejections and escalations to support are the cheapest quality measure available. Add a small standing sample that a named person reads on a regular rota.
Should prompts be stored in version control?
Yes. A prompt is configuration that changes behaviour, so it belongs in the repository beside the code that calls it and should be reviewed like any other change. That gives every wording change an author, a date, a diff and a rollback. Prompts edited directly in a vendor console leave no record, which is why systems appear to stop working with no corresponding deployment.
How much should an AI feature cost to run?
That depends on how much context you send and how often, so measure cost per call, per user and per feature from the first week rather than waiting for an invoice. Set a ceiling that triggers an action when reached. Oversized retrieval, where an entire knowledge base is pasted into every request, is the usual cause of both high spend and vaguer answers.
What is the difference between evaluation and monitoring for AI systems?
Evaluation runs before a change ships, against a fixed labelled set, and answers whether the new version is better than the old one. Monitoring runs continuously against real traffic and answers what is happening right now. Both are needed. Evaluation catches regressions before users meet them, while monitoring catches the input distribution drifting away from anything you originally tested.
When is a rule better than a model?
Whenever the input varies less than it appears to. Deterministic logic is cheaper, faster, testable and explainable, so a model should earn its place by handling variation a rule genuinely cannot enumerate. A useful check is to write the rule first. If it takes an afternoon and covers most cases, the model belongs to the remainder rather than to the whole task.
Facing this in your
own business?
Tell us where you’re headed — we’ll map the shortest honest route.