API Migration Testing for LLM Apps: A Practical Plan
API migration testing for an LLM app is different from testing a normal REST API, because the same prompt never returns the same text twice. You test properties of the answer, not exact strings.
Last updated

Summary
- Build a golden set of 50 to 200 real prompts with rule-based checks, not exact-match expectations.
- Fix temperature and seed to reduce noise, but never expect identical text.
- Record latency percentiles and cost per request on the old and new route.
- Write rollback triggers down before the rollout starts.
What you are actually testing
With a classic API you compare responses byte for byte. With a language model that breaks immediately. Instead, decide which properties of an answer your product depends on, and test those.
- Shape. Valid JSON, required keys, a tool call with the right arguments.
- Content. Mentions the required facts, follows the format, stays in the persona.
- Safety of the product. No refusal where an answer is expected, no leaked system prompt.
- Operations. Latency, error rate, rate limit behaviour, cost.
If you still need the switching steps themselves, the plan is in the migration guide on this site, and failures you hit along the way are covered in common LLM integration errors. This article covers verification.
Build the golden prompt set
Pull 50 to 200 real prompts from your logs (strip personal data first). Cover the common path, the awkward cases and the cases that broke before. Store each as a record with its messages, parameters and the checks it must pass.
{
"id": "invoice-json-03",
"messages": [
{"role": "system", "content": "Extract fields. Reply with JSON only."},
{"role": "user", "content": "Invoice 4471 from Acme, total 1,250 EUR, due 2025-03-01"}
],
"json": true,
"required_keys": ["number", "vendor", "total"],
"must_contain": ["4471"]
}
Tag cases by risk: tool calls, long context, streaming, non-English, very short and very long answers. When a bug appears in production later, add it to the set. The set grows into your regression suite.
Write a harness that records everything
One script should run the whole set against a route, apply the checks and save the numbers. Use temperature=0 and a fixed seed to cut variation. Both are passed through by this API, but treat them as noise reduction, not a guarantee.
import json, time
from openai import OpenAI
client = OpenAI(base_url="https://api.apimigration.com/v1", api_key="YOUR_KEY")
PRICE_IN, PRICE_OUT = 0.15, 0.50 # USD per 1M tokens
def run(case):
t0 = time.perf_counter()
raw = client.chat.completions.with_raw_response.create(
model="uncensored",
messages=case["messages"],
temperature=0,
seed=42,
)
elapsed = time.perf_counter() - t0
resp = raw.parse()
text = resp.choices[0].message.content or ""
u = resp.usage
return {
"id": case["id"],
"text": text,
"seconds": elapsed,
"cost": (u.prompt_tokens * PRICE_IN + u.completion_tokens * PRICE_OUT) / 1e6,
"balance": raw.headers.get("X-Balance-USD"),
"request_id": raw.headers.get("X-Request-Id"),
}
Keep the X-Request-Id for every failed case. It is what you quote when you ask support about a specific request.
Diff by rules, not by strings
Apply the checks from each record. Anything that fails goes to a short list for a human to read.
def check(case, text):
problems = []
if case.get("json"):
try:
obj = json.loads(text)
except ValueError:
return ["invalid json"]
for key in case.get("required_keys", []):
if key not in obj:
problems.append("missing key " + key)
for needle in case.get("must_contain", []):
if needle.lower() not in text.lower():
problems.append("missing text " + needle)
if len(text.strip()) < case.get("min_chars", 1):
problems.append("answer too short")
return problems
For free-form answers, add a cheap length band (for example within half to double of the old answer) and a short list of refusal phrases to flag. Then read 20 outputs side by side by eye. Rules catch regressions; reading catches tone. Do both.
For JSON outputs, set response_format to {"type":"json_object"} and still validate, since the checks above are cheap.
Test the failure paths on purpose
Happy-path prompts pass on almost any provider. Failures are where migrations hurt, so provoke them deliberately before users do.
- Bad key. Send a request with a wrong key and confirm your app shows a sane message and does not retry in a loop.
- Rate limits. Fire more than 8 requests at once on a full key (2 on a trial key) and confirm your code handles HTTP 429 with the
concurrencycode, then therate_limitedcode if you push past the per-minute cap. - Temporary overload. A 429 with
Retry-After: 5should lead to a wait of five seconds, not an immediate hammering retry. - Oversized input. Send a prompt near the 100,000-token window and one beyond it. Your app should trim or summarize, not crash.
- Big bodies. The limit is 8 MB. Confirm you reject larger payloads yourself.
- Dropped streams. Kill the connection mid-stream and confirm the concurrency slot comes back.
Each of these should have an automated test that runs in CI against a fake server, plus one manual pass against the real one.
Check streaming and tool calls
Streaming and tools change the shape of the response, so a golden set built on plain text misses them. Add a case for each.
tools = [{
"type": "function",
"function": {
"name": "get_order_status",
"description": "Look up an order by id",
"parameters": {
"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"],
},
},
}]
r = client.chat.completions.create(
model="uncensored",
messages=[{"role": "user", "content": "Where is order A-1042?"}],
tools=tools,
tool_choice="auto",
)
calls = r.choices[0].message.tool_calls or []
assert calls and calls[0].function.name == "get_order_status"
print(calls[0].function.arguments)
The assertion checks the call exists and that the arguments parse as the JSON you expect. Add a check that the second turn, with the result sent back as a message with role: "tool", produces a normal answer. For streams, assert that chunks arrive, that the final usage appears and that your code stops reading at the end of the stream.
Track latency and cost per route
Run the set on the old and new route and compare the same columns: median and 95th percentile seconds, error count, tokens, cost. This article gives no latency numbers for any provider, because yours will depend on your prompts and network. Measure your own.
Cost for the new route is token counts times the rates, $0.15 per 1M input and $0.50 per 1M output. Example: 200 golden prompts averaging 1,200 input and 300 output tokens.
- Input: 240,000 tokens x $0.15 / 1,000,000 = $0.036.
- Output: 60,000 tokens x $0.50 / 1,000,000 = $0.03.
- One full run: $0.066.
The $0.50 trial credit covers about seven such runs. A trial key allows 2 concurrent requests and 60 per minute, so 200 prompts take at least three and a half minutes. Failed or refused requests are not charged. You can also read the balance from the X-Balance-USD header after each call to confirm your own arithmetic.
Shadow traffic and staged rollout
Once the golden set passes, copy live requests to the new route without showing the answers. Log the results and compare them to the live responses with the same checks. Shadow runs need a concurrency cap: a full key allows 8 requests at once, so queue the mirrored calls instead of firing them all together.
Then move real users in steps, such as 5%, 25%, 50% and 100%. Hold each step until error rate, failed checks and complaints stay flat.
Decide the rollback rules in advance
Write thresholds down before the first user is moved, so the decision in the moment is mechanical.
| Signal | Example trigger |
|---|---|
| Check failures on live traffic | More than your agreed share fail rule checks |
| HTTP 429 or 5xx rate | Sustained above the old route's baseline |
| Latency p95 | Above your product's timeout budget |
| Cost per request | Above the estimate by a set margin |
| User reports | A cluster about the same behaviour |
Rollback should be a config change, not a deploy. Keep the old route funded and configured for at least a few days after 100%. When you are ready to run the tests, get an API key and claim the trial, check pricing for larger runs, and use the docs for endpoint details.
More guides
Common questions
How many golden prompts do I need?
Fifty to two hundred real prompts covers most apps. Quality matters more than count, so include the awkward and previously broken cases.
Can I compare outputs with an exact-match diff?
Not reliably. Even at temperature 0 with a fixed seed, treat outputs as variable and test rules such as valid JSON, required facts and length.
How do I track cost during testing?
Multiply usage tokens by $0.15 per 1M input and $0.50 per 1M output, and watch the X-Balance-USD header for the actual balance.
What should trigger a rollback?
Define thresholds first: failed checks, 429 or 5xx rates, latency against your timeout, and cost against the estimate.
Get your API key
Register, copy the key, update the base URL. Setup is complete.