Skip to content
Kien

The sandbox is not the supplier

· 3 min read

cert green · prod different

A supplier's test environment passes the whole certification catalog, and production behaves differently — inventory that never sells out, errors that are not in the document, rate limits that exist on only one side, and latency you cannot tune a timeout against. What certification actually proves, and what the first real bookings are for.

  • integrations
  • testing
  • suppliers
  • operations
On this page
  1. Green is a smaller claim than it looks
  2. The inventory never sells out
  3. The documented errors are a subset
  4. Rate limits exist on one side only
  5. Latency distributions are not comparable
  6. Treat the first production bookings as the real test
  7. The rule

Green is a smaller claim than it looks

Integrating a supplier ends with certification: a per-provider testcase catalog driven through the whole flow, with each testcase's request and response and the upstream call crawled into a named folder. That folder is the artifact the partner reviews, and I would not give it up. But it is worth being precise about what a fully green catalog asserts. It asserts that against their test environment, on their scripted cases, our module said the right things and understood the answers.

Production is a different system with the same schema. The ways it diverges are not mysterious once you have watched a few integrations go live; they are the same four every time.

The inventory never sells out

Sandbox inventory is static. It is seeded once so the test cases stay reproducible, which is exactly what a test environment should do, and it means the room you selected is always still there when you go to book it.

So the whole family of code paths that handles the offer you selected is gone is never executed. Not tested badly — never executed. The gap between search and prebook is where a real supplier tells you the rate was withdrawn, the price moved, the allocation went to someone else. In certification that gap is always empty, and the branch we wrote for it has, at go-live, the status of an untested guess written months earlier by someone reading a PDF.

The documented errors are a subset

Sandbox returns the errors the integration guide describes, because the same team wrote both. Production adds the ones nobody wrote down: an upstream timeout surfaced in the supplier's own wrapper, a maintenance page, a credential state that only exists for live accounts, a validation message in a language the document does not use.

This lands harder where a supplier answers HTTP 200 with the error inside the body — a real pattern, not an edge case. An unrecognised error shape inside a 200 does not fail loudly; it parses to something empty and travels on as a successful-looking result.

Rate limits exist on one side only

Nobody rate-limits a sandbox seriously. Hotel and flight search fan out one job per supplier across a queue, and the concurrency the sandbox absorbed without comment is precisely the number production starts counting. The first time you learn the real ceiling is usually under real search traffic, which is also the worst time.

Latency distributions are not comparable

A sandbox serves canned answers from a small dataset. Production searches real inventory under real load. The medians are different, and the tails — the part a timeout is actually about — are not related at all. Hotel search sits under an absolute 30-second deadline with a per-supplier budget beneath it, and any budget set from certification timings is a guess wearing a number's clothes.

Treat the first production bookings as the real test

The practice that follows is unexciting and works. Keep the evidence pack; it still answers the partner's question. Then treat the first bookings on production as the actual test: the cheapest offer with free cancellation, run by a person who is watching, not by a schedule. And log the request and response of the very first production call — every provider call is logged with both, and a failed one can be re-fired as the exact curl it sent, which is the difference between a first surprise that is diagnosable and one that is a story.

I do not have a count of how many integrations produced an error shape in their first production week that was absent from certification. Nobody tracked it. The pattern is strong enough that I plan around it anyway, and I would rather say that plainly than attach a number I never collected.

The rule

Certification proves you can talk to them. It does not prove you can work with them. The first is a conversation under laboratory conditions; the second includes sold-out rooms, undocumented failures, a limit you have not met yet, and a tail latency nobody showed you.