The setup
At VNTrip we attached flight-delay insurance to flight orders. The insurer issues the actual policy on their side, asynchronously: we submit the request, they answer "received", and confirmation of issuance comes later. Their integration offers the standard shape for this — they call a webhook of ours when the policy is issued.
The first version of the design treated that callback as the mechanism. Policy confirmed when the callback lands; done. It is the obvious design, it is the one the insurer's documentation implies, and it is wrong in three specific ways, each of which is invisible on the day you ship it.
Three ways a callback lies
It can be lost. Their outbound call crosses the public internet into our ingress. A deploy on our side, a network blip on either side, a retry policy on theirs that gives up — and the message is simply gone. Nothing errors on our side, because from our side nothing happened. The order sits in "pending" forever, which the customer eventually experiences as insurance they paid for and cannot see.
It can arrive twice. Retries exist precisely because delivery is unreliable, so the same confirmation can land two or more times. If applying it twice does anything — double-writes a row, re-fires a customer email — the retry that was supposed to save you creates the incident.
It can arrive before your own write is visible. This is the one that bites people who have handled the first two. We create the local policy record inside a transaction, call the insurer, commit. On a fast day their callback comes back before our commit is visible to the connection handling the webhook. The handler looks up the order, finds nothing, and now the most correct, most prompt message the insurer will ever send us gets dropped — by us.
How often each of these actually happened in production, I never counted, and that is rather the point: once the design below was in place, the question stopped mattering.
The poll is the design
The fix was not a better webhook. It was demoting the webhook. A scheduled job polls the insurer for every order still awaiting confirmation and applies whatever state it finds. That job is the source of truth: if the callback never comes, the poll gets the answer; if the callback came too early and was dropped, the poll gets the answer; if it came twice, see below. The webhook stays, because customers like fast — but its only job now is making the answer arrive sooner than the next poll tick. Losing it costs latency, not correctness.
That inversion forces two properties on the apply step:
Idempotency. Callback and poll can now both deliver the same confirmation, so applying it must be safe to do twice. The apply is a state transition — pending to confirmed with the insurer's policy number — and a transition that has already happened is a no-op, not an error. Once that holds, retries, duplicates and the callback-vs-poll race all collapse into the same harmless case.
Reconciliation. The poll answers "what does the insurer say"; it does not answer "do we agree". A separate report walks both sides — our rows, their policy states — and lists every order where the two disagree, in either direction. It trusts neither side; it compares. Mismatches export to CSV so a human can audit a period, and one order can be re-synced on demand instead of waiting for a batch. That report found real disagreements; how many it would have found in the months before it existed, I cannot know, because nothing was looking.
The rule
If losing one message silently changes a customer-visible state, that state needs a poll, not a louder webhook. Harden the callback all you want — retries, queues, signatures — it remains a push across a boundary you do not control, and the failure mode of push is silence. The poll turns silence back into a question you ask on your own schedule. The webhook is an optimisation. It is a good one. That is all it is.