A queue usually enters a codebase as a performance decision, and that framing hides the trade. You are not buying speed. You are buying a different failure surface, and the bill arrives in instalments, on different days, long after the person who chose the queue has moved on to something else.
Here is the ledger, from a system that made the choice.
What it buys
Hotel, flight and car search on our gateway fan out one job per supplier onto a queue; the HTTP layer aggregates whatever has landed and answers under an absolute 30-second deadline. The mechanics are a separate piece; what matters here is the single property that justified it.
Without the queue, a search is a loop and the loop's latency is the slowest supplier in it — one supplier having a bad afternoon becomes every customer's bad afternoon, including on searches where that supplier had nothing to sell. With the queue, the slow supplier is slow by itself: its job holds its own worker and misses the deadline alone, while everyone else's results are already on the page.
That is the purchase: a slow participant stops being a shared problem. If that is not what you wanted, the rest of this essay is a list of prices paid for nothing.
Ordering is gone unless you buy it back
The first charge lands immediately. A queue delivers in roughly the order things arrived, and "roughly" is doing all the work: with more than one consumer, arrival order and processing order are different sequences. Two messages about the same entity can be applied backwards and nothing notices, because each was individually valid.
The fix is never "make the queue ordered". It is to decide, per key, what must serialise, and pay for that specifically. Our per-supplier jobs are each guarded by a Redis lock, which buys exactly one thing: one worker on this supplier for this search — nothing about ordering across searches, and it was never meant to. The approval engine I built at VNTrip pays differently: optimistic locking on the approval step, so a stale write loses rather than overwrites. Same admission either way — order is a property you add back where you need it, not one the transport hands you.
A retry makes idempotency your problem
The second charge is the one I see misread most often. A queue promises at-least-once delivery, and "the queue will retry it" gets read as a reliability feature. It is not. It says your consumer may run twice on the same message, and whether that is safe is a property of your code, not of the queue's promise.
The test is not "does it retry", it is what a second run does. A worker that writes its supplier's results into a cache hash under that supplier's key is safe by shape: run it twice, the second write lands on the same key with the same shape. A worker that appends is not. A worker that sends money is emphatically not — which is why a retry policy on a job that books something is a design decision argued out loud, not a number copied from the last service's config.
A message that fails forever takes the consumer with it
The third charge is the one that turns into an incident. Somewhere there is a message that will never succeed: a payload from an older schema, a supplier code that no longer exists, a null in a place the parser assumed could not be null.
With retries on and nowhere for it to go, that message comes back, fails, comes back, fails. The consumer is busy, the queue depth looks alive, nothing progresses — and the metric that would say so is work completed, not messages handled, which is usually the one nobody plotted. The message needs a dead-letter destination that keeps the payload and the failure that sent it there. A dead-letter queue nobody reads is a delete with extra steps, so it needs an owner too: an organisational cost, not a technical one.
The state of the work now lives in three places
The fourth charge is permanent. Before the queue, "is this done?" was a question about a function that had returned or not. After it, the answer is distributed: the queue knows what will happen, the cache holds partial results, the database holds committed truth. Three stores, three answers, and the failure modes are the disagreements between them — a job acknowledged whose cache write never landed; results in the cache with no record marking them complete; a row committed while the job that should have followed it never ran.
This is why the completion marker exists at all. A results hash cannot separate "this supplier has not answered yet" from "this supplier answered with nothing", and those mean opposite things to whoever is reading the page. So a worker writes a second, meta record when it finishes: completion becomes a positive fact rather than an inference from silence. The progress fraction is read from that record, and so is the answer to "who did not come back before the deadline" — the difference between telling a partner "no availability" and telling them "one supplier timed out".
And the stack trace becomes a correlation exercise
Synchronously, a failure arrives attached to the thing that caused it: one stack, one request, one place to look. A queue cuts that in half. The producer's stack ends at enqueued, the consumer's begins at received, and they are joined only by an identifier somebody decided to carry in the payload. Without that decision the two halves of the story sit in two log streams with no column to join on — and the identifier has to exist before the incident, because it cannot be added to logs already written.
I have no before-and-after latency number for the queue on our search path — nobody instrumented the synchronous version before it was replaced — so the case here is structural rather than measured, and I would rather say so than round a plausible figure into a claim.
The rule
A queue moves the failure. It does not remove it. The slow supplier is still slow, now somewhere that blocks nobody else. The error still happens, now in a process with no user attached — which is why it needs a dead letter and a correlation id to be visible at all.
Adopt one when that move is precisely what you wanted — when isolating a slow participant is worth ordering, idempotency, poison handling and a state machine spread over three stores. Adopt it because concurrency sounds modern and you will pay every instalment while collecting nothing.