Skip to content
Kien
ENVI
All lessons

One hotel search wrote 1.2 GB into Redis

· 4 min read

1.2 GB → ~200 MB

An audit of a hotel engine's search path: why the worker kept getting restarted at the 3 GB mark, why the event loop stalled for 6–20 seconds per job, and the three changes that took Redis from 1.2 GB to 150–250 MB per search.

  • Node.js
  • Redis
  • performance
  • event loop
On this page
  1. The setup
  2. Three things killing it
  3. Three changes
  4. The CPU half: 100 million date parses
  5. Proving it
  6. What I took away

The setup

One hotel search fans out to roughly 20 suppliers. Each returns about 2,000 hotels, each hotel about 250 offers. That is 10 million offers for a single search.

The symptoms were the familiar ones: the worker kept being restarted by PM2 at max_memory_restart 3G, and the API got slower whenever a big job was running.

Three things killing it

1. Peak memory scaled with hotels × offers. One job held around 500,000 offers as JavaScript objects — 1–2 GB. Worse, the persist step ran Promise.all over 4 chunks of 500 hotels concurrently, with no cap, multiplying peak memory by about 4. That is what hit the 3 GB ceiling.

2. Synchronous compression, all inside one process.nextTick.

// Old version, simplified: 2,000 iterations, never yielding
for (const hotel of hotels) {
  const json = JSON.stringify(hotel)     // ~300–800 KB per hotel
  pipeline.hset(key, hotel.id, gzipSync(json))   // ~2–5 ms per hotel
}

JSON.stringify produced roughly 600 MB – 1.6 GB of string churn per job, and gzipSync across 2,000 hotels blocked the event loop for 6–10 seconds. All of it sat inside a single process.nextTick, so for that whole stretch the process answered nothing.

3. The read path scanned the entire hash on every poll. The "cheapest" list was stored as one field per hotel, so filtering by supplier meant scanning a 40,000-field hash — about 800,000 comparisons inside single-threaded Redis. Node then did 40,000 unzipSync calls plus 40,000 JSON.parse calls, another 1–3 seconds of blocking per call. The frontend polls until the search completes.

Three changes

Batch, and yield. The 2,000-hotel loop became batches of 100 with await setImmediate() between them. Not faster in total, but the process stops going unresponsive mid-job.

Move compression to the threadpool. zlib.gzipSync became promisify(zlib.gzip) running on the libuv threadpool, bounded by p-limit(4). Compression leaves the event loop, and the cap of 4 keeps peak memory from exploding the way the uncapped Promise.all did.

Change the key structure instead of optimising the scan. Rather than one field per hotel, write one field per supplier — the whole 2,000-entry slim array gzipped into roughly 100–200 KB. The full-hash scan disappears because there is nothing left to scan: you read the one field you want.

Redis went from ~1.2 GB to ~150–250 MB per search.

The CPU half: 100 million date parses

The same audit turned up a second problem. The function that merges cancellation policies was called once per offer, doing about 200 moment parses each time — on the order of 100 million moment operations for one supplier's job, and 70–80% of the worker's CPU.

The fix was not clever: compare numbers instead of moment objects. Parse each string once, cache it as milliseconds, and run every boundary, segment and gap comparison on numbers. moment stays only as the fallback for unusual input.

Measured: the merge function got 18× faster, the enrich function 30× faster, saving roughly 2–10 seconds of CPU on a large search-by-id request.

Proving it

This kind of refactor fails quietly — faster and wrong. So before changing anything I built 1,300+ parity test cases: same input, old output against new output. Then an end-to-end run against 762 real production logs captured from one wholesaler: 37 of 38 passed (97.4%).

One caveat about the numbers above. The 18× and 30× are measured benchmarks. The "100 million operations" and "600 MB – 1.6 GB of string churn" are order-of-magnitude estimates derived from reading the code, not measurements. I labelled them that way in the audit document itself, because an estimate that a later reader mistakes for a measurement is the fastest way to lose trust in the whole document.

What I took away

What takes a server down is rarely the algorithm — it's the code that refuses to yield. None of the functions above were wrong. They were synchronous, at a scale nobody had in mind when they were written. Node gives you one thread, and anything that holds it too long is an incident waiting for a busy day.

Changing the data structure beats optimising the loop. I could have spent the week tuning that Lua scan. Moving from one field per hotel to one field per supplier deleted the scan entirely. The first question is not "how do I make this faster" but "why is this work happening at all?"

max_memory_restart is a bandage, not a fix. The 3 GB ceiling hid the problem for months: the process died, came back, and the graphs looked fine. It was only when I worked out peak memory as hotels × offers × concurrency that it became obvious the number was never going to be enough.

A performance refactor without parity tests is a guess. The longest part of this work was not writing the code — it was building the 1,300 cases proving the new path returned exactly what the old one did. Faster and wrong is worse than slow.