Skip to content
Kien

One feature, eleven repos, and two production incidents

When a feature spans eleven repositories, the unit of release is the set, not the repo — attaching a flight-delay insurance product to a live booking flow, where missing a single repository means undercharging customers with no exception thrown anywhere.

Period
2026
Role
Feature owner end-to-end — backend, consumer web, admin, accounting, mail

Stack

  • NestJS
  • Express 4
  • Next.js
  • React
  • PostgreSQL
  • TypeORM
  • Sequelize
  • Kafka
  • Redis
  • Bull
  • GitLab CI

The problem

A flight-delay insurance product had to attach to a live booking flow: opt-in on the consumer site, summed into the order total, a policy created with the underwriter, visible in the admin panel, reflected in accounting and in mail.

Missing a single repository means the system undercharges customers and throws no error at all. So the unit of release here is the set of repos, not any one repo — and the go-live had to be run that way.

The approach

The go-live ran as an auditable cherry-pick campaign: a release branch per repo, a per-repo note listing every commit picked — SHA, source MR, conflicts and how each was resolved, commits deliberately skipped — plus a summary table, with the MRs the original hand-written note had missed marked in bold.

Incident 1 — undercharging. Four days after go-live, production was charging less than the UI showed. I traced it to one repo that had never been on the list, proved it by diffing the same total-price expression on master versus test and comparing two order rows in the database — and ruled out the "production is running an old image" theory using deploy timestamps 17 hours ahead of the affected order.

Incident 2 — silent death. The entire feature stopped: no policy created, cancellation broken, the admin panel reporting not-authenticated — and zero errors anywhere. The cause: a shared service token written into config without quotes, so its special characters were stripped. I proved it with a matrix of live curls showing the stripped token authenticating against one service and the intact token against the other — exactly inverted. The caller swallowed the 401 in a catch and returned null, so three separate features had been failing quietly.

The result

11 repos, 69 commits cherry-picked, 18 conflicts, 45 MRs covered.

The fix shipped as three MRs and was verified on production: three endpoints went 401 → 200, and the service's permission rows went 0 → 23.

I also caught that the fix branch was missing from the test branch, meaning the next test-to-master merge would have silently deleted the production fix, and shipped a fourth MR to resync it. I then swept all 36 repos in the workspace for the same pattern, documented that every other hit was a false positive, and turned the principle into a reusable multi-repo go-live checker — a tool that treats the set, not the repo, as the thing being released.