An alerting library is a data-egress boundary

Where it started

Every service had its own version of the same forty lines: catch an error, build a message, POST it to a chat webhook. Same idea, twelve slightly different implementations. Different field names, different formatting, different behaviour when the webhook itself failed — and one of those twelve would take down the request it was supposed to be reporting on.

So I pulled it out into a package and published it: @kien2k1/multi-channel-alert. Fifteen or so destinations — Telegram, Slack, Discord, Mattermost, email, and the rest — behind one interface, plus a MultiChannelAlert that broadcasts to several at once.

The channel count is the least interesting thing about it. Here is what actually took thought.

1. The security realisation, which I had late

The natural way to write an alert call is:

alert.error(err)     // or worse: alert.error({ ...context, err })

Now think about what is actually in that object on a bad day. The upstream request body. Headers. A bearer token. An account's credentials, because the request you were assembling had to carry them. A customer's personal details.

An alerting library is not a logging concern. It is an egress path — a sanctioned, always-on channel that copies data out of your process into a chat room that has more members than your production database has users. The library decides what goes through it, so the library needs an opinion.

So the shape it landed on is an allow-list:

new TelegramAlert({
  strictMode: true,               // nothing leaves unless it is named below
  specific: [
    { key: 'trace_id',  title: 'Trace' },
    { key: 'error_code', title: 'Code' },
    { key: 'message',   title: 'Message' },
  ],
})

With strictMode on, fields not in specific are dropped before the message is built — not redacted at render time, dropped from the object. It can be set per channel too, which matters: the same incident can go to an internal engineering topic in full and to a wider room stripped down.

If I were designing it again the one thing I would change is the default. strictMode defaults to false — send everything — because that was the behaviour of the code I was replacing and I did not want to break call sites on upgrade. That is a real reason and I still think it was the wrong call. The safe mode should be the one you get by not thinking about it.

2. Fail-soft is not a nice-to-have

An alert that throws turns one incident into two, and the second one is your fault. So the failure behaviour is settled in two places.

In the library: failSilently (default true) — if three of five channels are down, you get the delivery report, not an exception.

And again at every call site, because a library default is not a guarantee:

class TelegramProvider {
  constructor({ botToken, chatId }) {
    if (!botToken || !chatId) {
      this.client = null
      console.warn('⚠️ TelegramProvider disabled: botToken/chatId missing')
      return                       // missing config disables alerting, never boot
    }
    this.client = new TelegramClient({ botToken, chatId, timeout: 10000 })
  }
 
  async send(options) {
    if (!this.client) return
    try {
      await this.client.sendMessage(options)
    } catch (error) {
      console.error('#telegram send failed', error)   // swallowed on purpose
    }
  }
}

Two decisions in there worth naming. Missing credentials disable the alerting, they do not stop the service booting — otherwise a rotated token takes down production. And the timeout, because a chat provider having a slow morning must not become your latency.

3. Zero dependencies, deliberately

The package has an empty dependencies block. It is installed in a dozen services; anything it pulls in, all twelve carry, forever, including into their audit reports.

The only thing it really needed was an HTTP client, so it has one — about 180 lines over node:http/node:https, exposing get/post/put/patch/delete with an axios-shaped response ({ data, status, headers }) so the call sites read exactly as they would have. Timeouts destroy the socket, non-2xx rejects with the response attached, JSON parses opportunistically and falls back to text.

That is not a hard piece of code, and writing it is worth it when the alternative is adding a transitive dependency tree to twelve production services to save an afternoon.

4. The bug that makes the alert about the failure fail

Telegram's legacy Markdown parser rejects the whole message if *, _, ` or [ appear unescaped. Which means a stack trace — the exact payload you most want to send — gets you a 400 instead of a notification. You find out about the outage some other way, and you never find out that your alerting was silently broken.

The fix is four lines of escaping. The lesson is that the delivery path needs its own tests with hostile input, because it is the one code path that by definition runs when everything else is already going wrong.

5. Routing, so alerts stay readable

Alerts all landing in one room is the same as no alerts. Each provider maps service → kind → operation onto a per-product message thread, so a search failure and a booking failure do not interleave in the same feed. Cheap to build, and it is the difference between a channel people read and a channel people mute.

The mistake I left in public

Writing this up, I re-read the package's own README and found two screenshots of real alerts in the "Live Demo" section — captured from an actual deployment, with the real bot name and a real payload visible. A third had already been commented out for being stale, which tells you nobody removed them on purpose.

Nothing catastrophic. Also not nothing: it is internal service naming and a live payload shape, published to npm, indexed, and mirrored by every package proxy on the internet.

A screenshot of production is production data. I would not paste that payload into a public issue, and pasting it into a README is the same act with better lighting. Fixing it is on my list; I am leaving it written down here because a lesson I only kept for the flattering parts would not be worth publishing.

What I took away

Anything that sends data out of your process is a boundary, so give it a policy. Logging, alerting, error reporting, analytics — they all feel like plumbing and they are all egress. An allow-list of fields is a two-line change that closes the whole class.

Defaults are the design. Nobody reads the options table. The behaviour you get by not configuring anything is the behaviour you have shipped, so the safe option is the one that should be free.

Observability must never be able to break what it observes. Fail-soft, timeout everything, and let missing configuration degrade the feature rather than the service.

Zero dependencies is a feature when the thing is installed everywhere. Cost of writing an HTTP client: one afternoon. Cost of a transitive dependency across a dozen production services: permanent.