A client that cannot connect
The standard mysql client cannot reach the gateway database. Not slow, not
intermittent — it cannot open a connection at all, for reasons that live in how
the network is arranged rather than in anything a client flag can fix.
So for a while, every query was a small ceremony. One person had a GUI that happened to work. One kept a connection snippet in a scratch file. One would paste SQL into chat and ask whoever's setup was alive that day to run it. Everyone got their answer eventually, and every answer travelled a slightly different path. When the way in is inconvenient, people do not stop querying — they improvise, and improvisation has no standard behaviour you can reason about.
One script instead of five workarounds
The replacement was deliberately boring: a small script that takes SQL and returns rows. It became the standard tool not by decree but by being easier than every workaround it replaced — type the query, get the result, done.
The part worth writing about is not the connection handling. It is what got built into the script rather than left to discipline.
SELECT runs immediately. No prompt, no flag, no ritual. Reads are the
overwhelming majority of what anyone wants from this database, and the tool
treats them that way.
A write does not run. The script recognises it, stops, and prints two things to the human before anything happens: the hostname it is about to write to, and the server's read_only flag. Then it asks for explicit confirmation. Only after a yes does the statement execute.
Why those two lines exist
Because a config on a developer machine can point at production, and nothing
about the terminal tells you so. A committed config file, a copied .env, a
shell export left over from last week's incident — any of them can quietly aim
your session at the wrong database, and with a plain client the first hint is
the damage. A read against the wrong host is embarrassing. A write against the
wrong host is the story every team ends up telling in the past tense.
The hostname line answers where am I actually connected. The read_only line
answers is this the primary — which also catches the mistake in the other
direction, writing into a replica and wondering why nothing sticks. Neither
check is clever. Both are exactly the checks a careful person would run by
hand, every single time, if careful people ran checks every single time.
Guardrails in tools versus guardrails in people
They do not, and that is the whole lesson. The old defence was discipline — always check which host you are on before you write — and discipline is a guardrail that lives in a human. It holds on a calm morning and fails late in an incident, when the person typing the UPDATE is the same tired person who has been staring at logs for hours. The script does not care what kind of day it is. It prints the hostname to the tired person too.
I should be honest about the evidence. There is no count of averted mistakes — nothing logs the moments someone read the hostname line, went pale, and pressed no. I cannot tell you the check has saved us N times, because N was never measured and I will not invent it. The reason to believe in it is the design, not a statistic: the check runs on every write, the human sees the two facts that matter before anything is committed, and the failure it targets is one that every team with more than one database eventually meets.
The rule
Make the safe path the easy path. The script wins because SELECT — the
common case — carries less friction than any of the workarounds did, so
people reach for it without being told; the friction lands only on the rare,
dangerous case, sized to what that case deserves. A guardrail that lives in the
tool survives a bad day. A guardrail that lives in a human does not — and the
bad day is precisely the one it was for.