Monitoring that does not nag
Thirty alerts a day and not a single one read — that is what my monitoring looked like after two weeks. Here is what two incidents changed, how I separated measurement from notification, and why fewer alerts end up meaning more safety.
My first attempt at monitoring was a success on paper: everything that mattered to me reported in. After two weeks I had stopped reading the alerts. That is exactly where monitoring stops working — not when a value is missing, but when too many values are shouting. I have built it the other way round ever since.
The first alert was the start of the problem
I began the way many people do: one service after another brought into view, each with its own threshold. The network hiccups — an alert. A device stops answering — an alert. Storage grows — an alert. It worked. First twenty alerts a day, then thirty, and at some point I was only swiping them away without reading a single one.
That is not a character flaw, it is arithmetic. Anyone who gets thirty alerts a day and finds twenty-eight of them inconsequential learns within two weeks that alerts are inconsequential. Your eye gets faster, your attention shorter — and the two real cases disappear in the very same motion.
The night nobody noticed anything
One case I do not forget. A service failed shortly after three in the morning, and the alert arrived neatly and on time. It just sat in a mailbox I do not open at night, and the second route it was meant to take as well had been muted months earlier because it was too loud. When I sat down at half past seven, it was one entry among six others that I cleared with a swipe. I only found out about the outage when someone asked why something was not working.
The outage itself was not the problem. It lasted a few minutes, and the recovery was a single gesture. The problem was that I had built an alert which reaches nobody at the one moment it matters — and that a pattern like this can stay unnoticed for months.
The second case: the value that never stood out
Different, but just as instructive: one value was unremarkable for months. I had set a threshold well above what was normal — generously, so that it would not beep all the time. That put it so high that it only triggers once it is too late. The value crept up slowly, week after week, always below the line. When it finally crossed, the problem was no longer small: storage was full, and I had to clean up properly instead of looking into a minor detail.
Both cases have the same root. I had asked myself how I would notice a fault — and not who can act when it matters and when I want to know about it at all.
What I monitor — and what I do not
The change was mostly a decision about what is not monitored. Today only what has real consequences stays in view: whether services answer, whether storage runs short, whether unknown devices appear on the network, whether the backups have run. Everything else may be a chart, but not an alert.
- An alert needs a reason to arrive now — not eventually.
- It has to say what to do; otherwise it is a question, not a signal.
- Anything nobody would read at night should not alert at night.
- A value that only grows slowly needs no threshold but a trend with a look ahead.
The readings still come from every direction: AdGuard Home for name resolution, Authelia for sign-ins, NetAlertX for the devices on the network, Proxmox VE for the machines, TrueNAS for storage, plus Home Assistant and a few Shelly plugs. They are judged in one place, not in six.
The filter sits between measurement and message
Between the measurement and the message belongs a stage that summarises. It knows the past few days, works out whether something is new, and stays quiet when nothing new has happened. The same knowledge checks whether a pattern is really there: a single dropout is not an outage, and anything back five minutes later was not one either. Only what persists becomes an alert.
What matters is that this stage does not make anything disappear. Everything is still written down; the stage only decides what gets woken up. If a minor detail turns out to matter later, I can look it up — I have the history, not just the last alarm.
The second part is delivery. There is exactly one route that genuinely gets read at night, and it is reserved for cases where someone has to act. Everything else arrives with the morning report: collected, with time and frequency, in one place. Since that separation exists, the report is useful again, because it contains only what actually happened.
Two rules that have proven themselves
The first rule: every alert must be switchable off without losing anything. If that is not possible, the alert is cut the wrong way. The second: whoever adds an alert also provides for the opposite case — a test that has really triggered it once. An alert that has never fired is an assumption.
What has become better
The difference is not that less happens. Roughly the same amount happens. The difference is that I read what arrives again, and that a value growing slowly is noticed in time. When something comes in at night, for the past few months there has genuinely been something every time.
What I take from it
Monitoring is not a field of view that covers everything, but an agreement with myself: few alerts, each with a reason, each with a next step — and a calm report for everything in between. Whoever reports everything ends up reporting nothing.