In this Field Note
A backhaul fails. Forty devices disappear behind it. The monitoring system sends forty alerts, then forty recovery messages when the link returns. Somewhere in that eighty-message performance is the fact the operator actually needed.
More notifications have not created more understanding. They have assigned someone a sorting job at an inconvenient hour.
A useful monitoring system should help answer three questions quickly: what work is affected, what evidence explains it, and what should someone do next? Build the alert policy around those questions before polishing the dashboard.
Model how you reach the device
Start with one branch or remote site. Trace the monitoring path through its router, backhaul, switch, and any relevant firewall or power dependency. The dependency map should describe what can make the observation fail. Grouping devices by manufacturer does not do that.
Then write a few failure expectations. Here is a small hypothetical site with a primary backhaul, a secondary connection, and several downstream devices:
| Observed condition | Useful notification | Evidence to retain |
|---|---|---|
| Primary path fails; service works over backup | Site is on reduced-capacity backup; assess urgency against the agreed operating plan. | Failure time, route change, backup load, and service-test results. |
| Both transport paths fail | Site is unreachable through both monitored paths. | Downstream failed checks, clearly marked as consequences or unknown state. |
| One downstream switch fails; its upstream path works | The switch and the services it supports need attention. | Last power/temperature data, interface state, and recent changes. |
This is notification design, not proof of a root cause. “Unreachable through both paths” does not establish whether a site has lost power. Say what you observed and what you infer separately.
Tools can implement some of the grouping. LibreNMS device dependencies, for example, suppress alert delivery for a dependent device when all its configured parent devices are down. That “all” matters when a device has alternate paths. Check that the parents represent real reachability dependencies, not convenient diagram neighbors.
Suppression should quiet redundant messages while preserving state and history. When the upstream path recovers, a downstream device that remains broken still needs to become visible. Include that case in your test.
Give interruption a price
A notification is a request to spend somebody’s attention. Reserve an immediate interruption for a condition that needs action within the recipient’s response window. Put slower-moving work in a ticket or review queue.
A failed backup configuration job may need attention during the next working period. Loss of the only usable connection to an occupied site may need action now. A short utilization spike may need no message at all. The business impact, remaining protection, and available response determine the choice.
Use duration and separate trigger/recovery thresholds where they suit the signal. Suppose a noncritical capacity warning is intended to flag a sustained busy link. You might test a policy that opens after five minutes above an agreed threshold and clears only after a sustained return below a lower threshold. Those durations are examples; a service that cannot tolerate five minutes of loss needs a different policy.
Do not let a percentage make the decision alone. An interface averaging 60 percent over a long interval can still have short bursts that fill a queue. Conversely, high utilization during a planned file transfer may be perfectly acceptable if the important applications remain healthy. Correlate load, discards, delay, and the affected service.
Google’s monitoring chapter provides a useful distinction between observing a system from inside and testing its externally visible behavior. A switch can be reachable while the service passing through it is unusable. Both views earn their place.
Write the alert before the rule
Try composing the notification in plain English before building its expression:
North site is using its backup connection. The primary path failed at 14:32. The approved service check is passing over backup, but available capacity is reduced. Check the primary-path dashboard and recovery runbook. Owner: network operations. Escalate if the service check fails or backup demand exceeds the agreed limit.
This example forces several useful decisions. Which test proves that backup service works? Where is the runbook? Who receives the message? What changes its urgency? If those answers do not exist, the alert rule is not finished.
Include stable device/site identifiers and relevant links. Keep credentials, sensitive configurations, and unnecessary customer information out of notifications; email and chat have their own access and retention arrangements.
A named escalation contact should know they are named. Software can deliver an alarm at two in the morning. It cannot invent an on-call agreement.
Keep evidence from before the incident
Record enough history to distinguish a sudden event from deterioration: interface errors and discards, radio quality, device temperature, supply condition, and meaningful service timings. Check clock synchronization so events can be compared across devices.
Configuration history helps answer a different question: what changed? The LibreNMS–Oxidized integration can expose configuration history and differences alongside device information. Restrict access to those archives; they may contain operationally sensitive information.
Monitor backup freshness as well as device availability. A successful poll does not establish that the latest configuration was captured, and a captured configuration still needs a tested restoration method.
For application checks, choose an approved, non-destructive transaction. A safe read-only request or dedicated test endpoint may be appropriate. Do not create orders, move money, or send control commands merely to keep an indicator green.
Test the alerting system, too
Start with one service and rehearse its agreed failure cases in a lab or coordinated maintenance window. Verify the initial message, suppression behavior, escalation, recovery message, and reappearance of a still-broken dependent device. Check that scheduled maintenance ends when intended.
Test delivery independently of the network being monitored. If the monitoring host loses internet access along with the site, its excellent diagnosis may remain a private thought. An appropriate independent observer or delivery-health check can reveal that blind spot.
Review a sample of recent alerts with the people who received them. Which led to an action? Which were duplicates? Which lacked enough context? Retire rules that have no useful response, and investigate the gaps that prevented a response.
Try the interactive alert-dependency lab
Open the alert-dependency workshop. Work through three guided incident challenges, see why each observation can or cannot join an upstream group, and test what happens when a parent recovers or a backup path answers. Independent alarms and every original event remain visible. Save a learning note or the complete result when you finish.
Every example is synthetic. Change one observation at a time, predict the result, then check the groups and retained events. An unknown parent is not a healthy parent; a recovered parent does not magically repair its children. This is a small teaching model, not a monitoring product or proof of root cause.
Download all three browser workshops for offline use. Extract the collection and open the index in a browser with JavaScript enabled. The exercises need no account, software installation or internet connection. The separate GNS3/CHR exercise still uses router software.
Oaktown Tech’s monitoring work can help connect those decisions to the network’s actual dependencies. You can also begin with one noisy site and the table above. A quieter phone is welcome; a clearer account of what needs attention is the real improvement.

Join the discussion