Write the Network Runbook for the Next Person on Call

Network diagrams, laptop, fiber patch leads, and switches arranged for infrastructure documentation
In this Field Note
  1. Give the reader a front door
  2. Write down why the odd thing exists
  3. A recovery page you can adapt
  4. A configuration archive needs an owner
  5. Test the copy you will use during an outage
  6. Let someone else find the gaps

The network manual has a cover page, a revision number, and a diagram large enough to require its own printer. What it does not have is the location of the spare firewall.

During a handoff meeting, that omission is easy to overlook. During an outage, it becomes the plot.

Documentation should let another competent person understand the design, make a safe change, and recover a failed component. That person may be a new employee. It may also be you in six months, wondering why VLAN 417 seemed so memorable at the time.

Give the reader a front door

Most people do not begin an outage by reading a manual from page one. They have a symptom and need somewhere useful to start.

Put a short operational index at the front: important services, the sites they depend on, the relevant recovery procedures, and the person responsible. Link an alert’s device name to the same name in the inventory, configuration archive, and rack label. That chain should not require translation between “Main Router,” “GW02,” and “the little gray one.”

Keep design reference material separate from recovery instructions. A routing diagram explains the intended paths. A recovery procedure explains how to replace the device, which path should return first, and what would make you stop. They support each other, but they are used at different speeds.

Write down why the odd thing exists

The configuration tells you that a route has a particular preference. It may not tell you that the alternative circuit has a usage charge, or that one application fails over badly and requires a planned transition.

For a meaningful exception, capture five things: the reason, the affected service, the owner, the evidence, and the condition for removing it. For example: “Vendor maintenance rule; permits the approved support host to the application gateway during scheduled work; reviewed with operations; remove when replacement support method is accepted.” Use the real identifiers in the controlled document.

A review date helps, but a date without an owner is decorative. Tie the review to a person or work item that can actually resolve it.

Label uncertainty honestly. “Carrier diversity claimed; separate building entrances not yet confirmed” is more useful than drawing two beautifully independent lines. Documentation must not quietly promote a sales description into a verified engineering fact.

A recovery page you can adapt

The following is a specimen for a hypothetical branch firewall. It deliberately contains no commands or credentials: the equipment-specific steps belong in a tested procedure for the installed platform.

Section What the finished page should say
Purpose and impact Restore the branch gateway. Identify affected services and anything that should keep operating locally.
Authority and stop conditions Name who approves replacement. Stop for evidence of compromise, uncertain cabling, an unsafe physical condition, or a mismatch with the tested spare.
Access State the console method, required adapter, emergency access process, and credential-vault reference.
Replacement set Identify the spare, compatible software, restore files, licenses, certificates, optics, and power supply.
Before connecting Isolate the spare, verify its identity and software, apply the approved recovery procedure, and confirm interface mapping.
Cutover Identify labeled connections and the approved order. Prevent duplicate active addressing or unintended simultaneous operation.
Acceptance Verify management access, the agreed business transaction, required VPNs, and intended security boundaries.
If acceptance fails State the rollback decision time, supported recovery steps, and escalation contact.
Evidence and owner Record who rehearsed the procedure, on which equipment/software, on what date, and with what limitations.

The important part is the specificity behind each cell. “Restore the backup” is not a restore sequence. “Check the internet” is not an acceptance test for a gateway that also carries phones, a site-to-site VPN, and a specialist application.

Give the reader observable checkpoints. After a step, what should the screen show? How long does startup normally take in the tested configuration? Which result means proceed, and which means stop? That keeps uncertainty from turning into improvised changes.

A configuration archive needs an owner

Automated collection is valuable. Oxidized, for example, supports configuration retrieval, version history, and differences. Those features help you find what changed. They do not decide which revision is a known-good restore candidate or prove that a spare can accept it.

Record the relationship between device, saved configuration, software, and test result. Protect the archive as sensitive material: network configurations can contain credentials, keys, addressing, and information about security boundaries.

Check the platform’s backup semantics. MikroTik’s backup documentation includes restoration and encryption considerations. Other platforms have their own dependencies. A procedure copied between products without checking those differences can look reassuring right up to the restore.

Keep passwords in the approved credential store rather than embedding them in a broadly shared manual. Document how authorized staff reach that store during an identity-service or network outage. “The password is in the vault” is incomplete if opening the vault requires the failed gateway.

Test the copy you will use during an outage

Provide a controlled recovery copy outside the failure domain it describes. That might be an appropriately protected offline package or another approved access route. Decide who can hold it, how it is refreshed, and how obsolete copies are retired.

Test the actual access arrangement, including authentication, encryption keys, and any required application. A document downloaded to a laptop is not necessarily usable if its attachments remain links to the unreachable server.

This is a practical confidentiality problem as well as an availability problem. Avoid distributing complete network maps and access information merely to make them convenient. Give the recovery team what it needs through an arrangement the organization can control.

Let someone else find the gaps

Choose a representative failure and ask a competent colleague who did not write the procedure to walk through it in a safe test environment. The author should observe rather than narrate the missing steps.

Record where the reader hesitates, which reference is inaccessible, which label does not match, and what has to be supplied from memory. Those are documentation defects, not shortcomings in the reader. Fix them and repeat the difficult portion.

Afterward, make updating the record part of completing a network change. A new carrier needs revised circuit details; a firewall replacement needs revised recovery evidence. Sampling a few critical records periodically is more revealing than admiring the folder structure.

If the documentation is extensive but recovery still depends on one person’s memory, an assessment and handoff review can close that gap. A useful first step you can take yourself is smaller: pick one essential device and give its recovery page to somebody else. The questions they ask in daylight are the ones you want answered before the phone rings.

Share this article

LinkedInFacebookXEmail

Field discussion

What would you test next?

Ask a question, challenge an assumption, or share a lesson from the field. Keep customer names, live addresses, credentials, and sensitive network details out of public comments. Charles reviews every comment before it appears. See the privacy notice for how comment information is handled.

Join the discussion

Your comment will appear after Charles reviews it. Required fields are marked .

A pseudonym is fine.

Working through a similar problem?

Describe the network, the difficulty, and the outcome you need. Charles reviews the brief and helps establish the right scope.