Rehearse the Network Change While Failure Is Still Cheap

Network engineer studying primary and backup paths in a virtual routing lab beside a rack of network equipment.
In this Field Note
  1. Write the question before drawing the topology
  2. Make one hypothesis easy to disprove
  3. Read the evidence before admiring the timer
  4. Use the right virtual device for the question
  5. Choose failures that challenge the design
  6. Put a boundary around the claim

The backup route is in the table. Small pings work. The large transfer does not. You have successfully demonstrated three different things, only one of which the customer wanted.

A network change is a poor time to discover that distinction. GNS3 gives it somewhere cheaper to happen: a lab where you can break the relevant path, keep the evidence, and put everything back without asking a paying customer to appreciate the educational value.

Write the question before drawing the topology

“Model the network” is an invitation to spend three weeks arranging icons. A useful brief is narrower: Will this subscriber service survive loss of the primary aggregation path, with isolation and management still working?

That question tells you what belongs in the model: both paths, their service handoffs, addressing, routing, policy, packet-size requirements, and an endpoint at each end. Include DHCP, DNS, RADIUS, or another dependency when its behavior can change the answer. Include the return path; packets do not become less interesting on their way home.

Write the simplifications beside the diagram. For an FTTH design, a Linux Ethernet endpoint may stand in for the OLT handoff. That is useful for testing VLAN transport. It does not simulate optical loss, PON scheduling, or the OLT’s provisioning system.

Make one hypothesis easy to disprove

Consider a synthetic service that crosses a preferred path and an alternate path. The hypothesis is that the alternate carries the same customer traffic. A rehearsal can proceed like this:

Stage Action Evidence
Baseline Send small probes, large packets, and a representative transaction. Working routes, policies, packet sizes, application result, and capture locations.
Fault Disable one lab link, or separately drop traffic while carrier remains up. A timestamped fault marker and the actual interface state.
Detection Watch the adjacency or configured health mechanism. When that mechanism reports failure; not when you expected it to.
Recovery Keep measuring at the service endpoints. First successful transaction, subsequent stability, and management access.
Restoration Return to the saved baseline. Normal path preference and all original checks passing again.

Now deliberately constrain the alternate path’s supported packet size in the lab. A small probe may recover while a larger transaction remains broken. That failure is valuable: it tells you the route change and the customer’s recovery are not the same event.

Record the actual platform behavior around Path MTU Discovery, encapsulation, and fragmentation. Do not infer a service MTU from one successful ping, and do not assume every device interprets its MTU setting the same way.

Read the evidence before admiring the timer

Suppose a synthetic test records the injected fault at 10.0 seconds, a routing alarm at 10.3, a route change at 10.7, and successful application samples at 9 and 16 seconds. Application samples at 10 through 15 failed.

You can say the alarm was recorded 0.3 seconds after the fault marker. You can say the first observed application success after the fault arrived six seconds later. You cannot honestly report a 0.3-second customer outage. Nor can one-second sampling identify the exact instant recovery occurred between the last failed sample and the successful one.

Clock synchronization and capture location matter. When events come from different hosts, record their time sources and known uncertainty. A neat spreadsheet can make misaligned clocks look remarkably authoritative.

Open the failure-evidence workshop. Compare route events with application observations, work through three evidence challenges, and change the sample interval and offset in a controlled experiment. Inspect six scenarios, open your own small CSV, and save the observations or report. Inputs stay in your browser; no probes are sent.

Download all three browser workshops for offline use. Extract the collection and open the index in a browser with JavaScript enabled. The exercises need no account, software installation or internet connection. The separate GNS3/CHR exercise still uses router software.

Use the exercise to separate detection, forwarding changes, and observed application recovery. Then adapt the evidence sheet to a real, authorized rehearsal. The sample measurements are invented teaching data, not results from an Oaktown client.

Use the right virtual device for the question

GNS3 can combine supported virtual network operating systems and test endpoints. If the decision depends on a vendor’s behavior, use that vendor’s appropriate virtual image under a valid license. A standards-based router is useful for a protocol lesson without proving how a production appliance implements the feature.

The GNS3 appliance guide distinguishes a template from the software image it runs. Record both versions, image provenance and checksum, relevant settings, and host resources. A marketplace template is not a license.

Keep the lab isolated from production. Imported configurations can contain real addresses, credentials, and routes. Use synthetic values and no production bridge unless the connection is specifically designed and approved. For shared or remote labs, protect the controller and consoles; GNS3’s remote-server guidance calls out the exposure risk. Check the supported procedure for your current release rather than copying historical installation commands.

Choose failures that challenge the design

A link-down test is a beginning. It does not exercise a path that remains electrically up while silently losing traffic. GNS3’s packet filters provide useful impairment options; confirm their direction and behavior in the version you use.

  • Silent failure: Does the chosen detection mechanism notice, and does the service actually move?
  • Service failure: What happens when DNS, address assignment, or authentication disappears while routing remains healthy?
  • Isolation failure: Can a customer reach management, or one tenant reach another? Retain evidence of the blocked path and the permitted path.
  • Management failure: Can you perform rollback using only the access that would survive the fault?
  • Return-to-normal failure: Does restoring the preferred path interrupt sessions again, create a loop, or leave a route in the wrong place?

Change one variable at a time and return to a known baseline. If you also upgrade software, alter timers, and change policy, a better result will be difficult to explain; and a worse one harder to undo.

Put a boundary around the claim

A virtual lab can establish observed routing and policy behavior for the versions and conditions tested. It cannot certify radio propagation, optical loss, PoE compatibility, ASIC forwarding capacity, or production timing on different hardware. A BFD experiment can demonstrate detection behavior without promising the same interruption under field conditions.

Carry those gaps into the hardware test or pilot. Keep captures and configuration exports private: they can reveal customer topology, secrets, or application data. Retain the raw evidence, not just screenshots of green indicators.

The handoff should be small enough to use: topology and versions, baseline configurations, fault-and-restoration steps, expected versus observed results, remaining gaps, and the revised implementation and rollback plan.

Oaktown Tech can build that rehearsal around a proposed change. Bring the decision, the service that must survive, and the failure you are worried about. A good lab ends with fewer assumptions and a better procedure. Ideally, the production evening then becomes considerably less interesting.

Share this article

LinkedInFacebookXEmail

Field discussion

What would you test next?

Ask a question, challenge an assumption, or share a lesson from the field. Keep customer names, live addresses, credentials, and sensitive network details out of public comments. Charles reviews every comment before it appears. See the privacy notice for how comment information is handled.

Join the discussion

Your comment will appear after Charles reviews it. Required fields are marked .

A pseudonym is fine.

Working through a similar problem?

Describe the network, the difficulty, and the outcome you need. Charles reviews the brief and helps establish the right scope.