Skip to main content

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
---
title: "Per-Hop Packet Loss: The Engineering Beauty of Low-Level Diagnostics"  
date: 2026-04-22
description: "Parsing raw mtr output, identifying true network bottlenecks, and proving that fundamental systems programming still holds immense value."
tags: ["packet-loss", "network-diagnostics", "mtr", "low-level", "python", "systems-engineering"]
categories: ["Development"]  
---

## April 22, 2026 – Engineering Log

> *Note: This diagnostic engine has evolved into the broader [NMPL](https://www.getnmpl.com) project.*

### Can a Minimal CLI Tool Uncover the Hidden Realities of a Network Path?

"There's no issue on our side." Every network engineer and home lab enthusiast eventually hits this wall with an ISP. You experience lag spikes, packet loss, or dropped SSH sessions, but standard high-level tools tell you everything is fine. 

Standard ping utilities give you aggregate statistics: *"5% packet loss."* But where is it dropping? At your router? At your ISP's edge? Or on an upstream transit provider's backbone? 

To solve this, I built a lightweight diagnostic parser in pure Python to sit directly on top of `mtr` (my traceroute). The goal was simple: turn unformatted terminal stream output into precise, per-hop mathematical evidence without relying on heavy third-party libraries or bloated dependencies.

---

### The Findings: What Raw Network Data Actually Teaches Us

Testing real-world network paths—such as a route from **Malmö, Sweden (Telenor ISP)** to Google's public DNS (`8.8.8.8`)—reveals fascinating realities about how the internet actually routes packets.

```bash
Hop  IP Address       Loss%   Avg Latency   Diagnosis
1    192.168.1.1      0.0%    0.8ms         Local Gateway (OK)
...
4    81.228.84.95     70.0%   8.2ms         Control-Plane ICMP Rate-Limit
...
12   8.8.8.8          0.0%    9.9ms         Final Destination (OK)

Finding 1: The “False Positive” Loss Illusion

The most critical finding from hop-by-hop analysis is that a high loss percentage on an intermediate hop does not mean the link is failing.

In our Malmö trace, Hop 4 (81.228.84.95) showed a massive 70.0% packet loss. To a naive parser or an untrained eye, Hop 4 looks like a broken router. But looking at Hop 12 (8.8.8.8), the loss was strictly 0.0%.

  • Why this happens: Core backbone routers prioritize forwarding payload data plane traffic. When an ICMP “Time Exceeded” probe hits the router’s control plane, the CPU intentionally drops or rate-limits those response packets.
  • The Rule: If loss does not propagate down to subsequent hops, it is an ICMP rate-limiting artifact, not a network bottleneck. True path failure propagates all the way to the destination.

Finding 2: Structured Evidence Moves Support Tickets

ISPs ignore raw screenshots or vague complaints. But when you hand a support team a structured JSON diagnostic or a CSV time-series log isolating an unpropagated 0% final loss vs. a true propagating 40% loss across specific IP coordinates, the conversation shifts immediately from script-reading to tier-2 escalation.


The Elegance of Low-Level Engineering

In an industry increasingly dominated by heavy runtimes, bloated container images, and multi-gigabyte dependency trees, there is genuine power in building small, sharp systems tools.

1
2
3
4
5
6
7
8
# Raw stream -> hop-by-hop isolation -> zero-dependency evidence engine
def parse_mtr_stream(raw_output: str) -> list[HopMetrics]:
    """
    Parses raw mtr stream streams across variable hop counts,
    handling partial packet lines and ICMP rate-limit edge cases.
    """
    # Deterministic string processing without external parser dependencies
    ...

1. Zero Runtime Dependencies

This tool relies strictly on the Python Standard Library. No pip install, no virtual environments, no third-party network wrappers, no C-extension compilation errors. You drop a single file onto a bare Linux server or macOS terminal, and it runs instantly.

2. Deterministic Performance & Low Footprint

At roughly 15KB, a lightweight diagnostic utility executes in milliseconds and consumes negligible RAM. When your network or system is undergoing severe degradation, your diagnostic tools must not compete with your infrastructure for CPU cycles or memory.

3. Mechanics Over Abstraction

Operating at this layer forces a direct relationship with OS and network primitives: handling POSIX subprocess streams, parsing raw ICMP/UDP socket behavior, managing terminal TTY states, and interpreting rate-limiting signatures. You aren’t merely consuming someone else’s abstracted API—you are observing physical state on the wire.


There Is Still Work to Be Done in the Fundamentals

It is tempting to believe that foundational infrastructure challenges are solved, or that modern software engineering has reduced to chaining high-level cloud APIs.

That assumption is wrong.

Critical, high-value work remains down in the plumbing:

  • Observability: Rendering complex underlying systems transparent without imposing runtime overhead.
  • Resilient Diagnostics: Building tools that gracefully handle edge cases, rate-limiting noise, and malformed stream outputs in production.
  • Efficiency: Replacing over-engineered monitoring frameworks with fast, lightweight tooling that operates identically on an AWS EC2 instance or a low-cost edge gateway in Malmö.

Mastering raw data streams, respecting system efficiency, and understanding first-principles networking will always be a competitive edge. Building lean, reliable tools isn’t an exercise in nostalgia—it is how resilient infrastructure gets built.