Automated VPN Egress Validation — Streaming Reachability, Leak Detection & Performance Grading at Fleet Scale

Continuous Control Validation for a Global VPN Fleet — Every server, every night: connect, verify no leak, measure performance, probe seven streaming platforms, grade the node, and email a colour-coded report. Fully isolated per-server testing via Linux network namespaces.

Project Overview

Project Summary

A VPN fleet is a set of security controls, and like every security control it degrades silently. A node that was leaking nothing last month starts leaking after a config change. A node that reached a streaming platform last week gets its egress range blocklisted. A node that delivered 200 Mbps now delivers 12. None of that shows up in an uptime check — the tunnel is up, the ping responds, and the control is quietly broken.

I built an automated validation harness that treats every server in the fleet as a control to be continuously verified, not assumed. Each night it connects to every node in the inventory, and for each one: verifies the tunnel actually carries the traffic (no IP leak), measures latency, jitter, packet loss and throughput, computes a weighted quality score, classifies the node’s fitness for gaming / streaming / browsing, probes seven streaming platforms for reachability and region correctness, runs a three-method DNS/IP leak test, and writes the result into a colour-coded HTML report emailed to the team with a full text report alongside it.

The architectural decision that makes the whole thing possible is per-server network namespace isolation — each tunnel is established inside its own namespace with its own routing table and its own resolver, so N servers can be validated in sequence on one host without a single change to the host’s own routing.

Confidentiality note: The implementation is proprietary to Kolpolok Limited and is not published. This write-up covers architecture, detection methodology, and engineering decisions. Code blocks are logic descriptions, not source. Platform-specific detection signatures are deliberately omitted.

The Problem

Before automation, verification meant an engineer connecting to servers manually, opening streaming sites in a browser, eyeballing a speed test, and reporting anecdotally. That does not scale past a handful of nodes, is not repeatable, and produces no history — so degradation trends were invisible.

The Core Architectural Decision — Network Namespace Isolation

Establishing a VPN tunnel normally rewrites the host’s default route. That makes testing multiple servers a serial nightmare: connect, test, tear down, restore host routing, verify you actually restored it, repeat. One failed teardown leaves the monitoring host itself routed through a VPN.

Instead, each server under test gets a dedicated Linux network namespace:

• a fresh namespace is created per server, named from a sanitised server identifier
• a veth pair connects the namespace to the host, with a dedicated private subnet
• the namespace gets its own default route pointing at the host end of the pair
• an iptables NAT masquerade rule provides upstream connectivity for that subnet
• a namespace-scoped resolver configuration is written, so DNS inside the namespace is independent of the host’s — which is precisely what makes DNS leak testing meaningful
• the VPN client is launched inside the namespace, and every probe — ping, throughput, streaming, leak checks — executes inside it
• on completion, all namespace processes are killed, the namespace is deleted, the veth pair removed, and the NAT rule withdrawn

The host’s routing table is never touched. Ever.

        PER-SERVER ISOLATION MODEL

  ┌──────────────────── MONITORING HOST ────────────────────┐
  │                                                         │
  │  host routing table  ── UNTOUCHED ──▸ internet          │
  │                                                         │
  │   ┌───────────────── NAMESPACE: ns-<server> ──────────┐ │
  │   │  veth1 ──┐                                        │ │
  │   │          │  default route → host veth0            │ │
  │   │  own resolv.conf (isolated DNS)                   │ │
  │   │          │                                        │ │
  │   │  VPN client ──▸ tunnel iface ──▸ remote node      │ │
  │   │          │                                        │ │
  │   │  ALL PROBES RUN HERE:                             │ │
  │   │   · egress IP verification    · throughput        │ │
  │   │   · latency / jitter / loss   · 7× streaming      │ │
  │   │   · 3-method leak test                            │ │
  │   └────────────┬──────────────────────────────────────┘ │
  │                │ veth pair + NAT masquerade             │
  │   teardown: kill procs → delete ns → remove veth → drop NAT
  └─────────────────────────────────────────────────────────┘

Validation Pipeline — Per Server

# LOGIC ONLY — implementation withheld (proprietary, Kolpolok Limited)

detect host egress IP and geolocation      # baseline for leak comparison

FOR each server in inventory CSV:
    create isolated namespace + veth + NAT + scoped resolver

    # 1 — trust establishment
    server_cert_pin ← probe_server_certificate_pin(address)
    IF no pin: RECORD failure(no-pin); teardown; CONTINUE

    # 2 — connect, pinned
    launch VPN client INSIDE namespace, pinned to server_cert_pin
    WAIT for tunnel interface, bounded timeout
    IF timeout: RECORD failure(timeout); teardown; CONTINUE

    # 3 — the check that matters most
    egress_ip ← query_public_ip_from_inside_namespace()
    IF egress_ip == host_egress_ip OR invalid:
        RECORD failure(IP LEAK); teardown; CONTINUE

    # 4 — where does the world think we are
    geo ← resolve_geolocation(egress_ip)     # 3-provider chain + cache
    country ← extract_country_code(geo)

    # 5 — performance
    latency, jitter, loss ← ping_metrics(samples=10)
    throughput            ← timed_bulk_download()
    score                 ← weighted(throughput, latency, jitter)
    fitness               ← classify(gaming | streaming | browsing)

    # 6 — reachability, region-aware
    FOR each of 7 streaming platforms:
        result ← probe(platform, country)    # WORKING | BLOCKED | REGION_MISMATCH | UNREACHABLE

    # 7 — leak verification, 3 independent methods
    leak ← leak_test(namespace, egress_ip, host_egress_ip)

    RECORD row; teardown namespace completely

generate text report + colour-coded HTML report
email both with per-server logs retained

Leak Detection — Three Independent Methods

A single leak-test provider is a single point of both failure and false positive. The leak test uses a three-method cascade, each independently sufficient, with the earliest conclusive answer winning:

Method 1 — Edge-network trace. A request from inside the namespace to a major CDN’s diagnostic endpoint returns the source address that CDN’s edge observed. If that address equals the monitoring host’s own egress IP, traffic bypassed the tunnel — confirmed leak. Any other valid address means the traffic was tunnelled. This is the primary method because it requires nothing but an HTTP request and reports ground truth from outside the host entirely.

Method 2 — Dedicated leak-test API. A purpose-built leak-detection service queried from inside the namespace, with the same host-IP comparison.

Method 3 — Dual independent echo cross-check. Two unrelated IP-echo services queried from inside the namespace. If either returns the host’s egress IP, the leak is confirmed. Using two independent services eliminates the single-provider false positive — a service being down is not the same as a leak.

If all three methods fail to return any address, the result is recorded as UNKNOWN rather than PASS. A test that could not run is not a test that passed — silently grading an untested control as healthy is how monitoring systems lie to you.

Streaming Reachability — Four States, Not Two

Naive checking asks “did the page load?” That question produces false confidence, because a blocked user still gets a page — just the wrong one. Detection therefore classifies into four states:

Each platform probe combines a header/redirect-chain inspection with a bounded body inspection, and evaluates several independent conditions rather than one string match — because platform anti-VPN responses vary by region, change without notice, and are frequently not an HTTP error code.

Two engineering details that made detection reliable:

• Region-aware probing. The node’s resolved country code drives a country-specific request path, so the probe asks the platform what a user in that country would see. Locale-style codes that look like country codes but are not have to be handled explicitly, or every node in certain regions reports a false mismatch.
• Byte-bounded body reads with binary sanitisation. Body inspection is range-limited to the first few kilobytes — enough for the signals, a fraction of the bandwidth and time — and null bytes are stripped so that compressed or binary responses cannot corrupt downstream text processing.

Performance Grading

Raw numbers are not a decision. The harness converts measurements into a composite quality score and a fitness classification, so the report answers “what is this node good for?” rather than “here are four numbers”.

Composite quality score — a weighted normalisation with throughput dominant, latency secondary, jitter a tiebreaker (60 / 30 / 10). Every component is clamped so a single pathological measurement cannot produce a nonsense score.

Fitness classification — the node is labelled by what it can actually deliver:

• Gaming — latency and jitter both inside real-time thresholds
• Streaming — throughput above the sustained-HD threshold
• Browsing — functional, but not for either of the above

Performance bands (HIGH / GOOD / MOD / LOW / FAIL) drive the colour coding in the report, so a 40-server report is readable in five seconds instead of five minutes.

Geolocation — Three-Provider Consensus with Caching

Geolocation providers disagree, rate-limit, and return malformed responses. Since the country code drives region-aware streaming probes, a wrong country produces a cascade of false mismatches.

The resolver therefore uses:

• a manual CIDR override map — checked first, authoritative for ranges whose true location is known and which providers get wrong
• a three-provider fallback chain — primary, secondary, tertiary, each parsed defensively with empty and null handling
• an in-memory cache keyed by IP — the same address is never queried twice in a run, which matters when providers rate-limit
• empty-input guarding — an unset IP must never reach the cache lookup, or the run dies on an array subscript error at 3 a.m.

Reporting

Every run produces two artefacts and one email:

• a plain-text report — greppable, diffable, archivable
• a colour-coded HTML report — status-tinted rows, per-platform result cells, performance-banded throughput values, and run summary statistics
• an email with the HTML rendered inline in the body (not as an attachment nobody opens), built as a correctly structured multipart MIME message and delivered over authenticated TLS SMTP to a recipient list

Per-server connection logs are written to a dedicated log directory, one file per server, overwritten each run — the current state is always one cat away, without a timestamped-file graveyard to clean up.

Testing & Verification

• Leak detection validated by deliberately inducing a bypass and confirming a LEAK verdict from each of the three methods independently
• Namespace teardown verified by asserting the host routing table, veth inventory, and NAT rule set are identical before and after a full run
• Certificate-pinning path verified against a node presenting an unexpected certificate — connection correctly refused
• Streaming classification validated against nodes known to be blocked, known to be working, and known to be region-mismatched, on each of the seven platforms
• Geolocation resolver tested with each provider individually disabled, confirming clean fallback and correct cache behaviour
• Timeout paths exercised: unreachable node, node accepting connections but never establishing a tunnel, node establishing a tunnel with no upstream
• Reporting validated for correct MIME structure and inline HTML rendering across mail clients

Outcomes

Honest Limitations & Next Iteration

• Streaming detection is inherently brittle. Platform anti-VPN responses change without notice, which is why the harness went through four versions and why detection uses multiple independent conditions per platform rather than one signature. A signature registry decoupled from code is the right next step, so a platform change becomes a config edit rather than a release.
• Serial execution. Nodes are validated one at a time. Namespace isolation makes parallel execution architecturally safe already — the remaining work is bounded concurrency and per-namespace subnet allocation, which would cut a fleet run substantially.
• No trend storage. Results are per-run artefacts. Writing each run to a time-series store would turn point-in-time verification into degradation alerting — flag the node that dropped 60% week over week, not just the node that is failing today.
• Threshold tuning is static. Fitness thresholds are fixed constants. Per-region baselines would be fairer, since acceptable latency to a node three continents away is legitimately different from one next door.