IT Security Workflow Automation — Turning Manual Fleet Checks Into Unattended Reporting Pipelines

Automation as a Security Control — A family of unattended monitoring pipelines covering DNS resolution integrity, in-memory datastore health, and service-state verification across a 170+ server multi-region fleet — each delivering a colour-coded report to the security inbox with zero human intervention.

Project Overview

Project Summary

Every one of these pipelines started the same way: an engineer doing something repetitive, at the same time every day, and occasionally forgetting.

Manual checks are not a monitoring strategy. They are a monitoring strategy that fails silently the first week someone takes leave. Across a 170+ server multi-region fleet, I replaced recurring manual verification with a family of unattended automation pipelines built on a shared architectural pattern: inventory-driven, self-healing, fail-safe, and reporting into a single artefact per run.

This write-up covers two representative pipelines in detail — DNS resolution integrity monitoring and datastore health & growth tracking — plus the reusable engineering patterns that every pipeline in the family shares. Together they answer questions that uptime monitoring cannot: is this resolver returning the right answer, is this service actually running, and did this datastore stop growing?

Confidentiality note: These are internal operational tools owned by Kolpolok Limited and are not published. This write-up covers architecture, methodology, and engineering patterns. Code blocks are logic descriptions, not source. Hostnames, addresses, and target domains are omitted.

Pipeline 1 — DNS Resolution Integrity & Service-State Monitoring

The problem

A DNS server that is up and a DNS server that is correct are two different things, and only one of them matters. A resolver can respond to every query, pass every port check, and still be returning the wrong answer — because a zone was edited, a service silently died and something else answered, an upstream changed, or the resolution path was tampered with. Meanwhile a companion SNI proxy service can be dead while the resolver answers perfectly, breaking the delivery path with no DNS symptom at all.

Port-open monitoring cannot detect any of this. Answer-correctness monitoring can.

The approach — validate the answer, not the port

For every server in a CSV inventory, the pipeline:

1. Resolves geolocation for the server’s address via a three-provider consensus vote — used to confirm the node is actually where the inventory claims it is
2. Verifies SSH reachability with a bounded, non-interactive, key-based connection before attempting anything else
3. Ensures the DNS query tool exists — and if it does not, installs it (see self-healing, below)
4. Executes a DNS query against the server’s own local resolver for a known target domain, and compares the returned address against the expected answer for that specific node
5. Verifies the companion proxy service state via the service manager
6. Records a per-node verdict with an explanatory remark for every failure mode

The critical design point is step 4. The expected answer is not a single global constant — it is derived per node, so each server is validated against what that server should be returning. A resolver returning a valid-looking address that belongs to a different node is a misconfiguration, and this check catches it. A generic “did DNS respond” check does not.

# LOGIC ONLY — implementation withheld (proprietary, Kolpolok Limited)

FOR each row in inventory CSV (sn, name, address, ssh_user, ssh_port):
    country ← geo_consensus(address)              # 3 providers, 2-of-3 vote

    IF NOT ssh_reachable(address, user, port):
        RECORD status=Failed, remark="SSH unreachable"; CONTINUE

    IF query_tool_missing(remote):
        os ← detect_remote_os_family(remote)
        install_query_tool(remote, package_for(os))   # self-healing
        IF still missing:
            RECORD status=Failed, remark="tool install failed"; CONTINUE

    answer ← remote_dns_query(remote, local_resolver, target_domain)

    IF answer is empty:            status ← Failed; remark ← "no answer"
    ELSE IF answer == expected_for_this_node:  status ← Active
    ELSE:                          status ← Failed; remark ← "mismatch: got X, expected Y"

    proxy_state ← remote_service_state(remote, companion_service)
    IF proxy_state is not active:  status ← Failed; append remark

    RECORD row

build_html_dashboard(rows)    # summary counters + colour-coded table
email(html_inline + csv_attachment)

Self-healing dependencies

A monitoring check that fails because the target is missing a tool has produced a false alarm, and false alarms are how monitoring gets ignored. So when the query tool is absent on a node, the pipeline:

• reads the remote OS identity from the standard release metadata
• maps it to the correct package name across Debian/Ubuntu, RHEL/Fedora/Rocky/Alma, Arch, and Alpine families
• installs non-interactively, capturing output for the log
• re-verifies and only then proceeds
• falls back to a best-effort multi-package-manager attempt for unrecognised distributions

The pipeline repairs the precondition, records that it did so, and continues. A genuine failure is reported as a genuine failure; a missing tool is not.

Geolocation consensus voting

Geolocation providers disagree and rate-limit. Rather than trusting one, the pipeline queries three, validates each response (non-empty, plausible length, expected character set — rejecting error pages, rate-limit messages, and stray JSON), and applies a two-of-three agreement vote. With no consensus, it falls back through a fixed provider priority order. Only if every provider fails does it report Unknown.

This is the same principle as multi-source threat intelligence: a single source is a single point of failure, and validating the shape of a response before trusting its content is not optional.

Reporting

Each run produces a summary CSV and a self-contained HTML dashboard — total / active / failed counters, colour-coded status pills, per-node country and remark columns, run metadata — delivered as a properly structured multipart MIME email with the HTML inline and the CSV attached, over authenticated TLS SMTP.

Pipeline 2 — Datastore Health & Growth Tracking

The problem

An in-memory datastore that is reachable and responding is not necessarily healthy. The failure mode that actually hurts is a datastore that stopped growing — the process is up, connections succeed, and the pipeline feeding it broke three days ago. Reachability checks report green throughout.

Detecting that requires comparing today’s state to yesterday’s, which means the monitor needs memory.

The approach — stateful delta tracking

The pipeline maintains a persistent state file that carries forward the previous run’s key count and check date for every node. Each run:

1. Reads the previous run’s values per node from the state file
2. Probes each node with a bounded, timeout-guarded authenticated key-count query
3. Validates the response is genuinely numeric before accepting it as a result — a connection error, an auth failure, or a truncated response must never be recorded as a count
4. Writes the new row including the carried-forward previous value, so every report line shows current state next to prior state
5. Atomically replaces the state file with the new one
6. Archives a timestamped copy and prunes the archive to a rolling retention window
7. Emails a colour-coded HTML table with the CSV attached

Point 3 is where naive scripts break. Checking a command’s exit status is not sufficient when a tool prints diagnostics to standard output and still exits cleanly. Validating the shape of the response — not just the exit code — is what prevents “OK, 0 keys” being reported for a datastore that was actually unreachable.

Rolling archive retention

Every run archives a timestamped snapshot and prunes to the most recent N reports. Bounded retention by design: enough history to answer “when did this change?”, no unbounded directory growth to discover at 100% disk.

Shared Engineering Patterns

Every pipeline in this family is built on the same reusable patterns. This is the part that actually transfers between projects.

Why This Is Security Work, Not Just Ops Work

Every pipeline here maps directly onto security operations practice:

• DNS answer validation is integrity monitoring of a resolution path — the same control class as detecting DNS tampering or hijack
• Service-state verification is control-presence verification: proving a security-relevant service is actually running, not merely installed
• Datastore growth tracking is anomaly detection on a data pipeline — a flatline is an incident signal
• Geolocation consensus is multi-source corroboration with response validation before trust
• Fail-safe verdicts encode the discipline that an unverified control is an unhealthy control
• Automated reporting produces the evidence trail that ISO 27001 audit readiness requires — dated, per-node, per-run, retained

And the meta-point, which is the one that matters most: alert fatigue is an engineering problem, not an analyst problem. Every design decision above — response-shape validation, self-healing dependencies, explanatory remarks, geolocation consensus — exists to make sure that when a report says something is broken, it is broken. A monitoring system that cries wolf gets filtered into a folder nobody opens, and then it is worse than nothing, because it produces the feeling of coverage without the fact of it.

Testing & Verification

• Fault injection per pipeline: node offline, node reachable but service dead, node returning an incorrect answer, node missing the required tooling
• Inventory edge cases: CRLF line endings, blank rows, comment rows, missing optional columns, whitespace-padded fields
• Verified the dedicated-descriptor read processes every row when remote commands are invoked inside the loop
• Response-shape validation confirmed to reject error strings, auth failures, and empty output rather than record them as values
• Self-healing install path exercised across multiple distribution families
• Geolocation consensus tested with each provider individually failing and returning malformed content
• Retention pruning verified at and beyond the retention boundary
• Email structure validated for correct multipart MIME and inline HTML rendering
• Full runs timed against the fleet to confirm bounded total runtime with unreachable nodes present

Outcomes

Next Iteration

• Bounded parallelism. Fleet runs are serial. Concurrency with a worker pool would cut wall-clock time substantially at 170+ nodes.
• Time-series backend. Reports are point-in-time. Writing results to a time-series store converts them into trend alerting — “this node degraded 40% this week” rather than “this node failed today”.
• Convergence on a single reporting library. Each pipeline builds its own HTML. One shared reporting module would make every new pipeline cheaper to write and visually consistent.
• Migration of the pipeline family to Python. Bash was the right tool for orchestrating system commands, and remains so for the enforcement paths. But state handling, response parsing, structured output, and testability all argue for Python at this level of complexity — and it makes unit testing the parsing and validation logic genuinely practical.
• Secret management. Credentials belong in a secret store or environment injection, not in configuration blocks — the correct fix regardless of file permissions.