A hub and edge pattern for metrics using ZeroTier

Remote monitoring of hosts on a far side site comes with the challenge of protecting the far side from an inadvertent security posturing as a result of wanting remote monitoring.

There are lots of solutions available on the market for monitoring edge or remote hardware and services, of course.

Solutions are typically integrated: acme vendor has both edge-side agents, collectors and then has a wonderful cloud-side all-in-one central hub and dashboard, so the components used are all provided by them.

In March 2026, I wanted to see if I could develop a really simple pattern that makes use of existing either free or libre software tools, and achieves a similar result.

The architecture is the same, edge agents and a central hub, but in this exercise I control all of the components, define how they reach each other and the specificity of ‘signals’ (metrics, logs, tests) are defined explicitly by me, in a readable way (a repo).

In many cases depending on industry, edge targets can be quite varied in architecture and power or resource constraints.

However, for a client office with an on-site consumer NAS, there is usually enough compute (metrics doesn’t require much) and an architecture match (x86), but there is, let’s say, political limitations, in the format of application you can deploy.

There may indeed be constraints in the fact that a utility server or similar can’t really be used because none exists without being bought & provisioned first.

Synology, QNAP and TrueNAS being closed appliances, treat binaries and posix or systemd kind of orchestration with hostility, blocking apt, overwriting crontabs on reboot, and so on. Instead they welcome in packaged applications for their platform via their own app stores, but also Docker containers.

So this pattern benefits from being shaped as an OCI container, with deployment config shaped as docker compose, which is accessible on all 3 platforms.

Topology

Edge:

• Devices
  • NAS servers
  • Out of band BMC
  • Metrics node (independent, always on)
• Services
  • NAS storage (SMB)
  • Log export capability (syslog)
  • Node metrics (performance, stats)
  • Uptime or availability checkpoint (an open port, a /health endpoint, or a real service port capable of performing a test system transaction)

Hub:

• Devices
  • Server (VPS, or other compute)
• Services
  • Metrics aggregator
  • Logs aggregator
  • Metrics visualiser
  • Uptime poller (check an open port, request a health endpoint, or do a test transaction)
  • Alert orchestrator

Architecture

As a worked example:

A TrueNAS server runs a metrics exporter for performance stats. Its underlying hardware has a proprietary BMC, which can expose metrics. Both have logs to export.

The edge “node” contains a metrics scraper (prometheus, victoriametrics…), which scrapes the TrueNAS metrics exporter, the BMC exporter. It contains a log receiver agent, which receives incoming syslog lines from those two hosts.

The edge node also runs a ‘blackbox’ exporter: checks ports for open. It could also perform a synthetic transaction (such as authenticate over SMB and successfully write and read a test file).

A hub is deployed, running a metrics aggregator, a logs aggregator, metrics visualiser,… and you get the picture.

The hub’s services make it capable of aggregating these signals, having a place to define reasonable amounts and therefore alerts, and exist in a consistent place on rented cloud space, in lieu of real company-owned hardware.

And then as a map:

Implementation

I have liked ZeroTier as a VPN, as a SDN tool, and for its holepunching mechanism & relay servers that are available with reasonable limitations in the free plan. On Linux it uses a tun device to create virtual interfaces on the host, corresponding with virtual ZT networks.

It is deployable as a container, with persistent storage for the secret key. That’s the VPN layer sorted.

VictoriaMetrics is an excellent project with very versatile and swappable components.

I use vmagent as a scraper on the edge, configured to remote push to a hub-situated victoriametrics instance. vlagent is a logs agent, configured to remote push to a hub-situated victorialogs instance.

But in this pattern, I have devised a slightly different path that the signals actually travel on. The containers that collect metrics and logs on the edge (vmagent and vlagent) are actually unaware of the ZeroTier layer and don’t connect to it directly.

An ‘uplink’ is a sidecar container, which IS part of the ZT network, receives traffic because of a Docker network alias creating a DNS record only within that small segment. That record for hub hostname (victoriametrics.hub.mon.corp.internal) means that the raw TCP stream is directed at uplink and then handled via socat inside it, pointing at the hub’s ZT remote IP address. Since socat only forwards bytes, the TLS session, client certificate and Host header all arrive at the hub as if the agent had connected directly.

The benefit is that neither the NAS nor the agents ever join the overlay network, and the VPN identity is located in one container, not affecting the rest of the stack.

Additionally, mTLS is used between edge and hub. The secrets are generated at deployment time and are persisted so that they can remain in place indefinitely, until re-generated on some other kind of schedule.

Detail

Dockerfile

FROM ghcr.io/zyclonite/zerotier:1.16.0
RUN apk add --no-cache bash socat tini
COPY entrypoint-wrapper.sh /usr/local/bin/entrypoint-wrapper.sh
RUN chmod +x /usr/local/bin/entrypoint-wrapper.sh
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/entrypoint-wrapper.sh"]

Entrypoint excerpt:

# Use a fixed identity from a Docker secret, so the node ID survives rebuilds
ln -sf "$ZEROTIER_IDENTITY_SECRET_PATH" /var/lib/zerotier-one/identity.secret

# Join the network by dropping an empty conf file into networks.d
mkdir -p /var/lib/zerotier-one/networks.d
touch "/var/lib/zerotier-one/networks.d/${ZEROTIER_NETWORK_ID}.conf"

# Forward 443 as raw TCP to the destination
run_socat() {
    socat TCP-LISTEN:"$1",fork,reuseaddr \
        "TCP:${SOCAT_FORWARD_DEST}:$1,connect-timeout=${SOCAT_CONNECT_TIMEOUT}" &
}
run_socat 443

/usr/sbin/entrypoint.sh -U   # start zerotier-one

Edge compose service

---
name: corp-monitoring-edge
services:
  # [...]
  uplink:
    image: ghcr.io/corp/uplink:0.2.0
    cap_add:
      - NET_ADMIN
    cap_drop:
      - ALL
    devices:
      - /dev/net/tun
    environment:
      SOCAT_FORWARD_DEST: ${HUB_REMOTE_IPV4_ADDRESS}
      ZEROTIER_IDENTITY_SECRET_PATH: /run/secrets/zerotier-secret-key
      ZEROTIER_NETWORK_ID: ${ZEROTIER_NETWORK_ID}
    networks:
      uplink:
        aliases:
          - ${HUB_VICTORIALOGS_HOSTNAME}
          - ${HUB_VICTORIAMETRICS_HOSTNAME}
    secrets:
      - zerotier-secret-key
    security_opt:
      - no-new-privileges:true
    volumes:
      - type: tmpfs
        target: /var/lib/zerotier-one
  vmagent:
    # Metrics collector. Scrapes targets defined in --promscrape.config, forwards to --remoteWrite.url
    image: victoriametrics/vmagent:v1.134.0
    configs:
      - source: vmagent-prometheus-yml
        target: /etc/prometheus/prometheus.yml
    command:
      - "--promscrape.config=/etc/prometheus/prometheus.yml"
      - "--remoteWrite.url=https://${HUB_VICTORIAMETRICS_HOSTNAME}/api/v1/write"
      - "--remoteWrite.tlsCAFile=/run/secrets/monitoring-rootca"
      - "--remoteWrite.tlsCertFile=/run/secrets/vmagent-client-cert"
      - "--remoteWrite.tlsKeyFile=/run/secrets/vmagent-client-key"
    dns:
      - "${TARGETS_GATEWAY}"
    depends_on:
      - uplink
    ports:
      - "${TARGETS_GATEWAY}:8429:8429" # expose vmagent's internals
    #read_only: true
    networks:
      default:
        aliases:
          - vmagent-internal
      targets:
        ipv4_address: "${TARGETS_VMAGENT_IPV4_ADDRESS}"
      uplink:
    restart: unless-stopped
    secrets:
      - monitoring-rootca
      - hub-server-cert
      - vmagent-client-cert
      - vmagent-client-key
    volumes:
      - vmagent-data:/vmagentdata

HOST=nas2.corp.internal
HUB_SERVER_CERT_HOSTNAME=hub.mon.corp.internal
HUB_REMOTE_IPV4_ADDRESS=10.0.13.22
HUB_VICTORIALOGS_HOSTNAME=victorialogs.hub.mon.corp.internal
HUB_VICTORIAMETRICS_HOSTNAME=victoriametrics.hub.mon.corp.internal
TARGETS_GATEWAY=10.0.130.7
TARGETS_VMAGENT_IPV4_ADDRESS=10.0.130.2
ZEROTIER_NETWORK_ID=4cd3fdeadbeefe4b

Hub compose service

---
name: corp-monitoring-hub
services:
  # [...]
  inbound-proxy:
    image: haproxy:3.3.2-alpine
    command: >
      haproxy -W -db -f /etc/haproxy/haproxy.cfg
    configs:
      - source: inbound-proxy-haproxy-cfg
        target: /etc/haproxy/haproxy.cfg
    networks:
      - default
      - inbound-uplink
    read_only: false
    secrets:
      - source: monitoring-rootca
        target: /etc/haproxy/certs/monitoring-rootca.crt
      - source: hub-server-cert
        target: /etc/haproxy/certs/hub-server.crt
      - source: hub-server-key
        target: /etc/haproxy/certs/hub-server.key
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /run/haproxy:mode=0700,uid=99,gid=99
    user: haproxy
  inbound-uplink:
    image: ghcr.io/corp/uplink:0.2.0
    cap_add:
      - NET_ADMIN
    cap_drop:
      - ALL
    devices:
      - /dev/net/tun
    environment:
      SOCAT_FORWARD_DEST: inbound-proxy
      ZEROTIER_IDENTITY_SECRET_PATH: /run/secrets/inbound-zerotier-secret-key
      ZEROTIER_NETWORK_ID: ${ZEROTIER_NETWORK_ID}
    networks:
      - inbound-uplink
    secrets:
      - inbound-zerotier-secret-key
    security_opt:
      - no-new-privileges:true
    volumes:
      - type: tmpfs
        target: /var/lib/zerotier-one
  victoriametrics:
    # Storing metrics and serve read requests.
    image: victoriametrics/victoria-metrics:v1.134.0
    command:
      - "--storageDataPath=/storage"
      - "--httpListenAddr=:8428"
      - "--vmalert.proxyURL=http://vmalert:8880"
      - "--selfScrapeInterval=10s"
    restart: always
    networks:
      - default
    volumes:
      - victoriametrics-data:/storage

HUB_SERVER_CERT_HOSTNAME=hub.mon.corp.internal
HUB_GRAFANA_HOSTNAME=dash.mon.corp.internal
HUB_VICTORIALOGS_HOSTNAME=victorialogs.hub.mon.corp.internal
HUB_VICTORIAMETRICS_HOSTNAME=victoriametrics.hub.mon.corp.internal
ZEROTIER_NETWORK_ID=4cd3fdeadbeefe4b

mTLS and HAProxy routing

On the hub, everything coming in from the edge arrives at a single HAProxy frontend. This instance is in charge of doing mTLS.

It only accepts a connection if the client presents a certificate signed by the monitoring CA. Then requests are differentiated based on hostname of course. As mentioned, ZT already encrypts the traffic between sites, so mTLS here is a second layer, that would prevent some other kind of network intrusion from writing data into the hub without a valid certificate.

frontend edge_ingest
    bind *:443 ssl crt /etc/haproxy/certs/hub-server.crt ca-file /etc/haproxy/certs/hub-server.crt verify required
    log-format "... %[ssl_c_s_dn]"
    use_backend victorialogs     if { hdr_end(host) -i victorialogs.hub.mon.corp.internal }
    use_backend victoriametrics  if { hdr_end(host) -i victoriametrics.hub.mon.corp.internal }

mTLS certificate generation

A justfile is used to automate these tasks, using openssl:

• Generate a root CA.
• Sign a new key & CSR for the hub, with SAN hostname definition for the victoriametrics and logs hostnames.
• Per edge client, sign a new key & certificate (such as vmagent.nas2., vlagent.nas2.), and verify the result files.
• Package the secret and certificate string values into a single SOPS-encrypted JSON.
• Deploy to the compose targets via ssh/scp as a temp value, passing each specific value to the correct docker compose secret, with those values finishing up only in the container filesystem and not visible on container inspection.*

• A hand-drawn runtime secrets injection, if you will.

Edge scrape config

The edge’s vmagent config is where every signal for the site is spelled out, in one readable file that lives in the repo. Exporters that run on the hosts themselves, like node, smartctl and ZFS, are scraped directly. Anything that needs an intermediary, such as a blackbox probe, the iDRAC or the UPS, goes through the standard Prometheus relabelling pattern. The target and labels are adjusted so that the transport address is actually scraped and the ‘display’ address is shown in the metrics, so dashboards read truthfully with respect to the real hosts.

- job_name: "blackbox_smb"
  metrics_path: /probe
  params:
    module: [tcp_connect]
  static_configs:
    - targets:
        - "nas2.corp.internal:445"
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter-internal:9115

Not discussed

Configuration of the alerts, and further steps.

Interruption handling, and the impact of a failure at different stages of the components, and how such a failure might manifest depending on which bit.

Such as, if the uplink container can’t find a remote peer on zerotier, but both edge and hub have good functioning WAN links, what is the resulting display on our metrics dashboard? Do our alerts still fire? Do we have vitals still running that are independent to that?

Alternatives

ZeroTier offering a Layer 2-based protocol leaves a lot of room open for risk, and also host modification (in the form of capabilities for NET_ADD and /dev/net/tun). An L3 based overlay would be appropriate to explore.

Tailscale has userspace WireGuard that can run in very low-privilege containers.