Written on 2023-11-05

Running traefik in a less braindead way

Introduction

This is a second post in the High Altitude Water Aerosols series with the aim to get “cloud native” work for internal (web-)services.

Problem statement

As mentioned in the intro post, I want some sort of containerized reverse proxy to pass through traffic to the individual backend services.

One obvious choice is traefik, which, in their words, is “The Cloud Native Application Proxy”.

So instead of using haproxy to direct traffic, I’m going to use the cloud native hotness¹. Because one of the main selling points is the use of container labels to auto-configure backend routing.

Road to solution

If you naively go with their 5-Minute Quickstart, slap the:

version: '3'

services:
  reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.10
    # Enables the web UI and tells Traefik to listen to docker
    command: --api.insecure=true --providers.docker
    ports:
      # The HTTP port
      - "80:80"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock

in your docker-compose.yaml and call it a day, you’re in for a world of pain.

What’s wrong with it, you ask?

The http proxy runs under root by default²
You give it unrestricted access to the docker API

The first issue can be easily mitigated by not running your podman under root, in which case you won’t be binding any port under 1024 without additional stupid hacks³. But whatever.

The second issue is worse, and I’m far from the first to figure it out. The Exposing Docker socket to Traefik container is a serious security risk issue on traefik’s GH is open since November 2018. But what’s 5 years between friends?

So, to run traefik semi-securely we’ll need to run it in a slightly different way. By running haproxy in another container that will shrink the allowed API to a safe(r) subset⁴.

Because it turns out that traefik’s docker plugin only needs GET access to the following PATH prefixes to work:

/v1.24/version
/v1.24/events
/v1.24/containers
/v1.24/_ping

Which disallows launching new containers (all writes, actually), messing with secrets, etc.

Now, in an ideal world, I’d also run this haproxy container under a non-root user, but podman forces umask 0177 on the podman.sock. So until that gets fixed, haproxy as root it is⁵.

Solution

Based on previous section, the solution ends up being:

expected endstate

And thus we need two containers⁶.

Socket proxy

This is mostly taken from docker-socket-proxy:

# compose.yml
services:
# [...]
  sp:
    build: sp
    no_cache: true
    restart: always
    stop_signal: SIGTERM
    volumes:
      - /run/podman/podman.sock:/var/run/docker.sock:ro

with:

Dockerfile (click to expand)

FROM haproxy:2.2-alpine

EXPOSE 2375
ENV ALLOW_RESTARTS=0 \
    AUTH=0 \
    BUILD=0 \
    COMMIT=0 \
    CONFIGS=0 \
    CONTAINERS=1 \
    DISTRIBUTION=0 \
    EVENTS=1 \
    EXEC=0 \
    GRPC=0 \
    IMAGES=0 \
    INFO=0 \
    LOG_LEVEL=info \
    NETWORKS=0 \
    NODES=0 \
    PING=1 \
    PLUGINS=0 \
    POST=0 \
    SECRETS=0 \
    SERVICES=0 \
    SESSION=0 \
    SOCKET_PATH=/var/run/docker.sock \
    SWARM=0 \
    SYSTEM=0 \
    TASKS=0 \
    VERSION=1 \
    VOLUMES=0
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

haproxy.cfg (click to expand)

global
    log stdout format raw daemon "${LOG_LEVEL}"

    pidfile /run/haproxy.pid
    maxconn 4000

    # Turn on stats unix socket
    server-state-file /var/lib/haproxy/server-state

    ## you can't drop perms;
    ## you wouldn't be able to connect to podman sock otherwise.
    #user haproxy
    #group haproxy

defaults
    mode http
    log global
    option httplog
    option dontlognull
    option http-server-close
    option redispatch
    retries 3
    timeout http-request 10s
    timeout queue 1m
    timeout connect 10s
    timeout client 10m
    timeout server 10m
    timeout http-keep-alive 10s
    timeout check 10s
    maxconn 3000

    # Allow seamless reloads
    load-server-state-from-file global

    # Use provided example error pages
    errorfile 400 /usr/local/etc/haproxy/errors/400.http
    errorfile 403 /usr/local/etc/haproxy/errors/403.http
    errorfile 408 /usr/local/etc/haproxy/errors/408.http
    errorfile 500 /usr/local/etc/haproxy/errors/500.http
    errorfile 502 /usr/local/etc/haproxy/errors/502.http
    errorfile 503 /usr/local/etc/haproxy/errors/503.http
    errorfile 504 /usr/local/etc/haproxy/errors/504.http

backend dockerbackend
    server dockersocket $SOCKET_PATH

frontend dockerfrontend
    bind :2375
    http-request deny unless METH_GET || { env(POST) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers/[a-zA-Z0-9_.-]+/((stop)|(restart)|(kill)) } { env(ALLOW_RESTARTS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/auth } { env(AUTH) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/build } { env(BUILD) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/commit } { env(COMMIT) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/configs } { env(CONFIGS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers } { env(CONTAINERS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/distribution } { env(DISTRIBUTION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/events } { env(EVENTS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/exec } { env(EXEC) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/grpc } { env(GRPC) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/images } { env(IMAGES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/info } { env(INFO) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/networks } { env(NETWORKS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/nodes } { env(NODES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/_ping } { env(PING) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/plugins } { env(PLUGINS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/secrets } { env(SECRETS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/services } { env(SERVICES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/session } { env(SESSION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/swarm } { env(SWARM) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/system } { env(SYSTEM) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/tasks } { env(TASKS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/version } { env(VERSION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/volumes } { env(VOLUMES) -m bool }
    http-request deny
    default_backend dockerbackend

Traefik itself

To spin up traefik, I’m using official image and throwing in a bit of config:

# compose.yml
services:
# [...]
  traefik:
    image: traefik:v2.10.4
    ports:
      - "10.X.Y.Z:80:8000"   # FIXME
      - "10.X.Y.Z:8080:8080" # FIXME
      - "10.X.Y.Z:443:4443"  # FIXME
    volumes:
      - ./traefik:/etc/traefik:ro
    restart: always
    user: "10xx:10xx" # FIXME

The noteworthy bits are:

ports: that map the individual $IP:80 to internal 8000 (etc)
user: that forces the service to run as given uid:gid pair
volumes: that provides additional config as a read-only mount

The additional config consists of two files:

traefik.yml

This is the static global config read at start, which configures “providers”, ports, etc.

global:
  checkNewVersion: false
  # Go spy on someone else...
  sendAnonymousUsage: false

providers:
  docker:
    watch: true
    exposedByDefault: false
    defaultRule: "Host(`{{ index .Labels \"com.docker.compose.service\"}}.int.wejn.org`)"
    # Here's the socket proxy endpoint:
    endpoint: 'tcp://sp:2375'

  file:
    watch: true
    # Dynamic config, see the dynamic.yml below...
    filename: /etc/traefik/dynamic.yml

api:
  dashboard: true
  insecure: true

entrypoints:
  http:
    address: :8000

  https:
    address: :4443
    http:
      tls: true

dynamic.yml

Dynamic file config, that is used for middleware and SSL certificates.

tls:
  stores:
    default:
      defaultCertificate:
        # Hello default wildcard SSL certificate :)
        certFile: /etc/traefik/int.wejn.org.crt
        keyFile: /etc/traefik/int.wejn.org.key

http:
  middlewares:
    redirect-https:
      redirectScheme:
        scheme: "https"
        permanent: true

The certificates I generated myself, using my own root certificate authority setup. More on this in a future post.

Test backend

In order to run this as an unprivileged user, I’m again setting user: directive and also mapping the listening port to 3000. The rest is pretty much vanilla traefik routing – redirect http to https, and set the backend name to whoami (resolved via internal DNS).

# compose.yml
services:
# [...]
  whoami:
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      # http
      - "traefik.http.routers.whoami-http.rule=Host(`whoami.int.wejn.org`)"
      - "traefik.http.routers.whoami-http.entrypoints=http"
      - "traefik.http.routers.whoami-http.middlewares=redirect-https@file"
      - "traefik.http.routers.whoami-http.service=whoami"
      # https
      - "traefik.http.routers.whoami-https.rule=Host(`whoami.int.wejn.org`)"
      - "traefik.http.routers.whoami-https.entrypoints=https"
      - "traefik.http.routers.whoami-https.service=whoami"
      # backend (port)
      - "traefik.http.services.whoami.loadbalancer.server.port=3000"
    restart: always
    user: "10xx:10xx" # FIXME
    environment:
      "WHOAMI_PORT_NUMBER": 3000

Testing

With all three containers in place:

$ podman-compose ps 2>/dev/null | sed -r 's/^(.{80}).*/\1/'
CONTAINER ID  IMAGE                                COMMAND               CREATED
2160c2412a69  localhost/int_sp:latest              haproxy -f /usr/l...  7 weeks
9b9519a2c826  docker.io/library/traefik:v2.10.4    traefik               7 weeks
db397824072b  docker.io/traefik/whoami:latest                            7 weeks

It’s time to test:

$ curl -L http://whoami.int.wejn.org/
Hostname: db397824072b
IP: 127.0.0.1
IP: ::1
IP: 10.xx.y.zz
IP: fe80::aaaa:bbbb:cccc:dddd
RemoteAddr: 10.xx.y.50:50800
GET / HTTP/1.1
Host: whoami.int.wejn.org
User-Agent: curl/7.xx.y
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.x.y.zzz
X-Forwarded-Host: whoami.int.wejn.org
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 9b9519a2c826
X-Real-Ip: 10.x.y.zzz

Which looks like a great success to me.

Closing words

This was a relatively quick intro into configuring traefik for internal services.

The irony of running yet another reverse proxy (haproxy) to make the setup less horrible from security PoV isn’t lost on me, though.

Next up: the CA web server.

Or maybe insecure by default “hot mess”? You’ll see. ↩
Since our podman runs under root. ↩
sysctl -w net.ipv4.ip_unprivileged_port_start=80 is a terrible idea. Always was. ↩
Because that’s how you solve things in a ~~Cloud Native~~ High Altitude Water Aerosol way, I guess? You add more dead weight. ↩
I guess I trust haproxy smidge more than traefik. IAC, perfect enemy of good and all that. ↩
Well, three, if we want to test it. The last one for sample backend. ↩

← → ⇈