Running traefik in a less braindead way


This is a second post in the High Altitude Water Aerosols series with the aim to get “cloud native” work for internal (web-)services.

Problem statement

As mentioned in the intro post, I want some sort of containerized reverse proxy to pass through traffic to the individual backend services.

One obvious choice is traefik, which, in their words, is “The Cloud Native Application Proxy”.

So instead of using haproxy to direct traffic, I’m going to use the cloud native hotness1. Because one of the main selling points is the use of container labels to auto-configure backend routing.

Road to solution

If you naively go with their 5-Minute Quickstart, slap the:

version: '3'

    # The official v2 Traefik docker image
    image: traefik:v2.10
    # Enables the web UI and tells Traefik to listen to docker
    command: --api.insecure=true --providers.docker
      # The HTTP port
      - "80:80"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock

in your docker-compose.yaml and call it a day, you’re in for a world of pain.

What’s wrong with it, you ask?

  1. The http proxy runs under root by default2
  2. You give it unrestricted access to the docker API

The first issue can be easily mitigated by not running your podman under root, in which case you won’t be binding any port under 1024 without additional stupid hacks3. But whatever.

The second issue is worse, and I’m far from the first to figure it out. The Exposing Docker socket to Traefik container is a serious security risk issue on traefik’s GH is open since November 2018. But what’s 5 years between friends?

So, to run traefik semi-securely we’ll need to run it in a slightly different way. By running haproxy in another container that will shrink the allowed API to a safe(r) subset4.

Because it turns out that traefik’s docker plugin only needs GET access to the following PATH prefixes to work:

Which disallows launching new containers (all writes, actually), messing with secrets, etc.

Now, in an ideal world, I’d also run this haproxy container under a non-root user, but podman forces umask 0177 on the podman.sock. So until that gets fixed, haproxy as root it is5.


Based on previous section, the solution ends up being:

expected endstate

And thus we need two containers6.

Socket proxy

This is mostly taken from docker-socket-proxy:

# compose.yml
# [...]
    build: sp
    no_cache: true
    restart: always
    stop_signal: SIGTERM
      - /run/podman/podman.sock:/var/run/docker.sock:ro


Dockerfile (click to expand)

FROM haproxy:2.2-alpine

    AUTH=0 \
    BUILD=0 \
    COMMIT=0 \
    CONFIGS=0 \
    EVENTS=1 \
    EXEC=0 \
    GRPC=0 \
    IMAGES=0 \
    INFO=0 \
    LOG_LEVEL=info \
    NETWORKS=0 \
    NODES=0 \
    PING=1 \
    PLUGINS=0 \
    POST=0 \
    SECRETS=0 \
    SERVICES=0 \
    SESSION=0 \
    SOCKET_PATH=/var/run/docker.sock \
    SWARM=0 \
    SYSTEM=0 \
    TASKS=0 \
    VERSION=1 \
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

haproxy.cfg (click to expand)

    log stdout format raw daemon "${LOG_LEVEL}"

    pidfile /run/
    maxconn 4000

    # Turn on stats unix socket
    server-state-file /var/lib/haproxy/server-state

    ## you can't drop perms;
    ## you wouldn't be able to connect to podman sock otherwise.
    #user haproxy
    #group haproxy

    mode http
    log global
    option httplog
    option dontlognull
    option http-server-close
    option redispatch
    retries 3
    timeout http-request 10s
    timeout queue 1m
    timeout connect 10s
    timeout client 10m
    timeout server 10m
    timeout http-keep-alive 10s
    timeout check 10s
    maxconn 3000

    # Allow seamless reloads
    load-server-state-from-file global

    # Use provided example error pages
    errorfile 400 /usr/local/etc/haproxy/errors/400.http
    errorfile 403 /usr/local/etc/haproxy/errors/403.http
    errorfile 408 /usr/local/etc/haproxy/errors/408.http
    errorfile 500 /usr/local/etc/haproxy/errors/500.http
    errorfile 502 /usr/local/etc/haproxy/errors/502.http
    errorfile 503 /usr/local/etc/haproxy/errors/503.http
    errorfile 504 /usr/local/etc/haproxy/errors/504.http

backend dockerbackend
    server dockersocket $SOCKET_PATH

frontend dockerfrontend
    bind :2375
    http-request deny unless METH_GET || { env(POST) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers/[a-zA-Z0-9_.-]+/((stop)|(restart)|(kill)) } { env(ALLOW_RESTARTS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/auth } { env(AUTH) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/build } { env(BUILD) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/commit } { env(COMMIT) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/configs } { env(CONFIGS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/containers } { env(CONTAINERS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/distribution } { env(DISTRIBUTION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/events } { env(EVENTS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/exec } { env(EXEC) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/grpc } { env(GRPC) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/images } { env(IMAGES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/info } { env(INFO) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/networks } { env(NETWORKS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/nodes } { env(NODES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/_ping } { env(PING) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/plugins } { env(PLUGINS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/secrets } { env(SECRETS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/services } { env(SERVICES) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/session } { env(SESSION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/swarm } { env(SWARM) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/system } { env(SYSTEM) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/tasks } { env(TASKS) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/version } { env(VERSION) -m bool }
    http-request allow if { path,url_dec -m reg -i ^(/v[\d\.]+)?/volumes } { env(VOLUMES) -m bool }
    http-request deny
    default_backend dockerbackend

Traefik itself

To spin up traefik, I’m using official image and throwing in a bit of config:

# compose.yml
# [...]
    image: traefik:v2.10.4
      - "10.X.Y.Z:80:8000"   # FIXME
      - "10.X.Y.Z:8080:8080" # FIXME
      - "10.X.Y.Z:443:4443"  # FIXME
      - ./traefik:/etc/traefik:ro
    restart: always
    user: "10xx:10xx" # FIXME

The noteworthy bits are:

The additional config consists of two files:


This is the static global config read at start, which configures “providers”, ports, etc.

  checkNewVersion: false
  # Go spy on someone else...
  sendAnonymousUsage: false

    watch: true
    exposedByDefault: false
    defaultRule: "Host(`{{ index .Labels \"com.docker.compose.service\"}}`)"
    # Here's the socket proxy endpoint:
    endpoint: 'tcp://sp:2375'

    watch: true
    # Dynamic config, see the dynamic.yml below...
    filename: /etc/traefik/dynamic.yml

  dashboard: true
  insecure: true

    address: :8000

    address: :4443
      tls: true


Dynamic file config, that is used for middleware and SSL certificates.

        # Hello default wildcard SSL certificate :)
        certFile: /etc/traefik/
        keyFile: /etc/traefik/

        scheme: "https"
        permanent: true

The certificates I generated myself, using my own root certificate authority setup. More on this in a future post.

Test backend

In order to run this as an unprivileged user, I’m again setting user: directive and also mapping the listening port to 3000. The rest is pretty much vanilla traefik routing – redirect http to https, and set the backend name to whoami (resolved via internal DNS).

# compose.yml
# [...]
    image: traefik/whoami
      - "traefik.enable=true"
      # http
      - "traefik.http.routers.whoami-http.rule=Host(``)"
      - "traefik.http.routers.whoami-http.entrypoints=http"
      - "traefik.http.routers.whoami-http.middlewares=redirect-https@file"
      - "traefik.http.routers.whoami-http.service=whoami"
      # https
      - "traefik.http.routers.whoami-https.rule=Host(``)"
      - "traefik.http.routers.whoami-https.entrypoints=https"
      - "traefik.http.routers.whoami-https.service=whoami"
      # backend (port)
      - ""
    restart: always
    user: "10xx:10xx" # FIXME
      "WHOAMI_PORT_NUMBER": 3000


With all three containers in place:

$ podman-compose ps 2>/dev/null | sed -r 's/^(.{80}).*/\1/'
CONTAINER ID  IMAGE                                COMMAND               CREATED
2160c2412a69  localhost/int_sp:latest              haproxy -f /usr/l...  7 weeks
9b9519a2c826    traefik               7 weeks
db397824072b                            7 weeks

It’s time to test:

$ curl -L
Hostname: db397824072b
IP: ::1
IP: 10.xx.y.zz
IP: fe80::aaaa:bbbb:cccc:dddd
RemoteAddr: 10.xx.y.50:50800
GET / HTTP/1.1
User-Agent: curl/7.xx.y
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.x.y.zzz
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 9b9519a2c826
X-Real-Ip: 10.x.y.zzz

Which looks like a great success to me.

Closing words

This was a relatively quick intro into configuring traefik for internal services.

The irony of running yet another reverse proxy (haproxy) to make the setup less horrible from security PoV isn’t lost on me, though.

Next up: the CA web server.

  1. Or maybe insecure by default “hot mess”? You’ll see.

  2. Since our podman runs under root.

  3. sysctl -w net.ipv4.ip_unprivileged_port_start=80 is a terrible idea. Always was.

  4. Because that’s how you solve things in a Cloud Native High Altitude Water Aerosol way, I guess? You add more dead weight.

  5. I guess I trust haproxy smidge more than traefik. IAC, perfect enemy of good and all that.

  6. Well, three, if we want to test it. The last one for sample backend.