Getting Started With Traefik & Podman

Why Podman?

Primarily, because it's completely Open Source and runs very well on Linux. I also like a lot of the semantics of Podman versus OTHER container runtimes. In addition, Podman is pretty much a drop-in replacement for the incumbent container runtime. Compose files, commands, etc... They're all pretty much the same. Finally, Podman has a lot of features which other container runtimes do NOT have, like . . . Pods . . . instead of just containers.

Why Traefik?

I have used HAProxy for YEARS when I needed a reverse proxy. It worked well, but it was also a bit of a struggle to automate and reconfigure at runtime. When one of the people I follow on Mastodon started talking about Traefik I did some reading and realized that my preferred workflow of using containers and LetsEncrypt would be made a LOT easier using Traefik. Beyond that, Traefik has a large ecosystem of extensions/plugins to provide more capabilities than what I could easily add to HAProxy.

Getting Started

Installing Podman

If you are running a modern Linux distribution, you can almost certainly install Podman using your native package manager. On MacOS and Windows you can use package managers like Homebrew and Nuget to install Podman as well. Once you have it installed, on Linux there are no other steps. On Windows and MacOS, you will need to create a "podman machine", which is a small Linux virtual machine inside of which the real Podman runs.

Running Services With Podman

If you are coming from another container runtime, you may be used to running containers with a configuration like --restart unless-stopped and the runtime will handle keeping the container running. This is one area where Podman differs significantly. Podman uses the systemd capabilities of Linux itself instead. Podman provides a capability called Quadlets. A simple container Quadlet is shown below:

[Unit]
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=valkey
Image=docker.io/valkey/valkey:8.1
Environment=VALKEY_EXTRA_FLAGS='--maxmemory 512mb --maxmemory-policy noeviction'
PublishPort=6379:6379

[Service]
Restart=always

[Install]
WantedBy=default.target

This Quadlet would pull and start the container image docker.io/valkey/valkey:8.1 while setting an environment variable and name it valkey. The [Unit] section allows us to define service dependencies, so you can see that this container depends on the system's network connection to be up. The [Service] section can be used to define a Restart policy, but you can also use it to define a service as Type=oneshot meaning that the service is run once on boot, and as long as it exits with a 0 exit code, it will complete and not run it again. Finally, there is the [Install] section, which in this case is being used to request that this container is launched on boot.

You could install this Quadlet to run either rootless (by default Podman containers do not run with elevated privleges) or rootful.

Rootless Podman Services

As an unprivileged user, you would write that file to ${HOME}/.config/containers/systemd/valkey.container. Then you would have systemd reload it's configuration by running systemctl --user daemon-reload. You could then start the service using the command systemctl --user start valkey.service. Podman, when installed on a systemd enabled Linux distribution, adds a generator to systemd which reads quadlet definitions and automatically registers them as systemd services.

Rootful Podman Services

Similarly, as root on a Linux host, you would write that Quadlet to /etc/containers/systemd/valkey.container and reload the systemd services with systemctl daemon-reload (Without --user this time). Start the service using systemctl start valkey.service.

Choosing Between Rootful And Rootless Podman

The simple answer is that running rootless is FAR more secure than rootful. The container never has elevated privileges when running rootless as an unprivileged user. This means that if there were some malicious method used to try to jailbreak the container, even if successful it would not allow elevated privileges. The more complex answer is about system capabilities. Rootless podman containers cannot bind to ports <= 1024. Additionally, rootless containers cannot mount volumes which the unprivileged user does not have permissions to. Both of these limitations are easily addressed. For the privileged ports ( <= 1024) you can create a NAT rule using tools like firewalld/iptables/nftables. For the volume permissions, just be sure to put the files in directories which the unprivileged users has permissions to.

Enabling Podman Socket

Sometimes, you want to run a container which has the ability to read from (and possibly write to) the container runtime API. You will often see this pattern where a single container is set up to start other containers or read information about running containers. This is a pattern used extensively for Traefik. Podman supports this pattern by creating two (2) new systemd units which are enabled as follows:

bash

systemctl [--user] enable --now podman.socket
systemctl [--user] enable --now podman.service

Once this is done, you can now access the podman API via a UNIX socket:

Rootful: /run/podman/podman.sock
Rootless: /run/user/<User ID>/podman/podman.sock

Running Traefik In Podman

Traefik can be easily run using Podman using a quadlet like the one shown below:

[Unit]
After=network-online.target
Wants=network-online.target

[Container]
ContainerName=traefik
Image=docker.io/traefik:latest
Environment=VALKEY_EXTRA_FLAGS='--maxmemory 512mb --maxmemory-policy noeviction'
PublishPort=8080:80
PublishPort=8443:443
Volume=/home/myuser/services/traefik:/etc/traefik:z
Volume=/home/myuser/services/traefik/logs:/var/log/traefik:z
Volume=/run/podman/podman.sock:/var/run/docker.sock:ro,z
NoNewPrivileges=true
SecurityLabelType=container_runtime_t
AutoUpdate=registry
AddCapability=CAP_NET_BIND_SERVICE

[Service]
Restart=always

[Install]
WantedBy=default.target

Let's review what we are seeing here:

PublishPort - Exposes a port from the container via the host system's network
Volume - Binds a directory from the podman host to a directory inside of the container. The characters after the last colon (:) are mount options for things like handling SELinux relabeling or setting read-only (ro).
NoNewPrivileges - Tells podman to run the container and give up the ability to request additional privileges after the service starts (Not needed for Rootless)
SecurityLabelType=container_runtime_t - This is for systems which use SELinux to set the right security labels
AutoUpdate=registry - When the service starts, check to see if a newer version of the container image is available and pull it before starting
AddCapability=CAP_NET_BIND_SERVICE - Allow the container to bind to a privileged port (Not needed for Rootless)

The directory /home/myuser/services/traefik binding is where we would keep our Traefik configuration, both static and dynamic. I will explain the difference a little further.

Configuring Traefik

The static Traefik configuration may look something like this:

yaml

global:
  checkNewVersion: true
  sendAnonymousUsage: false

# Define the ports which Traefik will handle traffic for
entryPoints:
  http:
    address: ":80"
    # Redirect HTTP requests to the HTTPS equivalent
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"

# Define the dynamic configuration providers
# Configuration from these providers is automatically applied at runtime
# without restarting Traefik
providers:
  file:
    filename: /etc/traefik/config.yml
    watch: true
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

# Configure Traefik so that it an automatically request and configure
# TLS certificates using ACME and LetsEncrypt
certificatesResolvers:
  traefiktls:
    acme:
      email: you@example.com
      storage: /etc/traefik/letsencrypt/acme.json
      httpChallenge:
        entryPoint: http

# Define a store where the certificates for Traefik will be resolved from
tls:
  stores:
    default:
      defaultGeneratedCert:
        resolver: traefiktls
        domain:
          main: defaulthost.yourdomain.com

The syntax of this configuration file is explained in detail on the Traefik Docs Site. The configuration above is saved as /home/myuser/services/traefik/traefik.yml and mounted in the container as /etc/traefik/traefik.yml.

You will notice the providers.file section above where the /etc/traefik/config.yml (mounted from /home/myuser/services/traefik/config.yml) file is defined as a provider. This means that changes we make to that file will be watched and applied dynamically at runtime without restarting Traefik. Here's what an example config.yml might look like:

yaml

http:
  middlewares:
    secure-headers:
      headers:
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        customFrameOptionsValue: "SAMEORIGIN"
        contentTypeNosniff: true
        browserXssFilter: true
        referrerPolicy: "strict-origin-when-cross-origin"
        permissionsPolicy: "geolocation=(), microphone=(), camera=()"

The http.middlewares.secure-headers item allows us to define headers which can be injected into HTTP/HTTPS requests to enforce HSTS among other security best-practices. We can add other middleware items dynamically at runtime by modifying this configuration file.

Proxying Podman Containers Through Traefik

This is the part that I really love about using Traefik with Podman. The the Traefik static configuration above, there is a provider called "docker". This provider watches the podman API socket and based on the running container information (specifically the labels) it will dynamically configure the Traefik reverse proxy settings for that container/pod. Let's look at an example in the form of a Quadlet:

[Unit]
Wants=traefik.service
Requires=frontend-network.service
After=network-online.target
Requires=network-online.target

[Container]
ContainerName=webmail
Image=ghcr.io/root-fr/jmap-webmail:latest
Network=frontend
Environment="APP_NAME=JMAP Webmail"
Environment="JMAP_SERVER_URL=https://mail.example.com/"
Environment="SESSION_SECRET=REDACTED"
Label="traefik.enable=true"
Label="traefik.docker.network=frontend"
Label="traefik.http.routers.webmail.rule=Host(`webmail.example.com`)"
Label="traefik.http.routers.webmail.entrypoints=https"
Label="traefik.http.routers.webmail.service=webmail-http"
Label="traefik.http.routers.webmail.tls.certresolver=traefiktls"
Label="traefik.http.routers.webmail.middlewares=secure-headers@file"
Label="traefik.http.services.webmail-http.loadbalancer.server.port=3000"

[Service]
Restart=always

[Install]
WantedBy=default.target

You will notice the Label items in the Quadlet. These are labels attached to the container when it is running, but because of the format and syntax, it is also the dynamic configuration in Traefik. The labels above translates to the equivalent YAML config below:

traefik:
  enable: "true"
  http:
    routers:
      webmail:
        rule: "Host(`webmail.example.com`)"
        entrypoints: https
        service: webmail-http
        tls:
          certresolver: traefiktls
        middlewares: secure-headers@file
    services:
      webmail-http:
        loadbalancer:
          server:
            port: "3000"

Adding these labels to the container tells Traefik to do the following:

Accept requests for the domain webmail.example.com on the https entrypoint (e.g. 8443)
Use LetsEncypt to request and assign a TLS certificate for that host
Inject our security related headers to all requests/responses as appropriate
Proxy requests to the associated container on port 3000

It's also worth noting that Traefik will automatically inject the relevant proxy request headers like X-Forward-Host and X-Real-IP to the container.

Summary

Traefik, when paired with a container runtime like Podman, provides a dynamic and powerful way to declaritively set up reverse proxy and LetsEncrypt TLS for all of the services you want to run inside (or even outside) of containers. Feel free to reach out via Mastodon and share your journey or tell me how crazy I am for doing all of this.

Getting Started With Traefik & Podman ​

Why Podman? ​

Why Traefik? ​

Getting Started ​

Installing Podman ​

Running Services With Podman ​

Rootless Podman Services ​

Rootful Podman Services ​

Choosing Between Rootful And Rootless Podman ​

Enabling Podman Socket ​

Running Traefik In Podman ​

Configuring Traefik ​

Proxying Podman Containers Through Traefik ​

Summary ​