Caddy and TLS certs


Caddy provides a reverse proxy with TLS termination for all internal services at the shop.
You must be on the shop LAN or on the shop VPN to access it. Working with certbot, we can get wildcard certs for all services.

Caddy runs as a container on the proxmox server on container named caddy with IP

Finally provides DNS with bind to match FQDN <-> TLS CN. See below to update DNS entries.



Assuming Ubuntu 22, following their install docs:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf '' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf '' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

systemctl enable caddy
systemctl start caddy


With the way proxmox does LXC containers, the normal install docs don't work. Instead we had to use pip and friends to achieve the same result:

sudo apt update && sudo apt install python3 python3-venv libaugeas0 python3-pip
sudo python3 -m venv /opt/certbot/
sudo /opt/certbot/bin/pip install --upgrade pip
sudo /opt/certbot/bin/pip3 install certbot requests
sudo ln -s /opt/certbot/bin/certbot /usr/bin/certbot
sudo ln -s /usr/bin/python3 /usr/bin/python

Then add a cronjob to check for renawals:

echo "0 0,12 * * * root /opt/certbot/bin/python -c 'import random; import time; time.sleep(random.random() * 3600)' && sudo /usr/bin/certbot renew -q" | sudo tee /etc/cron.d/certbot > /dev/null

Get the python script and make it executable. This uses the DNS service with some CNAME trickery:

mkdir -p /etc/letsencrypt/
curl -o /etc/letsencrypt/
chmod +x /etc/letsencrypt/
ln -s /usr/bin/python3 /usr/bin/python


Before trusting that the cronjob above will run, as root, make sure you test the renew command with a dry run. This will ensure you have everything installed and workine correcty:

/usr/bin/certbot renew --dry-run

The last time we rebuilt the caddy/certbot container we hit errors that went unnoticed until the cert expired. Running the above test would have exposed them had we tested them right at install time.

The errors were:

  • /usr/bin/env: ‘python’: No such file or directory - fixed by symlinking /usr/bin/python3 -> /usr/bin/python
  • The error was: expected /etc/letsencrypt/live/ to be a symlink - fixed by moving the files in /etc/letsencrypte/live/* and making them be symlinks to /etc/letsencrypte/archived/, then running certbot update_symlinks
  • ModuleNotFoundError: No module named 'requests' - fixed by instlling requests with pip3

First time cert generation w/ DNS update

You only have to do this ONCE!

On the Caddy box, initiate a request for * and domains, with a manual DNS validation:

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/ --preferred-challenges dns --debug-challenges -d \* -d

You'll then be prompted to create a DNS entry, something like:    CNAME

Create this by:

  1. SSH to
  2. sudo su -
  3. vim /etc/bind/master/
  4. edit serial number at top to be today's date
  5. add new line for above DNS entry
  6. restart DNS with rndc reload

Credentials are now stored in JSON in /etc/letsencrypt/acmedns.json. These are backed up in keepass just in case. Though you could go through above steps again if they're lost.

Back on the certbot box, hit return to continue the validation process. Certs should be created in /etc/letsencrypt/live/

To ensure the wildcard cert reloads every week, add a cronjob as the root user on the caddy box:

# restart caddy to reload certs once a week on 8.05 every sunday
5   8  *   *   0     /usr/bin/systemctl restart caddy

Set default cert in Caddy

In /etc/caddy/Caddyfile declare the top most host as shown below. All subsequent hosts will inherit this cert:

# this host just declared to define default cert all other hosts inherit
:443 {
   tls /etc/letsencrypt/live/ /etc/letsencrypt/live/
   root * /usr/share/caddy/

Adding service

Configure Caddy

Assuming you had a new service at called, you would:

  1. ssh into caddy box
  2. vim /etc/caddy/Caddyfile
  3. add new host entry (and see "Variations on Caddyfile entries" below). Because we declared a default host above, we can just add 3 lines which include the host and IP. It implicitly uses port 80 for IPv4 hosts: {
  4. restart caddy: systemctl restart caddy

Configure DNS Entry on

NOTE! - There is a wildcard CNAME entry for * to point to Caddy. You only need to make a DNS entry if you want it to NOT point to Caddy.

Set up new DNS entry:

  1. SSH into and sudo su - to become root
  2. vim /etc/bind/master/
  3. Add a new A recrod entry for your new service, looking at existing ones for a template.
  4. Modify the serial number (SOA) at top to be today's date + a unique 2 digit integer (it looks something like 2023090116; serial, todays date + serial # )
  5. Restart DNS with rndc reload.

Variations on Caddyfile entries

Step 3 above in "Configure Caddy" can have other options to support self signed certs and IPv6 hosts (or both!).

Self signed cert

We go from 3 lines to 9. The main difference is that we're telling it which IP with https:// and to ignore self signed certs with tls_insecure_skip_verify: {
   reverse_proxy {
      transport http {

IPv6 entry

Note the use of brackets around the IP [] and port at the end :80. {
        reverse_proxy [fd42:7c97:9426:8f29:216:3eff:fe0a:71c9]:80