Phoenix with Apache on FreeBSD

I already had Apache set up to host some (mostly) static websites on a FreeBSD server. I thought I'd like to also have it perform TLS termination and name-based virtual hosting for a few of my Phoenix web apps too.

Installation

I knew I'd need Elixir as well as PostgreSQL (which seems to be a very good fit for Phoenix web apps). When I was using OpenBSD, I got into using rsync and tmux to deploy web apps. Sounds a little weird, but follow along and you'll get to see how it works. Anyway, I liked how that worked so I figured I'd install tmux and rsync too.

root@vultr1:~ # pkg install elixir \
	postgresql13-server postgresql13-client elixir rsync tmux

I initialized the database cluster, added a line for PostgreSQL to rc.conf, and started the PostgreSQL server. The initialization should be performed as the postgres user. I set the default encoding to UTF-8 since Elixir uses it natively.

root@vultr1:~ # su - postgres
$ initdb -A scram-sha-256 -D /var/db/postgres/data13 -E UTF8 -W
$ exit
root@vultr1:~ # echo postgresql_enable="YES" >> /etc/rc.conf
root@vultr1:~ # service postgresql start

Certificates

Before setting up Apache to do TLS termination for my apps, I needed certificates for them. I use certbot for requesting and renewing certificates from Let's Encrypt.

root@vultr1:~ # certbot certonly --webroot -w /usr/local/www/certbot/ \
    --cert-name skilman.com \
    -d skilman.com -d www.skilman.com
root@vultr1:~ # certbot certonly --webroot -w /usr/local/www/certbot/ \
    --cert-name co2guru.org \
    -d co2guru.org -d www.co2guru.org
root@vultr1:~ # certbot certonly --webroot -w /usr/local/www/certbot/ \
    --cert-name lofi.limo \
    -d lofi.limo -d www.lofi.limo

Apache Configuration

I figured I could use Apache's Proxy module along with the Proxy HTTP and Proxy WS Tunnel modules to do what I needed. I added a new macro to my Apache configuration for web applications. It assumes that the only WebSocket will be on /socket, which is how I set my apps up. But you could add additional paths or do something clever with the Rewrite module to catch all WebSocket upgrades and run them through the Proxy WS Tunnel module.

My web applications depend on the x-forwarded-proto and x-forwarded-host headers to know when and how to do their own HTTP to HTTPS redirection. The Proxy HTTP module automatically includes x-forwarded-host, but I added my own x-forwarded-proto header (depending on which VirtualHost handled the request) using the Headers module.

The first ProxyPass directive to match gets the request, so I added a directive before the others to catch requests for ACME authentication and prevent them from being proxied. I have Apache serve those tokens from a directory I set up for certbot to put them in.

I enable the rewrite engine so that a rewrite rule in the base server configuration can do www.example.com to example.com redirection.

# SO module
LoadModule headers_module libexec/apache24/mod_headers.so
LoadModule proxy_module libexec/apache24/mod_proxy.so
LoadModule proxy_http_module libexec/apache24/mod_proxy_http.so
LoadModule proxy_wstunnel_module libexec/apache24/mod_proxy_wstunnel.so

# Web applications

<Macro WebApplication $name $port>

<VirtualHost *:80>
ServerName $name
ServerAlias www.$name
RequestHeader set x-forwarded-proto http
ProxyPass /.well-known/acme-challenge !
ProxyPass /socket ws://localhost:$port/socket
ProxyPass / http://localhost:$port/
RewriteEngine on
</VirtualHost>

<VirtualHost *:443>
ServerName $name
ServerAlias www.$name
RequestHeader set x-forwarded-proto https
ProxyPass /socket ws://localhost:$port/socket
ProxyPass / http://localhost:$port/
RewriteEngine on
SSLCertificateFile ${cert_dir}/$name/fullchain.pem
SSLCertificateKeyFile ${cert_dir}/$name/privkey.pem
SSLEngine on
</VirtualHost>

</Macro>

Use WebApplication skilman.com 4210
Use WebApplication co2guru.org 4220
Use WebApplication lofi.limo 4230

Deployment

Before deploying each application, I created a PostgreSQL user and database for that application:

createuser -P -U postgres skilman
createdb -O skilman -U postgres skilman

On my development workstation, I made a deploy script that looks something like this. On the first deployment (or if the app is otherwise not running), the tmux line fails. But otherwise it will restart the app after the rsync finishes.

#!/bin/bash
host=${1:-scratch}
rsync -r config lib mix.exs mix.lock priv $host:skilman
ssh $host "tmux respawn-window -t skilman:1 -k"

To run the application on the server, I made a little script on the server. This script in turn creates a set of three scripts, each to be run in its own tmux window: the application itself, an interactive Elixir session attached to the app for administration and debugging, and a psql session connected to the app's database. The parent script creates a tmux session and starts windows in it running these three scripts as well as a shell in the application directory. Having this session already set up makes it very convenient to log into the server and perform debugging or administrative work.

#!/bin/sh
app_name=skilman
db_pass=***
cd ~/$app_name
touch tmux_server.sh
chmod 600 tmux_server.sh
cat > tmux_server.sh <<-EOF
	export MIX_ENV=prod
	export SECRET_KEY_BASE=***
	export HOST=skilman.com
	export REPO_URL=ecto://$app_name:$db_pass@localhost/$app_name
	export SENDGRID_API_KEY=***
	export EASYPOST_KEY=***
	export STRIPE_PUBLISHABLE_KEY=***
	export STRIPE_RESTRICTED_KEY=***
	mix deps.get
	mix phx.digest
	mix ecto.migrate
	elixir --sname $app_name@localhost -S mix phx.server
EOF
touch tmux_iex.sh
chmod 600 tmux_iex.sh
cat > tmux_iex.sh <<-EOF
	iex --sname $app_name-iex@localhost --remsh $app_name@localhost
EOF
touch tmux_psql.sh
chmod 600 tmux_psql.sh
cat > tmux_psql.sh <<-EOF
	export PGPASSWORD=$db_pass
	psql $app_name $app_name
EOF
tmux new-session -d -n shell -s $app_name
tmux set-option -g -t $app_name remain-on-exit on
tmux new-window -d -n server -t $app_name:1 sh tmux_server.sh
tmux new-window -d -n iex -t $app_name:2 sh tmux_iex.sh
tmux new-window -d -n psql -t $app_name:3 sh tmux_psql.sh

To start each app when the server boots, I added a line like this to user's crontab:

@reboot ~/skilman.sh

Backups

For backups I've been happy with tarsnap and figured I'd use it for this project as well. There's a FreeBSD port, so installing it was as simple as pkg install tarsnap. I wanted to run it as user and use a key-file and cache-directory in user's home directory. I generated a key, made a directory for the cache, and made a directory in which I could stage files I'd like to back up.

I built a little script to copy the files I wanted to back up into the staging directory and to dump the PostgreSQL databases I wanted to back up. Then it backs the staging directory up to tarsnap. Once the backup is complete, the script enumerates the existing backups and removes all but the newest ninety. Finally, it gets the timestamp from the newest backup, converts it to a human-friendly format and puts it into a file. I added a command to ~/.profile to display this file on login so I can easily see that my backups are being performed regularly. Of course I also perform a test-restore from time to time.

#!/bin/sh

# Stage backups
cp backup.sh skilman.sh co2_guru.sh lofi_limo.sh ~/backup_staging
PGPASSWORD=*** pg_dump \
	-f ~/backup_staging/skilman.sql \
	-U skilman skilman
PGPASSWORD=*** pg_dump \
	-f ~/backup_staging/co2_guru.sql \
	-U co2_guru co2_guru
PGPASSWORD=*** pg_dump \
	-f ~/backup_staging/lofi_limo.sql \
	-U lofi_limo lofi_limo
cp -Rp ~/lofi_limo/media ~/backup_staging

# Create Tarsnap archive
tarsnap --keyfile tarsnap.key --cachedir tarsnap_cache --no-print-stats \
	-cf $(date +%s) backup_staging

# Purge old Tarsnap archives
tarsnap --list-archives --keyfile tarsnap.key --cachedir tarsnap_cache \
		| sort -nr | sed '1,90d' | while read name ; do
	tarsnap --keyfile tarsnap.key --cachedir tarsnap_cache \
		--no-print-stats -df $name
done

# List archives and note most recent
latest_timestamp=`tarsnap --list-archives \
	--keyfile tarsnap.key --cachedir tarsnap_cache \
	| sort -nr | head -n1`
latest_date=`date -r "$latest_timestamp" +'%A %B %e'`
echo "Most recent backup $latest_date" > backup_status

To run the script every day, I added a @daily ~/backup.sh to user's crontab.

Wrapping Up

I hope that this article has been helpful to you. If this is the kind of thing that interests you, you may enjoy my other work. If you have any questions, corrections, or comments please drop me a line!

Aaron D. Parks
aparks@aftermath.net