Database Improvements

[charliemirabile]

In preparation for switching from raw sql queries to an orm, we should clean up the existing database schema and make sure the existing code works with that schema, so that the schema can be more easily translated into a strictly typed class.

Each commit in this pr involves changing the schema of the database which means that the the binary blob containing a gzip'd tar of an empty orbit db file in the tests must be updated. This script will be helpful when reviewing. It takes arguments of two git objects (commit shas, or expressions that evaluate to one like master or v0.1 or HEAD or HEAD^), and shows a diff between the schemas (both in a traditional line base diff, and a word diff) and also calls the sqldiff utility from sqlite to show if any values changed (no values should change at any point since the db is and should remain empty).

#!/bin/sh

tmp1=$(mktemp)
tmp2=$(mktemp)
git show $1:test.sh | sed -n "/'EOF'/,/EOF/p" | head -n -1 | tail -n +2 | xxd -r | tar -zxO orbit.db > $tmp1
git show $2:test.sh | sed -n "/'EOF'/,/EOF/p" | head -n -1 | tail -n +2 | xxd -r | tar -zxO orbit.db > $tmp2

diff -U3 --color=auto <(sqlite3 $tmp1 .schema) <(sqlite3 $tmp2 .schema)
wdiff -w "$(printf '\033[31m')" -x "$(printf '\033[0m')" -y "$(printf '\033[32m')" -z "$(printf '\033[0m')" <(sqlite3 $tmp1 .schema) <(sqlite3 $tmp2 .schema)

sqldiff $tmp1 $tmp2

rm $tmp1 $tmp2

[charliemirabile]

This script was helpful to me for updating the binary blob in the tests. Not needed for reviewing the changes, but useful for anyone else who will change the schema.

#!/bin/sh
set -ex

#create a new empty db from the schema using version of sqlite from alpine
podman run \
	-i \
	--rm \
	alpine:3.19 \
	sh -c '
		apk add sqlite >/dev/null 2>/dev/null
		sqlite3 db
		cat db
	' \
	< orbit/init-db.sql \
	> /tmp/orbit.db

trap 'rm /tmp/orbit.db' EXIT

#create hexdump of db. Use options suggested here https://reproducible-builds.org/docs/archives/ to make it idempotent
tar \
	--sort=name \
	--mtime='@0' \
	--owner=100 \
	--group=100 \
	--numeric-owner \
	--pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
	-C /tmp/ \
	-cz \
	orbit.db \
| xxd \
> /tmp/orbit.db.hex

trap 'rm /tmp/orbit.db /tmp/orbit.db.hex' EXIT

# delete current hexdump and replace with new one
# commands are:
# 1) seek to line after one matching /'EOF'/
# 2) delete from curr line to one before one matching /EOF/
# 3) read in hex file starting at line before the one matching /EOF/
# 4) write and quit
ed -s test.sh <<'EOF'
/'EOF'/+
.,/EOF/- d
- r /tmp/orbit.db.hex
wq
EOF

[charliemirabile]

this script however will be useful. It does a full cycle of bringing containers up/down rebuilding and testing. The real innovation here is that it knows how to wait until synapse is finished initializing before starting the tests.

#!/bin/sh
set -ex

podman-compose down || true # not a big deal if this fails since they might not be up
yes | podman volume prune #slightly scary in case you have other volumes you care about. FIXME: could name them explicitly
podman-compose build
podman-compose up -d
# wait until synapse is done initializing
podman-compose logs -f submatrix 2>&1 | sed '/Synapse now listening on TCP port 8008/ q'
./test.sh
podman-compose down