atoll PDS

Running a server

Production deployment runbook

A step-by-step path from an empty server to a PDS federating on the live network, followed by the production checklist. The documents in this directory cover every feature and option in depth; this runbook is the ordered walkthrough. Commands assume a Linux host with systemd and a domain you control. Steps 1–8 build and verify the server in isolation; steps 9–10 touch the live network and are deliberately last.

Two facts shape the order of this runbook. First, Atoll refuses to boot in production without its core secrets, so configuration comes before the first start. Second, creating an account registers a did:plc identity whose rotation keys control it permanently, so live signup comes last, after every local check, and starts with a throwaway account.

1. Server, DNS, and prerequisites#

  • A host with a public IPv4/IPv6 address, systemd, and a non-root deploy user.
  • PostgreSQL 16+ (18 recommended; backups need matching client binaries on PATH — pg_dump 14 cannot dump an 18 server).
  • Elixir/OTP matching mix.exs on the build machine (build on the same OS/arch as the server, or on the server itself).
  • DNS A/AAAA records for the PDS hostname (for example pds.example.com) and, if you will host user handles, a wildcard for the handle domain (*.users.example.com) pointing at the same proxy.
  • Ports 80/443 reachable; the application itself listens only on localhost.

2. PostgreSQL#

sudo -u postgres createuser --pwprompt atoll
sudo -u postgres createdb --owner=atoll atoll_prod

Use TLS (?ssl=true in DATABASE_URL) whenever the database is not on the same host. The application needs an ordinary owner role; no extensions beyond plpgsql are required.

Optional asynchronous read replica#

Leave READ_DATABASE_URL unset to use the primary for all database access. To enable a separate read pool, add these settings alongside DATABASE_URL:

DATABASE_URL=ecto://atoll:PRIMARY_PASSWORD@primary.example.com/atoll_prod?ssl=true
READ_DATABASE_URL=ecto://atoll_reader:READER_PASSWORD@replica.example.com/atoll_prod?ssl=true
READ_POOL_SIZE=10

READ_POOL_SIZE defaults to POOL_SIZE, or 10 when neither is set. The reader uses the same ECTO_IPV6 setting as the primary, but has its own URL and credentials. Restart Atoll after changing the settings. Provision replication separately: Atoll does not create or manage PostgreSQL replicas. Use a role with SELECT access to the application tables and schema usage. Reader sessions set default_transaction_read_only=on, and the read repository exposes no Ecto insert/update/delete functions.

Ordinary Ecto reads (get, all, one, exists?, aggregates, reloads and preloads) use the reader. Writes use the primary. Transactions, row-locking reads, migrations, authentication/authorization checks, current account availability, signing-key custody, and record takedown checks retain primary consistency. Repository exports and other reads that depend on shared locks also stay on the primary; those locks cannot be moved to an asynchronous replica. Inventory, account search, and database metrics use explicit read transactions where safe.

Writes, ordinary primary transactions and raw primary SQL pin subsequent reads in the same BEAM process to the primary. This preserves read-after-write behavior within that process, including after a transaction commits. The pin does not expire during the process lifetime and does not propagate to spawned processes. Fresh requests/processes can see replication lag. The AppView read-after-write reconciliation path explicitly uses the primary even in a fresh process. Replica reads do not guarantee that a just-created account, handle, record or blob is immediately visible to other requests.

Only Atoll.Repo is in ecto_repos; migrate the primary and let PostgreSQL replicate the schema before serving reads. /health/ready probes both pools when the reader is configured. Reader failures are reported instead of silently rerouting reads to the primary. Existing database query metrics include both pools. To disable the reader, remove READ_DATABASE_URL and restart.

Application code continues to call Atoll.Repo. Use primary: true on an individual Ecto read or Atoll.Repo.with_primary(fn -> ... end) for a group of reads that must be current. Repo.query/3 and Repo.query!/3 are always primary; use Repo.read_query/3 or Repo.read_query!/3 only for audited read-only SQL. Repo.read_transaction/2 keeps a read callback and its streams on one connection; nested read callbacks join the outer transaction and rollback aborts that outer transaction. Use Repo.read_query!/3 for transaction-local settings within these callbacks. Writes, ordinary transactions and raw primary SQL are rejected inside read callbacks. Do not use read transactions for authorization or locking reads.

3. Build the release#

On the build machine, from a clean checkout of the exact commit you validated in CI:

MIX_ENV=prod mix deps.get --only prod
MIX_ENV=prod mix assets.deploy
MIX_ENV=prod mix release

Copy _build/prod/rel/atoll to the server (for example /opt/atoll/releases/<git-sha> with a /opt/atoll/current symlink). Record the git commit hash; recovery sets ask for it.

4. Generate and store the secrets#

mix phx.gen.secret            # SECRET_KEY_BASE
openssl rand -base64 32       # ATOLL_KEY_ENCRYPTION_KEY
openssl rand -base64 32       # ATOLL_SESSION_SIGNING_KEY
openssl rand -base64 32       # ATOLL_OAUTH_NONCE_SECRET
openssl rand -base64 32       # ATOLL_PDS_SIGNING_KEY (secp256k1 service key)

Also choose a strong ATOLL_ADMIN_PASSWORD for the operator endpoints.

Before anything else, put ATOLL_KEY_ENCRYPTION_KEY and ATOLL_PDS_SIGNING_KEY in your durable secret store (and an offline copy). The key-encryption key wraps every account's repository and PLC signing key: if it is lost, custody in the database is unrecoverable and every hosted identity is stranded. Back it up with the same care as the database, and keep old keys listed in ATOLL_PREVIOUS_KEY_ENCRYPTION_KEYS until a rewrap completes (see keys.md).

Store the environment in a root-owned file such as /etc/atoll/atoll.env with mode 0600.

5. Configure the environment#

Minimum viable production environment:

# Boot-required
PHX_SERVER=true
PORT=4000
DATABASE_URL=ecto://atoll:PASSWORD@localhost/atoll_prod
SECRET_KEY_BASE=...
ATOLL_KEY_ENCRYPTION_KEY=...
ATOLL_SESSION_SIGNING_KEY=...
PHX_HOST=pds.example.com
ATOLL_PDS_DID=did:web:pds.example.com
ATOLL_AVAILABLE_USER_DOMAINS=.users.example.com

# Required for a functioning PDS in practice
ATOLL_PDS_SIGNING_KEY=...        # signs /.well-known/did.json service identity
ATOLL_OAUTH_NONCE_SECRET=...     # OAuth (modern clients) is unusable without it
ATOLL_ADMIN_PASSWORD=...         # operator/admin endpoints
ATOLL_TRUSTED_PROXY_CIDRS=127.0.0.1/32   # your reverse proxy's address(es)

# Federation
ATOLL_APPVIEW_PROXY=did:web:api.bsky.app#bsky_appview
ATOLL_RELAY_URLS=https://bsky.network
ATOLL_RELAY_CRAWL_ENABLED=true
ATOLL_MOD_SERVICE_PROXY=did:plc:ar7c4by46qjdydhdevvrndac#atproto_labeler
ATOLL_REPORT_SERVICE_PROXY=did:plc:ar7c4by46qjdydhdevvrndac#atproto_labeler
ATOLL_IMAGE_CDN_URL_PATTERN=https://cdn.bsky.app/img/%s/plain/%s/%s@jpeg

# Signup policy (leave signup off until step 10)
ATOLL_SIGNUP_ENABLED=false
ATOLL_INVITE_CODE_REQUIRED=true

# Bind only to loopback when the reverse proxy shares the host
ATOLL_LISTEN_IP=127.0.0.1

Self-service signup refuses operational first labels (www, admin, mail, pds, cdn, ...) beneath the handle domains; tune the list with ATOLL_RESERVED_HANDLES (see docs/accounts.md).

Verify the moderation-service DIDs yourself before relying on them; service DIDs are operator configuration, not protocol constants.

Recommended from day one: ATOLL_METRICS_ENABLED=true (with monitoring from ops/prometheus), the background workers (ATOLL_BLOB_CLEANUP_ENABLED, ATOLL_ACCOUNT_CLEANUP_ENABLED, ATOLL_IDENTITY_REFRESH_ENABLED, ATOLL_EVENT_RETENTION_ENABLED with ATOLL_EVENT_RETENTION_SECONDS, ATOLL_OAUTH_KEY_CHECKS_ENABLED), and ATOLL_MEDIA_VALIDATION=images.

Optional subsystems, each documented under docs/: S3 blob storage (ATOLL_BLOB_STORAGE=s3 plus the ATOLL_S3_* settings), Redis-shared rate limits for multi-node deployments (ATOLL_RATE_LIMIT_BACKEND=redis, ATOLL_REDIS_URL), the Cloudflare email Worker (ATOLL_EMAIL_WORKER_URL/ATOLL_EMAIL_WORKER_TOKEN — without it email confirmation, password reset, and email-authorized PLC signing cannot send; a ready-to-deploy Worker lives in ops/email-worker), passkeys (ATOLL_PASSKEYS_ENABLED=true), verified PLC resolution (ATOLL_PLC_RESOLUTION_MODE=audit), quotas, cache TTLs, rate limits, ATOLL_PRIVACY_POLICY_URL, ATOLL_TERMS_OF_SERVICE_URL, and ATOLL_CONTACT_EMAIL.

6. Reverse proxy#

For Rocksky handles, set ATOLL_AVAILABLE_USER_DOMAINS=.rocksky.social and point wildcard DNS for *.rocksky.social at the proxy with HTTPS configured for those hosts. Preserve the original Host header when proxying to Atoll. The homepage redirects completed local accounts such as https://canary.rocksky.social/ to https://rocksky.app/profile/canary.rocksky.social with a non-cached HTTP 302. Only / redirects; /.well-known/atproto-did and the PDS API routes continue to work. Unknown handles and the configured PDS host keep the PDS landing page.

TLS terminates at the proxy; Atoll listens on plain HTTP behind it, and production releases enforce HTTPS themselves through the compile-time force_ssl in config/prod.exs (HSTS on, localhost excluded). The proxy must pass WebSocket upgrades (the firehose) and preserve client addresses. Caddy does both by default:

pds.example.com, *.users.example.com {
	reverse_proxy 127.0.0.1:4000
}

The wildcard site needs either a wildcard certificate (Caddy: DNS challenge plugin for your DNS provider) or on-demand TLS guarded by Atoll's GET /tls-check endpoint, which approves only the server host and completed hosted-handle hosts:

{
	on_demand_tls {
		ask http://127.0.0.1:4000/tls-check
	}
}

pds.example.com, *.users.example.com {
	tls {
		on_demand
	}
	reverse_proxy 127.0.0.1:4000
}

Every user handle host must serve both /.well-known/atproto-did and the XRPC routes. If you use nginx instead, enable proxy_http_version 1.1 with Upgrade/Connection headers for /xrpc/com.atproto.sync.subscribeRepos, forward X-Forwarded-For and X-Forwarded-Proto, and allow request bodies of at least 10 MB. List the proxy's address in ATOLL_TRUSTED_PROXY_CIDRS so rate limits see real client addresses.

7. systemd unit#

/etc/systemd/system/atoll.service:

[Unit]
Description=Atoll PDS
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
User=atoll
Group=atoll
EnvironmentFile=/etc/atoll/atoll.env
WorkingDirectory=/opt/atoll/current
ExecStartPre=/opt/atoll/current/bin/atoll eval "Atoll.Release.migrate()"
ExecStart=/opt/atoll/current/bin/atoll start
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now atoll
journalctl -u atoll -f

A missing required secret fails ExecStart immediately with a message naming the variable — that is the boot guard working, not a packaging problem.

8. Smoke-test locally (no federation yet)#

To export traces, logs and metrics to an OTLP collector, configure the service environment as described in OpenTelemetry before restarting.

All of these must pass before touching the live network:

curl -s https://pds.example.com/xrpc/_health          # {"version":"..."}
curl -s https://pds.example.com/health/ready           # {"status":"ok"}
curl -s https://pds.example.com/xrpc/com.atproto.server.describeServer
curl -s https://pds.example.com/.well-known/did.json   # service identity
curl -s -u admin:$ATOLL_ADMIN_PASSWORD https://pds.example.com/metrics | head
curl -s https://pds.example.com/robots.txt
# Firehose upgrade through the proxy (expect a successful WebSocket handshake):
curl -si -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: aaaaaaaaaaaaaaaaaaaaaa==' \
  'https://pds.example.com/xrpc/com.atproto.sync.subscribeRepos?cursor=0' | head -3

Also confirm a wildcard host answers: curl -s https://anything.users.example.com/xrpc/_health.

9. First live-network contact, read-only#

did:web:pds.example.com needs no registration — publishing /.well-known/did.json (step 8) is the whole ceremony. Confirm the PDS DID resolves from outside, then verify outbound reachability to the services you configured:

curl -s https://plc.directory/_health
curl -s https://api.bsky.app/xrpc/_health
curl -s https://bsky.network/xrpc/_health

Nothing has written to the network yet.

10. First account — throwaway first#

Account creation registers a did:plc identity at the directory. Do this first with an account you are prepared to abandon.

  1. Issue one invite code (keep ATOLL_INVITE_CODE_REQUIRED=true): mix atoll.invites.create on a deploy host, or POST /xrpc/com.atproto.server.createInviteCode with the admin credential.

  2. Set ATOLL_SIGNUP_ENABLED=true and restart.

  3. Create the account with any atproto client, or:

    curl -s https://pds.example.com/xrpc/com.atproto.server.createAccount \
  -H 'content-type: application/json' \
  -d '{"handle":"canary.users.example.com","password":"...","inviteCode":"..."}'

  4. Validate federation end to end:

    • https://plc.directory/<did> shows the new identity with your PDS endpoint.
    • com.atproto.identity.resolveHandle for the handle returns the DID, and https://canary.users.example.com/.well-known/atproto-did serves it.
    • Relay crawl: either wait for the periodic announcement or run mix atoll.relays.request_crawl; the durable audit records the outcome.
    • Log into the real Bluesky app against https://pds.example.com, post, upload an image, follow someone, open a custom feed, and check the profile appears on bsky.app within a minute. This exercises sessions, the AppView proxy, feed-generator tokens, and read-after-write against the live network.

  5. If a submission ends ambiguously, use the PLC journal tooling (mix atoll.accounts.reconcile_signup, mix atoll.plc.reconcile_*) described in keys.md rather than retrying blindly.

Only after the canary behaves for a while should real accounts, or a migration of an identity you care about, follow — and before any migration, run the backup and restore drill against this deployment (ops/backup).

Production checklist#

Boot and identity

  • [ ] CI green on the deployed commit; release built from that exact commit.
  • [ ] DATABASE_URL, SECRET_KEY_BASE, ATOLL_KEY_ENCRYPTION_KEY, ATOLL_SESSION_SIGNING_KEY, PHX_HOST, ATOLL_PDS_DID set (boot refuses without them); ATOLL_PDS_SIGNING_KEY, ATOLL_OAUTH_NONCE_SECRET, ATOLL_ADMIN_PASSWORD set.
  • [ ] ATOLL_KEY_ENCRYPTION_KEY and ATOLL_PDS_SIGNING_KEY in durable, offline secret storage; environment file root-owned, mode 0600; secrets absent from shell history and version control.
  • [ ] /.well-known/did.json and /xrpc/_health correct from the public internet, including a wildcard user host.

Transport

  • [ ] TLS at the proxy with a valid wildcard certificate (releases enforce HTTPS/HSTS at compile time, localhost excluded).
  • [ ] WebSocket upgrade verified through the proxy on subscribeRepos.
  • [ ] ATOLL_TRUSTED_PROXY_CIDRS lists exactly the proxy addresses.
  • [ ] Database connections use TLS if the database is remote.

Federation

  • [ ] ATOLL_APPVIEW_PROXY, ATOLL_RELAY_URLS + ATOLL_RELAY_CRAWL_ENABLED=true, mod/report service DIDs verified and configured, ATOLL_IMAGE_CDN_URL_PATTERN set.
  • [ ] Canary account created, visible on plc.directory, crawled by the relay, and usable from the official app (post, image, follow, custom feed).

Policy

  • [ ] Signup posture chosen: disabled, or enabled with invites required.
  • [ ] Repository and blob quotas reviewed (ATOLL_REPO_MAX_ACCOUNT_*, ATOLL_BLOB_MAX_ACCOUNT_*); rate limits reviewed; ATOLL_MEDIA_VALIDATION=images if you want structural image checks.
  • [ ] describeServer metadata set: privacy policy, terms, contact email.
  • [ ] Email Worker configured if accounts will use email confirmation, password reset, or email-authorized PLC operations.

Operations

  • [ ] Background workers enabled: blob cleanup, account cleanup, identity refresh, event retention (with a retention window), OAuth key checks.
  • [ ] ATOLL_METRICS_ENABLED=true; Prometheus scraping /metrics with the operator credential; alert rules and runbook from ops/prometheus loaded and firing to a real destination.
  • [ ] Nightly recovery sets via scripts/recovery_set.py under ATOLL_READ_ONLY=true quiescence (see ops/backup), stored encrypted off-host with the deployment revision and keyring reference recorded.
  • [ ] One full restore drill executed against THIS deployment's archive on an isolated target before real accounts depend on it.
  • [ ] journalctl log retention configured; logs confirmed free of secrets.
  • [ ] Upgrade procedure rehearsed: deploy new release directory, flip symlink, restart (migrations run via ExecStartPre); rollback is the previous symlink plus Atoll.Release.rollback/2 if a migration must be undone.
  • [ ] Multi-node only: ATOLL_RATE_LIMIT_BACKEND=redis with a shared Redis, one shared database, and every node quiesced together for backups.

Custody discipline (ongoing)

  • [ ] Master-key rotation uses mix atoll.keys.rotate_* with previous keys retained until mix atoll.keys.rewrap completes.
  • [ ] PLC journal health checked after any ambiguous directory submission (atoll.plc.verify, then the reconcile tasks as the signup-recovery decision tree in accounts.md directs).
  • [ ] Operator actions reviewed via the moderation audit history (mix atoll.moderation.history).

Type to search across every page.