Operations runbooks

Step-by-step procedures for operating a standalone Graphorin server: backup, restore, upgrade, rollback and key rotation. Each runbook uses only shipped commands, states its preconditions, and says what proves it in CI. Scope: a single-node server (the supported deployment shape today - see Road to 1.0 for what HA/scale-out evidence is still owed).

The Storage page carries the command safety matrix (read-only vs write-lock-contending vs requires-a-stopped-server); this page assumes it.

What is on disk

Artifact	Default location	Notes
Main database	storage.path (./.graphorin/data.db; /data/data.db in the container)	Plus WAL sidecars data.db-wal / data.db-shm while a writer is up. Holds memory, sessions, checkpoints, triggers, auth tokens, idempotency state.
Audit log	audit.path (defaults to audit.db next to the main DB)	Always encrypted, separate hash chain.
Secrets	keyring, or the encrypted-file bundle (~/.graphorin/secrets.enc / mounted files)	The bundle is AES-256-GCM at rest - safe to copy as a file.
Config	--config path (/etc/graphorin/config.json in the container)	Plain file; keep it in your config management, not in backups of the data dir.

Backup

graphorin storage backup has two honest modes, selected by the config's storage.encryption.enabled:

• Plaintext store: an online, page-level backup - safe under a live writer (the server keeps running) and preserves rowids so the FTS5 mapping survives.
• Encrypted store: a consistent stopped-server byte copy. The cipher driver cannot key either side of an online page-level transfer, so the command checkpoints the WAL, proves no other holder is live (refusing with a live-writer error otherwise), byte-copies the file and verifies the copy's cipher integrity. The copy stays encrypted with the same key.

Either way it is the only supported backup - never use VACUUM INTO (it renumbers rowids and corrupts the FTS mapping on restore).

# Plaintext store - run on a schedule, the server may stay up.
# Encrypted store - run in the maintenance window with the server stopped.
graphorin storage backup /backups/data-$(date +%F).db --config /etc/graphorin/config.json

# Retention for the backups the server knows about:
graphorin storage cleanup-backups --config /etc/graphorin/config.json

# Reclaim free pages periodically (write-lock-contending, prefer quiet hours):
graphorin storage compact --config /etc/graphorin/config.json

The audit log is a separate database with its own passphrase. Two supported options:

# Online, append-friendly archival (JSONL, resumable by sequence):
graphorin audit export --to /backups/audit-$(date +%F).jsonl --config /etc/graphorin/config.json

or a plain file copy of audit.db while the server is stopped. For the secrets bundle, a plain file copy is enough - it is encrypted at rest.

Treat data.db + audit.db + the secrets bundle as one backup set: restoring them from different points in time works, but audit references and token state may skew across the gap.

Restore

Preconditions: a backup file produced by storage backup; the server stopped.

1. Stop the server.
2. Move the damaged database AND its sidecars out of the way: data.db, data.db-wal, data.db-shm (stale sidecars from a different database generation must not be replayed into the restored file).
3. Copy the backup into place as storage.path and restore tight permissions: chmod 0600 data.db.
4. Start the server. Migrations are idempotent and checksummed - a backup taken on an older patch/minor upgrades itself on boot.
5. Verify: /v1/health answers ok, and graphorin doctor --all --config ... is clean.

Caveats:

• Backups are point-in-time. Erasure/deletion requests honored after the backup was taken must be repeated after a restore.
• Raw auth tokens minted after the backup point no longer verify (their hashes are not in the restored DB); tokens minted before it work again. graphorin token list shows what the restored state believes.
• A fresh audit.db starts a new chain if you do not restore the old one alongside; the old chain stays independently verifiable from its own backup.

The weekly Docker smoke workflow runs this drill end-to-end: boots the shipped image (encrypted store), mints a token, stops the server, takes the stopped-server backup, destroys the container AND the data volume, restores the backup into a fresh volume, reboots, and asserts the token survived.

Upgrade

Graphorin releases are lockstep: upgrade every @graphorin/* package to the same version at once.

1. Read the target version's section in Migration (pre-1.0) - pre-1.0 minors may carry breaking changes; patches never do.
2. Take a backup first (see above). The schema is forward-only, so this backup IS your rollback path.
3. Install the new lockstep set (pin exact versions in your manifest/lockfile; for the container image, rebuild from the new tag).
4. Apply migrations offline: graphorin migrate --config ... on the stopped server, or let the server migrate itself on first boot - both run the same atomic, idempotent runner. Read-only CLI commands against a newer-schema database refuse rather than auto-upgrade, so a half-rolled fleet fails loud, not quiet.
5. Start, then verify /v1/health and graphorin doctor --all.

Rollback

There is deliberately no schema downgrade. Rolling back a release:

1. Stop the server.
2. Reinstall the previous lockstep version set.
3. Restore the pre-upgrade backup (runbook above) - this is why step 2 of Upgrade is not optional.
4. Start and verify.

Data written between the upgrade and the rollback lives only in the rolled-back-from database; session exports taken from it remain readable by the older release within the N-2 export compatibility band.

Key rotation

Four distinct keys, four runbooks - they do not substitute for each other:

Database passphrase (encrypted store)

# Requires a stopped server; passphrases travel as SecretRef URIs, never argv:
graphorin storage rekey \
  --old-passphrase-from file:/run/secrets/graphorin/db-passphrase \
  --new-passphrase-from file:/run/secrets/graphorin/db-passphrase-next \
  --config /etc/graphorin/config.json

Then update the config's storage.encryption.passphraseRef source to serve the new value and start the server. The rekey verifies cipher integrity before reporting success.

Server pepper (token hashing)

Only token hashes are stored, keyed by the pepper - a new pepper invalidates every existing token, and nothing can re-hash them. The order matters:

# 1. Write the new pepper into the secrets source the config points at
#    (auth.pepperRef), e.g.:
openssl rand -hex 32 | graphorin secrets set graphorin_server_pepper --from-stdin

# 2. Re-issue every active token under the new pepper:
graphorin token rekey --config /etc/graphorin/config.json

# 3. Redistribute the freshly printed raw tokens to clients.

A live server's verifier cache honors old tokens for up to ~60 s after rotation - plan the redistribution window accordingly.

Secrets bundle passphrase (encrypted-file store)

# Re-encrypts the whole bundle (fresh KDF salt), values unchanged:
GRAPHORIN_MASTER_PASSPHRASE='<current>' graphorin secrets rekey \
  --secrets-source encrypted-file \
  --new-passphrase-from env:GRAPHORIN_NEW_MASTER_PASSPHRASE

Afterwards start supplying the new value as GRAPHORIN_MASTER_PASSPHRASE. Sources without a bundle passphrase (keyring, env, memory) answer exit code 2 (unsupported).

Individual secret values

graphorin secrets rotate <key> --from-stdin

Same store write as set, surfaced separately so audit logs distinguish a rotation from an initial write. Rotate the deployment pepper and provider API keys on your normal credential schedule.

Scaling model

SQLite is a single-writer store, and the architecture is honest about it: exactly one server process owns the database file. WAL mode keeps concurrent reads cheap inside that process; a second process on the same file is refused by the write lock, not silently corrupted.

What that means operationally:

• Scale up, not out. More CPU/RAM on the one node; the k8s template's replicas: 1 + strategy: Recreate is a design statement, not a placeholder.
• Scale by partition, not by replica. Multiple tenants or workloads = multiple server instances, each with its own volume, secrets, and tokens. Nothing is shared, so nothing needs consensus.
• Availability = fast restore, not failover. The backup/restore runbook above plus the crash-resume behaviour below are the availability story today; crash recovery time is budgeted and measured weekly (see What CI proves). A shared-backend HA topology would need a different store backend and is tracked on the road to 1.0.
• Agent instances are single-flight. One run at a time per registered instance (409 agent-busy otherwise) - deploy an instance pool sized to your target concurrency (see the production golden path).

Load and soak SLOs

The weekly soak workflow (soak.yml) drives sustained, paced load through the full server turn path - auth, agent runtime, provider streaming, run tracking, SQLite - using a deterministic stub provider, and fails the run when any published budget is violated:

SLO	Budget
Transport errors / 5xx / any non-200	zero
Latency p95 (full POST /v1/agents/:id/run turn, loopback, stub provider)	< 1000 ms
Server RSS peak	< 1 GB
Memory growth	last-quarter mean RSS < 2x first-quarter mean + 64 MB

The budgets are deliberately generous - shared CI runners are noisy neighbours; the gate exists to catch order-of-magnitude regressions and leaks, not 10% drift. The driver paces to ~300 requests/second (still far above real LLM-bound traffic) rather than hammering at max throughput: run-tracker bookkeeping retains terminal records for five minutes by design, so memory scales with request RATE, and an unpaced stub can push it anywhere.

Published soak runs

Every run below is a public GitHub Actions run of soak.yml on an ubuntu-latest runner (2 vCPU class), reproducible via Actions -> "Server soak (stub provider)" -> Run workflow with the listed inputs. The weekly schedule runs the 600 s variant; the 3600 s row is the long-soak envelope, re-dispatched after operationally significant changes rather than weekly.

Date	Duration	Concurrency / target rps	Requests (all 200)	p50 / p95 / p99	RSS first-quarter -> peak	Verdict
2026-07-22	3600 s	8 / 300	1,101,486	2.5 / 4.1 / 6.1 ms	234 -> 290 MB	PASS
2026-07-22	300 s	8 / 300	91,534	2.6 / 4.3 / 6.4 ms	156 -> 225 MB	PASS

The hour-long run is the long-soak envelope: over a million full agent turns with zero non-200s, flat tail latencies, and last-quarter mean RSS (280 MB) within 20% of the first quarter - the run-tracker's five-minute retention window plateaus exactly as designed instead of growing with total request count.

Real-provider soak

The stub rows above prove the server; this run proves the same turn path against a LIVE cloud provider - TLS, streaming, retry middleware, real latency and error behaviour - paced at realistic LLM-bound traffic. It is maintainer-gated (costs real money, needs OPENAI_API_KEY), runs from a workstation rather than CI (the repository deliberately carries no provider secret), and reproduces with scripts/soak-real.sh; the compose module is .github/smoke/soak-real.app.mjs (agent pool over the openai-compatible adapter wrapped in withRetry, output capped at 128 tokens/turn).

Date	Subject	Duration	Target rps	Requests (all 200)	p50 / p95 / p99	RSS quarters -> peak	Verdict
2026-07-22	gpt-4.1-nano (api.openai.com)	3600 s (+ drain)	2	6,322	831 / 1,786 / 5,347 ms	94 -> 97 -> 132 MB	PASS

SLO envelope for this variant: zero non-200/transport errors, p95 < 20 s (cloud latency, not the stub's 1 s), same RSS ceiling and growth bound. One turn peaked at ~16.7 min: the retry middleware absorbed a provider-side stall and the request still returned 200 with every concurrent request unaffected (p99 stayed at 5.3 s) - which is exactly the behaviour the middleware exists to prove. Achieved rate was 1.5 rps against the 2 rps target: pacing slots are skipped, not queued, while a worker waits out a slow turn.

What CI proves

• Weekly Docker smoke: image build, boot with the shipped config (encrypted store + token auth), online backup, full destroy-and-restore drill, health + token survival - plus an image vulnerability gate on fixable high/critical findings.
• Crash-resume drill (same workflow): a workflow run parked on a durable 15 s timer survives docker kill (SIGKILL - no shutdown hooks) and completes after restart with NO operator action; the timer daemon's post-boot sweep fires the due timer read back from SQLite. Recovery time is measured and gated on every run, both clocks starting at the SIGKILL (so container teardown + boot are included): SIGKILL to /v1/health OK under 60 s, SIGKILL to the parked run completed under 150 s (measured 2 s / 14 s on a shared ubuntu-latest runner, 2026-07-23, run 30005490068). Budgets are generous for the same shared-runner reason as the soak SLOs; each run's measured numbers land in its step summary.
• Weekly soak: the SLO table above, enforced.
• Per-PR template validation: the Kubernetes manifest under kubeconform -strict; the systemd unit under systemd-analyze verify (any warning fails) plus an offline security-exposure score gated below 5.
• Package test suites: online backup mode/permissions, encrypt + rekey + swap live-writer guards, token rekey semantics, secrets bundle rekey round-trip (old passphrase must fail afterwards), migration idempotence + checksum tamper refusal, export schema-band enforcement, multi-process checkpoint durability.

Not covered yet

HA topologies with automatic failover, rolling upgrades across replicas, and chaos/failure-injection beyond the SIGKILL drill are not yet proven - they are the operational tail tracked on the road to 1.0. Until then, size a single node with Performance & scale and treat the restore runbook as the availability story.