Deployment

Two images, one chart. The API image carries the UI; the worker only dials the API. For a single Docker host see Install with Docker Compose. Both are distroless static, non-root (65532), read-only, and satisfy the restricted Pod Security Standard.

Artifact	Where
ghcr.io/lusoris/snatcharr-api	Go control plane + embedded SPA (HTTP 8080, gRPC 9090)
ghcr.io/lusoris/snatcharr-worker	Rust snatch-worker (HTTP 8081: /livez, /readyz, /metrics)
oci://ghcr.io/lusoris/charts/snatcharr	Helm chart

Every image and the chart are signed with cosign (keyless, GitHub OIDC) and carry a SLSA build provenance attestation in the registry. The GitHub release holds the chart archive, an SPDX SBOM per image, a checksum file snatcharr-<version>-SHA256SUMS over those files, and snatcharr-<version>-provenance.sigstore.json: one provenance attestation that names every file of the page, the checksum file included, as its subject. Tags are semver only: there is no latest.

Verify an artifact before you deploy it. The check needs only cosign (2.4 or later) and no forge login. It requires the signing certificate to name one of this repository's release workflows, run on the release tag at the commit of that tag, so an artifact built from any other commit is refused. After 0.2.1 the images and the chart are built, signed and attested in .github/workflows/release-build.yml, a reusable workflow that only builds and restores no cache (SLSA Build Level 3); the files of the release page and every earlier release are signed by .github/workflows/release.yml. The check accepts these two workflows and no other. Look the commit up once:

git ls-remote https://github.com/lusoris/SnatchArr 'refs/tags/v0.2.1^{}' refs/tags/v0.2.1

The last line is the commit (an annotated tag prints two lines; the ^{} one is the commit). Then check both images and the chart in the registry:

COMMIT=<commit of the tag>
tools/release/verify.sh ghcr.io/lusoris/snatcharr-api:0.2.1 lusoris/SnatchArr refs/tags/v0.2.1 "$COMMIT"
tools/release/verify.sh ghcr.io/lusoris/snatcharr-worker:0.2.1 lusoris/SnatchArr refs/tags/v0.2.1 "$COMMIT"
tools/release/verify.sh ghcr.io/lusoris/charts/snatcharr:0.2.1 lusoris/SnatchArr refs/tags/v0.2.1 "$COMMIT"

and the files of the release page, downloaded from the web page or with gh release download into one directory:

tools/release/verify-page.sh page 0.2.1 lusoris/SnatchArr refs/tags/v0.2.1 "$COMMIT"

verify.sh prints the identity and commit it verified and runs these commands, which also work without a checkout:

cosign verify ghcr.io/lusoris/charts/snatcharr:0.2.1 \
  --certificate-identity-regexp '^https://github\.com/lusoris/SnatchArr/\.github/workflows/release(-build)?\.yml@refs/tags/v0\.2\.1$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-github-workflow-sha "$COMMIT"
cosign verify-attestation --new-bundle-format --type https://slsa.dev/provenance/v1 \
  ghcr.io/lusoris/charts/snatcharr:0.2.1 \
  --certificate-identity-regexp '^https://github\.com/lusoris/SnatchArr/\.github/workflows/release(-build)?\.yml@refs/tags/v0\.2\.1$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-github-workflow-sha "$COMMIT"

For a file of the page, verify-page.sh checks the checksum file, refuses a file it does not list, and runs cosign verify-blob-attestation --new-bundle-format --bundle snatcharr-<version>-provenance.sigstore.json with the same identity flags on each file.

The release workflow (.github/workflows/release.yml, job verify) runs the same checks on both images, the chart and the page, and fails the release when one is not covered. Before that it proves the refusals: an unattested chart, another commit, another git ref and another repository, and a prefix of the git ref or a ref whose dots would match any character (tools/release/verify-selftest.sh), and for the page a wrong commit and a changed chart with a rewritten checksum file.

Older releases: the release page carries checksum file and provenance bundle from 0.2.1 on; earlier pages are not covered. 0.2.0 published nothing (no images, chart or page files; its release notes say so): use 0.2.1, which carries the same changes. The chart carries an attestation from 0.1.5 on; the charts up to 0.1.4 are signed only. 0.1.1 was rebuilt by hand from main: verify it with refs/heads/main and commit e0b3e17b1fd0cc81ed8756e43da2bb690fe3e53e. A rebuild runs on the tag it builds (gh workflow run release.yml --ref v0.2.1 -f tag=v0.2.1); the workflow refuses to start on any other ref.

Helm

Requirements: Kubernetes 1.29+, and the CloudNativePG operator unless you bring your own Postgres.

helm install snatcharr oci://ghcr.io/lusoris/charts/snatcharr \
  --namespace snatcharr --create-namespace \
  --set config.publicUrl=https://snatcharr.example.com \
  --set api.ingress.enabled=true \
  --set 'api.ingress.hosts[0].host=snatcharr.example.com' \
  --set 'api.ingress.hosts[0].paths[0].path=/' \
  --set 'api.ingress.hosts[0].paths[0].pathType=Prefix' \
  --set networkPolicy.enabled=true

The first visit runs the setup wizard. helm status prints the port-forward command when no Ingress or HTTPRoute is enabled.

When exposing the API behind an Ingress or reverse proxy, configure config.trustedProxies (or APP_SNATCHARR_PROXY_TRUSTED) with the proxy or ingress controller CIDRs. The API uses this list to trust X-Forwarded-For and resolve the client IP. Without trusted proxies, all forwarded requests appear to arrive from the proxy's IP address, so 5 failed login attempts by any client will lock out the proxy IP and block all users from logging in.

CI installs the chart on every pull request that touches deploy/ or a Dockerfile (make e2e-chart-install, tools/e2e/chart_install.sh): the defaults on CloudNativePG and an upgrade from the last published chart, each until the API and worker Deployments are Available and /readyz answers 200, and the upgrade must keep the generated Secret. A planted install whose Secret holds the published development crypto key must be refused first, with the API's error named.

What the chart renders

Resource	Notes
Deployment -api	probes on /startupz, /livez, /readyz (readiness: Postgres and the leader gate, which is down while the election fails or a periodic loop failed to start); /tmp is a memory-backed emptyDir; replicaCount may exceed 1, periodic work runs on the elected replica
Deployment -worker	dials <release>-api:9090; liveness probe on /livez (port worker.httpPort, 8081), which fails once no API call succeeded for worker.staleSecs (180 s); readiness probe on /readyz, which also fails until the first successful API call (a lease long-poll returns within worker.leaseWaitSecs), so the Deployment is Available only once its workers reached the API; /metrics there counts failed API calls per call in snatcharr_worker_api_failures_total; optional HPA 1..4 on CPU, or a KEDA ScaledObject on queued runs (below)
Service -api	http 8080, grpc 9090 (h2c)
Ingress or HTTPRoute	api.ingress.* or api.httpRoute.*
CNPG Cluster -db	postgres.cnpg.*; the API reads the DSN from the operator's -db-app Secret
Secret -secrets	crypto key, CSRF secret, API-key HMAC secret, worker token
NetworkPolicy	worker may only reach the API, networkPolicy.arrCidrs and networkPolicy.arrPeers (both empty allow no *arr egress, never all of it); gRPC only from the worker
ServiceMonitor	serviceMonitor.enabled scrapes /metrics on the API
PodDisruptionBudget	per component, opt-in

The image tags default to the chart's appVersion, so chart and images move together. A pinned release tag (image.api.tag, image.worker.tag) older than the templates need fails the render: the worker probes need a worker image newer than 0.2.1. The minimum per component lives in snatcharr.minImageVersion (templates/_helpers.tpl); tests/image_test.yaml holds the planted case, chart 0.1.8 with worker image 0.1.5. A tag that is no release, such as a commit build, is not compared.

Worker autoscaling

The worker runs worker.replicaCount replicas unless one of two autoscalers owns the count, between worker.autoscaling.minReplicas and maxReplicas (1..4):

• worker.autoscaling.enabled=true: a HorizontalPodAutoscaler on CPU (targetCPUUtilizationPercentage, 70). A worker waits on *arr most of the time, so CPU rises late.
• worker.autoscaling.keda.enabled=true: a KEDA ScaledObject (templates/worker-scaledobject.yaml) on queued runs. Prerequisites: KEDA in the cluster and a Prometheus that scrapes the API /metrics (for example serviceMonitor.enabled=true with the Prometheus Operator). Set worker.autoscaling.keda.serverAddress to that Prometheus. The default query is max(snatcharr_runs{status="queued",namespace="<release namespace>"}): the elected API replica exports the gauge every 30 seconds, and max keeps a stale series of a former leader from being added. keda.query replaces it, for example when your Prometheus labels the namespace differently. The worker count targets keda.threshold queued runs per replica, by default worker.concurrency. KEDA reads Prometheus, never the database, so it needs no database credentials.

The two are exclusive: KEDA creates its own HorizontalPodAutoscaler, and the chart refuses to render both. With either one the Deployment leaves replicas to the autoscaler.

make kind-keda-proof (tools/e2e/keda_scale.sh, after make kind-up, inside the incus VM only) installs KEDA and a minimal Prometheus (deploy/kustomize/kind/prometheus.yaml) and applies the chart with deploy/kustomize/kind/values-keda.yaml. With 50 runs queued it first requires the worker to stay at one replica while the query is planted to read 0, then to scale up within 120 seconds on the chart's query, and back to one replica within 600 seconds after every run is cancelled.

Metrics

The API and worker both serve Prometheus metrics on /metrics. See the comprehensive Metrics and Observability reference for metric names, types, labels, scrape configuration (ServiceMonitor and PodMonitor), Grafana dashboard setup, and leader-only domain state gauge behaviors.

NetworkPolicy and in-cluster *arr apps

networkPolicy.arrCidrs allows egress by IP range. That reaches *arr apps outside the cluster, but some CNIs never apply CIDR rules to traffic between pods they manage: Cilium states that "CIDR rules do not apply to traffic where both sides of the connection are either managed by Cilium" (its layer 3 policy docs). If your *arr apps, Seerr, Prowlarr or download clients run as pods in the same cluster, select them in networkPolicy.arrPeers:

networkPolicy:
  enabled: true
  arrPeers:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: media

An entry without ports allows every port of the pods it selects. To allow one port per app, give the entry a ports list; the chart then renders an egress rule of its own for it:

networkPolicy:
  enabled: true
  arrCidrs: []
  arrPeers:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: lidarr
      ports:
        - port: 8686
          protocol: TCP

A podSelector without a namespaceSelector selects pods in the release namespace only. Each ports entry needs a port (number or port name) and may set protocol (TCP by default) and endPort. arrPeers accepts podSelector, namespaceSelector and ports only; IP ranges belong in arrCidrs, which always excludes the cloud metadata address and allows every port. A rule for one port of an IP range belongs in networkPolicy.extraEgress.

Secrets

The chart needs four secrets. Leave them empty and Helm generates them once, reusing the existing Secret on upgrades (helm.sh/resource-policy: keep). Tools that render without cluster access, such as ArgoCD, cannot reuse them, so create the Secret first:

kubectl -n snatcharr create secret generic snatcharr-secrets \
  --from-literal=crypto-key="$(openssl rand -hex 32)" \
  --from-literal=csrf-secret="$(openssl rand -hex 32)" \
  --from-literal=apikey-secret="$(openssl rand -base64 36)" \
  --from-literal=worker-token="$(openssl rand -base64 36)"

and set secrets.existingSecret=snatcharr-secrets. The crypto key encrypts every *arr API key and download-client password at rest; losing it means re-entering them.

The Helm chart generates a random bootstrap secret (setup-secret) into the Secret by default, locking POST /auth/setup on new clusters. Retrieve it for the first-run setup wizard:

kubectl -n snatcharr get secret snatcharr-secrets -o jsonpath="{.data.setup-secret}" | base64 -d; echo

If supplying an existingSecret, add --from-literal=setup-secret="$(openssl rand -base64 36)" (or set secrets.setupSecret). When existingSecret is used without setup-secret, the deployment marks the environment variable optional so existing clusters continue running without breaking.

Postgres

postgres.cnpg.enabled=true (default) renders a CloudNativePG Cluster with postgres.cnpg.instances replicas and postgres.cnpg.storage.size per instance; postgres.cnpg.extraSpec is merged verbatim for backups, affinity or parameters, and postgres.cnpg.bootstrap replaces the default initdb bootstrap, for a restore (see Backup and restore). For an external database set postgres.cnpg.enabled=false and point postgres.external.dsnSecretRef at a Secret holding the DSN. Only the API connects to Postgres.

sslmode=require in the DSN encrypts the connection without checking the server certificate. To verify it against the cluster CA (verify-full), mount the CA and point the API at it. The settings override the DSN, and the CA file is re-read for every new connection, so a rotated CA applies without a restart. With CloudNativePG the CA lives in the <cluster>-ca Secret; mount only its ca.crt, never ca.key:

config:
  extraEnv:
    - name: APP_DB_SSL_MODE
      value: verify-full
    - name: APP_DB_SSL_ROOTCERT
      value: /etc/snatcharr/db-ca/ca.crt
api:
  extraVolumes:
    - name: db-ca
      secret:
        secretName: pg-cluster-ca
        items:
          - key: ca.crt
            path: ca.crt
  extraVolumeMounts:
    - name: db-ca
      mountPath: /etc/snatcharr/db-ca
      readOnly: true

verify-full also checks the host name, so the DSN must name a host on the server certificate, such as CloudNativePG's <cluster>-rw.<namespace>.svc.

The schema migrations that run at start use the same settings: a migrator that inherits APP_DB_DSN also takes the APP_DB_SSL_* values and, when set, the password from the db.password_file config key. A path in APP_DB_SSL_* must not contain a space. Before 0.1.3 the migrator ignored these settings and the API crash-looped on x509: certificate signed by unknown authority; on 0.1.2 and older, add sslrootcert=<path> to the DSN itself. An explicit APP_DB_MIGRATE_DSN is used exactly as given: it carries its own password and TLS parameters. TestMigratorConnection in api/internal/app checks both rules against a real server.

Backup and restore

With CloudNativePG, back up through the Barman Cloud plugin (the operator's in-tree barmanObjectStore is deprecated). Install cert-manager and the plugin in the operator's namespace as the plugin's installation docs describe, create an ObjectStore for your S3 bucket, and turn on WAL archiving in the chart:

postgres:
  cnpg:
    extraSpec:
      plugins:
        - name: barman-cloud.cloudnative-pg.io
          isWALArchiver: true
          parameters:
            barmanObjectName: snatcharr-backups

Take base backups with a ScheduledBackup or an on-demand Backup of method: plugin. Keep a copy of the <release>-secrets Secret with the backups: its crypto key decrypts the *arr API keys and download-client passwords in the database, and a restored database without it means re-entering them. The chart keeps this Secret when the release is uninstalled.

To restore, install the chart with a recovery bootstrap instead of initdb, pointing an external cluster at the store and at the serverName of the lost cluster (by default its name, <release>-db):

postgres:
  cnpg:
    bootstrap:
      recovery:
        source: backup
        database: snatcharr
        owner: snatcharr
    extraSpec:
      externalClusters:
        - name: backup
          plugin:
            name: barman-cloud.cloudnative-pg.io
            parameters:
              barmanObjectName: snatcharr-backups
              serverName: snatcharr-db

database and owner make the operator create the <release>-db-app Secret the API reads its DSN from. The recovered cluster archives nothing until you add the plugins entry again; give it a new serverName parameter (for example snatcharr-db-restored) so it does not write into the archive it was restored from. A bootstrap key inside extraSpec is refused, since it would repeat the chart's own.

make e2e-cnpg-backup-restore runs these steps in CI on a kind cluster with the operator, cert-manager, the plugin and a RustFS store (tools/e2e/cnpg_backup_restore.sh). It completes the setup, takes a base backup, uninstalls the release and its database, and restores. It then checks that every table reads back unchanged, that archiving runs again under the new serverName, and that the API starts with the setup complete and the old password. A planted reinstall without the recovery bootstrap must be refused first.

Postgres major upgrade

CI, the release checks and the dev compose run Postgres 18. An installed database keeps the major it was initialised with until you move it; SnatchArr itself needs no change for the move, only a restart onto the upgraded database, after which its migrations bring the schema to the current version.

CloudNativePG (postgres.cnpg.*): the operator upgrades in place from CloudNativePG 1.26.0 on (upstream docs/src/postgres_upgrades.md, "Offline In-Place Major Upgrades"). Set postgres.cnpg.imageName to a Postgres 18 operand image built on the same OS distribution as the running one and apply the chart. The operator then:

• stops every instance, replicas included, so the database is down for the whole upgrade;
• runs a job named after the primary with the suffix -major-upgrade, which calls pg_upgrade --link;
• restarts the primary and clones the replicas again, which takes long on a large database.

If the job fails, set imageName back to the previous image: the operator removes the failed job and starts the old major again, with the data unchanged. With an empty imageName the operator's default image applies, and a new operator release can change its major; pin imageName to control when the upgrade happens. Afterwards run ANALYZE (statistics are not carried over) and take a new base backup: point-in-time recovery does not cross the major upgrade. With the Barman Cloud plugin, change serverName together with imageName so the new timeline does not write into the old archive. Postgres 17.0 to 17.5 refuse the upgrade while max_slot_wal_keep_size is set to anything but -1: upgrade to 17.6 or later first.

With the chart the whole upgrade is one value change:

helm upgrade snatcharr oci://ghcr.io/lusoris/charts/snatcharr --reuse-values \
  --set postgres.cnpg.imageName=ghcr.io/cloudnative-pg/postgresql:18.6-system-trixie
kubectl wait cluster/snatcharr-db --timeout=30m \
  --for=jsonpath='{.status.pgDataImageInfo.majorVersion}'=18

make e2e-cnpg-pg-upgrade runs this in CI on a kind cluster with the operator. It installs the chart on 17.10-system-trixie, completes the setup and changes the image to 18.6-system-trixie. It then checks that the operator reports major 18, that every table reads back unchanged, that the migration version is at least 7, and that the API starts with the setup complete and the old password. A planted chart upgrade without the image change must be refused first (tools/e2e/cnpg_pg_upgrade.sh). CI runs it on the API and worker images built from the checkout (UPGRADE_API_IMAGE, UPGRADE_WORKER_IMAGE), because a release pull request pins an appVersion whose images are not published yet; without them the test runs the released images of the chart's appVersion.

External Postgres (postgres.external.dsnSecretRef): dump and restore with the client tools of the new major. Stop the API first, so nothing writes during the dump (kubectl scale deployment <fullname>-api --replicas=0; the full name is the release name when it contains snatcharr, else <release>-snatcharr), then run against the old and the new server:

pg_dump --format=custom --dbname="$OLD_DSN" \
  | pg_restore --no-owner --exit-on-error --dbname="$NEW_DSN"

Point the DSN Secret at the new server and scale the API up again. The new database must exist and be owned by the role in NEW_DSN. TestPostgresMajorUpgradeByDumpRestore in api/internal/store runs exactly this command from a Postgres 17 database holding a full 0.1.8 dataset into Postgres 18, then migrates and reads every row back.

Homelab compose (deploy/homelab/compose.yaml): dump with the old image, move the old data directory aside inside the volume, start the new image and restore. The commands are in install-compose.md, and make e2e-compose-pg-upgrade runs them from a Postgres 17 install with a data read-back.

Dev compose (deploy/compose/docker-compose.yml): the volume is mounted at /var/lib/postgresql, where the 18 image keeps its data in a per-major directory. A volume first initialised by a 17 image does not start on 18: the container exits with Error: in 18+, these Docker images are configured to store database data in a format which is compatible with "pg_ctlcluster" and points at the old data it found. Dev data is disposable: docker compose -f deploy/compose/docker-compose.yml down --volumes removes the volume, and the next make dev initialises it on 18.

Configarr

Already running Configarr? Mount the same files and SnatchArr imports every instance from them (linked mode):

configarr:
  enabled: true
  existingConfigMap: configarr-config   # key config.yml
  existingSecret: configarr-secrets     # key secrets.yml (optional)

Coexistence with Cleanuparr

SnatchArr no longer integrates Cleanuparr: operators who keep Cleanuparr must not let it act on the same queues as SnatchArr's queue management.

ArgoCD

deploy/argocd/ holds an AppProject and an Application that track main with automated prune and self-heal, server-side apply and retries. Override values under spec.source.helm.valuesObject; keep secrets.existingSecret set, see above.

Local cluster

make kind-up builds both images, starts an incus VM (tools/incus-kind-vm.sh, 6 CPUs and 12 GiB by default), creates a kind cluster inside it, installs CloudNativePG and applies deploy/kustomize/kind (restricted namespace, NetworkPolicy on, throwaway secrets). kind never runs in host Docker: its kubelet changes kernel settings such as vm.overcommit_memory that are not namespaced, so on the host it would change the workstation's kernel (#61).

The host reaches the cluster through a dedicated kubeconfig:

export KUBECONFIG=$PWD/.kind/snatcharr.kubeconfig
kubectl -n snatcharr port-forward svc/snatcharr-api 8080:8080

make kind-down deletes the VM, its profile and the kubeconfig. The VM needs incus with KVM on the host; SNATCHARR_KIND_VM_CPUS and SNATCHARR_KIND_VM_MEMORY size it. For plain local development make dev starts Postgres with Docker Compose and runs the three processes on the host.

End-to-end image smoke

make e2e-image-smoke validates the built container images locally and in CI before release images are pushed:

• Builds snatcharr-api:e2e (with embedded SvelteKit SPA) and snatcharr-worker:e2e (Rust musl binary).
• Runs tools/e2e/smoke.sh in image mode against a temporary Postgres container (tools/e2e/postgres.sh).
• Proves distroless non-root execution (user 65532:65532, --read-only, --cap-drop ALL, no-new-privileges).
• Proves the api image carries the CA bundle (/etc/ssl/certs/ca-certificates.crt) its outbound HTTPS calls read; the worker embeds its root certificates.
• Proves the embedded SvelteKit SPA serves index.html (text/html with SvelteKit markers) and hashed static assets (/_app/immutable/...).
• Proves runtime IANA time zone database availability by creating a disabled schedule window in Europe/Berlin: the API validates the zone with the image's zone data, since the binary embeds none.
• Proves multi-worker run leasing, stamina caps, afterglow deduplication, and liveness probes.
• Runs the api and both workers under the chart's resources.limits.memory (docker run --memory, no swap) and fails when a container's peak (cgroup memory.peak) passes 80 % of that limit (MEM_SHARE, tools/e2e/mem_limits.sh, cases in tools/e2e/mem_limits_test.sh). Measured on 2026-10-17: api 171 MiB of 256 MiB (67 %), each worker 8 MiB of 128 MiB. Raise a limit in values.yaml before the smoke fails, not after a pod is killed.

The release workflow runs the same smoke on amd64 before it pushes images, then builds both images for arm64 and runs tools/e2e/arm64-check.sh. The check reads the ELF header of each entrypoint and refuses a binary that is not aarch64, because both Dockerfiles cross-compile on the build platform and an image labelled arm64 can carry an amd64 binary. It then runs each binary under QEMU: the API must stop on one of the config errors the check lists, the worker must log its start. CI's image smoke runs the same check on the amd64 images (CHECK_ARCH=amd64), so a changed startup error fails the pull request, not the release; tools/e2e/arm64-check_test.sh holds the cases of the list.

To run the image smoke locally:

make e2e-image-smoke

To run against pre-built images without re-building:

SMOKE_SKIP_BUILD=1 E2E_API_IMAGE=<api-image> E2E_WORKER_IMAGE=<worker-image> make e2e-image-smoke

Soak

make e2e-soak keeps the image smoke running for SOAK_SECONDS (default 600) after its checks. The API rotates every gRPC connection after SOAK_CONNECTION_AGE (default 15s, instead of the production 2 minutes), so the workers' lease long-polls (30 s, the production default) cross many rotations. Every SOAK_ROUND_SECONDS (default 20) the soak starts a missing run on Sonarr and Radarr, then fails the round on:

• any failed worker API call (snatcharr_worker_api_failures_total on either worker grew);
• any failed run;
• a last worker contact (worker_last_contact_at in /api/v1/system/status) older than SOAK_MAX_CONTACT_AGE seconds (default 45: the long-poll plus 15).

At the end, at least two runs per round must be done. The workflow .github/workflows/soak.yml runs it nightly and on demand; a failure on main opens or comments on the issue "Soak fails on main". The soak catches lease calls cut at a connection rotation and calls a closing connection refused before sending them, which a shorter smoke run does not reach.

SOAK_SECONDS=1800 make e2e-soak