{"id":"69dbce074e5bc2f6ca3c5755","title":"2026 04 12 Secrets Deployment And Cloudflare Config","path":"how-to/technical-articles/2026-04-12-secrets-deployment-and-cloudflare-config","contentMarkdown":"# 09-Jul-2026 — Secrets, Deployment, Integration Worker, Fly Tokens, and Cloudflare Configuration [#114](https://github.com/nbarrett/ngx-ramblers/issues/114) [#125](https://github.com/nbarrett/ngx-ramblers/issues/125) [#304](https://github.com/nbarrett/ngx-ramblers/issues/304)\n\n_____\n\nThis article explains how secrets flow through NGX Ramblers — from storage, through the deployment pipeline, to runtime use by the backend and frontend. It covers the three-tier secret storage model, the Fly.io deployment process, the shared integration worker (browser automation, HTML fetch, Flickr scraping, document conversion, migration jobs and Serenity walks uploads, with report upload and lifecycle audit events), the per-environment Fly API token encrypted into the `FLY_CONFIG` bundle (powering machine restart and memory-diagnostics metrics), and how Cloudflare email routing configuration reaches managed environments.\n\n## Overview\n\nNGX Ramblers uses a layered approach to secret management. Secrets live in three places, with a clearly-defined precedence: CI reads from MongoDB, local development reads from `server/.env` and `non-vcs/secrets/` files, and GitHub Secrets carry only the handful of bootstrap values the deploy script itself needs.\n\nA shared integration worker runs as a separate Fly.io app alongside the origin environments. Its secrets follow the same DB-first model but are synced to GitHub as a separate secret (`INTEGRATION_WORKER_SECRETS_JSON`) alongside the main `CONFIGS_JSON`.\n\n```mermaid\nflowchart LR\n GH@{ icon: \"logos:github-icon\", label: \"GitHub Secrets\", pos: \"b\", h: 48 }\n DB@{ icon: \"ngx:mongodb\", label: \"MongoDB Atlas\", pos: \"b\", h: 48 }\n LOCAL@{ icon: \"ngx:file\", label: \"Local Files\", pos: \"b\", h: 48 }\n\n GH --- DB --- LOCAL\n```\n\n| Tier | Location | Used By | Contents |\n|------|----------|---------|----------|\n| **GitHub Secrets** | GitHub Actions | CI/CD pipelines | `CONFIGS_JSON`, `INTEGRATION_WORKER_SECRETS_JSON`, `FLY_API_TOKEN`, `ADMIN_MONGODB_URI`, Docker credentials |\n| **MongoDB Atlas** | `config` collection | Runtime + deployment | Per-environment secrets, integration worker config, per-environment Fly token, Cloudflare config, AWS credentials, MongoDB URIs |\n| **Local Files** | `server/.env`, `non-vcs/secrets/` | Local development only | Developer overrides and per-app `.env` files, all git-ignored |\n\n## Deployment Flow\n\nWhen code is pushed to `main`, or a manual deployment is triggered, the following sequence occurs:\n\n```mermaid\nflowchart LR\n GHA@{ icon: \"logos:github-actions\", label: \"GitHub Actions\", pos: \"b\", h: 48 }\n TS@{ icon: \"logos:typescript-icon\", label: \"Deploy Script\", pos: \"b\", h: 48 }\n QUERY@{ icon: \"ngx:mongodb\", label: \"Query MongoDB\", pos: \"b\", h: 48 }\n DEPLOY@{ icon: \"logos:fly-icon\", label: \"Deploy to Fly.io\", pos: \"b\", h: 48 }\n\n GHA -->|\"Run deploy\"| TS\n TS -->|\"ADMIN_MONGODB_URI set\"| QUERY\n QUERY --> MERGE[\"Encrypt & merge secrets\"]\n MERGE --> IMPORT[\"flyctl secrets import\"]\n IMPORT --> DEPLOY\n```\n\n### How the deploy runner reads from MongoDB\n\nThe GitHub Actions runner opens a short-lived connection to the staging MongoDB cluster during each deploy. The connection string lives in the GitHub secret `ADMIN_MONGODB_URI`; the deploy script aliases it to `MONGODB_URI` and uses it to read the `ENVIRONMENTS` config document from the `config` collection. Per-environment secrets never pass through `CONFIGS_JSON` — they travel directly from MongoDB into `flyctl secrets import` on the target Fly app.\n\n### Secret Building Priority\n\nDuring deployment, `buildSecretsFromDatabaseConfig()` merges secrets in a specific order. Later values override earlier ones:\n\n1. **Global secrets** from `globalConfig.secrets`\n2. **Global AWS config** from `globalConfig.aws`\n3. **Per-environment AWS config** (overrides global if present)\n4. **MongoDB URI** built from `envConfig.mongo` credentials\n5. **`NGX_LITE`** flag derived from `envConfig.ngxLite`\n6. **Per-environment secrets** from `envConfig.secrets`\n7. **Integration worker callback base URL** — defaulted to `https://.fly.dev` when not already set\n8. **Fly config** — `{apiToken, appName}` encrypted with AES-256-GCM and stored as `FLY_CONFIG` (from `envConfig.flyio.apiKey`)\n9. **Cloudflare config** — encrypted with AES-256-GCM and stored as `CLOUDFLARE_CONFIG`\n\n### Database vs File Import\n\nThe deploy script has exactly two modes in `importSecrets()`:\n\n- **CI mode** (the only mode exercised on a GitHub runner): `ADMIN_MONGODB_URI` is always set, so `importSecretsFromDatabase()` runs. The runner has no access to `non-vcs/secrets/` — those files are git-ignored and simply don't exist on the runner, so the file path can never execute.\n- **Local-developer mode**: when a developer runs `deploy-to-environments.ts` from a laptop without `ADMIN_MONGODB_URI` set, `importSecretsFromFile()` reads `non-vcs/secrets/secrets..env` from disk. This is a convenience for local testing of the deploy script itself — it is not a production fallback.\n\n```mermaid\nflowchart LR\n START@{ icon: \"ngx:ramblers\", label: \"deploy script\", pos: \"b\", h: 48 }\n QUERY@{ icon: \"ngx:mongodb\", label: \"Query MongoDB\", pos: \"b\", h: 48 }\n FILE@{ icon: \"ngx:file\", label: \"Read non-vcs/secrets/*.env\", pos: \"b\", h: 48 }\n IMPORT@{ icon: \"logos:fly-icon\", label: \"flyctl secrets import\", pos: \"b\", h: 48 }\n\n START --> CI{\"ADMIN_MONGODB_URI set?\"}\n CI -->|\"Yes (CI)\"| QUERY\n QUERY --> IMPORT\n CI -->|\"No (local dev)\"| FILE\n FILE --> IMPORT\n```\n\nIf `importSecretsFromDatabase()` returns empty for an environment, the CI run simply continues without importing — Fly keeps whatever secrets are already on the app. There is no \"file fallback\" on the runner because there are no files.\n\n## Integration Worker Deployment\n\nThe integration worker is a general machine-to-machine worker. It runs Serenity browser automation for Ramblers walks uploads, but also handles synchronous browser operations (HTML fetch, Flickr album scraping, migration scrapes), image resizing, and document conversion. It is deployed as a separate Fly.io app (`ngx-ramblers-integration-worker`) that shares the same Docker image as the origin but runs `npm run worker-server` (which starts `integration-worker-server`) instead of `npm run server`.\n\nThe worker exposes several route groups, all under `/api/integration-worker`:\n\n| Route group | Purpose |\n|-------------|---------|\n| `/jobs`, `/progress`, `/result` | Ramblers walks upload jobs (queued Serenity runs with signed progress and result callbacks) |\n| `/browser/html-fetch`, `/browser/flickr-user-albums` | Synchronous headless-browser HTML fetch and Flickr album scraping |\n| `/migration/*` | Legacy-site migration scrapes |\n| `/resize/*` | Image resize jobs |\n| `/document-conversion/convert` | Document conversion |\n\nThe walks-upload flow below is the most involved of these and is documented in detail; the other route groups follow the same signed request/response pattern.\n\n```mermaid\nflowchart LR\n PUSH@{ icon: \"logos:github-icon\", label: \"Push to main\", pos: \"b\", h: 48 }\n BUILD@{ icon: \"logos:docker-icon\", label: \"Build image\", pos: \"b\", h: 48 }\n ORIGIN@{ icon: \"logos:fly-icon\", label: \"Deploy origin\", pos: \"b\", h: 48 }\n FILTER[\"Worker change detection\"]\n WORKER@{ icon: \"logos:fly-icon\", label: \"Deploy worker\", pos: \"b\", h: 48 }\n\n PUSH --> BUILD\n BUILD --> ORIGIN\n BUILD --> FILTER\n FILTER -->|\"Runtime changes in worker graph\"| WORKER\n FILTER -->|\"No change\"| SKIP[\"Skip — keep warm\"]\n```\n\n### How Worker Deployment Works\n\n- The main build workflow (`build-push-and-deploy-ngx-ramblers-docker-image.yml`) runs a **Detect worker-relevant changes** step (`server/deploy/detect-worker-changes.ts`) that decides whether the worker needs redeploying\n- When worker-relevant runtime changes are found, the workflow builds and pushes a `ngx-ramblers:integration-worker-` image and a conditional **Deploy Ramblers upload worker** step runs `server/deploy/deploy-integration-worker.ts` with that image tag\n- When nothing worker-relevant changed, the build and deploy steps are skipped entirely — the worker stays warm with no restart\n- After a successful worker deploy, the workflow advances a `worker-deployed` git ref (force-pushed to `HEAD`) so the next run can diff against the last thing actually deployed to the worker, not merely the previous commit\n- A standalone `workflow_dispatch`-only workflow (`build-and-deploy-integration-worker.yml`) exists as a manual escape hatch for redeploying a chosen image tag\n\n### Worker Change Detection\n\nRather than a static path glob, `detect-worker-changes.ts` walks the worker's real dependency graph. It runs `madge` from the worker entry point (`server/lib/ramblers/integration-worker-server.ts`) to resolve every `.ts` file the worker actually imports, then unions that set with a fixed list of infrastructure files. A change is only treated as worker-relevant if it falls inside that set **and** changes the emitted JavaScript — type-only edits (identical transpiled output) are ignored.\n\nThe dependency graph plus the following explicit infrastructure files make up the detection scope:\n\n| Category | Paths |\n|----------|-------|\n| Worker dependency graph | Every `.ts` file reachable from `server/lib/ramblers/integration-worker-server.ts` (resolved at build time via `madge`) |\n| Container + Fly config | `Dockerfile`, `fly.integration-worker.toml` |\n| Server packages | `server/package.json`, `server/package-lock.json`, `server/playwright.config.ts` |\n| Deploy + detection scripts | `server/deploy/deploy-integration-worker.ts`, `server/deploy/detect-worker-changes.ts` |\n| Workflows | `.github/workflows/build-push-and-deploy-ngx-ramblers-docker-image.yml`, `.github/workflows/build-and-deploy-integration-worker.yml` |\n\nChanges are compared against the `worker-deployed` ref when it exists, otherwise against `HEAD~1`.\n\n### Routing Uploads to the Worker\n\nEnvironments route uploads to the worker by setting `INTEGRATION_WORKER_URL` in their per-environment secrets. When set, the dispatcher sends signed job requests to the worker instead of spawning Serenity locally. The worker exposes three walks-upload endpoints under `/api/integration-worker`:\n\n| Endpoint | Direction | Purpose |\n|----------|-----------|---------|\n| `POST /jobs` | Origin → Worker | Submit a signed, AES-encrypted job. Worker runs it immediately or queues behind the active job, returning `{queued, queuePosition, activeJobId}` |\n| `POST /progress` | Worker → Origin | Stream per-line stdout events, parsed test steps, and lifecycle markers back to the origin |\n| `POST /result` | Worker → Origin | Announce completion, exit status, and the S3 report key prefix |\n\n```mermaid\nflowchart LR\n subgraph origin[\"Origin (e.g. EKWG)\"]\n DISPATCH@{ icon: \"ngx:express\", label: \"Dispatcher\", pos: \"b\", h: 48 }\n end\n subgraph worker[\"Worker\"]\n RECEIVE@{ icon: \"ngx:express\", label: \"Job receiver\", pos: \"b\", h: 48 }\n SERENITY[\"Serenity spawn\"]\n end\n\n DISPATCH -->|\"POST /jobs — HMAC signed, AES-encrypted credentials\"| RECEIVE\n RECEIVE --> SERENITY\n SERENITY -->|\"POST /progress & /result — signed callbacks\"| DISPATCH\n\n style origin fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n style worker fill:#EEE8F5,stroke:#AB9BC8,stroke-width:2px,rx:12,ry:12,color:#404143\n```\n\nCredentials are AES-encrypted in the job request and decrypted on the worker side. The worker never connects to MongoDB — `Login.performAs` short-circuits its Mongo lookup when `RAMBLERS_USERNAME` and `RAMBLERS_PASSWORD` are set in the environment (they always are on the worker, because the runner writes them from the decrypted payload before spawning Serenity).\n\n### Serenity Report Upload to S3\n\nEvery upload run — success or failure — uploads the complete Serenity HTML report to the origin's own S3 bucket so that operators can open the detailed test output from the Walk Export UI.\n\n```mermaid\nflowchart LR\n subgraph worker[\"Worker\"]\n RUN[\"Serenity run\"] --> GEN[\"Report generated at target/site/serenity\"]\n GEN --> UP@{ icon: \"ngx:aws\", label: \"uploadSerenityReportToS3\", pos: \"b\", h: 48 }\n end\n subgraph origin[\"Origin\"]\n RESULT@{ icon: \"ngx:express\", label: \"POST /result\", pos: \"b\", h: 48 }\n AUDIT@{ icon: \"ngx:mongodb\", label: \"recordReportLocation\", pos: \"b\", h: 48 }\n WS[\"WebSocket broadcast\"]\n UI@{ icon: \"ngx:ramblers\", label: \"View Report button\", pos: \"b\", h: 48 }\n end\n S3@{ icon: \"ngx:aws\", label: \"S3 ramblers-upload-reports//\", pos: \"b\", h: 48 }\n\n UP --> S3\n UP -->|\"reportBucket + reportKeyPrefix\"| RESULT\n RESULT --> AUDIT\n AUDIT --> WS --> UI\n UI -->|\"/api/aws/s3/.../index.html\"| S3\n\n style origin fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n style worker fill:#EEE8F5,stroke:#AB9BC8,stroke-width:2px,rx:12,ry:12,color:#404143\n```\n\nEnd-to-end flow:\n\n1. Origin's dispatcher computes `reportKeyPrefix = ramblers-upload-reports/` and sends it, together with AES-encrypted S3 credentials (reusing the environment's own `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`) in the `/jobs` request.\n2. After the Serenity subprocess exits, `finishJob()` calls `uploadSerenityReportToS3()` which walks `target/site/serenity` and uploads every file via `uploadDirectoryToS3`, using the centralised `contentTypeForExtension` / `fileContentTypeMappings` helpers in `projects/ngx-ramblers/src/app/models/content-metadata.model.ts` (shared with the origin's AWS utilities).\n3. The worker then posts `/result` with `reportBucket` and `reportKeyPrefix` populated.\n4. The origin's `recordReportLocation` persists the prefix onto the audit session and broadcasts an audit row over the WebSocket so the Walk Export UI renders a **View Report** button next to **Back To Walks Admin**.\n5. The UI fetches the report through the existing `/api/aws/s3/.../index.html` proxy — no public bucket access, no extra credentials.\n\n### Lifecycle Audit Events\n\nBeyond the per-step `SUCCESS` / `TEST_STEP` rows parsed from stdout, the worker emits additional lifecycle markers via the `IntegrationWorkerEventType.LIFECYCLE` type. The origin's `recordLifecycleEvent` writes each one as a `SUMMARY` audit row and pushes it over the WebSocket:\n\n| Marker | When |\n|--------|------|\n| `Scenario execution complete` | First time the Serenity report writer prints `[report]` to stderr |\n| `Serenity report generation starting` | Immediately after, before the report writer runs |\n| `Serenity report generation complete, uploading to S3` | After Serenity exits, before `uploadSerenityReportToS3` is invoked |\n| `Report upload to S3 complete` | After the report directory has been uploaded to S3, before the final result callback |\n\nThese make the \"silent\" stretch between the last scenario step and the final result callback visible in the audit log.\n\n### Local Developer Ergonomics\n\n`ngx-cli local dev` honours four worker-related overrides in `server/.env`:\n\n- `INTEGRATION_WORKER_URL`\n- `INTEGRATION_WORKER_CALLBACK_BASE_URL`\n- `INTEGRATION_WORKER_SHARED_SECRET`\n- `INTEGRATION_WORKER_ENCRYPTION_KEY`\n\nThe `local.ts` command reads these values via `readServerEnvValue` and applies them on top of the secrets resolved from the database (see `SERVER_ENV_OVERRIDE_KEYS`). A developer can therefore flip between a locally-running worker container and the Fly-hosted worker purely by editing `.env`, without touching the database or redeploying anything.\n\n## Worker Secret Management\n\nWorker configuration lives in the `uploadWorker` field of the environments config document — alongside `environments[]`, not inside it. This ensures \"deploy all environments\" never touches the worker.\n\n```mermaid\nflowchart LR\n UI@{ icon: \"ngx:ramblers\", label: \"Admin UI\", pos: \"b\", h: 48 }\n DB@{ icon: \"ngx:mongodb\", label: \"MongoDB config\", pos: \"b\", h: 48 }\n PUSH[\"Push to GitHub\"]\n GH1@{ icon: \"logos:github-icon\", label: \"CONFIGS_JSON\", pos: \"b\", h: 48 }\n GH2@{ icon: \"logos:github-icon\", label: \"INTEGRATION_WORKER_SECRETS_JSON\", pos: \"b\", h: 48 }\n\n UI -->|\"Save\"| DB\n DB --> PUSH\n PUSH --> GH1\n PUSH --> GH2\n```\n\n| Config location | What it holds | Managed via |\n|----------------|---------------|-------------|\n| `value.uploadWorker` in environments config | App name, HMAC secret, AES key, memory, scale, org-scoped Fly `apiKey` | Global Settings tab in admin UI |\n| `value.environments[].secrets` per routing env | `INTEGRATION_WORKER_URL`, `_SHARED_SECRET`, `_ENCRYPTION_KEY` | Per-environment settings in admin UI |\n| GitHub secret `INTEGRATION_WORKER_SECRETS_JSON` | JSON blob with HMAC + AES keys | \"Push to GitHub\" button (synced alongside `CONFIGS_JSON`) |\n| GitHub variable `INTEGRATION_WORKER_APP_NAME` | Fly app name | \"Push to GitHub\" button |\n\n### Key Rotation\n\nTo rotate the worker's HMAC or AES key:\n\n1. Edit the key in the **Global Settings** tab (Upload Worker Configuration section)\n2. Update the matching key in every routing environment's secrets (currently EKWG only)\n3. Click **Save Configuration**\n4. Go to **GitHub Secrets** tab and click **Push to GitHub** — this updates both `CONFIGS_JSON` and `INTEGRATION_WORKER_SECRETS_JSON` atomically\n5. Redeploy the worker and the routing environment(s)\n\n## Cloudflare Configuration\n\nCloudflare configuration deserves special attention because it contains a sensitive API token that must never reach the frontend, yet the frontend needs certain non-sensitive fields to render the committee email routing UI.\n\n### Encryption at Deploy Time\n\nDuring deployment, the Cloudflare config is encrypted using AES-256-GCM before being stored as a Fly.io secret:\n\n```mermaid\nflowchart LR\n DBCF@{ icon: \"ngx:mongodb\", label: \"MongoDB config\", pos: \"b\", h: 48 }\n ENVVAR@{ icon: \"logos:fly-icon\", label: \"CLOUDFLARE_CONFIG\", pos: \"b\", h: 48 }\n\n DBCF --> ENCRYPT[\"AES-256-GCM encrypt\"] --> ENVVAR\n```\n\nThe encrypted payload structure is:\n\n| Bytes | Content |\n|-------|---------|\n| 0-12 | Initialisation vector (IV) |\n| 12-28 | Authentication tag |\n| 28+ | Ciphertext |\n\n### Runtime: Backend to Frontend\n\nAt runtime, the backend decrypts `CLOUDFLARE_CONFIG` and uses the full config (including `apiToken`) for Cloudflare API calls. The frontend only receives non-sensitive fields:\n\n```mermaid\nflowchart LR\n subgraph backend[\"Backend\"]\n DECRYPT[\"Decrypt\"] --> FULL[\"Full config\"]\n FULL --> API@{ icon: \"ngx:cloudflare\", label: \"Cloudflare API\", pos: \"b\", h: 48 }\n end\n\n subgraph frontend[\"Frontend\"]\n SAFE[\"Safe config\"] --> UI@{ icon: \"ngx:ramblers\", label: \"Committee UI\", pos: \"b\", h: 48 }\n end\n\n FULL -->|\"Non-sensitive only\"| SAFE\n UI -->|\"Via backend API\"| FULL\n```\n\n### Non-Sensitive Config Fields\n\nThe `/api/cloudflare/email-routing/config` endpoint returns only:\n\n| Field | Type | Purpose |\n|-------|------|---------|\n| `configured` | `boolean` | Whether Cloudflare is set up for this environment |\n| `accountId` | `string` | Used to build Cloudflare dashboard links |\n| `zoneId` | `string` | Used to build Cloudflare dashboard links |\n| `baseDomain` | `string` | The domain for email routing (e.g. `nwkramblers.org.uk`) |\n| `ownsZone` | `boolean` | Whether this site owns its Cloudflare zone or is a subdomain of a shared zone (drives the apex-vs-subdomain inbox setup) |\n| `zoneName` | `string` | The resolved Cloudflare zone name for the domain |\n\nThe `apiToken` is **never** sent to the frontend. All Cloudflare operations go through authenticated backend endpoints.\n\n## Fly API Token & the `FLY_CONFIG` Bundle\n\nThe Fly API token follows the identical encrypted-bundle pattern as the Cloudflare config. Each environment carries one org-scoped Fly API token in `flyio.apiKey` (alongside `flyio.appName`) in the `ENVIRONMENTS` config document in MongoDB. That single token does everything Fly-related for the environment: it authenticates the deploy (`flyctl deploy` and `flyctl secrets import`), it restarts a stuck machine from the NGX admin UI, and it reads Fly's Prometheus metrics for the Memory Diagnostics charts.\n\n### Encryption at Deploy Time\n\nDuring deployment, `buildSecretsFromDatabaseConfig()` encrypts `{apiToken, appName}` into a `FLY_CONFIG` Fly secret using AES-256-GCM keyed by `ENVIRONMENT_SETUP_API_KEY` — the same mechanism, same encryption key, and same payload layout used for `CLOUDFLARE_CONFIG`.\n\n```mermaid\nflowchart LR\n DBFLY@{ icon: \"ngx:mongodb\", label: \"flyio.apiKey + appName\", pos: \"b\", h: 48 }\n ENVVAR@{ icon: \"logos:fly-icon\", label: \"FLY_CONFIG\", pos: \"b\", h: 48 }\n\n DBFLY --> ENCRYPT[\"AES-256-GCM encrypt\"] --> ENVVAR\n```\n\n### Runtime Token Resolution\n\nAt runtime, `flyRuntimeConfig()` resolves the token in a fixed precedence order and stops at the first source that yields both a token and an app name:\n\n1. **Fly-injected environment variables** — Fly sets `FLY_APP_NAME` and `FLY_MACHINE_ID` on every deployed machine; combined with a `FLY_API_TOKEN` env var this is the fast path\n2. **Decrypted `FLY_CONFIG` bundle** — decrypted with `ENVIRONMENT_SETUP_API_KEY`, this is how deployed machines normally obtain the token\n3. **Environments config in the database** — matched on the Mongo database name, used as a last-resort fallback\n\n```mermaid\nflowchart LR\n START@{ icon: \"logos:fly-icon\", label: \"flyRuntimeConfig()\", pos: \"b\", h: 48 }\n ENV@{ icon: \"logos:fly-icon\", label: \"FLY_* env vars\", pos: \"b\", h: 48 }\n BUNDLE@{ icon: \"logos:fly-icon\", label: \"Decrypt FLY_CONFIG\", pos: \"b\", h: 48 }\n DB@{ icon: \"ngx:mongodb\", label: \"Environments config\", pos: \"b\", h: 48 }\n TOKEN[\"Resolved token + appName\"]\n\n START --> ENV\n ENV -->|\"token + appName present\"| TOKEN\n ENV -->|\"missing\"| BUNDLE\n BUNDLE -->|\"decrypted\"| TOKEN\n BUNDLE -->|\"absent\"| DB\n DB --> TOKEN\n```\n\n### What the Token Powers\n\n| Use | Where | Notes |\n|-----|-------|-------|\n| Deploy + `flyctl secrets import` | CI deploy runner | The per-environment `apiKey` is loaded into `FLY_API_TOKEN` for that environment's `flyctl` commands |\n| Restart a stuck machine | Admin UI (#304) | `restartCurrentMachine()` calls the Fly Machines API with the resolved token |\n| Read memory / metric history | Memory Diagnostics charts (#304) | `fly-metrics` queries Fly's Prometheus API with the resolved token |\n\nAn org-scoped token is required for the metrics path: an app-scoped token can manage and restart the app but returns 403 from Fly's Prometheus API.\n\nOriginally five environments carried a separate app-scoped deploy token plus a distinct org-scoped `metricsToken`. On 9 Jul 2026 this was consolidated to a single org-scoped `apiKey` per environment and the `metricsToken` field was removed, so one token now covers deploy, restart and metrics.\n\n## Required Environment Variables\n\n### Origin Environments\n\n| Variable | Required | Purpose |\n|----------|----------|---------|\n| `AUTH_SECRET` | Yes | JWT authentication secret |\n| `AWS_ACCESS_KEY_ID` | Yes | S3 file storage (also reused to upload Serenity reports) |\n| `AWS_SECRET_ACCESS_KEY` | Yes | S3 file storage |\n| `AWS_REGION` | Yes | S3 region (e.g. `eu-west-2`) |\n| `AWS_BUCKET` | Yes | S3 bucket name |\n| `MONGODB_URI` | Yes | Database connection string |\n| `CLOUDFLARE_CONFIG` | No | Encrypted Cloudflare config (enables email routing) |\n| `FLY_CONFIG` | No | Encrypted per-environment Fly token + app name (enables machine restart + metrics) |\n| `ENVIRONMENT_SETUP_API_KEY` | No | Encryption key for both `CLOUDFLARE_CONFIG` and `FLY_CONFIG` |\n| `INTEGRATION_WORKER_URL` | No | Worker URL (enables remote upload routing) |\n| `INTEGRATION_WORKER_SHARED_SECRET` | If worker URL set | HMAC signing key for worker requests |\n| `INTEGRATION_WORKER_ENCRYPTION_KEY` | If worker URL set | AES key for credential encryption |\n\n### Worker App\n\n| Variable | Required | Purpose |\n|----------|----------|---------|\n| `INTEGRATION_WORKER_SHARED_SECRET` | Yes | HMAC signature validation |\n| `INTEGRATION_WORKER_ENCRYPTION_KEY` | Yes | Credential + report-credential decryption |\n| `CHROME_BIN` | Built-in | Chrome path (baked into Docker image via `ENV`) |\n| `CHROMEDRIVER_PATH` | Built-in | Chromedriver path (baked into Docker image via `ENV`) |\n\nThe worker deliberately has no `MONGODB_URI`. `RAMBLERS_USERNAME` and `RAMBLERS_PASSWORD` are set by the runner from the decrypted job payload immediately before Serenity spawns, so `Login.performAs` takes its env-var fast path and never attempts a Mongo lookup.\n\n### Deploy-Only Variables\n\n| Variable | Required | Purpose |\n|----------|----------|---------|\n| `FLY_API_TOKEN` | Deploy only | GitHub secret — the deploy runner's ambient Fly authentication. Distinct from the per-environment `flyio.apiKey`, which the deploy loads into `FLY_API_TOKEN` per environment and which is also baked into that environment's `FLY_CONFIG` secret |\n| `ADMIN_MONGODB_URI` | Deploy only | Staging database for secrets import |\n| `INTEGRATION_WORKER_SECRETS_JSON` | Deploy only | GitHub secret — worker HMAC + AES keys as JSON |\n| `INTEGRATION_WORKER_APP_NAME` | Deploy only | GitHub variable — worker Fly app name |\n\n## Issue History\n\nThe secrets and deployment infrastructure was built across several issues:\n\n```mermaid\nflowchart BT\n I146@{ icon: \"logos:github-icon\", label: \"#146 Env Setup\", pos: \"b\", h: 48 }\n I156@{ icon: \"ngx:cloudflare\", label: \"#156 DNS/SSL\", pos: \"b\", h: 48 }\n I125@{ icon: \"ngx:cloudflare\", label: \"#125 Email Routing\", pos: \"b\", h: 48 }\n I82[\"#82 Brevo Auth\"]\n I130[\"#130 Replies Rejected\"]\n I114@{ icon: \"logos:fly-icon\", label: \"#114 Integration Worker\", pos: \"b\", h: 48 }\n I304@{ icon: \"logos:fly-icon\", label: \"#304 Machine Restart & Metrics\", pos: \"b\", h: 48 }\n TODAY@{ icon: \"logos:fly-icon\", label: \"Today: Worker, Fly Tokens & Secrets\", pos: \"b\", h: 48 }\n\n I146 -->|\"secrets\"| I125\n I146 -->|\"config\"| I156\n I156 -->|\"Cloudflare\"| I125\n I156 -->|\"DNS\"| I82\n I130 -->|\"motivated\"| I125\n I125 --> TODAY\n I82 --> TODAY\n I114 -->|\"worker + reports + lifecycle\"| TODAY\n I304 -->|\"Fly token + restart + metrics\"| TODAY\n```\n\n| Issue | Title | Key Contribution |\n|-------|-------|-----------------|\n| [#146](https://github.com/nbarrett/ngx-ramblers/issues/146) | Automated environment creation | Database-first secrets loading, GitHub secrets sync, global AWS |\n| [#156](https://github.com/nbarrett/ngx-ramblers/issues/156) | DNS/SSL automation | Cloudflare DNS and certificate management in environment setup |\n| [#130](https://github.com/nbarrett/ngx-ramblers/issues/130) | Email replies rejected | The original problem that drove Cloudflare email routing work |\n| [#125](https://github.com/nbarrett/ngx-ramblers/issues/125) | Cloudflare email routing | Committee email forwarding, worker scripts, email logs UI |\n| [#82](https://github.com/nbarrett/ngx-ramblers/issues/82) | Brevo domain authentication | DKIM/SPF DNS records via Cloudflare for email deliverability |\n| [#114](https://github.com/nbarrett/ngx-ramblers/issues/114) | Distributed integration worker | Shared worker (Serenity uploads, browser fetch, Flickr scraping, migration, resize, document conversion), queued job execution, dependency-graph deploy detection, S3 Serenity report upload, lifecycle audit events, env-var login short-circuit, `.env` overrides for local dev |\n| [#304](https://github.com/nbarrett/ngx-ramblers/issues/304) | Machine restart & memory diagnostics | Per-environment org-scoped Fly token encrypted into `FLY_CONFIG`, admin restart of a stuck Fly machine, Fly Prometheus metric-history charts, consolidation of deploy + metrics tokens into one `apiKey` |\n\n## Why No Code Changes Were Needed (Original Cloudflare Work)\n\nThe application code already handled Cloudflare gracefully — showing a warning when config was missing and enabling features when config was present. The missing piece was the deployment pipeline: managed environments had no `CLOUDFLARE_CONFIG` secret because the deploy script did not import secrets from the database. Adding `importSecretsFromDatabase()` to the deploy flow was the final link in the chain.\n\n## What Changed for the Integration Worker\n\nThe integration worker required changes across several layers:\n\n- **Model**: `UploadWorkerConfig` added to `EnvironmentsConfig` as a sibling to `environments[]`, keeping it out of the deploy-all matrix; `IntegrationWorkerEventType.LIFECYCLE` added for non-stdout audit markers\n- **Worker runner**: Serenity subprocess uploads `target/site/serenity` to `s3:///ramblers-upload-reports//` via `uploadSerenityReportToS3`, and emits lifecycle progress events around report generation and upload\n- **Origin audit notifier**: new `recordLifecycleEvent` and `recordReportLocation` helpers persist the extra rows and broadcast the report location over the WebSocket so the UI can surface a \"View Report\" link\n- **Admin UI**: New section in the Global Settings tab for worker app name, org-scoped Fly `apiKey`, HMAC secret, AES key, memory, and scale; Walk Export UI renders the report link against the selected session\n- **GitHub sync**: the \"Push to GitHub\" action writes `INTEGRATION_WORKER_SECRETS_JSON` alongside `CONFIGS_JSON` when worker config exists\n- **CI workflow**: a dependency-graph deploy step (`detect-worker-changes.ts`) piggybacked on the main build, reusing the same Docker image with a different Fly config (`fly.integration-worker.toml`) via `deploy-integration-worker.ts`\n- **Content-type mapping**: `contentTypeForExtension` + `fileContentTypeMappings` centralised in `projects/ngx-ramblers/src/app/models/content-metadata.model.ts` and shared between the S3 report uploader and `aws-utils`\n- **Local dev**: `ngx-cli local dev` reads four worker env vars from `server/.env` so a developer can flip between a local worker and the Fly worker without redeploying\n","contentHtml":"

09-Jul-2026 — Secrets, Deployment, Integration Worker, Fly Tokens, and Cloudflare Configuration #114 #125 #304

\n
\n

This article explains how secrets flow through NGX Ramblers — from storage, through the deployment pipeline, to runtime use by the backend and frontend. It covers the three-tier secret storage model, the Fly.io deployment process, the shared integration worker (browser automation, HTML fetch, Flickr scraping, document conversion, migration jobs and Serenity walks uploads, with report upload and lifecycle audit events), the per-environment Fly API token encrypted into the FLY_CONFIG bundle (powering machine restart and memory-diagnostics metrics), and how Cloudflare email routing configuration reaches managed environments.

\n

Overview

\n

NGX Ramblers uses a layered approach to secret management. Secrets live in three places, with a clearly-defined precedence: CI reads from MongoDB, local development reads from server/.env and non-vcs/secrets/ files, and GitHub Secrets carry only the handful of bootstrap values the deploy script itself needs.

\n

A shared integration worker runs as a separate Fly.io app alongside the origin environments. Its secrets follow the same DB-first model but are synced to GitHub as a separate secret (INTEGRATION_WORKER_SECRETS_JSON) alongside the main CONFIGS_JSON.

\n
flowchart LR\n    GH@{ icon: "logos:github-icon", label: "GitHub Secrets", pos: "b", h: 48 }\n    DB@{ icon: "ngx:mongodb", label: "MongoDB Atlas", pos: "b", h: 48 }\n    LOCAL@{ icon: "ngx:file", label: "Local Files", pos: "b", h: 48 }\n\n    GH --- DB --- LOCAL\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
TierLocationUsed ByContents
GitHub SecretsGitHub ActionsCI/CD pipelinesCONFIGS_JSON, INTEGRATION_WORKER_SECRETS_JSON, FLY_API_TOKEN, ADMIN_MONGODB_URI, Docker credentials
MongoDB Atlasconfig collectionRuntime + deploymentPer-environment secrets, integration worker config, per-environment Fly token, Cloudflare config, AWS credentials, MongoDB URIs
Local Filesserver/.env, non-vcs/secrets/Local development onlyDeveloper overrides and per-app .env files, all git-ignored
\n

Deployment Flow

\n

When code is pushed to main, or a manual deployment is triggered, the following sequence occurs:

\n
flowchart LR\n    GHA@{ icon: "logos:github-actions", label: "GitHub Actions", pos: "b", h: 48 }\n    TS@{ icon: "logos:typescript-icon", label: "Deploy Script", pos: "b", h: 48 }\n    QUERY@{ icon: "ngx:mongodb", label: "Query MongoDB", pos: "b", h: 48 }\n    DEPLOY@{ icon: "logos:fly-icon", label: "Deploy to Fly.io", pos: "b", h: 48 }\n\n    GHA -->|"Run deploy"| TS\n    TS -->|"ADMIN_MONGODB_URI set"| QUERY\n    QUERY --> MERGE["Encrypt & merge secrets"]\n    MERGE --> IMPORT["flyctl secrets import"]\n    IMPORT --> DEPLOY\n
\n

How the deploy runner reads from MongoDB

\n

The GitHub Actions runner opens a short-lived connection to the staging MongoDB cluster during each deploy. The connection string lives in the GitHub secret ADMIN_MONGODB_URI; the deploy script aliases it to MONGODB_URI and uses it to read the ENVIRONMENTS config document from the config collection. Per-environment secrets never pass through CONFIGS_JSON — they travel directly from MongoDB into flyctl secrets import on the target Fly app.

\n

Secret Building Priority

\n

During deployment, buildSecretsFromDatabaseConfig() merges secrets in a specific order. Later values override earlier ones:

\n
    \n
  1. Global secrets from globalConfig.secrets
    2. \n
  2. Global AWS config from globalConfig.aws
    3. \n
  3. Per-environment AWS config (overrides global if present)
    4. \n
  4. MongoDB URI built from envConfig.mongo credentials
    5. \n
  5. NGX_LITE flag derived from envConfig.ngxLite
    6. \n
  6. Per-environment secrets from envConfig.secrets
    7. \n
  7. Integration worker callback base URL — defaulted to https://<appName>.fly.dev when not already set
    8. \n
  8. Fly config — {apiToken, appName} encrypted with AES-256-GCM and stored as FLY_CONFIG (from envConfig.flyio.apiKey)
    9. \n
  9. Cloudflare config — encrypted with AES-256-GCM and stored as CLOUDFLARE_CONFIG
    10. \n
\n

Database vs File Import

\n

The deploy script has exactly two modes in importSecrets():

\n
    \n
  • CI mode (the only mode exercised on a GitHub runner): ADMIN_MONGODB_URI is always set, so importSecretsFromDatabase() runs. The runner has no access to non-vcs/secrets/ — those files are git-ignored and simply don't exist on the runner, so the file path can never execute.
    • \n
  • Local-developer mode: when a developer runs deploy-to-environments.ts from a laptop without ADMIN_MONGODB_URI set, importSecretsFromFile() reads non-vcs/secrets/secrets.<appName>.env from disk. This is a convenience for local testing of the deploy script itself — it is not a production fallback.
    • \n
\n
flowchart LR\n    START@{ icon: "ngx:ramblers", label: "deploy script", pos: "b", h: 48 }\n    QUERY@{ icon: "ngx:mongodb", label: "Query MongoDB", pos: "b", h: 48 }\n    FILE@{ icon: "ngx:file", label: "Read non-vcs/secrets/*.env", pos: "b", h: 48 }\n    IMPORT@{ icon: "logos:fly-icon", label: "flyctl secrets import", pos: "b", h: 48 }\n\n    START --> CI{"ADMIN_MONGODB_URI set?"}\n    CI -->|"Yes (CI)"| QUERY\n    QUERY --> IMPORT\n    CI -->|"No (local dev)"| FILE\n    FILE --> IMPORT\n
\n

If importSecretsFromDatabase() returns empty for an environment, the CI run simply continues without importing — Fly keeps whatever secrets are already on the app. There is no "file fallback" on the runner because there are no files.

\n

Integration Worker Deployment

\n

The integration worker is a general machine-to-machine worker. It runs Serenity browser automation for Ramblers walks uploads, but also handles synchronous browser operations (HTML fetch, Flickr album scraping, migration scrapes), image resizing, and document conversion. It is deployed as a separate Fly.io app (ngx-ramblers-integration-worker) that shares the same Docker image as the origin but runs npm run worker-server (which starts integration-worker-server) instead of npm run server.

\n

The worker exposes several route groups, all under /api/integration-worker:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Route groupPurpose
/jobs, /progress, /resultRamblers walks upload jobs (queued Serenity runs with signed progress and result callbacks)
/browser/html-fetch, /browser/flickr-user-albumsSynchronous headless-browser HTML fetch and Flickr album scraping
/migration/*Legacy-site migration scrapes
/resize/*Image resize jobs
/document-conversion/convertDocument conversion
\n

The walks-upload flow below is the most involved of these and is documented in detail; the other route groups follow the same signed request/response pattern.

\n
flowchart LR\n    PUSH@{ icon: "logos:github-icon", label: "Push to main", pos: "b", h: 48 }\n    BUILD@{ icon: "logos:docker-icon", label: "Build image", pos: "b", h: 48 }\n    ORIGIN@{ icon: "logos:fly-icon", label: "Deploy origin", pos: "b", h: 48 }\n    FILTER["Worker change detection"]\n    WORKER@{ icon: "logos:fly-icon", label: "Deploy worker", pos: "b", h: 48 }\n\n    PUSH --> BUILD\n    BUILD --> ORIGIN\n    BUILD --> FILTER\n    FILTER -->|"Runtime changes in worker graph"| WORKER\n    FILTER -->|"No change"| SKIP["Skip — keep warm"]\n
\n

How Worker Deployment Works

\n
    \n
  • The main build workflow (build-push-and-deploy-ngx-ramblers-docker-image.yml) runs a Detect worker-relevant changes step (server/deploy/detect-worker-changes.ts) that decides whether the worker needs redeploying
    • \n
  • When worker-relevant runtime changes are found, the workflow builds and pushes a ngx-ramblers:integration-worker-<run_number> image and a conditional Deploy Ramblers upload worker step runs server/deploy/deploy-integration-worker.ts with that image tag
    • \n
  • When nothing worker-relevant changed, the build and deploy steps are skipped entirely — the worker stays warm with no restart
    • \n
  • After a successful worker deploy, the workflow advances a worker-deployed git ref (force-pushed to HEAD) so the next run can diff against the last thing actually deployed to the worker, not merely the previous commit
    • \n
  • A standalone workflow_dispatch-only workflow (build-and-deploy-integration-worker.yml) exists as a manual escape hatch for redeploying a chosen image tag
    • \n
\n

Worker Change Detection

\n

Rather than a static path glob, detect-worker-changes.ts walks the worker's real dependency graph. It runs madge from the worker entry point (server/lib/ramblers/integration-worker-server.ts) to resolve every .ts file the worker actually imports, then unions that set with a fixed list of infrastructure files. A change is only treated as worker-relevant if it falls inside that set and changes the emitted JavaScript — type-only edits (identical transpiled output) are ignored.

\n

The dependency graph plus the following explicit infrastructure files make up the detection scope:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CategoryPaths
Worker dependency graphEvery .ts file reachable from server/lib/ramblers/integration-worker-server.ts (resolved at build time via madge)
Container + Fly configDockerfile, fly.integration-worker.toml
Server packagesserver/package.json, server/package-lock.json, server/playwright.config.ts
Deploy + detection scriptsserver/deploy/deploy-integration-worker.ts, server/deploy/detect-worker-changes.ts
Workflows.github/workflows/build-push-and-deploy-ngx-ramblers-docker-image.yml, .github/workflows/build-and-deploy-integration-worker.yml
\n

Changes are compared against the worker-deployed ref when it exists, otherwise against HEAD~1.

\n

Routing Uploads to the Worker

\n

Environments route uploads to the worker by setting INTEGRATION_WORKER_URL in their per-environment secrets. When set, the dispatcher sends signed job requests to the worker instead of spawning Serenity locally. The worker exposes three walks-upload endpoints under /api/integration-worker:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
EndpointDirectionPurpose
POST /jobsOrigin → WorkerSubmit a signed, AES-encrypted job. Worker runs it immediately or queues behind the active job, returning {queued, queuePosition, activeJobId}
POST /progressWorker → OriginStream per-line stdout events, parsed test steps, and lifecycle markers back to the origin
POST /resultWorker → OriginAnnounce completion, exit status, and the S3 report key prefix
\n
flowchart LR\n    subgraph origin["Origin (e.g. EKWG)"]\n        DISPATCH@{ icon: "ngx:express", label: "Dispatcher", pos: "b", h: 48 }\n    end\n    subgraph worker["Worker"]\n        RECEIVE@{ icon: "ngx:express", label: "Job receiver", pos: "b", h: 48 }\n        SERENITY["Serenity spawn"]\n    end\n\n    DISPATCH -->|"POST /jobs — HMAC signed, AES-encrypted credentials"| RECEIVE\n    RECEIVE --> SERENITY\n    SERENITY -->|"POST /progress & /result — signed callbacks"| DISPATCH\n\n    style origin fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n    style worker fill:#EEE8F5,stroke:#AB9BC8,stroke-width:2px,rx:12,ry:12,color:#404143\n
\n

Credentials are AES-encrypted in the job request and decrypted on the worker side. The worker never connects to MongoDB — Login.performAs short-circuits its Mongo lookup when RAMBLERS_USERNAME and RAMBLERS_PASSWORD are set in the environment (they always are on the worker, because the runner writes them from the decrypted payload before spawning Serenity).

\n

Serenity Report Upload to S3

\n

Every upload run — success or failure — uploads the complete Serenity HTML report to the origin's own S3 bucket so that operators can open the detailed test output from the Walk Export UI.

\n
flowchart LR\n    subgraph worker["Worker"]\n        RUN["Serenity run"] --> GEN["Report generated at target/site/serenity"]\n        GEN --> UP@{ icon: "ngx:aws", label: "uploadSerenityReportToS3", pos: "b", h: 48 }\n    end\n    subgraph origin["Origin"]\n        RESULT@{ icon: "ngx:express", label: "POST /result", pos: "b", h: 48 }\n        AUDIT@{ icon: "ngx:mongodb", label: "recordReportLocation", pos: "b", h: 48 }\n        WS["WebSocket broadcast"]\n        UI@{ icon: "ngx:ramblers", label: "View Report button", pos: "b", h: 48 }\n    end\n    S3@{ icon: "ngx:aws", label: "S3 ramblers-upload-reports/<csvFileName>/", pos: "b", h: 48 }\n\n    UP --> S3\n    UP -->|"reportBucket + reportKeyPrefix"| RESULT\n    RESULT --> AUDIT\n    AUDIT --> WS --> UI\n    UI -->|"/api/aws/s3/.../index.html"| S3\n\n    style origin fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n    style worker fill:#EEE8F5,stroke:#AB9BC8,stroke-width:2px,rx:12,ry:12,color:#404143\n
\n

End-to-end flow:

\n
    \n
  1. Origin's dispatcher computes reportKeyPrefix = ramblers-upload-reports/<csvFileName without .csv> and sends it, together with AES-encrypted S3 credentials (reusing the environment's own AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) in the /jobs request.
    2. \n
  2. After the Serenity subprocess exits, finishJob() calls uploadSerenityReportToS3() which walks target/site/serenity and uploads every file via uploadDirectoryToS3, using the centralised contentTypeForExtension / fileContentTypeMappings helpers in projects/ngx-ramblers/src/app/models/content-metadata.model.ts (shared with the origin's AWS utilities).
    3. \n
  3. The worker then posts /result with reportBucket and reportKeyPrefix populated.
    4. \n
  4. The origin's recordReportLocation persists the prefix onto the audit session and broadcasts an audit row over the WebSocket so the Walk Export UI renders a View Report button next to Back To Walks Admin.
    5. \n
  5. The UI fetches the report through the existing /api/aws/s3/.../index.html proxy — no public bucket access, no extra credentials.
    6. \n
\n

Lifecycle Audit Events

\n

Beyond the per-step SUCCESS / TEST_STEP rows parsed from stdout, the worker emits additional lifecycle markers via the IntegrationWorkerEventType.LIFECYCLE type. The origin's recordLifecycleEvent writes each one as a SUMMARY audit row and pushes it over the WebSocket:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MarkerWhen
Scenario execution completeFirst time the Serenity report writer prints [report] to stderr
Serenity report generation startingImmediately after, before the report writer runs
Serenity report generation complete, uploading to S3After Serenity exits, before uploadSerenityReportToS3 is invoked
Report upload to S3 completeAfter the report directory has been uploaded to S3, before the final result callback
\n

These make the "silent" stretch between the last scenario step and the final result callback visible in the audit log.

\n

Local Developer Ergonomics

\n

ngx-cli local dev honours four worker-related overrides in server/.env:

\n
    \n
  • INTEGRATION_WORKER_URL
    • \n
  • INTEGRATION_WORKER_CALLBACK_BASE_URL
    • \n
  • INTEGRATION_WORKER_SHARED_SECRET
    • \n
  • INTEGRATION_WORKER_ENCRYPTION_KEY
    • \n
\n

The local.ts command reads these values via readServerEnvValue and applies them on top of the secrets resolved from the database (see SERVER_ENV_OVERRIDE_KEYS). A developer can therefore flip between a locally-running worker container and the Fly-hosted worker purely by editing .env, without touching the database or redeploying anything.

\n

Worker Secret Management

\n

Worker configuration lives in the uploadWorker field of the environments config document — alongside environments[], not inside it. This ensures "deploy all environments" never touches the worker.

\n
flowchart LR\n    UI@{ icon: "ngx:ramblers", label: "Admin UI", pos: "b", h: 48 }\n    DB@{ icon: "ngx:mongodb", label: "MongoDB config", pos: "b", h: 48 }\n    PUSH["Push to GitHub"]\n    GH1@{ icon: "logos:github-icon", label: "CONFIGS_JSON", pos: "b", h: 48 }\n    GH2@{ icon: "logos:github-icon", label: "INTEGRATION_WORKER_SECRETS_JSON", pos: "b", h: 48 }\n\n    UI -->|"Save"| DB\n    DB --> PUSH\n    PUSH --> GH1\n    PUSH --> GH2\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Config locationWhat it holdsManaged via
value.uploadWorker in environments configApp name, HMAC secret, AES key, memory, scale, org-scoped Fly apiKeyGlobal Settings tab in admin UI
value.environments[].secrets per routing envINTEGRATION_WORKER_URL, _SHARED_SECRET, _ENCRYPTION_KEYPer-environment settings in admin UI
GitHub secret INTEGRATION_WORKER_SECRETS_JSONJSON blob with HMAC + AES keys"Push to GitHub" button (synced alongside CONFIGS_JSON)
GitHub variable INTEGRATION_WORKER_APP_NAMEFly app name"Push to GitHub" button
\n

Key Rotation

\n

To rotate the worker's HMAC or AES key:

\n
    \n
  1. Edit the key in the Global Settings tab (Upload Worker Configuration section)
    2. \n
  2. Update the matching key in every routing environment's secrets (currently EKWG only)
    3. \n
  3. Click Save Configuration
    4. \n
  4. Go to GitHub Secrets tab and click Push to GitHub — this updates both CONFIGS_JSON and INTEGRATION_WORKER_SECRETS_JSON atomically
    5. \n
  5. Redeploy the worker and the routing environment(s)
    6. \n
\n

Cloudflare Configuration

\n

Cloudflare configuration deserves special attention because it contains a sensitive API token that must never reach the frontend, yet the frontend needs certain non-sensitive fields to render the committee email routing UI.

\n

Encryption at Deploy Time

\n

During deployment, the Cloudflare config is encrypted using AES-256-GCM before being stored as a Fly.io secret:

\n
flowchart LR\n    DBCF@{ icon: "ngx:mongodb", label: "MongoDB config", pos: "b", h: 48 }\n    ENVVAR@{ icon: "logos:fly-icon", label: "CLOUDFLARE_CONFIG", pos: "b", h: 48 }\n\n    DBCF --> ENCRYPT["AES-256-GCM encrypt"] --> ENVVAR\n
\n

The encrypted payload structure is:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
BytesContent
0-12Initialisation vector (IV)
12-28Authentication tag
28+Ciphertext
\n

Runtime: Backend to Frontend

\n

At runtime, the backend decrypts CLOUDFLARE_CONFIG and uses the full config (including apiToken) for Cloudflare API calls. The frontend only receives non-sensitive fields:

\n
flowchart LR\n    subgraph backend["Backend"]\n        DECRYPT["Decrypt"] --> FULL["Full config"]\n        FULL --> API@{ icon: "ngx:cloudflare", label: "Cloudflare API", pos: "b", h: 48 }\n    end\n\n    subgraph frontend["Frontend"]\n        SAFE["Safe config"] --> UI@{ icon: "ngx:ramblers", label: "Committee UI", pos: "b", h: 48 }\n    end\n\n    FULL -->|"Non-sensitive only"| SAFE\n    UI -->|"Via backend API"| FULL\n
\n

Non-Sensitive Config Fields

\n

The /api/cloudflare/email-routing/config endpoint returns only:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
FieldTypePurpose
configuredbooleanWhether Cloudflare is set up for this environment
accountIdstringUsed to build Cloudflare dashboard links
zoneIdstringUsed to build Cloudflare dashboard links
baseDomainstringThe domain for email routing (e.g. nwkramblers.org.uk)
ownsZonebooleanWhether this site owns its Cloudflare zone or is a subdomain of a shared zone (drives the apex-vs-subdomain inbox setup)
zoneNamestringThe resolved Cloudflare zone name for the domain
\n

The apiToken is never sent to the frontend. All Cloudflare operations go through authenticated backend endpoints.

\n

Fly API Token & the FLY_CONFIG Bundle

\n

The Fly API token follows the identical encrypted-bundle pattern as the Cloudflare config. Each environment carries one org-scoped Fly API token in flyio.apiKey (alongside flyio.appName) in the ENVIRONMENTS config document in MongoDB. That single token does everything Fly-related for the environment: it authenticates the deploy (flyctl deploy and flyctl secrets import), it restarts a stuck machine from the NGX admin UI, and it reads Fly's Prometheus metrics for the Memory Diagnostics charts.

\n

Encryption at Deploy Time

\n

During deployment, buildSecretsFromDatabaseConfig() encrypts {apiToken, appName} into a FLY_CONFIG Fly secret using AES-256-GCM keyed by ENVIRONMENT_SETUP_API_KEY — the same mechanism, same encryption key, and same payload layout used for CLOUDFLARE_CONFIG.

\n
flowchart LR\n    DBFLY@{ icon: "ngx:mongodb", label: "flyio.apiKey + appName", pos: "b", h: 48 }\n    ENVVAR@{ icon: "logos:fly-icon", label: "FLY_CONFIG", pos: "b", h: 48 }\n\n    DBFLY --> ENCRYPT["AES-256-GCM encrypt"] --> ENVVAR\n
\n

Runtime Token Resolution

\n

At runtime, flyRuntimeConfig() resolves the token in a fixed precedence order and stops at the first source that yields both a token and an app name:

\n
    \n
  1. Fly-injected environment variables — Fly sets FLY_APP_NAME and FLY_MACHINE_ID on every deployed machine; combined with a FLY_API_TOKEN env var this is the fast path
    2. \n
  2. Decrypted FLY_CONFIG bundle — decrypted with ENVIRONMENT_SETUP_API_KEY, this is how deployed machines normally obtain the token
    3. \n
  3. Environments config in the database — matched on the Mongo database name, used as a last-resort fallback
    4. \n
\n
flowchart LR\n    START@{ icon: "logos:fly-icon", label: "flyRuntimeConfig()", pos: "b", h: 48 }\n    ENV@{ icon: "logos:fly-icon", label: "FLY_* env vars", pos: "b", h: 48 }\n    BUNDLE@{ icon: "logos:fly-icon", label: "Decrypt FLY_CONFIG", pos: "b", h: 48 }\n    DB@{ icon: "ngx:mongodb", label: "Environments config", pos: "b", h: 48 }\n    TOKEN["Resolved token + appName"]\n\n    START --> ENV\n    ENV -->|"token + appName present"| TOKEN\n    ENV -->|"missing"| BUNDLE\n    BUNDLE -->|"decrypted"| TOKEN\n    BUNDLE -->|"absent"| DB\n    DB --> TOKEN\n
\n

What the Token Powers

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
UseWhereNotes
Deploy + flyctl secrets importCI deploy runnerThe per-environment apiKey is loaded into FLY_API_TOKEN for that environment's flyctl commands
Restart a stuck machineAdmin UI (#304)restartCurrentMachine() calls the Fly Machines API with the resolved token
Read memory / metric historyMemory Diagnostics charts (#304)fly-metrics queries Fly's Prometheus API with the resolved token
\n

An org-scoped token is required for the metrics path: an app-scoped token can manage and restart the app but returns 403 from Fly's Prometheus API.

\n

Originally five environments carried a separate app-scoped deploy token plus a distinct org-scoped metricsToken. On 9 Jul 2026 this was consolidated to a single org-scoped apiKey per environment and the metricsToken field was removed, so one token now covers deploy, restart and metrics.

\n

Required Environment Variables

\n

Origin Environments

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
VariableRequiredPurpose
AUTH_SECRETYesJWT authentication secret
AWS_ACCESS_KEY_IDYesS3 file storage (also reused to upload Serenity reports)
AWS_SECRET_ACCESS_KEYYesS3 file storage
AWS_REGIONYesS3 region (e.g. eu-west-2)
AWS_BUCKETYesS3 bucket name
MONGODB_URIYesDatabase connection string
CLOUDFLARE_CONFIGNoEncrypted Cloudflare config (enables email routing)
FLY_CONFIGNoEncrypted per-environment Fly token + app name (enables machine restart + metrics)
ENVIRONMENT_SETUP_API_KEYNoEncryption key for both CLOUDFLARE_CONFIG and FLY_CONFIG
INTEGRATION_WORKER_URLNoWorker URL (enables remote upload routing)
INTEGRATION_WORKER_SHARED_SECRETIf worker URL setHMAC signing key for worker requests
INTEGRATION_WORKER_ENCRYPTION_KEYIf worker URL setAES key for credential encryption
\n

Worker App

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
VariableRequiredPurpose
INTEGRATION_WORKER_SHARED_SECRETYesHMAC signature validation
INTEGRATION_WORKER_ENCRYPTION_KEYYesCredential + report-credential decryption
CHROME_BINBuilt-inChrome path (baked into Docker image via ENV)
CHROMEDRIVER_PATHBuilt-inChromedriver path (baked into Docker image via ENV)
\n

The worker deliberately has no MONGODB_URI. RAMBLERS_USERNAME and RAMBLERS_PASSWORD are set by the runner from the decrypted job payload immediately before Serenity spawns, so Login.performAs takes its env-var fast path and never attempts a Mongo lookup.

\n

Deploy-Only Variables

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
VariableRequiredPurpose
FLY_API_TOKENDeploy onlyGitHub secret — the deploy runner's ambient Fly authentication. Distinct from the per-environment flyio.apiKey, which the deploy loads into FLY_API_TOKEN per environment and which is also baked into that environment's FLY_CONFIG secret
ADMIN_MONGODB_URIDeploy onlyStaging database for secrets import
INTEGRATION_WORKER_SECRETS_JSONDeploy onlyGitHub secret — worker HMAC + AES keys as JSON
INTEGRATION_WORKER_APP_NAMEDeploy onlyGitHub variable — worker Fly app name
\n

Issue History

\n

The secrets and deployment infrastructure was built across several issues:

\n
flowchart BT\n    I146@{ icon: "logos:github-icon", label: "#146 Env Setup", pos: "b", h: 48 }\n    I156@{ icon: "ngx:cloudflare", label: "#156 DNS/SSL", pos: "b", h: 48 }\n    I125@{ icon: "ngx:cloudflare", label: "#125 Email Routing", pos: "b", h: 48 }\n    I82["#82 Brevo Auth"]\n    I130["#130 Replies Rejected"]\n    I114@{ icon: "logos:fly-icon", label: "#114 Integration Worker", pos: "b", h: 48 }\n    I304@{ icon: "logos:fly-icon", label: "#304 Machine Restart & Metrics", pos: "b", h: 48 }\n    TODAY@{ icon: "logos:fly-icon", label: "Today: Worker, Fly Tokens & Secrets", pos: "b", h: 48 }\n\n    I146 -->|"secrets"| I125\n    I146 -->|"config"| I156\n    I156 -->|"Cloudflare"| I125\n    I156 -->|"DNS"| I82\n    I130 -->|"motivated"| I125\n    I125 --> TODAY\n    I82 --> TODAY\n    I114 -->|"worker + reports + lifecycle"| TODAY\n    I304 -->|"Fly token + restart + metrics"| TODAY\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
IssueTitleKey Contribution
#146Automated environment creationDatabase-first secrets loading, GitHub secrets sync, global AWS
#156DNS/SSL automationCloudflare DNS and certificate management in environment setup
#130Email replies rejectedThe original problem that drove Cloudflare email routing work
#125Cloudflare email routingCommittee email forwarding, worker scripts, email logs UI
#82Brevo domain authenticationDKIM/SPF DNS records via Cloudflare for email deliverability
#114Distributed integration workerShared worker (Serenity uploads, browser fetch, Flickr scraping, migration, resize, document conversion), queued job execution, dependency-graph deploy detection, S3 Serenity report upload, lifecycle audit events, env-var login short-circuit, .env overrides for local dev
#304Machine restart & memory diagnosticsPer-environment org-scoped Fly token encrypted into FLY_CONFIG, admin restart of a stuck Fly machine, Fly Prometheus metric-history charts, consolidation of deploy + metrics tokens into one apiKey
\n

Why No Code Changes Were Needed (Original Cloudflare Work)

\n

The application code already handled Cloudflare gracefully — showing a warning when config was missing and enabling features when config was present. The missing piece was the deployment pipeline: managed environments had no CLOUDFLARE_CONFIG secret because the deploy script did not import secrets from the database. Adding importSecretsFromDatabase() to the deploy flow was the final link in the chain.

\n

What Changed for the Integration Worker

\n

The integration worker required changes across several layers:

\n
    \n
  • Model: UploadWorkerConfig added to EnvironmentsConfig as a sibling to environments[], keeping it out of the deploy-all matrix; IntegrationWorkerEventType.LIFECYCLE added for non-stdout audit markers
    • \n
  • Worker runner: Serenity subprocess uploads target/site/serenity to s3://<envBucket>/ramblers-upload-reports/<csvFileName>/ via uploadSerenityReportToS3, and emits lifecycle progress events around report generation and upload
    • \n
  • Origin audit notifier: new recordLifecycleEvent and recordReportLocation helpers persist the extra rows and broadcast the report location over the WebSocket so the UI can surface a "View Report" link
    • \n
  • Admin UI: New section in the Global Settings tab for worker app name, org-scoped Fly apiKey, HMAC secret, AES key, memory, and scale; Walk Export UI renders the report link against the selected session
    • \n
  • GitHub sync: the "Push to GitHub" action writes INTEGRATION_WORKER_SECRETS_JSON alongside CONFIGS_JSON when worker config exists
    • \n
  • CI workflow: a dependency-graph deploy step (detect-worker-changes.ts) piggybacked on the main build, reusing the same Docker image with a different Fly config (fly.integration-worker.toml) via deploy-integration-worker.ts
    • \n
  • Content-type mapping: contentTypeForExtension + fileContentTypeMappings centralised in projects/ngx-ramblers/src/app/models/content-metadata.model.ts and shared between the S3 report uploader and aws-utils
    • \n
  • Local dev: ngx-cli local dev reads four worker env vars from server/.env so a developer can flip between a local worker and the Fly worker without redeploying
    • \n
\n"}