09-Jul-2026 — Secrets, Deployment, Integration Worker, Fly Tokens, and Cloudflare Configuration #114 #125 #304

This article explains how secrets flow through NGX Ramblers — from storage, through the deployment pipeline, to runtime use by the backend and frontend. It covers the three-tier secret storage model, the Fly.io deployment process, the shared integration worker (browser automation, HTML fetch, Flickr scraping, document conversion, migration jobs and Serenity walks uploads, with report upload and lifecycle audit events), the per-environment Fly API token encrypted into the FLY_CONFIG bundle (powering machine restart and memory-diagnostics metrics), and how Cloudflare email routing configuration reaches managed environments.

Overview

NGX Ramblers uses a layered approach to secret management. Secrets live in three places, with a clearly-defined precedence: CI reads from MongoDB, local development reads from server/.env and non-vcs/secrets/ files, and GitHub Secrets carry only the handful of bootstrap values the deploy script itself needs.

A shared integration worker runs as a separate Fly.io app alongside the origin environments. Its secrets follow the same DB-first model but are synced to GitHub as a separate secret (INTEGRATION_WORKER_SECRETS_JSON) alongside the main CONFIGS_JSON.

flowchart LR
    GH@{ icon: "logos:github-icon", label: "GitHub Secrets", pos: "b", h: 48 }
    DB@{ icon: "ngx:mongodb", label: "MongoDB Atlas", pos: "b", h: 48 }
    LOCAL@{ icon: "ngx:file", label: "Local Files", pos: "b", h: 48 }

    GH --- DB --- LOCAL

Deployment Flow

When code is pushed to main, or a manual deployment is triggered, the following sequence occurs:

flowchart LR
    GHA@{ icon: "logos:github-actions", label: "GitHub Actions", pos: "b", h: 48 }
    TS@{ icon: "logos:typescript-icon", label: "Deploy Script", pos: "b", h: 48 }
    QUERY@{ icon: "ngx:mongodb", label: "Query MongoDB", pos: "b", h: 48 }
    DEPLOY@{ icon: "logos:fly-icon", label: "Deploy to Fly.io", pos: "b", h: 48 }

    GHA -->|"Run deploy"| TS
    TS -->|"ADMIN_MONGODB_URI set"| QUERY
    QUERY --> MERGE["Encrypt & merge secrets"]
    MERGE --> IMPORT["flyctl secrets import"]
    IMPORT --> DEPLOY

How the deploy runner reads from MongoDB

The GitHub Actions runner opens a short-lived connection to the staging MongoDB cluster during each deploy. The connection string lives in the GitHub secret ADMIN_MONGODB_URI; the deploy script aliases it to MONGODB_URI and uses it to read the ENVIRONMENTS config document from the config collection. Per-environment secrets never pass through CONFIGS_JSON — they travel directly from MongoDB into flyctl secrets import on the target Fly app.

Secret Building Priority

During deployment, buildSecretsFromDatabaseConfig() merges secrets in a specific order. Later values override earlier ones:

1. Global secrets from globalConfig.secrets
2. Global AWS config from globalConfig.aws
3. Per-environment AWS config (overrides global if present)
4. MongoDB URI built from envConfig.mongo credentials
5. NGX_LITE flag derived from envConfig.ngxLite
6. Per-environment secrets from envConfig.secrets
7. Integration worker callback base URL — defaulted to https://<appName>.fly.dev when not already set
8. Fly config — {apiToken, appName} encrypted with AES-256-GCM and stored as FLY_CONFIG (from envConfig.flyio.apiKey)
9. Cloudflare config — encrypted with AES-256-GCM and stored as CLOUDFLARE_CONFIG

Database vs File Import

The deploy script has exactly two modes in importSecrets():

• CI mode (the only mode exercised on a GitHub runner): ADMIN_MONGODB_URI is always set, so importSecretsFromDatabase() runs. The runner has no access to non-vcs/secrets/ — those files are git-ignored and simply don't exist on the runner, so the file path can never execute.
• Local-developer mode: when a developer runs deploy-to-environments.ts from a laptop without ADMIN_MONGODB_URI set, importSecretsFromFile() reads non-vcs/secrets/secrets.<appName>.env from disk. This is a convenience for local testing of the deploy script itself — it is not a production fallback.

flowchart LR
    START@{ icon: "ngx:ramblers", label: "deploy script", pos: "b", h: 48 }
    QUERY@{ icon: "ngx:mongodb", label: "Query MongoDB", pos: "b", h: 48 }
    FILE@{ icon: "ngx:file", label: "Read non-vcs/secrets/*.env", pos: "b", h: 48 }
    IMPORT@{ icon: "logos:fly-icon", label: "flyctl secrets import", pos: "b", h: 48 }

    START --> CI{"ADMIN_MONGODB_URI set?"}
    CI -->|"Yes (CI)"| QUERY
    QUERY --> IMPORT
    CI -->|"No (local dev)"| FILE
    FILE --> IMPORT

If importSecretsFromDatabase() returns empty for an environment, the CI run simply continues without importing — Fly keeps whatever secrets are already on the app. There is no "file fallback" on the runner because there are no files.

Integration Worker Deployment

The integration worker is a general machine-to-machine worker. It runs Serenity browser automation for Ramblers walks uploads, but also handles synchronous browser operations (HTML fetch, Flickr album scraping, migration scrapes), image resizing, and document conversion. It is deployed as a separate Fly.io app (ngx-ramblers-integration-worker) that shares the same Docker image as the origin but runs npm run worker-server (which starts integration-worker-server) instead of npm run server.

The worker exposes several route groups, all under /api/integration-worker:

The walks-upload flow below is the most involved of these and is documented in detail; the other route groups follow the same signed request/response pattern.

flowchart LR
    PUSH@{ icon: "logos:github-icon", label: "Push to main", pos: "b", h: 48 }
    BUILD@{ icon: "logos:docker-icon", label: "Build image", pos: "b", h: 48 }
    ORIGIN@{ icon: "logos:fly-icon", label: "Deploy origin", pos: "b", h: 48 }
    FILTER["Worker change detection"]
    WORKER@{ icon: "logos:fly-icon", label: "Deploy worker", pos: "b", h: 48 }

    PUSH --> BUILD
    BUILD --> ORIGIN
    BUILD --> FILTER
    FILTER -->|"Runtime changes in worker graph"| WORKER
    FILTER -->|"No change"| SKIP["Skip — keep warm"]

How Worker Deployment Works

• The main build workflow (build-push-and-deploy-ngx-ramblers-docker-image.yml) runs a Detect worker-relevant changes step (server/deploy/detect-worker-changes.ts) that decides whether the worker needs redeploying
• When worker-relevant runtime changes are found, the workflow builds and pushes a ngx-ramblers:integration-worker-<run_number> image and a conditional Deploy Ramblers upload worker step runs server/deploy/deploy-integration-worker.ts with that image tag
• When nothing worker-relevant changed, the build and deploy steps are skipped entirely — the worker stays warm with no restart
• After a successful worker deploy, the workflow advances a worker-deployed git ref (force-pushed to HEAD) so the next run can diff against the last thing actually deployed to the worker, not merely the previous commit
• A standalone workflow_dispatch-only workflow (build-and-deploy-integration-worker.yml) exists as a manual escape hatch for redeploying a chosen image tag

Worker Change Detection

Rather than a static path glob, detect-worker-changes.ts walks the worker's real dependency graph. It runs madge from the worker entry point (server/lib/ramblers/integration-worker-server.ts) to resolve every .ts file the worker actually imports, then unions that set with a fixed list of infrastructure files. A change is only treated as worker-relevant if it falls inside that set and changes the emitted JavaScript — type-only edits (identical transpiled output) are ignored.

The dependency graph plus the following explicit infrastructure files make up the detection scope:

Changes are compared against the worker-deployed ref when it exists, otherwise against HEAD~1.

Routing Uploads to the Worker

Environments route uploads to the worker by setting INTEGRATION_WORKER_URL in their per-environment secrets. When set, the dispatcher sends signed job requests to the worker instead of spawning Serenity locally. The worker exposes three walks-upload endpoints under /api/integration-worker:

flowchart LR
    subgraph origin["Origin (e.g. EKWG)"]
        DISPATCH@{ icon: "ngx:express", label: "Dispatcher", pos: "b", h: 48 }
    end
    subgraph worker["Worker"]
        RECEIVE@{ icon: "ngx:express", label: "Job receiver", pos: "b", h: 48 }
        SERENITY["Serenity spawn"]
    end

    DISPATCH -->|"POST /jobs — HMAC signed, AES-encrypted credentials"| RECEIVE
    RECEIVE --> SERENITY
    SERENITY -->|"POST /progress & /result — signed callbacks"| DISPATCH

    style origin fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style worker fill:#EEE8F5,stroke:#AB9BC8,stroke-width:2px,rx:12,ry:12,color:#404143

Credentials are AES-encrypted in the job request and decrypted on the worker side. The worker never connects to MongoDB — Login.performAs short-circuits its Mongo lookup when RAMBLERS_USERNAME and RAMBLERS_PASSWORD are set in the environment (they always are on the worker, because the runner writes them from the decrypted payload before spawning Serenity).

Serenity Report Upload to S3

Every upload run — success or failure — uploads the complete Serenity HTML report to the origin's own S3 bucket so that operators can open the detailed test output from the Walk Export UI.

flowchart LR
    subgraph worker["Worker"]
        RUN["Serenity run"] --> GEN["Report generated at target/site/serenity"]
        GEN --> UP@{ icon: "ngx:aws", label: "uploadSerenityReportToS3", pos: "b", h: 48 }
    end
    subgraph origin["Origin"]
        RESULT@{ icon: "ngx:express", label: "POST /result", pos: "b", h: 48 }
        AUDIT@{ icon: "ngx:mongodb", label: "recordReportLocation", pos: "b", h: 48 }
        WS["WebSocket broadcast"]
        UI@{ icon: "ngx:ramblers", label: "View Report button", pos: "b", h: 48 }
    end
    S3@{ icon: "ngx:aws", label: "S3 ramblers-upload-reports/<csvFileName>/", pos: "b", h: 48 }

    UP --> S3
    UP -->|"reportBucket + reportKeyPrefix"| RESULT
    RESULT --> AUDIT
    AUDIT --> WS --> UI
    UI -->|"/api/aws/s3/.../index.html"| S3

    style origin fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style worker fill:#EEE8F5,stroke:#AB9BC8,stroke-width:2px,rx:12,ry:12,color:#404143

End-to-end flow:

1. Origin's dispatcher computes reportKeyPrefix = ramblers-upload-reports/<csvFileName without .csv> and sends it, together with AES-encrypted S3 credentials (reusing the environment's own AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) in the /jobs request.
2. After the Serenity subprocess exits, finishJob() calls uploadSerenityReportToS3() which walks target/site/serenity and uploads every file via uploadDirectoryToS3, using the centralised contentTypeForExtension / fileContentTypeMappings helpers in projects/ngx-ramblers/src/app/models/content-metadata.model.ts (shared with the origin's AWS utilities).
3. The worker then posts /result with reportBucket and reportKeyPrefix populated.
4. The origin's recordReportLocation persists the prefix onto the audit session and broadcasts an audit row over the WebSocket so the Walk Export UI renders a View Report button next to Back To Walks Admin.
5. The UI fetches the report through the existing /api/aws/s3/.../index.html proxy — no public bucket access, no extra credentials.

Lifecycle Audit Events

Beyond the per-step SUCCESS / TEST_STEP rows parsed from stdout, the worker emits additional lifecycle markers via the IntegrationWorkerEventType.LIFECYCLE type. The origin's recordLifecycleEvent writes each one as a SUMMARY audit row and pushes it over the WebSocket:

These make the "silent" stretch between the last scenario step and the final result callback visible in the audit log.

Local Developer Ergonomics

ngx-cli local dev honours four worker-related overrides in server/.env:

• INTEGRATION_WORKER_URL
• INTEGRATION_WORKER_CALLBACK_BASE_URL
• INTEGRATION_WORKER_SHARED_SECRET
• INTEGRATION_WORKER_ENCRYPTION_KEY

The local.ts command reads these values via readServerEnvValue and applies them on top of the secrets resolved from the database (see SERVER_ENV_OVERRIDE_KEYS). A developer can therefore flip between a locally-running worker container and the Fly-hosted worker purely by editing .env, without touching the database or redeploying anything.

Worker Secret Management

Worker configuration lives in the uploadWorker field of the environments config document — alongside environments[], not inside it. This ensures "deploy all environments" never touches the worker.

flowchart LR
    UI@{ icon: "ngx:ramblers", label: "Admin UI", pos: "b", h: 48 }
    DB@{ icon: "ngx:mongodb", label: "MongoDB config", pos: "b", h: 48 }
    PUSH["Push to GitHub"]
    GH1@{ icon: "logos:github-icon", label: "CONFIGS_JSON", pos: "b", h: 48 }
    GH2@{ icon: "logos:github-icon", label: "INTEGRATION_WORKER_SECRETS_JSON", pos: "b", h: 48 }

    UI -->|"Save"| DB
    DB --> PUSH
    PUSH --> GH1
    PUSH --> GH2

Key Rotation

To rotate the worker's HMAC or AES key:

1. Edit the key in the Global Settings tab (Upload Worker Configuration section)
2. Update the matching key in every routing environment's secrets (currently EKWG only)
3. Click Save Configuration
4. Go to GitHub Secrets tab and click Push to GitHub — this updates both CONFIGS_JSON and INTEGRATION_WORKER_SECRETS_JSON atomically
5. Redeploy the worker and the routing environment(s)

Cloudflare Configuration

Cloudflare configuration deserves special attention because it contains a sensitive API token that must never reach the frontend, yet the frontend needs certain non-sensitive fields to render the committee email routing UI.

Encryption at Deploy Time

During deployment, the Cloudflare config is encrypted using AES-256-GCM before being stored as a Fly.io secret:

flowchart LR
    DBCF@{ icon: "ngx:mongodb", label: "MongoDB config", pos: "b", h: 48 }
    ENVVAR@{ icon: "logos:fly-icon", label: "CLOUDFLARE_CONFIG", pos: "b", h: 48 }

    DBCF --> ENCRYPT["AES-256-GCM encrypt"] --> ENVVAR

The encrypted payload structure is:

Runtime: Backend to Frontend

At runtime, the backend decrypts CLOUDFLARE_CONFIG and uses the full config (including apiToken) for Cloudflare API calls. The frontend only receives non-sensitive fields:

flowchart LR
    subgraph backend["Backend"]
        DECRYPT["Decrypt"] --> FULL["Full config"]
        FULL --> API@{ icon: "ngx:cloudflare", label: "Cloudflare API", pos: "b", h: 48 }
    end

    subgraph frontend["Frontend"]
        SAFE["Safe config"] --> UI@{ icon: "ngx:ramblers", label: "Committee UI", pos: "b", h: 48 }
    end

    FULL -->|"Non-sensitive only"| SAFE
    UI -->|"Via backend API"| FULL

Non-Sensitive Config Fields

The /api/cloudflare/email-routing/config endpoint returns only:

The apiToken is never sent to the frontend. All Cloudflare operations go through authenticated backend endpoints.

Fly API Token & the FLY_CONFIG Bundle

The Fly API token follows the identical encrypted-bundle pattern as the Cloudflare config. Each environment carries one org-scoped Fly API token in flyio.apiKey (alongside flyio.appName) in the ENVIRONMENTS config document in MongoDB. That single token does everything Fly-related for the environment: it authenticates the deploy (flyctl deploy and flyctl secrets import), it restarts a stuck machine from the NGX admin UI, and it reads Fly's Prometheus metrics for the Memory Diagnostics charts.

Encryption at Deploy Time

During deployment, buildSecretsFromDatabaseConfig() encrypts {apiToken, appName} into a FLY_CONFIG Fly secret using AES-256-GCM keyed by ENVIRONMENT_SETUP_API_KEY — the same mechanism, same encryption key, and same payload layout used for CLOUDFLARE_CONFIG.

flowchart LR
    DBFLY@{ icon: "ngx:mongodb", label: "flyio.apiKey + appName", pos: "b", h: 48 }
    ENVVAR@{ icon: "logos:fly-icon", label: "FLY_CONFIG", pos: "b", h: 48 }

    DBFLY --> ENCRYPT["AES-256-GCM encrypt"] --> ENVVAR

Runtime Token Resolution

At runtime, flyRuntimeConfig() resolves the token in a fixed precedence order and stops at the first source that yields both a token and an app name:

1. Fly-injected environment variables — Fly sets FLY_APP_NAME and FLY_MACHINE_ID on every deployed machine; combined with a FLY_API_TOKEN env var this is the fast path
2. Decrypted FLY_CONFIG bundle — decrypted with ENVIRONMENT_SETUP_API_KEY, this is how deployed machines normally obtain the token
3. Environments config in the database — matched on the Mongo database name, used as a last-resort fallback

flowchart LR
    START@{ icon: "logos:fly-icon", label: "flyRuntimeConfig()", pos: "b", h: 48 }
    ENV@{ icon: "logos:fly-icon", label: "FLY_* env vars", pos: "b", h: 48 }
    BUNDLE@{ icon: "logos:fly-icon", label: "Decrypt FLY_CONFIG", pos: "b", h: 48 }
    DB@{ icon: "ngx:mongodb", label: "Environments config", pos: "b", h: 48 }
    TOKEN["Resolved token + appName"]

    START --> ENV
    ENV -->|"token + appName present"| TOKEN
    ENV -->|"missing"| BUNDLE
    BUNDLE -->|"decrypted"| TOKEN
    BUNDLE -->|"absent"| DB
    DB --> TOKEN

What the Token Powers

An org-scoped token is required for the metrics path: an app-scoped token can manage and restart the app but returns 403 from Fly's Prometheus API.

Originally five environments carried a separate app-scoped deploy token plus a distinct org-scoped metricsToken. On 9 Jul 2026 this was consolidated to a single org-scoped apiKey per environment and the metricsToken field was removed, so one token now covers deploy, restart and metrics.

Required Environment Variables

Origin Environments

Worker App

The worker deliberately has no MONGODB_URI. RAMBLERS_USERNAME and RAMBLERS_PASSWORD are set by the runner from the decrypted job payload immediately before Serenity spawns, so Login.performAs takes its env-var fast path and never attempts a Mongo lookup.

Deploy-Only Variables

Issue History

The secrets and deployment infrastructure was built across several issues:

flowchart BT
    I146@{ icon: "logos:github-icon", label: "#146 Env Setup", pos: "b", h: 48 }
    I156@{ icon: "ngx:cloudflare", label: "#156 DNS/SSL", pos: "b", h: 48 }
    I125@{ icon: "ngx:cloudflare", label: "#125 Email Routing", pos: "b", h: 48 }
    I82["#82 Brevo Auth"]
    I130["#130 Replies Rejected"]
    I114@{ icon: "logos:fly-icon", label: "#114 Integration Worker", pos: "b", h: 48 }
    I304@{ icon: "logos:fly-icon", label: "#304 Machine Restart & Metrics", pos: "b", h: 48 }
    TODAY@{ icon: "logos:fly-icon", label: "Today: Worker, Fly Tokens & Secrets", pos: "b", h: 48 }

    I146 -->|"secrets"| I125
    I146 -->|"config"| I156
    I156 -->|"Cloudflare"| I125
    I156 -->|"DNS"| I82
    I130 -->|"motivated"| I125
    I125 --> TODAY
    I82 --> TODAY
    I114 -->|"worker + reports + lifecycle"| TODAY
    I304 -->|"Fly token + restart + metrics"| TODAY

Why No Code Changes Were Needed (Original Cloudflare Work)

The application code already handled Cloudflare gracefully — showing a warning when config was missing and enabling features when config was present. The missing piece was the deployment pipeline: managed environments had no CLOUDFLARE_CONFIG secret because the deploy script did not import secrets from the database. Adding importSecretsFromDatabase() to the deploy flow was the final link in the chain.

What Changed for the Integration Worker

The integration worker required changes across several layers:

• Model: UploadWorkerConfig added to EnvironmentsConfig as a sibling to environments[], keeping it out of the deploy-all matrix; IntegrationWorkerEventType.LIFECYCLE added for non-stdout audit markers
• Worker runner: Serenity subprocess uploads target/site/serenity to s3://<envBucket>/ramblers-upload-reports/<csvFileName>/ via uploadSerenityReportToS3, and emits lifecycle progress events around report generation and upload
• Origin audit notifier: new recordLifecycleEvent and recordReportLocation helpers persist the extra rows and broadcast the report location over the WebSocket so the UI can surface a "View Report" link
• Admin UI: New section in the Global Settings tab for worker app name, org-scoped Fly apiKey, HMAC secret, AES key, memory, and scale; Walk Export UI renders the report link against the selected session
• GitHub sync: the "Push to GitHub" action writes INTEGRATION_WORKER_SECRETS_JSON alongside CONFIGS_JSON when worker config exists
• CI workflow: a dependency-graph deploy step (detect-worker-changes.ts) piggybacked on the main build, reusing the same Docker image with a different Fly config (fly.integration-worker.toml) via deploy-integration-worker.ts
• Content-type mapping: contentTypeForExtension + fileContentTypeMappings centralised in projects/ngx-ramblers/src/app/models/content-metadata.model.ts and shared between the S3 report uploader and aws-utils
• Local dev: ngx-cli local dev reads four worker env vars from server/.env so a developer can flip between a local worker and the Fly worker without redeploying