# 09-Jul-2026 — Secrets, Deployment, Integration Worker, Fly Tokens, and Cloudflare Configuration [#114](https://github.com/nbarrett/ngx-ramblers/issues/114) [#125](https://github.com/nbarrett/ngx-ramblers/issues/125) [#304](https://github.com/nbarrett/ngx-ramblers/issues/304)

_____

This article explains how secrets flow through NGX Ramblers — from storage, through the deployment pipeline, to runtime use by the backend and frontend. It covers the three-tier secret storage model, the Fly.io deployment process, the shared integration worker (browser automation, HTML fetch, Flickr scraping, document conversion, migration jobs and Serenity walks uploads, with report upload and lifecycle audit events), the per-environment Fly API token encrypted into the `FLY_CONFIG` bundle (powering machine restart and memory-diagnostics metrics), and how Cloudflare email routing configuration reaches managed environments.

## Overview

NGX Ramblers uses a layered approach to secret management. Secrets live in three places, with a clearly-defined precedence: CI reads from MongoDB, local development reads from `server/.env` and `non-vcs/secrets/` files, and GitHub Secrets carry only the handful of bootstrap values the deploy script itself needs.

A shared integration worker runs as a separate Fly.io app alongside the origin environments. Its secrets follow the same DB-first model but are synced to GitHub as a separate secret (`INTEGRATION_WORKER_SECRETS_JSON`) alongside the main `CONFIGS_JSON`.

```mermaid
flowchart LR
    GH@{ icon: "logos:github-icon", label: "GitHub Secrets", pos: "b", h: 48 }
    DB@{ icon: "ngx:mongodb", label: "MongoDB Atlas", pos: "b", h: 48 }
    LOCAL@{ icon: "ngx:file", label: "Local Files", pos: "b", h: 48 }

    GH --- DB --- LOCAL
```

| Tier | Location | Used By | Contents |
|------|----------|---------|----------|
| **GitHub Secrets** | GitHub Actions | CI/CD pipelines | `CONFIGS_JSON`, `INTEGRATION_WORKER_SECRETS_JSON`, `FLY_API_TOKEN`, `ADMIN_MONGODB_URI`, Docker credentials |
| **MongoDB Atlas** | `config` collection | Runtime + deployment | Per-environment secrets, integration worker config, per-environment Fly token, Cloudflare config, AWS credentials, MongoDB URIs |
| **Local Files** | `server/.env`, `non-vcs/secrets/` | Local development only | Developer overrides and per-app `.env` files, all git-ignored |

## Deployment Flow

When code is pushed to `main`, or a manual deployment is triggered, the following sequence occurs:

```mermaid
flowchart LR
    GHA@{ icon: "logos:github-actions", label: "GitHub Actions", pos: "b", h: 48 }
    TS@{ icon: "logos:typescript-icon", label: "Deploy Script", pos: "b", h: 48 }
    QUERY@{ icon: "ngx:mongodb", label: "Query MongoDB", pos: "b", h: 48 }
    DEPLOY@{ icon: "logos:fly-icon", label: "Deploy to Fly.io", pos: "b", h: 48 }

    GHA -->|"Run deploy"| TS
    TS -->|"ADMIN_MONGODB_URI set"| QUERY
    QUERY --> MERGE["Encrypt & merge secrets"]
    MERGE --> IMPORT["flyctl secrets import"]
    IMPORT --> DEPLOY
```

### How the deploy runner reads from MongoDB

The GitHub Actions runner opens a short-lived connection to the staging MongoDB cluster during each deploy. The connection string lives in the GitHub secret `ADMIN_MONGODB_URI`; the deploy script aliases it to `MONGODB_URI` and uses it to read the `ENVIRONMENTS` config document from the `config` collection. Per-environment secrets never pass through `CONFIGS_JSON` — they travel directly from MongoDB into `flyctl secrets import` on the target Fly app.

### Secret Building Priority

During deployment, `buildSecretsFromDatabaseConfig()` merges secrets in a specific order. Later values override earlier ones:

1. **Global secrets** from `globalConfig.secrets`
2. **Global AWS config** from `globalConfig.aws`
3. **Per-environment AWS config** (overrides global if present)
4. **MongoDB URI** built from `envConfig.mongo` credentials
5. **`NGX_LITE`** flag derived from `envConfig.ngxLite`
6. **Per-environment secrets** from `envConfig.secrets`
7. **Integration worker callback base URL** — defaulted to `https://<appName>.fly.dev` when not already set
8. **Fly config** — `{apiToken, appName}` encrypted with AES-256-GCM and stored as `FLY_CONFIG` (from `envConfig.flyio.apiKey`)
9. **Cloudflare config** — encrypted with AES-256-GCM and stored as `CLOUDFLARE_CONFIG`

### Database vs File Import

The deploy script has exactly two modes in `importSecrets()`:

- **CI mode** (the only mode exercised on a GitHub runner): `ADMIN_MONGODB_URI` is always set, so `importSecretsFromDatabase()` runs. The runner has no access to `non-vcs/secrets/` — those files are git-ignored and simply don't exist on the runner, so the file path can never execute.
- **Local-developer mode**: when a developer runs `deploy-to-environments.ts` from a laptop without `ADMIN_MONGODB_URI` set, `importSecretsFromFile()` reads `non-vcs/secrets/secrets.<appName>.env` from disk. This is a convenience for local testing of the deploy script itself — it is not a production fallback.

```mermaid
flowchart LR
    START@{ icon: "ngx:ramblers", label: "deploy script", pos: "b", h: 48 }
    QUERY@{ icon: "ngx:mongodb", label: "Query MongoDB", pos: "b", h: 48 }
    FILE@{ icon: "ngx:file", label: "Read non-vcs/secrets/*.env", pos: "b", h: 48 }
    IMPORT@{ icon: "logos:fly-icon", label: "flyctl secrets import", pos: "b", h: 48 }

    START --> CI{"ADMIN_MONGODB_URI set?"}
    CI -->|"Yes (CI)"| QUERY
    QUERY --> IMPORT
    CI -->|"No (local dev)"| FILE
    FILE --> IMPORT
```

If `importSecretsFromDatabase()` returns empty for an environment, the CI run simply continues without importing — Fly keeps whatever secrets are already on the app. There is no "file fallback" on the runner because there are no files.

## Integration Worker Deployment

The integration worker is a general machine-to-machine worker. It runs Serenity browser automation for Ramblers walks uploads, but also handles synchronous browser operations (HTML fetch, Flickr album scraping, migration scrapes), image resizing, and document conversion. It is deployed as a separate Fly.io app (`ngx-ramblers-integration-worker`) that shares the same Docker image as the origin but runs `npm run worker-server` (which starts `integration-worker-server`) instead of `npm run server`.

The worker exposes several route groups, all under `/api/integration-worker`:

| Route group | Purpose |
|-------------|---------|
| `/jobs`, `/progress`, `/result` | Ramblers walks upload jobs (queued Serenity runs with signed progress and result callbacks) |
| `/browser/html-fetch`, `/browser/flickr-user-albums` | Synchronous headless-browser HTML fetch and Flickr album scraping |
| `/migration/*` | Legacy-site migration scrapes |
| `/resize/*` | Image resize jobs |
| `/document-conversion/convert` | Document conversion |

The walks-upload flow below is the most involved of these and is documented in detail; the other route groups follow the same signed request/response pattern.

```mermaid
flowchart LR
    PUSH@{ icon: "logos:github-icon", label: "Push to main", pos: "b", h: 48 }
    BUILD@{ icon: "logos:docker-icon", label: "Build image", pos: "b", h: 48 }
    ORIGIN@{ icon: "logos:fly-icon", label: "Deploy origin", pos: "b", h: 48 }
    FILTER["Worker change detection"]
    WORKER@{ icon: "logos:fly-icon", label: "Deploy worker", pos: "b", h: 48 }

    PUSH --> BUILD
    BUILD --> ORIGIN
    BUILD --> FILTER
    FILTER -->|"Runtime changes in worker graph"| WORKER
    FILTER -->|"No change"| SKIP["Skip — keep warm"]
```

### How Worker Deployment Works

- The main build workflow (`build-push-and-deploy-ngx-ramblers-docker-image.yml`) runs a **Detect worker-relevant changes** step (`server/deploy/detect-worker-changes.ts`) that decides whether the worker needs redeploying
- When worker-relevant runtime changes are found, the workflow builds and pushes a `ngx-ramblers:integration-worker-<run_number>` image and a conditional **Deploy Ramblers upload worker** step runs `server/deploy/deploy-integration-worker.ts` with that image tag
- When nothing worker-relevant changed, the build and deploy steps are skipped entirely — the worker stays warm with no restart
- After a successful worker deploy, the workflow advances a `worker-deployed` git ref (force-pushed to `HEAD`) so the next run can diff against the last thing actually deployed to the worker, not merely the previous commit
- A standalone `workflow_dispatch`-only workflow (`build-and-deploy-integration-worker.yml`) exists as a manual escape hatch for redeploying a chosen image tag

### Worker Change Detection

Rather than a static path glob, `detect-worker-changes.ts` walks the worker's real dependency graph. It runs `madge` from the worker entry point (`server/lib/ramblers/integration-worker-server.ts`) to resolve every `.ts` file the worker actually imports, then unions that set with a fixed list of infrastructure files. A change is only treated as worker-relevant if it falls inside that set **and** changes the emitted JavaScript — type-only edits (identical transpiled output) are ignored.

The dependency graph plus the following explicit infrastructure files make up the detection scope:

| Category | Paths |
|----------|-------|
| Worker dependency graph | Every `.ts` file reachable from `server/lib/ramblers/integration-worker-server.ts` (resolved at build time via `madge`) |
| Container + Fly config | `Dockerfile`, `fly.integration-worker.toml` |
| Server packages | `server/package.json`, `server/package-lock.json`, `server/playwright.config.ts` |
| Deploy + detection scripts | `server/deploy/deploy-integration-worker.ts`, `server/deploy/detect-worker-changes.ts` |
| Workflows | `.github/workflows/build-push-and-deploy-ngx-ramblers-docker-image.yml`, `.github/workflows/build-and-deploy-integration-worker.yml` |

Changes are compared against the `worker-deployed` ref when it exists, otherwise against `HEAD~1`.

### Routing Uploads to the Worker

Environments route uploads to the worker by setting `INTEGRATION_WORKER_URL` in their per-environment secrets. When set, the dispatcher sends signed job requests to the worker instead of spawning Serenity locally. The worker exposes three walks-upload endpoints under `/api/integration-worker`:

| Endpoint | Direction | Purpose |
|----------|-----------|---------|
| `POST /jobs` | Origin → Worker | Submit a signed, AES-encrypted job. Worker runs it immediately or queues behind the active job, returning `{queued, queuePosition, activeJobId}` |
| `POST /progress` | Worker → Origin | Stream per-line stdout events, parsed test steps, and lifecycle markers back to the origin |
| `POST /result` | Worker → Origin | Announce completion, exit status, and the S3 report key prefix |

```mermaid
flowchart LR
    subgraph origin["Origin (e.g. EKWG)"]
        DISPATCH@{ icon: "ngx:express", label: "Dispatcher", pos: "b", h: 48 }
    end
    subgraph worker["Worker"]
        RECEIVE@{ icon: "ngx:express", label: "Job receiver", pos: "b", h: 48 }
        SERENITY["Serenity spawn"]
    end

    DISPATCH -->|"POST /jobs — HMAC signed, AES-encrypted credentials"| RECEIVE
    RECEIVE --> SERENITY
    SERENITY -->|"POST /progress & /result — signed callbacks"| DISPATCH

    style origin fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style worker fill:#EEE8F5,stroke:#AB9BC8,stroke-width:2px,rx:12,ry:12,color:#404143
```

Credentials are AES-encrypted in the job request and decrypted on the worker side. The worker never connects to MongoDB — `Login.performAs` short-circuits its Mongo lookup when `RAMBLERS_USERNAME` and `RAMBLERS_PASSWORD` are set in the environment (they always are on the worker, because the runner writes them from the decrypted payload before spawning Serenity).

### Serenity Report Upload to S3

Every upload run — success or failure — uploads the complete Serenity HTML report to the origin's own S3 bucket so that operators can open the detailed test output from the Walk Export UI.

```mermaid
flowchart LR
    subgraph worker["Worker"]
        RUN["Serenity run"] --> GEN["Report generated at target/site/serenity"]
        GEN --> UP@{ icon: "ngx:aws", label: "uploadSerenityReportToS3", pos: "b", h: 48 }
    end
    subgraph origin["Origin"]
        RESULT@{ icon: "ngx:express", label: "POST /result", pos: "b", h: 48 }
        AUDIT@{ icon: "ngx:mongodb", label: "recordReportLocation", pos: "b", h: 48 }
        WS["WebSocket broadcast"]
        UI@{ icon: "ngx:ramblers", label: "View Report button", pos: "b", h: 48 }
    end
    S3@{ icon: "ngx:aws", label: "S3 ramblers-upload-reports/<csvFileName>/", pos: "b", h: 48 }

    UP --> S3
    UP -->|"reportBucket + reportKeyPrefix"| RESULT
    RESULT --> AUDIT
    AUDIT --> WS --> UI
    UI -->|"/api/aws/s3/.../index.html"| S3

    style origin fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style worker fill:#EEE8F5,stroke:#AB9BC8,stroke-width:2px,rx:12,ry:12,color:#404143
```

End-to-end flow:

1. Origin's dispatcher computes `reportKeyPrefix = ramblers-upload-reports/<csvFileName without .csv>` and sends it, together with AES-encrypted S3 credentials (reusing the environment's own `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`) in the `/jobs` request.
2. After the Serenity subprocess exits, `finishJob()` calls `uploadSerenityReportToS3()` which walks `target/site/serenity` and uploads every file via `uploadDirectoryToS3`, using the centralised `contentTypeForExtension` / `fileContentTypeMappings` helpers in `projects/ngx-ramblers/src/app/models/content-metadata.model.ts` (shared with the origin's AWS utilities).
3. The worker then posts `/result` with `reportBucket` and `reportKeyPrefix` populated.
4. The origin's `recordReportLocation` persists the prefix onto the audit session and broadcasts an audit row over the WebSocket so the Walk Export UI renders a **View Report** button next to **Back To Walks Admin**.
5. The UI fetches the report through the existing `/api/aws/s3/.../index.html` proxy — no public bucket access, no extra credentials.

### Lifecycle Audit Events

Beyond the per-step `SUCCESS` / `TEST_STEP` rows parsed from stdout, the worker emits additional lifecycle markers via the `IntegrationWorkerEventType.LIFECYCLE` type. The origin's `recordLifecycleEvent` writes each one as a `SUMMARY` audit row and pushes it over the WebSocket:

| Marker | When |
|--------|------|
| `Scenario execution complete` | First time the Serenity report writer prints `[report]` to stderr |
| `Serenity report generation starting` | Immediately after, before the report writer runs |
| `Serenity report generation complete, uploading to S3` | After Serenity exits, before `uploadSerenityReportToS3` is invoked |
| `Report upload to S3 complete` | After the report directory has been uploaded to S3, before the final result callback |

These make the "silent" stretch between the last scenario step and the final result callback visible in the audit log.

### Local Developer Ergonomics

`ngx-cli local dev` honours four worker-related overrides in `server/.env`:

- `INTEGRATION_WORKER_URL`
- `INTEGRATION_WORKER_CALLBACK_BASE_URL`
- `INTEGRATION_WORKER_SHARED_SECRET`
- `INTEGRATION_WORKER_ENCRYPTION_KEY`

The `local.ts` command reads these values via `readServerEnvValue` and applies them on top of the secrets resolved from the database (see `SERVER_ENV_OVERRIDE_KEYS`). A developer can therefore flip between a locally-running worker container and the Fly-hosted worker purely by editing `.env`, without touching the database or redeploying anything.

## Worker Secret Management

Worker configuration lives in the `uploadWorker` field of the environments config document — alongside `environments[]`, not inside it. This ensures "deploy all environments" never touches the worker.

```mermaid
flowchart LR
    UI@{ icon: "ngx:ramblers", label: "Admin UI", pos: "b", h: 48 }
    DB@{ icon: "ngx:mongodb", label: "MongoDB config", pos: "b", h: 48 }
    PUSH["Push to GitHub"]
    GH1@{ icon: "logos:github-icon", label: "CONFIGS_JSON", pos: "b", h: 48 }
    GH2@{ icon: "logos:github-icon", label: "INTEGRATION_WORKER_SECRETS_JSON", pos: "b", h: 48 }

    UI -->|"Save"| DB
    DB --> PUSH
    PUSH --> GH1
    PUSH --> GH2
```

| Config location | What it holds | Managed via |
|----------------|---------------|-------------|
| `value.uploadWorker` in environments config | App name, HMAC secret, AES key, memory, scale, org-scoped Fly `apiKey` | Global Settings tab in admin UI |
| `value.environments[].secrets` per routing env | `INTEGRATION_WORKER_URL`, `_SHARED_SECRET`, `_ENCRYPTION_KEY` | Per-environment settings in admin UI |
| GitHub secret `INTEGRATION_WORKER_SECRETS_JSON` | JSON blob with HMAC + AES keys | "Push to GitHub" button (synced alongside `CONFIGS_JSON`) |
| GitHub variable `INTEGRATION_WORKER_APP_NAME` | Fly app name | "Push to GitHub" button |

### Key Rotation

To rotate the worker's HMAC or AES key:

1. Edit the key in the **Global Settings** tab (Upload Worker Configuration section)
2. Update the matching key in every routing environment's secrets (currently EKWG only)
3. Click **Save Configuration**
4. Go to **GitHub Secrets** tab and click **Push to GitHub** — this updates both `CONFIGS_JSON` and `INTEGRATION_WORKER_SECRETS_JSON` atomically
5. Redeploy the worker and the routing environment(s)

## Cloudflare Configuration

Cloudflare configuration deserves special attention because it contains a sensitive API token that must never reach the frontend, yet the frontend needs certain non-sensitive fields to render the committee email routing UI.

### Encryption at Deploy Time

During deployment, the Cloudflare config is encrypted using AES-256-GCM before being stored as a Fly.io secret:

```mermaid
flowchart LR
    DBCF@{ icon: "ngx:mongodb", label: "MongoDB config", pos: "b", h: 48 }
    ENVVAR@{ icon: "logos:fly-icon", label: "CLOUDFLARE_CONFIG", pos: "b", h: 48 }

    DBCF --> ENCRYPT["AES-256-GCM encrypt"] --> ENVVAR
```

The encrypted payload structure is:

| Bytes | Content |
|-------|---------|
| 0-12 | Initialisation vector (IV) |
| 12-28 | Authentication tag |
| 28+ | Ciphertext |

### Runtime: Backend to Frontend

At runtime, the backend decrypts `CLOUDFLARE_CONFIG` and uses the full config (including `apiToken`) for Cloudflare API calls. The frontend only receives non-sensitive fields:

```mermaid
flowchart LR
    subgraph backend["Backend"]
        DECRYPT["Decrypt"] --> FULL["Full config"]
        FULL --> API@{ icon: "ngx:cloudflare", label: "Cloudflare API", pos: "b", h: 48 }
    end

    subgraph frontend["Frontend"]
        SAFE["Safe config"] --> UI@{ icon: "ngx:ramblers", label: "Committee UI", pos: "b", h: 48 }
    end

    FULL -->|"Non-sensitive only"| SAFE
    UI -->|"Via backend API"| FULL
```

### Non-Sensitive Config Fields

The `/api/cloudflare/email-routing/config` endpoint returns only:

| Field | Type | Purpose |
|-------|------|---------|
| `configured` | `boolean` | Whether Cloudflare is set up for this environment |
| `accountId` | `string` | Used to build Cloudflare dashboard links |
| `zoneId` | `string` | Used to build Cloudflare dashboard links |
| `baseDomain` | `string` | The domain for email routing (e.g. `nwkramblers.org.uk`) |
| `ownsZone` | `boolean` | Whether this site owns its Cloudflare zone or is a subdomain of a shared zone (drives the apex-vs-subdomain inbox setup) |
| `zoneName` | `string` | The resolved Cloudflare zone name for the domain |

The `apiToken` is **never** sent to the frontend. All Cloudflare operations go through authenticated backend endpoints.

## Fly API Token & the `FLY_CONFIG` Bundle

The Fly API token follows the identical encrypted-bundle pattern as the Cloudflare config. Each environment carries one org-scoped Fly API token in `flyio.apiKey` (alongside `flyio.appName`) in the `ENVIRONMENTS` config document in MongoDB. That single token does everything Fly-related for the environment: it authenticates the deploy (`flyctl deploy` and `flyctl secrets import`), it restarts a stuck machine from the NGX admin UI, and it reads Fly's Prometheus metrics for the Memory Diagnostics charts.

### Encryption at Deploy Time

During deployment, `buildSecretsFromDatabaseConfig()` encrypts `{apiToken, appName}` into a `FLY_CONFIG` Fly secret using AES-256-GCM keyed by `ENVIRONMENT_SETUP_API_KEY` — the same mechanism, same encryption key, and same payload layout used for `CLOUDFLARE_CONFIG`.

```mermaid
flowchart LR
    DBFLY@{ icon: "ngx:mongodb", label: "flyio.apiKey + appName", pos: "b", h: 48 }
    ENVVAR@{ icon: "logos:fly-icon", label: "FLY_CONFIG", pos: "b", h: 48 }

    DBFLY --> ENCRYPT["AES-256-GCM encrypt"] --> ENVVAR
```

### Runtime Token Resolution

At runtime, `flyRuntimeConfig()` resolves the token in a fixed precedence order and stops at the first source that yields both a token and an app name:

1. **Fly-injected environment variables** — Fly sets `FLY_APP_NAME` and `FLY_MACHINE_ID` on every deployed machine; combined with a `FLY_API_TOKEN` env var this is the fast path
2. **Decrypted `FLY_CONFIG` bundle** — decrypted with `ENVIRONMENT_SETUP_API_KEY`, this is how deployed machines normally obtain the token
3. **Environments config in the database** — matched on the Mongo database name, used as a last-resort fallback

```mermaid
flowchart LR
    START@{ icon: "logos:fly-icon", label: "flyRuntimeConfig()", pos: "b", h: 48 }
    ENV@{ icon: "logos:fly-icon", label: "FLY_* env vars", pos: "b", h: 48 }
    BUNDLE@{ icon: "logos:fly-icon", label: "Decrypt FLY_CONFIG", pos: "b", h: 48 }
    DB@{ icon: "ngx:mongodb", label: "Environments config", pos: "b", h: 48 }
    TOKEN["Resolved token + appName"]

    START --> ENV
    ENV -->|"token + appName present"| TOKEN
    ENV -->|"missing"| BUNDLE
    BUNDLE -->|"decrypted"| TOKEN
    BUNDLE -->|"absent"| DB
    DB --> TOKEN
```

### What the Token Powers

| Use | Where | Notes |
|-----|-------|-------|
| Deploy + `flyctl secrets import` | CI deploy runner | The per-environment `apiKey` is loaded into `FLY_API_TOKEN` for that environment's `flyctl` commands |
| Restart a stuck machine | Admin UI (#304) | `restartCurrentMachine()` calls the Fly Machines API with the resolved token |
| Read memory / metric history | Memory Diagnostics charts (#304) | `fly-metrics` queries Fly's Prometheus API with the resolved token |

An org-scoped token is required for the metrics path: an app-scoped token can manage and restart the app but returns 403 from Fly's Prometheus API.

Originally five environments carried a separate app-scoped deploy token plus a distinct org-scoped `metricsToken`. On 9 Jul 2026 this was consolidated to a single org-scoped `apiKey` per environment and the `metricsToken` field was removed, so one token now covers deploy, restart and metrics.

## Required Environment Variables

### Origin Environments

| Variable | Required | Purpose |
|----------|----------|---------|
| `AUTH_SECRET` | Yes | JWT authentication secret |
| `AWS_ACCESS_KEY_ID` | Yes | S3 file storage (also reused to upload Serenity reports) |
| `AWS_SECRET_ACCESS_KEY` | Yes | S3 file storage |
| `AWS_REGION` | Yes | S3 region (e.g. `eu-west-2`) |
| `AWS_BUCKET` | Yes | S3 bucket name |
| `MONGODB_URI` | Yes | Database connection string |
| `CLOUDFLARE_CONFIG` | No | Encrypted Cloudflare config (enables email routing) |
| `FLY_CONFIG` | No | Encrypted per-environment Fly token + app name (enables machine restart + metrics) |
| `ENVIRONMENT_SETUP_API_KEY` | No | Encryption key for both `CLOUDFLARE_CONFIG` and `FLY_CONFIG` |
| `INTEGRATION_WORKER_URL` | No | Worker URL (enables remote upload routing) |
| `INTEGRATION_WORKER_SHARED_SECRET` | If worker URL set | HMAC signing key for worker requests |
| `INTEGRATION_WORKER_ENCRYPTION_KEY` | If worker URL set | AES key for credential encryption |

### Worker App

| Variable | Required | Purpose |
|----------|----------|---------|
| `INTEGRATION_WORKER_SHARED_SECRET` | Yes | HMAC signature validation |
| `INTEGRATION_WORKER_ENCRYPTION_KEY` | Yes | Credential + report-credential decryption |
| `CHROME_BIN` | Built-in | Chrome path (baked into Docker image via `ENV`) |
| `CHROMEDRIVER_PATH` | Built-in | Chromedriver path (baked into Docker image via `ENV`) |

The worker deliberately has no `MONGODB_URI`. `RAMBLERS_USERNAME` and `RAMBLERS_PASSWORD` are set by the runner from the decrypted job payload immediately before Serenity spawns, so `Login.performAs` takes its env-var fast path and never attempts a Mongo lookup.

### Deploy-Only Variables

| Variable | Required | Purpose |
|----------|----------|---------|
| `FLY_API_TOKEN` | Deploy only | GitHub secret — the deploy runner's ambient Fly authentication. Distinct from the per-environment `flyio.apiKey`, which the deploy loads into `FLY_API_TOKEN` per environment and which is also baked into that environment's `FLY_CONFIG` secret |
| `ADMIN_MONGODB_URI` | Deploy only | Staging database for secrets import |
| `INTEGRATION_WORKER_SECRETS_JSON` | Deploy only | GitHub secret — worker HMAC + AES keys as JSON |
| `INTEGRATION_WORKER_APP_NAME` | Deploy only | GitHub variable — worker Fly app name |

## Issue History

The secrets and deployment infrastructure was built across several issues:

```mermaid
flowchart BT
    I146@{ icon: "logos:github-icon", label: "#146 Env Setup", pos: "b", h: 48 }
    I156@{ icon: "ngx:cloudflare", label: "#156 DNS/SSL", pos: "b", h: 48 }
    I125@{ icon: "ngx:cloudflare", label: "#125 Email Routing", pos: "b", h: 48 }
    I82["#82 Brevo Auth"]
    I130["#130 Replies Rejected"]
    I114@{ icon: "logos:fly-icon", label: "#114 Integration Worker", pos: "b", h: 48 }
    I304@{ icon: "logos:fly-icon", label: "#304 Machine Restart & Metrics", pos: "b", h: 48 }
    TODAY@{ icon: "logos:fly-icon", label: "Today: Worker, Fly Tokens & Secrets", pos: "b", h: 48 }

    I146 -->|"secrets"| I125
    I146 -->|"config"| I156
    I156 -->|"Cloudflare"| I125
    I156 -->|"DNS"| I82
    I130 -->|"motivated"| I125
    I125 --> TODAY
    I82 --> TODAY
    I114 -->|"worker + reports + lifecycle"| TODAY
    I304 -->|"Fly token + restart + metrics"| TODAY
```

| Issue | Title | Key Contribution |
|-------|-------|-----------------|
| [#146](https://github.com/nbarrett/ngx-ramblers/issues/146) | Automated environment creation | Database-first secrets loading, GitHub secrets sync, global AWS |
| [#156](https://github.com/nbarrett/ngx-ramblers/issues/156) | DNS/SSL automation | Cloudflare DNS and certificate management in environment setup |
| [#130](https://github.com/nbarrett/ngx-ramblers/issues/130) | Email replies rejected | The original problem that drove Cloudflare email routing work |
| [#125](https://github.com/nbarrett/ngx-ramblers/issues/125) | Cloudflare email routing | Committee email forwarding, worker scripts, email logs UI |
| [#82](https://github.com/nbarrett/ngx-ramblers/issues/82) | Brevo domain authentication | DKIM/SPF DNS records via Cloudflare for email deliverability |
| [#114](https://github.com/nbarrett/ngx-ramblers/issues/114) | Distributed integration worker | Shared worker (Serenity uploads, browser fetch, Flickr scraping, migration, resize, document conversion), queued job execution, dependency-graph deploy detection, S3 Serenity report upload, lifecycle audit events, env-var login short-circuit, `.env` overrides for local dev |
| [#304](https://github.com/nbarrett/ngx-ramblers/issues/304) | Machine restart & memory diagnostics | Per-environment org-scoped Fly token encrypted into `FLY_CONFIG`, admin restart of a stuck Fly machine, Fly Prometheus metric-history charts, consolidation of deploy + metrics tokens into one `apiKey` |

## Why No Code Changes Were Needed (Original Cloudflare Work)

The application code already handled Cloudflare gracefully — showing a warning when config was missing and enabling features when config was present. The missing piece was the deployment pipeline: managed environments had no `CLOUDFLARE_CONFIG` secret because the deploy script did not import secrets from the database. Adding `importSecretsFromDatabase()` to the deploy flow was the final link in the chain.

## What Changed for the Integration Worker

The integration worker required changes across several layers:

- **Model**: `UploadWorkerConfig` added to `EnvironmentsConfig` as a sibling to `environments[]`, keeping it out of the deploy-all matrix; `IntegrationWorkerEventType.LIFECYCLE` added for non-stdout audit markers
- **Worker runner**: Serenity subprocess uploads `target/site/serenity` to `s3://<envBucket>/ramblers-upload-reports/<csvFileName>/` via `uploadSerenityReportToS3`, and emits lifecycle progress events around report generation and upload
- **Origin audit notifier**: new `recordLifecycleEvent` and `recordReportLocation` helpers persist the extra rows and broadcast the report location over the WebSocket so the UI can surface a "View Report" link
- **Admin UI**: New section in the Global Settings tab for worker app name, org-scoped Fly `apiKey`, HMAC secret, AES key, memory, and scale; Walk Export UI renders the report link against the selected session
- **GitHub sync**: the "Push to GitHub" action writes `INTEGRATION_WORKER_SECRETS_JSON` alongside `CONFIGS_JSON` when worker config exists
- **CI workflow**: a dependency-graph deploy step (`detect-worker-changes.ts`) piggybacked on the main build, reusing the same Docker image with a different Fly config (`fly.integration-worker.toml`) via `deploy-integration-worker.ts`
- **Content-type mapping**: `contentTypeForExtension` + `fileContentTypeMappings` centralised in `projects/ngx-ramblers/src/app/models/content-metadata.model.ts` and shared between the S3 report uploader and `aws-utils`
- **Local dev**: `ngx-cli local dev` reads four worker env vars from `server/.env` so a developer can flip between a local worker and the Fly worker without redeploying