This article explains how the NGX Ramblers integration worker works: what it is, how the many websites talk to it, and — in rigorous, diagrammed detail — the security model that protects every message crossing the boundary between them. If you only read one part, read the Security model section further down: it is written and drawn so that anyone auditing this design can satisfy themselves it is a sound way to build a distributed, machine-to-machine architecture.
\nThe integration worker exists to keep the public websites small and fast. Each site runs on 512 MB Fly machines, sized for ordinary visitor traffic — serving pages, the API and WebSockets. Some operations are far too memory-hungry to run there: they would push the V8 heap towards its cap, slow every other request, or trip an out-of-memory kill. So any such operation is offloaded to the integration worker, a separate Fly.io application (ngx-ramblers-integration-worker) provisioned with 1024 MB — double the memory — and dedicated to heavy work.
Browser automation is the most visible example, but it is only one class. The worker handles two kinds of heavy job:
\nJobs that need a real browser (Chromium plus the Serenity-BDD Java tooling — expensive to load, and a large resident footprint):
\nJobs that are simply memory- and CPU-heavy, with no browser at all:
\nsharp library decodes full-resolution images into memory buffers to produce the site's rendition sizes. A batch of large images would blow the 512 MB budget on its own..docx via mammoth, PDFs via pdf-parse and styled extraction) into markdown. Whole documents are loaded and transformed in memory.The organising principle is memory, not browsers: the worker is where any job runs when running it on the 512 MB site would be unsafe.
\nflowchart TB\n subgraph site["A website — 512 MB"]\n VISITOR["Visitor traffic:<br/>pages, API, WebSockets"]\n end\n subgraph worker["Integration worker — 1024 MB — one heavy job at a time"]\n subgraph browser["Needs a real browser"]\n WALKS["Walks Manager upload"]\n MIG["Legacy site migration"]\n HTML["HTML fetch"]\n FLICKR["Flickr album scrape"]\n end\n subgraph heavy["Memory-heavy, no browser"]\n RESIZE["Image resizing (sharp)"]\n DOC["Document conversion<br/>(docx / pdf to markdown)"]\n end\n end\n site -->|"offload anything that<br/>would exceed 512 MB"| worker\n style site fill:#EAF2FB,stroke:#8FB5DE,stroke-width:2px,rx:12,ry:12,color:#404143\n style worker fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143\n style browser fill:#E8F5EE,stroke:#9BC8AB,rx:10,ry:10,color:#404143\n style heavy fill:#FBF0E4,stroke:#E4B876,rx:10,ry:10,color:#404143\nNGX Ramblers hosts many group websites — sixteen at the time of writing, each its own Fly application with its own database. They do not each run their own worker. Instead, all of them share a single integration worker. This is a many-to-one relationship: any site that needs heavy work submits a job to the one shared worker, which runs it and reports back to whichever site asked.
\nflowchart TB\n S1["EKWG"] --> WK\n S2["Canterbury"] --> WK\n S3["Kent"] --> WK\n S4["Bolton"] --> WK\n S5["… 16 sites in total"] --> WK\n WK@{ icon: "logos:fly-icon", label: "One shared integration worker", pos: "b", h: 48 }\n style WK fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143\nSharing one worker keeps the estate cheap — one 1024 MB machine instead of sixteen — and it is possible precisely because all sixteen sites are operated by the same team. They form a single administrative trust domain, which is what makes a shared worker and shared worker secrets acceptable — the Security model section below explains why.
\nIt works so comfortably because heavy jobs are rare. A walks upload happens when a volunteer chooses to publish; an image resize or document conversion only when content is added. Across the whole estate these are occasional, human-initiated actions, not a steady stream. The shared machine is therefore idle the overwhelming majority of the time — well into the high 99%s — so a single worker has ample headroom to serve all sixteen sites, and the chance of two jobs genuinely colliding is small. When it does happen, the queue simply runs them in turn.
\nBecause one worker serves every site, and heavy jobs are memory-hungry, the worker runs exactly one heavy job at a time. Walks uploads, image resizes and document conversions — from any site — all pass through a single shared queue. A new job either becomes the active job or waits its turn, so two memory-hungry operations never run concurrently and stack their footprints. This is what keeps a 1024 MB worker safely within budget, and why a submission may occasionally come back as "queued" rather than starting immediately.
\nGiven how rarely heavy jobs occur, serialising them costs almost nothing: the queue is empty the vast majority of the time, so in practice a job starts at once, and the "one at a time" rule is a cheap safety guarantee rather than a throughput limit.
\nflowchart LR\n J1["Walks upload<br/>(any site)"] --> Q\n J2["Image resize<br/>(any site)"] --> Q\n J3["Document conversion<br/>(any site)"] --> Q\n Q["Single heavy-job queue"] --> ACTIVE["One active job"]\n Q --> WAIT["Others wait<br/>(returned as queued)"]\n style ACTIVE fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n style WAIT fill:#F3F4F5,stroke:#B9BDC2,rx:10,ry:10,color:#404143\nThe worker is not a separate application. It runs the same NGX Ramblers server code as the websites, from the same repository and the same Dockerfile. That Dockerfile is a multi-stage build with two targets:
\nnode:24 — a lean runtime with no browser and no heavy toolchain.serenity-js/playwright base, which adds a headless Chromium and the Java runtime for Serenity-BDD, then copies in the identical application code and installs the same dependencies (including sharp and the document-conversion libraries).So the two images differ only in their base layer and therefore in what runtime tooling is present — not in the application they run. Both are the NGX server; the worker's image simply carries the extra machinery its heavy jobs need.
\nflowchart TB\n SRC["One codebase, one Dockerfile<br/>(the NGX Ramblers application)"]\n SRC --> T1["Website target<br/>base node:24 — lean, 512 MB"]\n SRC --> T2["Worker target<br/>base serenity-js/playwright<br/>adds Chromium + JVM; same app code"]\n style T1 fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n style T2 fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143\nThey are built and deployed by separate GitHub Actions workflows, so the two images ship independently. One consequence matters operationally: because a single worker serves every site, rebuilding and redeploying the worker updates the heavy-job behaviour for all sites at once. When automation code changes, deploying the websites does not touch the worker; the worker only runs the new code once its own target is rebuilt and redeployed. Until then, every site's heavy jobs run the previous worker version.
\nThe walks upload is the richest example. Here is the complete round trip, from the button press to the audit rows the user watches scroll past. Note that the browser automation runs on the worker, never in the visitor's browser — the client only submits the request and displays progress.
\nsequenceDiagram\n participant C as Site (Angular SPA)\n participant A as Site server\n participant W as Shared integration worker\n participant R as Ramblers Walks Manager\n\n C->>A: WebSocket: upload request\n A->>W: POST /jobs (signed body, encrypted credentials)\n Note over W: Verify HMAC signature. Reject 401 if invalid.\n W-->>A: 200 queued\n A-->>C: WebSocket: "Upload queued"\n Note over W: Decrypt credentials. Download images to /tmp.<br/>Wait for the shared queue, then run.\n W->>W: spawn Serenity subprocess (headless Chromium)\n W->>R: Log in, edit / upload / publish walks\n loop each Serenity step\n W->>A: POST /progress (signed)\n A->>A: Match jobId to active session\n A-->>C: WebSocket: audit row\n end\n W->>A: POST /result (signed)\n A-->>C: WebSocket: final status\nStep by step:
\n/api/integration-worker/jobs./tmp/ramblers/<jobId>/), the job waits for the shared heavy-job queue, and then a Serenity subprocess drives Walks Manager in a headless browser./result.So the shape is: client → (WebSocket) → site server → (signed HTTP) → worker runs the job → (signed HTTP callbacks) → site server → (WebSocket) → client.
\nThis is the section to scrutinise. The trust boundary is the network hop between a site and the shared worker — independently deployed services on the public internet. Every operational message that crosses it is protected on two axes: authenticity and integrity (is this message genuinely from a party that holds the secret, and has it been altered?) and, for credentials, confidentiality (can a third party read it?).
\nflowchart LR\n subgraph website["Site server"]\n direction TB\n W1["Job payload<br/>(not secret)"]\n W2["Credentials<br/>(secret)"]\n end\n subgraph boundary["Public internet — over TLS"]\n SIG["Every message:<br/>HMAC-SHA256 signature"]\n ENC["Credentials only:<br/>AES-256-GCM ciphertext"]\n end\n subgraph worker["Shared worker"]\n direction TB\n WK1["Verify signature<br/>(constant time)"]\n WK2["Decrypt credentials"]\n end\n W1 --> SIG\n W2 --> ENC\n W2 --> SIG\n SIG --> WK1\n ENC --> WK2\n style boundary fill:#FDECEC,stroke:#E0A2A2,stroke-width:2px,rx:12,ry:12,color:#404143\n style website fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n style worker fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143\nThe worker's job and callback endpoints are reachable over the internet, so the security of the whole system rests on the rule that every operational request must carry a valid signature or it is rejected.
\nflowchart TB\n NET["Public internet"]\n subgraph worker["Integration worker endpoints"]\n JOBS["POST /jobs — signed"]\n PROG["POST /progress — signed"]\n RES["POST /result — signed"]\n HTML["POST /browser/html-fetch — signed"]\n HEALTH["GET /api/health — open, no input, no secrets"]\n ROOT["GET / — open, static descriptor"]\n end\n NET --> JOBS\n NET --> PROG\n NET --> RES\n NET --> HTML\n NET --> HEALTH\n NET --> ROOT\n style JOBS fill:#EAF2FB,stroke:#8FB5DE,color:#404143\n style PROG fill:#EAF2FB,stroke:#8FB5DE,color:#404143\n style RES fill:#EAF2FB,stroke:#8FB5DE,color:#404143\n style HTML fill:#EAF2FB,stroke:#8FB5DE,color:#404143\n style HEALTH fill:#F3F4F5,stroke:#B9BDC2,color:#404143\n style ROOT fill:#F3F4F5,stroke:#B9BDC2,color:#404143\nConcretely, POST /jobs is verified against the shared secret and unsigned or mis-signed requests receive 401 and are never queued; the POST /progress and POST /result callbacks are verified against the callback secret; and the synchronous browser endpoints use the same signed request/response scheme. The only unauthenticated endpoints are GET /api/health and GET / — neither accepts input nor returns anything sensitive. There is no unauthenticated endpoint that performs work, accepts data, or discloses secrets.
Every signed request carries an x-ramblers-upload-signature header: an HMAC-SHA256 of the exact JSON request body, keyed on the shared secret. The receiver recomputes the HMAC over the body it received and compares.
sequenceDiagram\n participant S as Sender (holds secret)\n participant R as Receiver (holds secret)\n Note over S: signature = HMAC_SHA256(secret, body)\n S->>R: POST body + header x-ramblers-upload-signature\n Note over R: expected = HMAC_SHA256(secret, receivedBody)\n alt length differs OR timingSafeEqual is false\n R-->>S: 401 Unauthorized\n else signatures equal (constant-time compare)\n R-->>S: 200 — request accepted\n end\nTwo properties make this robust:
\ncrypto.timingSafeEqual, preceded by a length check, so the comparison takes the same time whether the first byte or the last byte differs. This closes the timing side-channel that a naive string equality (===) would open, where an attacker could otherwise recover a valid signature byte by byte by measuring response times.Only a party in possession of the secret can produce a signature that verifies. That is what authenticates a site to the worker, and the worker's callbacks back to the site.
\nThe signature proves who sent a message and that it is intact, but it does not hide the contents. The job body itself is not secret — it is walk data — and travels over TLS. The Ramblers login credentials and the AWS report-upload credentials, however, must never be readable in transit, so they are encrypted before they are placed in the request, independently of the transport.
\nflowchart LR\n CRED["Credentials<br/>(username, password)"] --> JSON["JSON.stringify"]\n EK["encryptionKey"] --> DK["key = SHA-256(encryptionKey)<br/>256-bit"]\n RB["crypto.randomBytes(12)"] --> IV["fresh iv per message"]\n JSON --> GCM["AES-256-GCM encrypt"]\n DK --> GCM\n IV --> GCM\n GCM --> TAG["authTag<br/>(16 bytes)"]\n GCM --> CT["ciphertext"]\n IV --> ENV\n TAG --> ENV\n CT --> ENV["base64( iv ‖ authTag ‖ ciphertext )"]\n style ENV fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n style GCM fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143\nSHA-256(encryptionKey), so a secret of any length produces a correctly sized key.base64(iv ‖ authTag ‖ ciphertext), carrying everything the receiver needs to authenticate and decrypt, and nothing more.The worker decrypts the credentials only in memory, only to hand them to the Serenity login step, and the per-job files are cleaned up afterwards.
\nSigning and encryption use two distinct secrets, never the same value. Separating them means a compromise or rotation of one concern does not implicate the other, and neither secret is overloaded across two cryptographic purposes.
\nflowchart TB\n SS["INTEGRATION_WORKER_SHARED_SECRET"] --> HMAC["HMAC-SHA256 signing<br/>authenticity + integrity<br/>(every message)"]\n EK["INTEGRATION_WORKER_ENCRYPTION_KEY"] --> AES["AES-256-GCM encryption<br/>confidentiality<br/>(credentials only)"]\n style HMAC fill:#EAF2FB,stroke:#8FB5DE,stroke-width:2px,rx:12,ry:12,color:#404143\n style AES fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143\nA signed callback is not blindly trusted just because it verifies. Each callback carries the jobId, and the site matches it to an active session. If no active session exists for that id — because the job never started, or already finished — the callback is rejected. This scopes callbacks to live work and means a replayed callback for a completed job has nowhere to land.
flowchart TB\n CB["Callback: POST /progress"] --> V{"Signature valid?"}\n V -->|"No"| R401["401 Unauthorized"]\n V -->|"Yes"| SESS{"jobId matches an<br/>active session?"}\n SESS -->|"No"| R404["404 — no session"]\n SESS -->|"Yes"| FWD["Forward as audit row<br/>over client WebSocket"]\n style FWD fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\n style R401 fill:#FDECEC,stroke:#E0A2A2,color:#404143\n style R404 fill:#FDECEC,stroke:#E0A2A2,color:#404143\nBecause a single worker serves every site, the two secrets are shared by the worker and all the sites that use it — they have to be, since the worker verifies and decrypts with one key. They are not per-site. They live in the central worker configuration and are injected as Fly secrets at deploy time; they are not committed to the repository, not written to .env files in the image, and not exported into a shared store.
This is safe because all the sites are within a single administrative trust domain — one team operates every site and the worker. A shared secret is only a weakness when the parties sharing it do not already trust one another. Here they are all facets of the same operator, so a site holding the secret can forge nothing it could not already do directly.
\nflowchart LR\n CFG@{ icon: "ngx:mongodb", label: "Central worker config", pos: "b", h: 48 }\n DEPLOY@{ icon: "logos:github-actions", label: "Deploy", pos: "b", h: 48 }\n subgraph domain["Single trust domain (one operator)"]\n SITES["All sites<br/>(shared secret + key)"]\n WK["Shared worker<br/>(same secret + key)"]\n end\n CFG --> DEPLOY\n DEPLOY -->|"inject as Fly secrets"| SITES\n DEPLOY -->|"inject as Fly secrets"| WK\n style domain fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143\nThe one caveat, stated plainly: if the worker were ever opened to mutually-distrusting tenants, this shared-secret model would no longer be appropriate and per-tenant secrets would be required. Today, within one operator, it is a standard and sound arrangement.
\nAll traffic is HTTPS (terminated by Fly, fronted by Cloudflare), so TLS already provides confidentiality and integrity on the wire. The application-level signing and encryption are deliberate defence in depth on top of TLS, not a substitute for it:
\nTo be precise about what is and is not claimed: the signature prevents forgery and tampering, not replay, in isolation — there is no per-message nonce or timestamp. Replay is mitigated by two other properties rather than by the signature: an attacker cannot capture a valid signed request in the first place without breaking TLS, and a replayed callback is rejected once its job's session is no longer active. If a future change ever moved either endpoint outside TLS, per-message anti-replay would need to be added; today, under TLS, the design does not depend on the recipient having seen a message before.
\n