15-Jul-2026 — Integration Worker: Distributed Architecture and Security

This article explains how the NGX Ramblers integration worker works: what it is, how the many websites talk to it, and — in rigorous, diagrammed detail — the security model that protects every message crossing the boundary between them. If you only read one part, read the Security model section further down: it is written and drawn so that anyone auditing this design can satisfy themselves it is a sound way to build a distributed, machine-to-machine architecture.

What the integration worker is — and why

The integration worker exists to keep the public websites small and fast. Each site runs on 512 MB Fly machines, sized for ordinary visitor traffic — serving pages, the API and WebSockets. Some operations are far too memory-hungry to run there: they would push the V8 heap towards its cap, slow every other request, or trip an out-of-memory kill. So any such operation is offloaded to the integration worker, a separate Fly.io application (ngx-ramblers-integration-worker) provisioned with 1024 MB — double the memory — and dedicated to heavy work.

Browser automation is the most visible example, but it is only one class. The worker handles two kinds of heavy job:

Jobs that need a real browser (Chromium plus the Serenity-BDD Java tooling — expensive to load, and a large resident footprint):

• Walks Manager upload — Ramblers has no bulk API, so walks are created, edited and published by driving its web forms with Playwright/Serenity.
• Legacy site migration — scraping a legacy group website page by page to import its content.
• HTML fetch — fetching and rendering a page's HTML for content import.
• Flickr album scraping — reading a Flickr user's albums for gallery import.

Jobs that are simply memory- and CPU-heavy, with no browser at all:

• Image resizing — the sharp library decodes full-resolution images into memory buffers to produce the site's rendition sizes. A batch of large images would blow the 512 MB budget on its own.
• Document conversion — turning uploaded committee documents (.docx via mammoth, PDFs via pdf-parse and styled extraction) into markdown. Whole documents are loaded and transformed in memory.

The organising principle is memory, not browsers: the worker is where any job runs when running it on the 512 MB site would be unsafe.

flowchart TB
    subgraph site["A website — 512 MB"]
        VISITOR["Visitor traffic:<br/>pages, API, WebSockets"]
    end
    subgraph worker["Integration worker — 1024 MB — one heavy job at a time"]
        subgraph browser["Needs a real browser"]
            WALKS["Walks Manager upload"]
            MIG["Legacy site migration"]
            HTML["HTML fetch"]
            FLICKR["Flickr album scrape"]
        end
        subgraph heavy["Memory-heavy, no browser"]
            RESIZE["Image resizing (sharp)"]
            DOC["Document conversion<br/>(docx / pdf to markdown)"]
        end
    end
    site -->|"offload anything that<br/>would exceed 512 MB"| worker
    style site fill:#EAF2FB,stroke:#8FB5DE,stroke-width:2px,rx:12,ry:12,color:#404143
    style worker fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143
    style browser fill:#E8F5EE,stroke:#9BC8AB,rx:10,ry:10,color:#404143
    style heavy fill:#FBF0E4,stroke:#E4B876,rx:10,ry:10,color:#404143

Many websites, one worker

NGX Ramblers hosts many group websites — sixteen at the time of writing, each its own Fly application with its own database. They do not each run their own worker. Instead, all of them share a single integration worker. This is a many-to-one relationship: any site that needs heavy work submits a job to the one shared worker, which runs it and reports back to whichever site asked.

flowchart TB
    S1["EKWG"] --> WK
    S2["Canterbury"] --> WK
    S3["Kent"] --> WK
    S4["Bolton"] --> WK
    S5["… 16 sites in total"] --> WK
    WK@{ icon: "logos:fly-icon", label: "One shared integration worker", pos: "b", h: 48 }
    style WK fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143

Sharing one worker keeps the estate cheap — one 1024 MB machine instead of sixteen — and it is possible precisely because all sixteen sites are operated by the same team. They form a single administrative trust domain, which is what makes a shared worker and shared worker secrets acceptable — the Security model section below explains why.

It works so comfortably because heavy jobs are rare. A walks upload happens when a volunteer chooses to publish; an image resize or document conversion only when content is added. Across the whole estate these are occasional, human-initiated actions, not a steady stream. The shared machine is therefore idle the overwhelming majority of the time — well into the high 99%s — so a single worker has ample headroom to serve all sixteen sites, and the chance of two jobs genuinely colliding is small. When it does happen, the queue simply runs them in turn.

One heavy job at a time

Because one worker serves every site, and heavy jobs are memory-hungry, the worker runs exactly one heavy job at a time. Walks uploads, image resizes and document conversions — from any site — all pass through a single shared queue. A new job either becomes the active job or waits its turn, so two memory-hungry operations never run concurrently and stack their footprints. This is what keeps a 1024 MB worker safely within budget, and why a submission may occasionally come back as "queued" rather than starting immediately.

Given how rarely heavy jobs occur, serialising them costs almost nothing: the queue is empty the vast majority of the time, so in practice a job starts at once, and the "one at a time" rule is a cheap safety guarantee rather than a throughput limit.

flowchart LR
    J1["Walks upload<br/>(any site)"] --> Q
    J2["Image resize<br/>(any site)"] --> Q
    J3["Document conversion<br/>(any site)"] --> Q
    Q["Single heavy-job queue"] --> ACTIVE["One active job"]
    Q --> WAIT["Others wait<br/>(returned as queued)"]
    style ACTIVE fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style WAIT fill:#F3F4F5,stroke:#B9BDC2,rx:10,ry:10,color:#404143

The same application, two build targets

The worker is not a separate application. It runs the same NGX Ramblers server code as the websites, from the same repository and the same Dockerfile. That Dockerfile is a multi-stage build with two targets:

• The website target builds on node:24 — a lean runtime with no browser and no heavy toolchain.
• The worker target builds on the serenity-js/playwright base, which adds a headless Chromium and the Java runtime for Serenity-BDD, then copies in the identical application code and installs the same dependencies (including sharp and the document-conversion libraries).

So the two images differ only in their base layer and therefore in what runtime tooling is present — not in the application they run. Both are the NGX server; the worker's image simply carries the extra machinery its heavy jobs need.

flowchart TB
    SRC["One codebase, one Dockerfile<br/>(the NGX Ramblers application)"]
    SRC --> T1["Website target<br/>base node:24 — lean, 512 MB"]
    SRC --> T2["Worker target<br/>base serenity-js/playwright<br/>adds Chromium + JVM; same app code"]
    style T1 fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style T2 fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143

They are built and deployed by separate GitHub Actions workflows, so the two images ship independently. One consequence matters operationally: because a single worker serves every site, rebuilding and redeploying the worker updates the heavy-job behaviour for all sites at once. When automation code changes, deploying the websites does not touch the worker; the worker only runs the new code once its own target is rebuilt and redeployed. Until then, every site's heavy jobs run the previous worker version.

How a job flows

The walks upload is the richest example. Here is the complete round trip, from the button press to the audit rows the user watches scroll past. Note that the browser automation runs on the worker, never in the visitor's browser — the client only submits the request and displays progress.

sequenceDiagram
    participant C as Site (Angular SPA)
    participant A as Site server
    participant W as Shared integration worker
    participant R as Ramblers Walks Manager

    C->>A: WebSocket: upload request
    A->>W: POST /jobs (signed body, encrypted credentials)
    Note over W: Verify HMAC signature. Reject 401 if invalid.
    W-->>A: 200 queued
    A-->>C: WebSocket: "Upload queued"
    Note over W: Decrypt credentials. Download images to /tmp.<br/>Wait for the shared queue, then run.
    W->>W: spawn Serenity subprocess (headless Chromium)
    W->>R: Log in, edit / upload / publish walks
    loop each Serenity step
        W->>A: POST /progress (signed)
        A->>A: Match jobId to active session
        A-->>C: WebSocket: audit row
    end
    W->>A: POST /result (signed)
    A-->>C: WebSocket: final status

Step by step:

1. Site client → site server (WebSocket). The SPA builds the request and sends it over a WebSocket. That is the client's entire outbound role: it downloads no images and launches no browser.
2. Site server → worker (signed HTTP). The server builds a job and POSTs it to the worker's /api/integration-worker/jobs.
3. The worker runs it. Source images are downloaded into a per-job temporary directory (/tmp/ramblers/<jobId>/), the job waits for the shared heavy-job queue, and then a Serenity subprocess drives Walks Manager in a headless browser.
4. Worker → site server (signed callbacks). The worker holds no WebSocket to the user — it is a different machine, and may be serving a different site's job moments later. As each step completes it POSTs progress to the site that submitted the job, at the callback URL carried in the job, and the final outcome to /result.
5. Site server → client (WebSocket). The server matches each callback to the WebSocket session that started the job and forwards it as an audit row. That is what populates the live "Walk upload audit" tab.

So the shape is: client → (WebSocket) → site server → (signed HTTP) → worker runs the job → (signed HTTP callbacks) → site server → (WebSocket) → client.

Security model

This is the section to scrutinise. The trust boundary is the network hop between a site and the shared worker — independently deployed services on the public internet. Every operational message that crosses it is protected on two axes: authenticity and integrity (is this message genuinely from a party that holds the secret, and has it been altered?) and, for credentials, confidentiality (can a third party read it?).

flowchart LR
    subgraph website["Site server"]
        direction TB
        W1["Job payload<br/>(not secret)"]
        W2["Credentials<br/>(secret)"]
    end
    subgraph boundary["Public internet — over TLS"]
        SIG["Every message:<br/>HMAC-SHA256 signature"]
        ENC["Credentials only:<br/>AES-256-GCM ciphertext"]
    end
    subgraph worker["Shared worker"]
        direction TB
        WK1["Verify signature<br/>(constant time)"]
        WK2["Decrypt credentials"]
    end
    W1 --> SIG
    W2 --> ENC
    W2 --> SIG
    SIG --> WK1
    ENC --> WK2
    style boundary fill:#FDECEC,stroke:#E0A2A2,stroke-width:2px,rx:12,ry:12,color:#404143
    style website fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style worker fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143

Trust boundary and authenticated surface

The worker's job and callback endpoints are reachable over the internet, so the security of the whole system rests on the rule that every operational request must carry a valid signature or it is rejected.

flowchart TB
    NET["Public internet"]
    subgraph worker["Integration worker endpoints"]
        JOBS["POST /jobs — signed"]
        PROG["POST /progress — signed"]
        RES["POST /result — signed"]
        HTML["POST /browser/html-fetch — signed"]
        HEALTH["GET /api/health — open, no input, no secrets"]
        ROOT["GET / — open, static descriptor"]
    end
    NET --> JOBS
    NET --> PROG
    NET --> RES
    NET --> HTML
    NET --> HEALTH
    NET --> ROOT
    style JOBS fill:#EAF2FB,stroke:#8FB5DE,color:#404143
    style PROG fill:#EAF2FB,stroke:#8FB5DE,color:#404143
    style RES fill:#EAF2FB,stroke:#8FB5DE,color:#404143
    style HTML fill:#EAF2FB,stroke:#8FB5DE,color:#404143
    style HEALTH fill:#F3F4F5,stroke:#B9BDC2,color:#404143
    style ROOT fill:#F3F4F5,stroke:#B9BDC2,color:#404143

Concretely, POST /jobs is verified against the shared secret and unsigned or mis-signed requests receive 401 and are never queued; the POST /progress and POST /result callbacks are verified against the callback secret; and the synchronous browser endpoints use the same signed request/response scheme. The only unauthenticated endpoints are GET /api/health and GET / — neither accepts input nor returns anything sensitive. There is no unauthenticated endpoint that performs work, accepts data, or discloses secrets.

Message authenticity and integrity — HMAC-SHA256, constant-time

Every signed request carries an x-ramblers-upload-signature header: an HMAC-SHA256 of the exact JSON request body, keyed on the shared secret. The receiver recomputes the HMAC over the body it received and compares.

sequenceDiagram
    participant S as Sender (holds secret)
    participant R as Receiver (holds secret)
    Note over S: signature = HMAC_SHA256(secret, body)
    S->>R: POST body + header x-ramblers-upload-signature
    Note over R: expected = HMAC_SHA256(secret, receivedBody)
    alt length differs OR timingSafeEqual is false
        R-->>S: 401 Unauthorized
    else signatures equal (constant-time compare)
        R-->>S: 200 — request accepted
    end

Two properties make this robust:

• Body binding. The signature is over the whole body, so altering a single byte — a walk id, a deletion target, a callback status — invalidates it. An attacker cannot tamper with a message in flight without the secret.
• Constant-time comparison. Verification uses Node's crypto.timingSafeEqual, preceded by a length check, so the comparison takes the same time whether the first byte or the last byte differs. This closes the timing side-channel that a naive string equality (===) would open, where an attacker could otherwise recover a valid signature byte by byte by measuring response times.

Only a party in possession of the secret can produce a signature that verifies. That is what authenticates a site to the worker, and the worker's callbacks back to the site.

Credential confidentiality — AES-256-GCM

The signature proves who sent a message and that it is intact, but it does not hide the contents. The job body itself is not secret — it is walk data — and travels over TLS. The Ramblers login credentials and the AWS report-upload credentials, however, must never be readable in transit, so they are encrypted before they are placed in the request, independently of the transport.

flowchart LR
    CRED["Credentials<br/>(username, password)"] --> JSON["JSON.stringify"]
    EK["encryptionKey"] --> DK["key = SHA-256(encryptionKey)<br/>256-bit"]
    RB["crypto.randomBytes(12)"] --> IV["fresh iv per message"]
    JSON --> GCM["AES-256-GCM encrypt"]
    DK --> GCM
    IV --> GCM
    GCM --> TAG["authTag<br/>(16 bytes)"]
    GCM --> CT["ciphertext"]
    IV --> ENV
    TAG --> ENV
    CT --> ENV["base64( iv ‖ authTag ‖ ciphertext )"]
    style ENV fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style GCM fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143

• Cipher: AES-256-GCM. A 256-bit key, authenticated encryption. GCM protects both confidentiality and integrity of the ciphertext: any tampering with the encrypted credentials causes decryption to fail rather than silently yielding altered values.
• Key derivation. The 256-bit key is derived as SHA-256(encryptionKey), so a secret of any length produces a correctly sized key.
• Fresh IV per message. A new random 12-byte initialisation vector is generated for every encryption, so encrypting the same credentials twice produces different ciphertext and no patterns leak.
• On-the-wire form. The output is base64(iv ‖ authTag ‖ ciphertext), carrying everything the receiver needs to authenticate and decrypt, and nothing more.

The worker decrypts the credentials only in memory, only to hand them to the Serenity login step, and the per-job files are cleaned up afterwards.

Separation of secrets

Signing and encryption use two distinct secrets, never the same value. Separating them means a compromise or rotation of one concern does not implicate the other, and neither secret is overloaded across two cryptographic purposes.

flowchart TB
    SS["INTEGRATION_WORKER_SHARED_SECRET"] --> HMAC["HMAC-SHA256 signing<br/>authenticity + integrity<br/>(every message)"]
    EK["INTEGRATION_WORKER_ENCRYPTION_KEY"] --> AES["AES-256-GCM encryption<br/>confidentiality<br/>(credentials only)"]
    style HMAC fill:#EAF2FB,stroke:#8FB5DE,stroke-width:2px,rx:12,ry:12,color:#404143
    style AES fill:#FDF3E7,stroke:#E4B876,stroke-width:2px,rx:12,ry:12,color:#404143

Session-bound callbacks

A signed callback is not blindly trusted just because it verifies. Each callback carries the jobId, and the site matches it to an active session. If no active session exists for that id — because the job never started, or already finished — the callback is rejected. This scopes callbacks to live work and means a replayed callback for a completed job has nowhere to land.

flowchart TB
    CB["Callback: POST /progress"] --> V{"Signature valid?"}
    V -->|"No"| R401["401 Unauthorized"]
    V -->|"Yes"| SESS{"jobId matches an<br/>active session?"}
    SESS -->|"No"| R404["404 — no session"]
    SESS -->|"Yes"| FWD["Forward as audit row<br/>over client WebSocket"]
    style FWD fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143
    style R401 fill:#FDECEC,stroke:#E0A2A2,color:#404143
    style R404 fill:#FDECEC,stroke:#E0A2A2,color:#404143

Shared secrets within one trust domain

Because a single worker serves every site, the two secrets are shared by the worker and all the sites that use it — they have to be, since the worker verifies and decrypts with one key. They are not per-site. They live in the central worker configuration and are injected as Fly secrets at deploy time; they are not committed to the repository, not written to .env files in the image, and not exported into a shared store.

This is safe because all the sites are within a single administrative trust domain — one team operates every site and the worker. A shared secret is only a weakness when the parties sharing it do not already trust one another. Here they are all facets of the same operator, so a site holding the secret can forge nothing it could not already do directly.

flowchart LR
    CFG@{ icon: "ngx:mongodb", label: "Central worker config", pos: "b", h: 48 }
    DEPLOY@{ icon: "logos:github-actions", label: "Deploy", pos: "b", h: 48 }
    subgraph domain["Single trust domain (one operator)"]
        SITES["All sites<br/>(shared secret + key)"]
        WK["Shared worker<br/>(same secret + key)"]
    end
    CFG --> DEPLOY
    DEPLOY -->|"inject as Fly secrets"| SITES
    DEPLOY -->|"inject as Fly secrets"| WK
    style domain fill:#E8F5EE,stroke:#9BC8AB,stroke-width:2px,rx:12,ry:12,color:#404143

The one caveat, stated plainly: if the worker were ever opened to mutually-distrusting tenants, this shared-secret model would no longer be appropriate and per-tenant secrets would be required. Today, within one operator, it is a standard and sound arrangement.

Transport, and an honest threat model

All traffic is HTTPS (terminated by Fly, fronted by Cloudflare), so TLS already provides confidentiality and integrity on the wire. The application-level signing and encryption are deliberate defence in depth on top of TLS, not a substitute for it:

• The HMAC signature guarantees end-to-end authenticity and integrity between a site and the worker at the application layer, independent of where TLS is terminated. Even a misconfigured proxy cannot forge or alter a message without the shared secret.
• The AES-GCM credential encryption means the most sensitive fields are unreadable even to anything that can see the decrypted TLS stream, narrowing where cleartext credentials ever exist to the two process memories that legitimately need them.

To be precise about what is and is not claimed: the signature prevents forgery and tampering, not replay, in isolation — there is no per-message nonce or timestamp. Replay is mitigated by two other properties rather than by the signature: an attacker cannot capture a valid signed request in the first place without breaking TLS, and a replayed callback is rejected once its job's session is no longer active. If a future change ever moved either endpoint outside TLS, per-message anti-replay would need to be added; today, under TLS, the design does not depend on the recipient having seen a message before.

Summary

• The integration worker is a shared heavy-job service on a bigger (1024 MB) machine, isolated from the lean 512 MB sites. It runs anything too memory-hungry for a site: browser automation (walks upload, migration, HTML fetch, Flickr scrape) and non-browser heavy work (image resizing, document conversion), one job at a time through a shared queue.
• It is a many-to-one arrangement: sixteen sites share one worker, cheaply, because they are one operator's estate.
• The worker is the same application as the sites — same code, same Dockerfile, a second build target whose base image adds the heavy toolchain — deployed independently, so rebuilding the worker updates heavy-job behaviour for every site at once.
• The client never runs the work — it requests and it observes.
• Every operational message across the boundary is HMAC-SHA256 signed and constant-time verified; credentials are AES-256-GCM encrypted with a separate key; the two secrets are shared within one administrative trust domain and never on disk; callbacks are session-bound; and the whole thing rides on TLS with the cryptography as defence in depth. This is a conservative, standard, auditable way to build a distributed machine-to-machine architecture.