11-Jun-2026 — PDF conversion moved to the integration worker, so big documents cannot crash the site ref #281
In plain English
- The committee section can turn an uploaded PDF into an editable page. Converting a big, photo-heavy PDF is hard work, and until now the website did that work itself. On a small production machine it could use up all the available memory and briefly bring the whole site down, showing visitors an error page instead. That actually happened on EKWG with a photo-heavy set of meeting notes.
- The heavy lifting now happens on a separate machine, the integration worker, which has twice the memory and can be given more if it ever needs it. However big the document, the website itself stays up and responsive.
- If the worker is unavailable for any reason, the website quietly does the conversion itself just as before, so the feature always works; it simply works more safely when the worker is there.
- Nothing changes in how you use it: Convert to editable document looks and behaves exactly the same.
The technical detail
Why this was needed
Converting a PDF on a production environment could kill the app: the EKWG machine (shared-cpu-1x, 512MB) was OOM-killed (exit 137) converting a photo-heavy meeting-notes PDF, and the Fly proxy showed the death as a 502. The conversion pipeline is memory-hungry by design: pdfjs loads the document, image extraction renders every page's embedded images to PNG buffers, and table-page detection renders whole pages to high-resolution screenshots. All of that ran in-process on top of the web app's baseline.
How it works now
Document conversion runs on the integration worker (1024MB, runtime-scalable via the uploadWorker config), using the same delegation pattern as the legacy-redirect scraper, Flickr album scraping and migration scrapes:
- the main app's two conversion endpoints route through
convertWithBestEngine: when INTEGRATION_WORKER_URL and the shared secret are configured, the file is sent to the worker as a signed request using the existing HMAC-SHA256 signature scheme; local development without a worker, or any worker failure, falls back to in-process conversion unchanged
- a new worker endpoint verifies the signature, converts, and returns markdown plus the extracted images, with image placeholders left intact for a second substitution pass
- the main app uploads the returned images to its own S3 bucket under
committeeFiles/converted-images, so credentials never leave the environment, then replaces the placeholders with real paths; this two-pass behaviour is now pinned by a spec
- the exchange is a simple synchronous request and response; the worker's 50MB body limit and ten-minute timeouts comfortably cover large documents
- fixed while testing: embedded images on table-captured pages were uploaded but never placed, because the whole page becomes a screenshot; they are now filtered out, saving eleven needless S3 objects on the Medway walks document alone
Deployment and verification
The worker has its own manual workflow (Build and Deploy Ramblers Upload Worker) and must be dispatched with the new image tag before it serves the endpoint; until then, the main apps fall back to in-process conversion exactly as today. 773 server tests pass, lint and TypeScript checks are clean, and the round trip was verified against a real photo-and-table-heavy walks PDF: eleven images collected, eleven placed, no leftover placeholders.