09-Sep-2026 — hold the worker awake for the whole job so the overrun stop can fire, and add admin #300
The problem
The 45-minute overrun stop is a timer inside the worker's Node process, but
Fly autosuspends the machine on excess capacity (zero concurrent requests),
which freezes the VM and stops that timer advancing. The previous keep-alive
pinged the health address every 60s, which left the machine idle almost the
whole minute, so Fly suspended it anyway. A running OS Maps conversion then
froze for hours, the queue jammed behind it, and the overrun stop never fired.
What is new
- The keep-alive now holds a single streaming connection open to the worker
for as long as a job runs (reopening if it drops), so Fly always sees
concurrency and does not suspend mid-job. Idle suspend is unchanged, so the
cost saving stays. The overrun watchdog remains the backstop and can now
actually run because the VM is never frozen while working.
- A GET /api/integration-worker/hold endpoint the keep-alive holds open.
- Admin controls for the worker queue, on the System Settings Diagnostics tab:
see the active job and anything queued, stop the active job, or clear the
queue, so a stuck job can be ended and someone else can run something without
a machine restart.
How it works
- Worker: the queue gains clearQueued(); the runner registers a cancel for the
active job (the same SIGKILL path as the watchdog) and exposes
cancelActiveWorkerJob(); signed /queue/status, /queue/cancel-active and
/queue/clear endpoints on the worker.
- Website: admin-only /api/health/worker-queue/* routes sign and proxy to the
worker, and an app-integration-worker-queue panel drives them.