09-Sep-2026 — hold the worker awake for the whole job so the overrun stop can fire, and add admin #300
build 885 — commit 570e098
The problem
The 45-minute overrun stop is a timer inside the worker's Node process, but Fly autosuspends the machine on excess capacity (zero concurrent requests), which freezes the VM and stops that timer advancing. The previous keep-alive pinged the health address every 60s, which left the machine idle almost the whole minute, so Fly suspended it anyway. A running OS Maps conversion then froze for hours, the queue jammed behind it, and the overrun stop never fired.
What is new
- The keep-alive now holds a single streaming connection open to the worker for as long as a job runs (reopening if it drops), so Fly always sees concurrency and does not suspend mid-job. Idle suspend is unchanged, so the cost saving stays. The overrun watchdog remains the backstop and can now actually run because the VM is never frozen while working.
- A GET /api/integration-worker/hold endpoint the keep-alive holds open.
- Admin controls for the worker queue, on the System Settings Diagnostics tab: see the active job and anything queued, stop the active job, or clear the queue, so a stuck job can be ended and someone else can run something without a machine restart.
How it works
- Worker: the queue gains clearQueued(); the runner registers a cancel for the active job (the same SIGKILL path as the watchdog) and exposes cancelActiveWorkerJob(); signed /queue/status, /queue/cancel-active and /queue/clear endpoints on the worker.
- Website: admin-only /api/health/worker-queue/* routes sign and proxy to the worker, and an app-integration-worker-queue panel drives them.