30-Apr-2026 — make Fly post-deploy scaling self-healing

build 632 — commit acc3049

Two of sixteen environments (medway, sevenoaks) failed in the all-environments deploy of 4cae672e with:

Error: could not update machine ...: failed to update VM ...: aborted: machine is replacing: concurrent update in progress

After flyctl deploy --strategy rolling returned success, the script fired flyctl scale memory immediately. Fly's machine-replacement state machine was still finalising the rolling swap behind the scenes - the scale call collided with that, and the existing 3-attempt 2s/4s retry backoff was too short to outlast the replacement window. By the second retry the old machine was gone and the new one had not yet appeared in the 'app' process group, producing the secondary "No active machines" error.

Two changes in server/lib/fly/fly-commands.ts that the deploy loop now relies on instead of issuing flyctl scale unconditionally:

• listAppMachines reads flyctl machines list -j and surfaces id, state, and current memory_mb. ensureScale compares this to the target scaleCount and memory from the environment config. When both already match (the common case - memory rarely changes between deploys) the scale commands are skipped entirely, which on its own would have rescued today's two failures.

• When scaling IS needed, waitForStableMachines polls flyctl machines list -j every 3s (90s timeout) until the expected count of machines are all in 'started' state with no transition states visible, then issues the scale command. Removes the race instead of retrying through it. After a count change the wait is repeated before the memory call so the two scale operations cannot collide with each other either.

deploy-to-environments.ts now calls ensureScale once per environment in place of the back-to-back flyctl scale count and flyctl scale memory calls.