{"id":"69f318d9e87d3c4451b248ee","title":"2026 04 30 Other","path":"how-to/committee/release-notes/2026-04-30-other","contentMarkdown":"# 30-Apr-2026 — make Fly post-deploy scaling self-healing\n\n## [build 632](https://github.com/nbarrett/ngx-ramblers/actions/runs/25156041311) — [commit acc3049](https://github.com/nbarrett/ngx-ramblers/commit/acc3049898948d7727a497b009b3c8f11fc18094)\n\n_____\n\nTwo of sixteen environments (medway, sevenoaks) failed in the\nall-environments deploy of 4cae672e with:\n\nError: could not update machine ...: failed to update VM ...:\naborted: machine is replacing: concurrent update in progress\n\nAfter flyctl deploy --strategy rolling returned success, the script\nfired flyctl scale memory immediately. Fly's machine-replacement state\nmachine was still finalising the rolling swap behind the scenes - the\nscale call collided with that, and the existing 3-attempt 2s/4s retry\nbackoff was too short to outlast the replacement window. By the second\nretry the old machine was gone and the new one had not yet appeared in\nthe 'app' process group, producing the secondary \"No active machines\"\nerror.\n\nTwo changes in server/lib/fly/fly-commands.ts that the deploy loop now\nrelies on instead of issuing flyctl scale unconditionally:\n\n- listAppMachines reads flyctl machines list -j and surfaces id,\nstate, and current memory_mb. ensureScale compares this to the\ntarget scaleCount and memory from the environment config. When\nboth already match (the common case - memory rarely changes\nbetween deploys) the scale commands are skipped entirely, which\non its own would have rescued today's two failures.\n\n- When scaling IS needed, waitForStableMachines polls flyctl\nmachines list -j every 3s (90s timeout) until the expected count\nof machines are all in 'started' state with no transition states\nvisible, then issues the scale command. Removes the race instead\nof retrying through it. After a count change the wait is repeated\nbefore the memory call so the two scale operations cannot collide\nwith each other either.\n\ndeploy-to-environments.ts now calls ensureScale once per environment\nin place of the back-to-back flyctl scale count and flyctl scale memory\ncalls.","contentHtml":"<h1>30-Apr-2026 — make Fly post-deploy scaling self-healing</h1>\n<h2><a href=\"https://github.com/nbarrett/ngx-ramblers/actions/runs/25156041311\">build 632</a> — <a href=\"https://github.com/nbarrett/ngx-ramblers/commit/acc3049898948d7727a497b009b3c8f11fc18094\">commit acc3049</a></h2>\n<hr>\n<p>Two of sixteen environments (medway, sevenoaks) failed in the\nall-environments deploy of 4cae672e with:</p>\n<p>Error: could not update machine ...: failed to update VM ...:\naborted: machine is replacing: concurrent update in progress</p>\n<p>After flyctl deploy --strategy rolling returned success, the script\nfired flyctl scale memory immediately. Fly&#39;s machine-replacement state\nmachine was still finalising the rolling swap behind the scenes - the\nscale call collided with that, and the existing 3-attempt 2s/4s retry\nbackoff was too short to outlast the replacement window. By the second\nretry the old machine was gone and the new one had not yet appeared in\nthe &#39;app&#39; process group, producing the secondary &quot;No active machines&quot;\nerror.</p>\n<p>Two changes in server/lib/fly/fly-commands.ts that the deploy loop now\nrelies on instead of issuing flyctl scale unconditionally:</p>\n<ul>\n<li><p>listAppMachines reads flyctl machines list -j and surfaces id,\nstate, and current memory_mb. ensureScale compares this to the\ntarget scaleCount and memory from the environment config. When\nboth already match (the common case - memory rarely changes\nbetween deploys) the scale commands are skipped entirely, which\non its own would have rescued today&#39;s two failures.</p>\n</li>\n<li><p>When scaling IS needed, waitForStableMachines polls flyctl\nmachines list -j every 3s (90s timeout) until the expected count\nof machines are all in &#39;started&#39; state with no transition states\nvisible, then issues the scale command. Removes the race instead\nof retrying through it. After a count change the wait is repeated\nbefore the memory call so the two scale operations cannot collide\nwith each other either.</p>\n</li>\n</ul>\n<p>deploy-to-environments.ts now calls ensureScale once per environment\nin place of the back-to-back flyctl scale count and flyctl scale memory\ncalls.</p>\n"}