{"id":"69e3f6e1b77beae64a7c1065","title":"2026 04 18","path":"how-to/committee/release-notes/2026-04-18","contentMarkdown":"# 18-Apr-2026 — Ramblers-upload-worker → Integration-worker [#114](https://github.com/nbarrett/ngx-ramblers/issues/114)\n\n## [build 614](https://github.com/nbarrett/ngx-ramblers/actions/runs/24613943307) — [commit 40e8d62](https://github.com/nbarrett/ngx-ramblers/commit/40e8d62202cac68a47709e707cdb38a165d2ca9d)\n\n_____\n\nRename the Ramblers upload worker to a generic integration worker, cut every\nin-process browser session out of the website, and replace the buffered\nSerenity StageCrewMember step-delivery path with a Playwright-native reporter\nthat posts each `test.step` as it ends. Clean up env-var wiring, settings\nsurface, and scripts along the way.\n\n## Rename: ramblers-upload-worker → integration-worker\n\nThe worker now handles Ramblers walks upload, Flickr album scraping,\narbitrary HTML fetches and full static-site migration - \"upload worker\"\nno longer describes it. Rename every piece of the worker plumbing;\nkeep walks-upload-specific files (dispatcher, queue, session registry,\naudit notifier, execution model) named `ramblers-upload-*` because they\ndescribe one *type* of integration job.\n\n- Fly app: `ngx-ramblers-upload-worker` → `ngx-ramblers-integration-worker`\n- Fly config: `fly.worker.toml` → `fly.integration-worker.toml`\n- Manual deploy workflow: `build-and-deploy-ramblers-upload-worker.yml` →\n `build-and-deploy-integration-worker.yml`\n- Docker image tags: `ngx-ramblers-worker:N` →\n `ngx-ramblers-integration-worker:N`; push tag\n `ngx-ramblers:worker-N` → `ngx-ramblers:integration-worker-N`;\n local tag `ngx-worker:local` → `ngx-integration-worker:local`;\n local container `ngx-worker-local` → `ngx-integration-worker-local`\n- Env vars: `RAMBLERS_UPLOAD_WORKER_*` → `INTEGRATION_WORKER_*`\n- Types: `RamblersUploadWorker*` → `IntegrationWorker*`\n- Files: 12 files in `server/lib/ramblers/`\n (`integration-worker-{client,server,routes,runner,callback-client,\nexecution-state,crypto,crypto.spec}`) plus\n `server/deploy/deploy-integration-worker.ts` and\n `projects/.../integration-worker.model.ts`\n- Route prefix: `/api/ramblers-upload-worker/*` →\n `/api/integration-worker/*`\n\n## Sync browser ops moved to the worker\n\n- `POST /api/integration-worker/browser/html-fetch`\n- launches Chromium,\n navigates, returns `{ html, finalUrl, baseHref }`. Used by\n `html-from-url`, `fetchHtmlFromUrl` and `search-venue-website` on\n the website.\n- `POST /api/integration-worker/browser/flickr-user-albums`\n- scrapes a\n Flickr albums page (TrustArc consent handled via Playwright\n `frameLocator`) and extracts `FlickrScrapedUserAlbumsData`.\n- New `integration-worker-browser-client.ts` on the website: signs with\n `signRamblersUploadBody`, POSTs to the worker, falls back to plain\n `fetch` if the worker errors out.\n- `flickr-provider.scrapeUserAlbumsPage` becomes a thin wrapper calling\n `scrapeFlickrUserAlbumsViaIntegrationWorker`; the ~180-line in-process\n page scrape + cookie-consent iframe dance moved to the worker.\n- `migration-routes.ts` drops `launchPlainBrowser()` entirely. The\n `USE_SERENITY_FOR_MIGRATION` feature flag and its env entry go with it.\n\n## Full site migration moved to the worker (async job + callbacks)\n\n- `POST /api/integration-worker/migration/jobs` accepts\n `{ jobId, siteConfig, persistData, uploadTos3, callback }`, runs\n `migrateStaticSite()` on the worker, streams progress back via signed\n callbacks. Refuses a second job while one is running.\n- New model types:\n `IntegrationWorkerMigrationJobRequest`,\n `IntegrationWorkerMigrationProgressCallback`,\n `IntegrationWorkerMigrationResultCallback`.\n- New backend callback endpoints at\n `/api/integration-worker/migration/progress` and `/result`. Each\n verifies the signature, looks up the active migration session, and\n forwards `{ level, message }` or the final `{ status, result,\nerrorMessage }` to the client websocket.\n- New `migration-session-registry.ts`\n- in-memory map of\n `jobId → { ws, siteIdentifier, siteName, historyId, startedAt }`.\n- New `migration-notifier.ts`\n- websocket send + audit-log /\n history-status persistence, split from the old ws handler so the\n callback route and initial handler share it.\n- `site-migration-ws-handler.ts` no longer imports\n `migrate-static-site-engine` or calls `migrateStaticSite` directly.\n It resolves the site config, creates the `migrationHistory` doc,\n generates a `jobId` via `crypto.randomUUID()`, registers a migration\n session, and submits the job to the worker via\n `submitMigrationJobToIntegrationWorker`.\n\n## Real-time step streaming: fix and DRY up the delivery path\n\nThe original `DomainEventPublisher` StageCrewMember (48aca026) worked\nunder WebdriverIO because WDIO fires `InteractionFinished` /\n`TaskFinished` synchronously as each action executes. Under\n`@serenity-js/playwright-test` those domain events are buffered by\n`PlaywrightEventBuffer` and only announced to the Stage at `onTestEnd`\n- so our crew member only ever saw them in one end-of-run batch, which\n is why the audit view only populated when the browser closed.\n\nReplace the whole per-step delivery path with a Playwright-native\n`Reporter` (bypasses Serenity's buffer):\n\n- `server/lib/serenity-js/reporters/realtime-step-reporter.ts` -\n listens to `Reporter.onStepEnd`, filters `category === \"test.step\"`,\n and POSTs each step to `/api/integration-worker/progress` with the\n existing `test-step` envelope (`eventData.timestamp/details.name/\noutcome.code`). Registered in `playwright.config.ts` alongside the\n Serenity reporter.\n\nBackend handler, session registry and audit notifier tightened so the\nreal-time stream behaves correctly:\n\n- `/progress` handler now responds `200 OK` immediately, then processes\n the audit asynchronously. Eliminates the 1-3 s per-request latency\n that was queueing test-step POSTs and bunching them at end-of-run.\n- `/progress` for `test-step` now unwraps the\n `{ eventData, finished }` envelope before handing the inner event to\n `parseTestStepEvent`, so the parser sees real `timestamp` /\n `details.name` / `outcome.code` again (the old path was passing the\n envelope, so every event came out with a NaN timestamp and got\n clustered into one write-time window).\n- `sendAudit` is now serialised per-session via a promise chain. Each\n background callback runs strictly after the previous one, so\n `session.record` increments monotonically instead of racing (no more\n runs of `rec=8, rec=8, rec=8`).\n- `updateRamblersUploadSession` mutates the session object in place\n (`Object.assign`) instead of replacing it in the map, so all captured\n `session` references stay in sync with the incremented record.\n\n## DRY: drop the old WDIO-era event transport entirely\n\nWith `RealtimeStepReporter` as the single per-step path, the old\nWebSocket-based transport and its wiring are dead weight:\n\n- Deleted:\n- `server/lib/serenity-js/screenplay/crew/DomainEventPublisher.ts`\n- `server/lib/ramblers/process-test-step-event.ts`\n- `server/lib/ramblers/integration-worker-websocket-server.ts`\n- `server/lib/websockets/websocket-client.ts`\n- Removed `EventType.TEST_STEP_REPORTER` from the shared\n `websocket.model.ts`.\n- Dropped the `TEST_STEP_REPORTER` handler from the backend\n `websocket-server.ts` and the `createIntegrationWorkerWebSocketServer`\n call from the worker server.\n\n## Environment enum: move to shared model, prune, enforce\n\n- `Environment` enum moved to\n `projects/ngx-ramblers/src/app/models/environment.model.ts` so the\n Angular app can use it too; deleted the server shim at\n `server/lib/env-config/environment-model.ts`; all 28 server importers\n rewritten to the shared path.\n- Added `ADMIN_MONGODB_URI` and `PLAYWRIGHT_HEADLESS` to the enum.\n- Every literal `process.env.X` / `env.X` / `secrets.X` access in the\n server now goes through `Environment.X` - deploy script, CLI\n commands, worker routes.\n- Frontend `environment-settings.ts` now uses `[Environment.X]`\n computed keys instead of string literals.\n\n## Local CLI ergonomics\n\n- New `--no-headless` flag on `ngx-cli local dev`. Runs the worker\n headed (visible browser) and implies `--no-docker-worker` (Docker\n always headless). Existing `--no-docker-worker` still works for\n fast-feedback headless iteration without an image rebuild.\n- `docker-worker` and `headless` threaded through `LocalRunConfig`.\n- Force-kills stale processes on backend / frontend / worker ports\n before each start (`SIGTERM` with 4 s grace, then `SIGKILL`);\n tears down any stale worker container.\n\n## Walk export UI\n\n- `autoTitle` added to the `` wrapper so the tab view has an\n H1 above the tabs, via the same mechanism used by Bookings / Member\n Bulk Load / Environment Setup.\n- Audit sort state persisted in the URL as `?sort=&sort-order=\n`:\n- New `StoredValue` entries `AUDIT_MESSAGE`, `AUDIT_TIME`,\n `DURATION`, `STATUS` (kebab-case values in the URL).\n- Reuse existing `StoredValue.SORT` + `SORT_ORDER` as the query\n keys and `SortDirection.ASC/DESC` from `sort.model.ts` for the\n values (no duplicate `\"asc\"`/`\"desc\"` constants).\n- `sortAuditsBy(field: StoredValue)` now flips order on the same\n column and resets to `DESC` on a new column, then persists via\n `updateUrl()`.\n- `AUDIT_SORT_FIELD_MAPPING` maps the URL/`StoredValue` kebab-case\n to the internal camelCase field the `sortBy` helper reads.\n- Duration column:\n- `durationMs` is now computed on every audit unconditionally\n (the `showDetail` gate was stopping the field from being added\n when the checkbox was off, which was why the Duration header sort\n did nothing).\n- `dateUtils.formatDuration` no longer short-circuits to \"0 secs\"\n when `fromTime === 0` - that falsy-zero guard was wrong for\n millisecond-delta inputs and was making every row render \"0 secs\"\n regardless of the real duration.\n- `applyFilter` handles the oldest row (no previous) cleanly\n instead of computing `auditTime - 0`.\n\n## Scripts cleanup\n\n`server/package.json`:\n- Dropped `test: \"failsafe serenity report\"`\n- duplicated `serenity`.\n- Dropped `lint` / `lintfix` / `lintversion` and the `tslint` dep +\n `server/tslint.json`. tslint has been unmaintained since 2019 and\n none of the hooks or CI workflows invoke them.\n\n`server/test-setenv.sh`: dropped stale `CHROME_BIN` /\n`CHROMEDRIVER_PATH` exports.\n\n`server/.env`: local key names renamed to `INTEGRATION_WORKER_*`;\nstale Chrome paths removed. (Local values only; remote Fly secrets\nre-imported under the new names separately.)\n\n## Dead code removal\n\n- `server/lib/migration/migrate-static-site.ts` (standalone NWK\n script, unreferenced).\n- `server/lib/migration/serenity-migration-engine.ts`,\n `serenity-migration-utils.ts`, `serenity-utils.ts` and the\n `screenplay/` scaffolding (parallel implementation, not imported\n by any live route).\n- `server/lib/migration/puppeteer-utils.ts` replaced by\n `server/lib/migration/browser-utils.ts` (only compiled into the\n worker image - never called from the website live path).\n\n## Fly secrets\n\nStaged but not deployed on the two apps available to the current\ntoken:\n- `ngx-ramblers-upload-worker`: new `INTEGRATION_WORKER_SHARED_SECRET`\n / `INTEGRATION_WORKER_ENCRYPTION_KEY` staged with matching digests\n to the old names.\n- `ngx-ramblers`: new `INTEGRATION_WORKER_SHARED_SECRET` /\n `INTEGRATION_WORKER_ENCRYPTION_KEY` / `INTEGRATION_WORKER_URL` /\n `INTEGRATION_WORKER_CALLBACK_BASE_URL` staged.\n\nStaging central DB `config.environments` for `staging`, `ekwg` and\n`ekwg-dev-2-staging` renamed the per-env secret keys from\n`RAMBLERS_UPLOAD_WORKER_*` to `INTEGRATION_WORKER_*` so future\ndeploys push the new names consistently.\n\nOld secrets remain Deployed on both apps until the next deploy\nactivates the new staged names.\n\n### **worker**: switch upload worker to Serenity/JS Playwright stack and cut the website off WebdriverIO/Puppeteer ([#114](https://github.com/nbarrett/ngx-ramblers/issues/114))\n\nMigrate the Ramblers upload worker off the WebdriverIO + Chrome/Chromedriver\nruntime onto the officially supported Serenity/JS Playwright image, harden the\nend-to-end upload path, and remove every WebdriverIO/Puppeteer usage from the\ncodebase so the website image no longer ships a browser stack.\n\n## Worker image\n\n- `Dockerfile` `worker` stage now builds from\n `ghcr.io/serenity-js/playwright:v1.58.1-noble` (Node 24 pinned via `n`,\n npm 11, OpenJDK, Playwright browsers pre-installed on Ubuntu 24.04).\n Chrome/Chromedriver/`@puppeteer/browsers` installs are removed - the\n Playwright image already ships the browsers the worker needs.\n- `npm run serenity-bdd-update` is pre-run so runtime doesn't re-download\n the Serenity-BDD JAR.\n- `.dockerignore` excludes `server/node_modules`, `cli-output/`, `target/`,\n `ts-gen/` and `dist/` so worker images stay lean.\n\n## Website image\n\n- Renamed `--target origin` to `--target website` to reflect its role as\n the pure web tier. Build arg `CHROME_VERSION`, OpenJDK 21 install and\n `serenity-bdd-update` are removed from this stage - the website no\n longer runs Serenity or any browser.\n- CI workflow now builds with `--target website` and no longer passes\n `--build-arg CHROME_VERSION`.\n- Dockerfile header comment updated to state the website stage ships\n neither a browser nor a JDK.\n\n## Test runner migration (WebdriverIO → Playwright)\n\n- `server/package.json` `serenity-run` now invokes\n `playwright test -c ./playwright.config.ts`. Adds `@playwright/test`,\n `playwright`, `@serenity-js/playwright` and `@serenity-js/playwright-test`.\n- New `server/playwright.config.ts` wires the Serenity crew, BDD reporter,\n `ArtifactArchiver` and `Photographer`, using `SerenityFixtures` /\n `SerenityWorkerFixtures`. Honours `PLAYWRIGHT_HEADLESS`,\n `RAMBLERS_FEATURE` and `BASE_URL`.\n- Feature specs (`contacts`, `process-command-args`, `walks-and-events-\nmanager`, `walks-programme`, `walks-upload`) migrated off mocha /\n `actorCalled` statics to the `@serenity-js/playwright-test` `actorCalled`\n fixture; `beforeEach` engagement is replaced with per-scenario\n `actorCalled(...)`.\n- `SaveBrowserSource` task replaces the WDIO `@wdio/globals` call with\n `BrowseTheWeb.currentPage().executeScript(...)` and now writes to\n `target/browser-source/` with a fallback page on capture failure.\n- New `SetFileInput` task drives file uploads via Playwright's native\n `setInputFiles`, replacing `Enter.theValue(...)` and fixing the\n \"Input of type file cannot be filled\" failure on the Ramblers upload\n modal.\n- Login wait restored to `Wait.upTo(DEFAULT_WAIT_TIMEOUT).until(...)`\n sourcing the 15 s budget from a single shared constant in\n `server/lib/serenity-js/config/serenity-timeouts.ts` (also consumed\n by `playwright.config.ts`). Fixes the 5 s `TimeoutExpiredError` on\n the post-login Create-menu dropdown check.\n\n## Bulk-action selector hardening\n\n- `WalksPageElements` tightens the `Delete` / `Unpublish` / `Publish` /\n `Uncancel` selectors to\n `#vbo-action-form-wrapper input[data-vbo='vbo-action'][value='...']`,\n eliminating ambiguous matches when the VBO action bar renders twice.\n- Adds a `loggedInWalksManagerPage` element (body CSS class check) used\n inside `AuthErrorCookieBannerOrCreateMenuDropdown` so post-login\n detection succeeds even before the Create menu renders.\n\n## Live audit streaming (in progress)\n\n- Server spawns the Serenity subprocess with\n `RAMBLERS_UPLOAD_WORKER_CALLBACK_BASE_URL`,\n `_CALLBACK_PROGRESS_PATH`, `_CALLBACK_RESULT_PATH`,\n `_CALLBACK_SECRET` and `_JOB_ID` in the environment (new\n `Environment` enum entries).\n- `DomainEventPublisher` posts `TEST_STEP` events directly to the worker\n callback endpoint when that env is present, with diagnostic logging on\n the transport branch actually taken, instead of going through the\n Serenity subprocess websocket (which intermittently dropped with\n close code 1006 and caused audit rows to bunch at end-of-run).\n- `processTestStepEvent` now writes step events straight into the active\n local session so the Walk upload audit tab updates while the browser\n is still running.\n- Frontend `WebSocketClientService` wraps `onopen` / `onmessage` /\n `onclose` / `onerror` in `NgZone.run(...)` so progress events trigger\n change detection. `walk-export.ts` calls\n `ChangeDetectorRef.detectChanges()` on updates, broadcasts\n `NamedEventType.REFRESH` on completion, and runs a 2 s audit refresh\n loop while the export is in progress.\n\n## Audit parser & timing clean-up\n\n- `ramblers-audit-parser` strips ANSI escape sequences via module-level\n `RegExp`s built from `String.fromCharCode(27)` (no inline control\n characters, passes `no-control-regex` without a disable) and filters\n separator / noise lines such as `Execution Summary`, `Scenarios:`,\n `Real time:`, `Total time:`, `1 passed`,\n `Succeeded with exit code 0 as all scripts passed`, `> playwright`,\n `@playwright/test`, `@serenity-js/playwright` / `-test`,\n `Running 1 test using 1 worker`, the `mkdir -p target/site/archive`\n line and all-`=` separator lines.\n- `walk-export.ts` `timing()` no longer has a special SUMMARY case that\n returned the whole-session duration; summary and step rows now both\n show true per-step durations derived from `durationMs` or the\n adjacent-row delta.\n- `ramblers-upload-walks.ts` progress line reports\n `PLAYWRIGHT_HEADLESS` and `BASE_URL` instead of the removed\n `CHROME_BIN` / `CHROMEDRIVER_PATH`.\n\n## Report storage (zipped artefacts)\n\n- `ramblers-upload-worker-runner.ts` uploads the Serenity report as a\n single `.zip` via `adm-zip`, emitting lifecycle progress\n events for archive start, completion (with file count, byte size and\n elapsed ms) and failure.\n- `recordReportLocation` stores the summary message as\n `s3:///.zip`.\n- New `GET /api/aws/report/:bucket/*` route in `aws-routes.ts` streams\n assets from the archive: `aws-controllers.ts` lazy-extracts each\n report under `target/report-cache//`, guards against directory\n traversal, dedupes concurrent extractions via an in-memory promise\n cache, and falls back to a chunked `streamToBuffer` for SDK response\n bodies that aren't already `Buffer`s.\n- `walk-export.ts` `openReport()` uses the new route with\n `bucket` + `keyPrefix` + `/_/` + `index.html` as the split marker;\n sessions without a stored bucket are no-ops.\n\n## WebdriverIO / Puppeteer removal across the codebase\n\n- `@serenity-js/webdriverio`, `webdriverio`, `puppeteer` and every\n `@wdio/*` runner/CLI package removed from `server/package.json`; the\n lockfile shrunk by ~4 200 lines.\n- Deleted `server/wdio.conf.ts`, `server/lib/serenity-js/features/config/\nactors.ts`, `server/lib/migration/puppeteer-utils.ts` and the parallel\n `serenity-migration-engine.ts` / `serenity-migration-utils.ts` /\n `serenity-utils.ts` stack plus their `screenplay/` scaffolding and the\n standalone `migrate-static-site.ts` script (none were imported by any\n live route).\n- New `server/lib/migration/browser-utils.ts` provides a Playwright\n `chromium.launch(...)` replacement for the old puppeteer helper.\n- `migrate-static-site-engine.ts` now imports `Browser`/`Page` from\n `playwright`, swaps `networkidle2` for `networkidle`, and restructures\n every `page.evaluate(fn, arg1, arg2, ...)` call for Playwright's\n single-argument contract (object-payload pattern).\n- `migration-routes.ts` collapses the dual WebdriverIO / Puppeteer\n branches, drops the `USE_SERENITY_FOR_MIGRATION` feature flag and its\n env entry, and uses Playwright-only paths.\n- `flickr-provider.ts` is on Playwright: `browser.url`/`waitUntil`/\n `execute`/`getPageSource`/`$` become `page.goto`/`waitForFunction`/\n `evaluate`/`content`/`locator`, and cookie-consent iframe handling is\n reimplemented with `page.frameLocator(...)` rather than\n `switchToFrame`.\n- `Environment` enum drops `CHROME_BIN` and `CHROMEDRIVER_PATH`; the\n `ramblers-audit-parser` noise list replaces the webdriverio marker\n with `@serenity-js/playwright`.\n\n## Local dev ergonomics\n\n- `bin/ngx-cli local dev` force-kills stale processes holding the\n backend / frontend / worker ports (`SIGTERM` with a 4 s grace, then\n `SIGKILL`), and tears down any old worker container before each run.\n- Docker worker rebuild trigger now watches\n `server/playwright.config.ts`, `server/lib/ramblers`,\n `server/lib/serenity-js` and `server/lib/env-config` recursively, via\n a new `newestMtimeMs` helper, so nested source edits trigger a\n rebuild.\n- When `PLAYWRIGHT_HEADLESS=false` is set, the in-process worker path\n is used (headed debugging) instead of the Docker worker, and\n `npm exec playwright install chromium` is ensured with output\n streamed to both stdout and the worker log.\n- Backend env injects\n `RAMBLERS_UPLOAD_WORKER_CALLBACK_BASE_URL=http://host.docker.internal:`\n when the Docker worker is used.\n- Worker Docker build logs are streamed to the terminal as well as the\n worker log file.\n\n## CI\n\n- `build-push-and-deploy-ngx-ramblers-docker-image.yml` builds a\n separate worker image tagged `worker-${run_number}` whenever\n worker-relevant files changed, deploys it with\n `--image-tag worker-${run_number}`, and now builds the website image\n with `--target website` (no Chrome build arg).\n- Manual `build-and-deploy-ramblers-upload-worker.yml` points at the\n `worker` build stage.\n- `server/deploy/detect-worker-changes.ts` treats\n `server/playwright.config.ts` as worker-relevant and drops\n `server/wdio.conf.ts` from its watch list.\n","contentHtml":"

18-Apr-2026 — Ramblers-upload-worker → Integration-worker #114

\n

build 614 — commit 40e8d62

\n
\n

Rename the Ramblers upload worker to a generic integration worker, cut every\nin-process browser session out of the website, and replace the buffered\nSerenity StageCrewMember step-delivery path with a Playwright-native reporter\nthat posts each test.step as it ends. Clean up env-var wiring, settings\nsurface, and scripts along the way.

\n

Rename: ramblers-upload-worker → integration-worker

\n

The worker now handles Ramblers walks upload, Flickr album scraping,\narbitrary HTML fetches and full static-site migration - "upload worker"\nno longer describes it. Rename every piece of the worker plumbing;\nkeep walks-upload-specific files (dispatcher, queue, session registry,\naudit notifier, execution model) named ramblers-upload-* because they\ndescribe one type of integration job.

\n
    \n
  • Fly app: ngx-ramblers-upload-worker → ngx-ramblers-integration-worker
    • \n
  • Fly config: fly.worker.toml → fly.integration-worker.toml
    • \n
  • Manual deploy workflow: build-and-deploy-ramblers-upload-worker.yml →\nbuild-and-deploy-integration-worker.yml
    • \n
  • Docker image tags: ngx-ramblers-worker:N →\nngx-ramblers-integration-worker:N; push tag\nngx-ramblers:worker-N → ngx-ramblers:integration-worker-N;\nlocal tag ngx-worker:local → ngx-integration-worker:local;\nlocal container ngx-worker-local → ngx-integration-worker-local
    • \n
  • Env vars: RAMBLERS_UPLOAD_WORKER_* → INTEGRATION_WORKER_*
    • \n
  • Types: RamblersUploadWorker* → IntegrationWorker*
    • \n
  • Files: 12 files in server/lib/ramblers/\n(integration-worker-{client,server,routes,runner,callback-client, execution-state,crypto,crypto.spec}) plus\nserver/deploy/deploy-integration-worker.ts and\nprojects/.../integration-worker.model.ts
    • \n
  • Route prefix: /api/ramblers-upload-worker/* →\n/api/integration-worker/*
    • \n
\n

Sync browser ops moved to the worker

\n
    \n
  • POST /api/integration-worker/browser/html-fetch
    • \n
  • launches Chromium,\nnavigates, returns { html, finalUrl, baseHref }. Used by\nhtml-from-url, fetchHtmlFromUrl and search-venue-website on\nthe website.
    • \n
  • POST /api/integration-worker/browser/flickr-user-albums
    • \n
  • scrapes a\nFlickr albums page (TrustArc consent handled via Playwright\nframeLocator) and extracts FlickrScrapedUserAlbumsData.
    • \n
  • New integration-worker-browser-client.ts on the website: signs with\nsignRamblersUploadBody, POSTs to the worker, falls back to plain\nfetch if the worker errors out.
    • \n
  • flickr-provider.scrapeUserAlbumsPage becomes a thin wrapper calling\nscrapeFlickrUserAlbumsViaIntegrationWorker; the ~180-line in-process\npage scrape + cookie-consent iframe dance moved to the worker.
    • \n
  • migration-routes.ts drops launchPlainBrowser() entirely. The\nUSE_SERENITY_FOR_MIGRATION feature flag and its env entry go with it.
    • \n
\n

Full site migration moved to the worker (async job + callbacks)

\n
    \n
  • POST /api/integration-worker/migration/jobs accepts\n{ jobId, siteConfig, persistData, uploadTos3, callback }, runs\nmigrateStaticSite() on the worker, streams progress back via signed\ncallbacks. Refuses a second job while one is running.
    • \n
  • New model types:\nIntegrationWorkerMigrationJobRequest,\nIntegrationWorkerMigrationProgressCallback,\nIntegrationWorkerMigrationResultCallback.
    • \n
  • New backend callback endpoints at\n/api/integration-worker/migration/progress and /result. Each\nverifies the signature, looks up the active migration session, and\nforwards { level, message } or the final { status, result, errorMessage } to the client websocket.
    • \n
  • New migration-session-registry.ts
    • \n
  • in-memory map of\njobId → { ws, siteIdentifier, siteName, historyId, startedAt }.
    • \n
  • New migration-notifier.ts
    • \n
  • websocket send + audit-log /\nhistory-status persistence, split from the old ws handler so the\ncallback route and initial handler share it.
    • \n
  • site-migration-ws-handler.ts no longer imports\nmigrate-static-site-engine or calls migrateStaticSite directly.\nIt resolves the site config, creates the migrationHistory doc,\ngenerates a jobId via crypto.randomUUID(), registers a migration\nsession, and submits the job to the worker via\nsubmitMigrationJobToIntegrationWorker.
    • \n
\n

Real-time step streaming: fix and DRY up the delivery path

\n

The original DomainEventPublisher StageCrewMember (48aca026) worked\nunder WebdriverIO because WDIO fires InteractionFinished /\nTaskFinished synchronously as each action executes. Under\n@serenity-js/playwright-test those domain events are buffered by\nPlaywrightEventBuffer and only announced to the Stage at onTestEnd

\n
    \n
  • so our crew member only ever saw them in one end-of-run batch, which\nis why the audit view only populated when the browser closed.
    • \n
\n

Replace the whole per-step delivery path with a Playwright-native\nReporter (bypasses Serenity's buffer):

\n
    \n
  • server/lib/serenity-js/reporters/realtime-step-reporter.ts -\nlistens to Reporter.onStepEnd, filters category === "test.step",\nand POSTs each step to /api/integration-worker/progress with the\nexisting test-step envelope (eventData.timestamp/details.name/ outcome.code). Registered in playwright.config.ts alongside the\nSerenity reporter.
    • \n
\n

Backend handler, session registry and audit notifier tightened so the\nreal-time stream behaves correctly:

\n
    \n
  • /progress handler now responds 200 OK immediately, then processes\nthe audit asynchronously. Eliminates the 1-3 s per-request latency\nthat was queueing test-step POSTs and bunching them at end-of-run.
    • \n
  • /progress for test-step now unwraps the\n{ eventData, finished } envelope before handing the inner event to\nparseTestStepEvent, so the parser sees real timestamp /\ndetails.name / outcome.code again (the old path was passing the\nenvelope, so every event came out with a NaN timestamp and got\nclustered into one write-time window).
    • \n
  • sendAudit is now serialised per-session via a promise chain. Each\nbackground callback runs strictly after the previous one, so\nsession.record increments monotonically instead of racing (no more\nruns of rec=8, rec=8, rec=8).
    • \n
  • updateRamblersUploadSession mutates the session object in place\n(Object.assign) instead of replacing it in the map, so all captured\nsession references stay in sync with the incremented record.
    • \n
\n

DRY: drop the old WDIO-era event transport entirely

\n

With RealtimeStepReporter as the single per-step path, the old\nWebSocket-based transport and its wiring are dead weight:

\n
    \n
  • Deleted:
    • \n
  • server/lib/serenity-js/screenplay/crew/DomainEventPublisher.ts
    • \n
  • server/lib/ramblers/process-test-step-event.ts
    • \n
  • server/lib/ramblers/integration-worker-websocket-server.ts
    • \n
  • server/lib/websockets/websocket-client.ts
    • \n
  • Removed EventType.TEST_STEP_REPORTER from the shared\nwebsocket.model.ts.
    • \n
  • Dropped the TEST_STEP_REPORTER handler from the backend\nwebsocket-server.ts and the createIntegrationWorkerWebSocketServer\ncall from the worker server.
    • \n
\n

Environment enum: move to shared model, prune, enforce

\n
    \n
  • Environment enum moved to\nprojects/ngx-ramblers/src/app/models/environment.model.ts so the\nAngular app can use it too; deleted the server shim at\nserver/lib/env-config/environment-model.ts; all 28 server importers\nrewritten to the shared path.
    • \n
  • Added ADMIN_MONGODB_URI and PLAYWRIGHT_HEADLESS to the enum.
    • \n
  • Every literal process.env.X / env.X / secrets.X access in the\nserver now goes through Environment.X - deploy script, CLI\ncommands, worker routes.
    • \n
  • Frontend environment-settings.ts now uses [Environment.X]\ncomputed keys instead of string literals.
    • \n
\n

Local CLI ergonomics

\n
    \n
  • New --no-headless flag on ngx-cli local dev. Runs the worker\nheaded (visible browser) and implies --no-docker-worker (Docker\nalways headless). Existing --no-docker-worker still works for\nfast-feedback headless iteration without an image rebuild.
    • \n
  • docker-worker and headless threaded through LocalRunConfig.
    • \n
  • Force-kills stale processes on backend / frontend / worker ports\nbefore each start (SIGTERM with 4 s grace, then SIGKILL);\ntears down any stale worker container.
    • \n
\n

Walk export UI

\n
    \n
  • autoTitle added to the <app-page> wrapper so the tab view has an\nH1 above the tabs, via the same mechanism used by Bookings / Member\nBulk Load / Environment Setup.
    • \n
  • Audit sort state persisted in the URL as `?sort=&sort-order=
    • \n
\n

<asc|desc>`:

\n
    \n
  • New StoredValue entries AUDIT_MESSAGE, AUDIT_TIME,\nDURATION, STATUS (kebab-case values in the URL).
    • \n
  • Reuse existing StoredValue.SORT + SORT_ORDER as the query\nkeys and SortDirection.ASC/DESC from sort.model.ts for the\nvalues (no duplicate "asc"/"desc" constants).
    • \n
  • sortAuditsBy(field: StoredValue) now flips order on the same\ncolumn and resets to DESC on a new column, then persists via\nupdateUrl().
    • \n
  • AUDIT_SORT_FIELD_MAPPING maps the URL/StoredValue kebab-case\nto the internal camelCase field the sortBy helper reads.
    • \n
  • Duration column:
    • \n
  • durationMs is now computed on every audit unconditionally\n(the showDetail gate was stopping the field from being added\nwhen the checkbox was off, which was why the Duration header sort\ndid nothing).
    • \n
  • dateUtils.formatDuration no longer short-circuits to "0 secs"\nwhen fromTime === 0 - that falsy-zero guard was wrong for\nmillisecond-delta inputs and was making every row render "0 secs"\nregardless of the real duration.
    • \n
  • applyFilter handles the oldest row (no previous) cleanly\ninstead of computing auditTime - 0.
    • \n
\n

Scripts cleanup

\n

server/package.json:

\n
    \n
  • Dropped test: "failsafe serenity report"
    • \n
  • duplicated serenity.
    • \n
  • Dropped lint / lintfix / lintversion and the tslint dep +\nserver/tslint.json. tslint has been unmaintained since 2019 and\nnone of the hooks or CI workflows invoke them.
    • \n
\n

server/test-setenv.sh: dropped stale CHROME_BIN /\nCHROMEDRIVER_PATH exports.

\n

server/.env: local key names renamed to INTEGRATION_WORKER_*;\nstale Chrome paths removed. (Local values only; remote Fly secrets\nre-imported under the new names separately.)

\n

Dead code removal

\n
    \n
  • server/lib/migration/migrate-static-site.ts (standalone NWK\nscript, unreferenced).
    • \n
  • server/lib/migration/serenity-migration-engine.ts,\nserenity-migration-utils.ts, serenity-utils.ts and the\nscreenplay/ scaffolding (parallel implementation, not imported\nby any live route).
    • \n
  • server/lib/migration/puppeteer-utils.ts replaced by\nserver/lib/migration/browser-utils.ts (only compiled into the\nworker image - never called from the website live path).
    • \n
\n

Fly secrets

\n

Staged but not deployed on the two apps available to the current\ntoken:

\n
    \n
  • ngx-ramblers-upload-worker: new INTEGRATION_WORKER_SHARED_SECRET\n/ INTEGRATION_WORKER_ENCRYPTION_KEY staged with matching digests\nto the old names.
    • \n
  • ngx-ramblers: new INTEGRATION_WORKER_SHARED_SECRET /\nINTEGRATION_WORKER_ENCRYPTION_KEY / INTEGRATION_WORKER_URL /\nINTEGRATION_WORKER_CALLBACK_BASE_URL staged.
    • \n
\n

Staging central DB config.environments for staging, ekwg and\nekwg-dev-2-staging renamed the per-env secret keys from\nRAMBLERS_UPLOAD_WORKER_* to INTEGRATION_WORKER_* so future\ndeploys push the new names consistently.

\n

Old secrets remain Deployed on both apps until the next deploy\nactivates the new staged names.

\n

worker: switch upload worker to Serenity/JS Playwright stack and cut the website off WebdriverIO/Puppeteer (#114)

\n

Migrate the Ramblers upload worker off the WebdriverIO + Chrome/Chromedriver\nruntime onto the officially supported Serenity/JS Playwright image, harden the\nend-to-end upload path, and remove every WebdriverIO/Puppeteer usage from the\ncodebase so the website image no longer ships a browser stack.

\n

Worker image

\n
    \n
  • Dockerfile worker stage now builds from\nghcr.io/serenity-js/playwright:v1.58.1-noble (Node 24 pinned via n,\nnpm 11, OpenJDK, Playwright browsers pre-installed on Ubuntu 24.04).\nChrome/Chromedriver/@puppeteer/browsers installs are removed - the\nPlaywright image already ships the browsers the worker needs.
    • \n
  • npm run serenity-bdd-update is pre-run so runtime doesn't re-download\nthe Serenity-BDD JAR.
    • \n
  • .dockerignore excludes server/node_modules, cli-output/, target/,\nts-gen/ and dist/ so worker images stay lean.
    • \n
\n

Website image

\n
    \n
  • Renamed --target origin to --target website to reflect its role as\nthe pure web tier. Build arg CHROME_VERSION, OpenJDK 21 install and\nserenity-bdd-update are removed from this stage - the website no\nlonger runs Serenity or any browser.
    • \n
  • CI workflow now builds with --target website and no longer passes\n--build-arg CHROME_VERSION.
    • \n
  • Dockerfile header comment updated to state the website stage ships\nneither a browser nor a JDK.
    • \n
\n

Test runner migration (WebdriverIO → Playwright)

\n
    \n
  • server/package.json serenity-run now invokes\nplaywright test -c ./playwright.config.ts. Adds @playwright/test,\nplaywright, @serenity-js/playwright and @serenity-js/playwright-test.
    • \n
  • New server/playwright.config.ts wires the Serenity crew, BDD reporter,\nArtifactArchiver and Photographer, using SerenityFixtures /\nSerenityWorkerFixtures. Honours PLAYWRIGHT_HEADLESS,\nRAMBLERS_FEATURE and BASE_URL.
    • \n
  • Feature specs (contacts, process-command-args, walks-and-events- manager, walks-programme, walks-upload) migrated off mocha /\nactorCalled statics to the @serenity-js/playwright-test actorCalled\nfixture; beforeEach engagement is replaced with per-scenario\nactorCalled(...).
    • \n
  • SaveBrowserSource task replaces the WDIO @wdio/globals call with\nBrowseTheWeb.currentPage().executeScript(...) and now writes to\ntarget/browser-source/ with a fallback page on capture failure.
    • \n
  • New SetFileInput task drives file uploads via Playwright's native\nsetInputFiles, replacing Enter.theValue(...) and fixing the\n"Input of type file cannot be filled" failure on the Ramblers upload\nmodal.
    • \n
  • Login wait restored to Wait.upTo(DEFAULT_WAIT_TIMEOUT).until(...)\nsourcing the 15 s budget from a single shared constant in\nserver/lib/serenity-js/config/serenity-timeouts.ts (also consumed\nby playwright.config.ts). Fixes the 5 s TimeoutExpiredError on\nthe post-login Create-menu dropdown check.
    • \n
\n

Bulk-action selector hardening

\n
    \n
  • WalksPageElements tightens the Delete / Unpublish / Publish /\nUncancel selectors to\n#vbo-action-form-wrapper input[data-vbo='vbo-action'][value='...'],\neliminating ambiguous matches when the VBO action bar renders twice.
    • \n
  • Adds a loggedInWalksManagerPage element (body CSS class check) used\ninside AuthErrorCookieBannerOrCreateMenuDropdown so post-login\ndetection succeeds even before the Create menu renders.
    • \n
\n

Live audit streaming (in progress)

\n
    \n
  • Server spawns the Serenity subprocess with\nRAMBLERS_UPLOAD_WORKER_CALLBACK_BASE_URL,\n_CALLBACK_PROGRESS_PATH, _CALLBACK_RESULT_PATH,\n_CALLBACK_SECRET and _JOB_ID in the environment (new\nEnvironment enum entries).
    • \n
  • DomainEventPublisher posts TEST_STEP events directly to the worker\ncallback endpoint when that env is present, with diagnostic logging on\nthe transport branch actually taken, instead of going through the\nSerenity subprocess websocket (which intermittently dropped with\nclose code 1006 and caused audit rows to bunch at end-of-run).
    • \n
  • processTestStepEvent now writes step events straight into the active\nlocal session so the Walk upload audit tab updates while the browser\nis still running.
    • \n
  • Frontend WebSocketClientService wraps onopen / onmessage /\nonclose / onerror in NgZone.run(...) so progress events trigger\nchange detection. walk-export.ts calls\nChangeDetectorRef.detectChanges() on updates, broadcasts\nNamedEventType.REFRESH on completion, and runs a 2 s audit refresh\nloop while the export is in progress.
    • \n
\n

Audit parser & timing clean-up

\n
    \n
  • ramblers-audit-parser strips ANSI escape sequences via module-level\nRegExps built from String.fromCharCode(27) (no inline control\ncharacters, passes no-control-regex without a disable) and filters\nseparator / noise lines such as Execution Summary, Scenarios:,\nReal time:, Total time:, 1 passed,\nSucceeded with exit code 0 as all scripts passed, > playwright,\n@playwright/test, @serenity-js/playwright / -test,\nRunning 1 test using 1 worker, the mkdir -p target/site/archive\nline and all-= separator lines.
    • \n
  • walk-export.ts timing() no longer has a special SUMMARY case that\nreturned the whole-session duration; summary and step rows now both\nshow true per-step durations derived from durationMs or the\nadjacent-row delta.
    • \n
  • ramblers-upload-walks.ts progress line reports\nPLAYWRIGHT_HEADLESS and BASE_URL instead of the removed\nCHROME_BIN / CHROMEDRIVER_PATH.
    • \n
\n

Report storage (zipped artefacts)

\n
    \n
  • ramblers-upload-worker-runner.ts uploads the Serenity report as a\nsingle <keyPrefix>.zip via adm-zip, emitting lifecycle progress\nevents for archive start, completion (with file count, byte size and\nelapsed ms) and failure.
    • \n
  • recordReportLocation stores the summary message as\ns3://<bucket>/<keyPrefix>.zip.
    • \n
  • New GET /api/aws/report/:bucket/* route in aws-routes.ts streams\nassets from the archive: aws-controllers.ts lazy-extracts each\nreport under target/report-cache/<sha1>/, guards against directory\ntraversal, dedupes concurrent extractions via an in-memory promise\ncache, and falls back to a chunked streamToBuffer for SDK response\nbodies that aren't already Buffers.
    • \n
  • walk-export.ts openReport() uses the new route with\nbucket + keyPrefix + /_/ + index.html as the split marker;\nsessions without a stored bucket are no-ops.
    • \n
\n

WebdriverIO / Puppeteer removal across the codebase

\n
    \n
  • @serenity-js/webdriverio, webdriverio, puppeteer and every\n@wdio/* runner/CLI package removed from server/package.json; the\nlockfile shrunk by ~4 200 lines.
    • \n
  • Deleted server/wdio.conf.ts, server/lib/serenity-js/features/config/ actors.ts, server/lib/migration/puppeteer-utils.ts and the parallel\nserenity-migration-engine.ts / serenity-migration-utils.ts /\nserenity-utils.ts stack plus their screenplay/ scaffolding and the\nstandalone migrate-static-site.ts script (none were imported by any\nlive route).
    • \n
  • New server/lib/migration/browser-utils.ts provides a Playwright\nchromium.launch(...) replacement for the old puppeteer helper.
    • \n
  • migrate-static-site-engine.ts now imports Browser/Page from\nplaywright, swaps networkidle2 for networkidle, and restructures\nevery page.evaluate(fn, arg1, arg2, ...) call for Playwright's\nsingle-argument contract (object-payload pattern).
    • \n
  • migration-routes.ts collapses the dual WebdriverIO / Puppeteer\nbranches, drops the USE_SERENITY_FOR_MIGRATION feature flag and its\nenv entry, and uses Playwright-only paths.
    • \n
  • flickr-provider.ts is on Playwright: browser.url/waitUntil/\nexecute/getPageSource/$ become page.goto/waitForFunction/\nevaluate/content/locator, and cookie-consent iframe handling is\nreimplemented with page.frameLocator(...) rather than\nswitchToFrame.
    • \n
  • Environment enum drops CHROME_BIN and CHROMEDRIVER_PATH; the\nramblers-audit-parser noise list replaces the webdriverio marker\nwith @serenity-js/playwright.
    • \n
\n

Local dev ergonomics

\n
    \n
  • bin/ngx-cli local dev force-kills stale processes holding the\nbackend / frontend / worker ports (SIGTERM with a 4 s grace, then\nSIGKILL), and tears down any old worker container before each run.
    • \n
  • Docker worker rebuild trigger now watches\nserver/playwright.config.ts, server/lib/ramblers,\nserver/lib/serenity-js and server/lib/env-config recursively, via\na new newestMtimeMs helper, so nested source edits trigger a\nrebuild.
    • \n
  • When PLAYWRIGHT_HEADLESS=false is set, the in-process worker path\nis used (headed debugging) instead of the Docker worker, and\nnpm exec playwright install chromium is ensured with output\nstreamed to both stdout and the worker log.
    • \n
  • Backend env injects\nRAMBLERS_UPLOAD_WORKER_CALLBACK_BASE_URL=http://host.docker.internal:<port>\nwhen the Docker worker is used.
    • \n
  • Worker Docker build logs are streamed to the terminal as well as the\nworker log file.
    • \n
\n

CI

\n
    \n
  • build-push-and-deploy-ngx-ramblers-docker-image.yml builds a\nseparate worker image tagged worker-${run_number} whenever\nworker-relevant files changed, deploys it with\n--image-tag worker-${run_number}, and now builds the website image\nwith --target website (no Chrome build arg).
    • \n
  • Manual build-and-deploy-ramblers-upload-worker.yml points at the\nworker build stage.
    • \n
  • server/deploy/detect-worker-changes.ts treats\nserver/playwright.config.ts as worker-relevant and drops\nserver/wdio.conf.ts from its watch list.
    • \n
\n"}