# 18-Apr-2026 — Ramblers-upload-worker → Integration-worker [#114](https://github.com/nbarrett/ngx-ramblers/issues/114)

## [build 614](https://github.com/nbarrett/ngx-ramblers/actions/runs/24613943307) — [commit 40e8d62](https://github.com/nbarrett/ngx-ramblers/commit/40e8d62202cac68a47709e707cdb38a165d2ca9d)

_____

Rename the Ramblers upload worker to a generic integration worker, cut every
in-process browser session out of the website, and replace the buffered
Serenity StageCrewMember step-delivery path with a Playwright-native reporter
that posts each `test.step` as it ends. Clean up env-var wiring, settings
surface, and scripts along the way.

## Rename: ramblers-upload-worker → integration-worker

The worker now handles Ramblers walks upload, Flickr album scraping,
arbitrary HTML fetches and full static-site migration - "upload worker"
no longer describes it. Rename every piece of the worker plumbing;
keep walks-upload-specific files (dispatcher, queue, session registry,
audit notifier, execution model) named `ramblers-upload-*` because they
describe one *type* of integration job.

- Fly app: `ngx-ramblers-upload-worker` → `ngx-ramblers-integration-worker`
- Fly config: `fly.worker.toml` → `fly.integration-worker.toml`
- Manual deploy workflow: `build-and-deploy-ramblers-upload-worker.yml` →
  `build-and-deploy-integration-worker.yml`
- Docker image tags: `ngx-ramblers-worker:N` →
  `ngx-ramblers-integration-worker:N`; push tag
  `ngx-ramblers:worker-N` → `ngx-ramblers:integration-worker-N`;
  local tag `ngx-worker:local` → `ngx-integration-worker:local`;
  local container `ngx-worker-local` → `ngx-integration-worker-local`
- Env vars: `RAMBLERS_UPLOAD_WORKER_*` → `INTEGRATION_WORKER_*`
- Types: `RamblersUploadWorker*` → `IntegrationWorker*`
- Files: 12 files in `server/lib/ramblers/`
  (`integration-worker-{client,server,routes,runner,callback-client,
execution-state,crypto,crypto.spec}`) plus
  `server/deploy/deploy-integration-worker.ts` and
  `projects/.../integration-worker.model.ts`
- Route prefix: `/api/ramblers-upload-worker/*` →
  `/api/integration-worker/*`

## Sync browser ops moved to the worker

- `POST /api/integration-worker/browser/html-fetch`
- launches Chromium,
  navigates, returns `{ html, finalUrl, baseHref }`. Used by
  `html-from-url`, `fetchHtmlFromUrl` and `search-venue-website` on
  the website.
- `POST /api/integration-worker/browser/flickr-user-albums`
- scrapes a
  Flickr albums page (TrustArc consent handled via Playwright
  `frameLocator`) and extracts `FlickrScrapedUserAlbumsData`.
- New `integration-worker-browser-client.ts` on the website: signs with
  `signRamblersUploadBody`, POSTs to the worker, falls back to plain
  `fetch` if the worker errors out.
- `flickr-provider.scrapeUserAlbumsPage` becomes a thin wrapper calling
  `scrapeFlickrUserAlbumsViaIntegrationWorker`; the ~180-line in-process
  page scrape + cookie-consent iframe dance moved to the worker.
- `migration-routes.ts` drops `launchPlainBrowser()` entirely. The
  `USE_SERENITY_FOR_MIGRATION` feature flag and its env entry go with it.

## Full site migration moved to the worker (async job + callbacks)

- `POST /api/integration-worker/migration/jobs` accepts
  `{ jobId, siteConfig, persistData, uploadTos3, callback }`, runs
  `migrateStaticSite()` on the worker, streams progress back via signed
  callbacks. Refuses a second job while one is running.
- New model types:
  `IntegrationWorkerMigrationJobRequest`,
  `IntegrationWorkerMigrationProgressCallback`,
  `IntegrationWorkerMigrationResultCallback`.
- New backend callback endpoints at
  `/api/integration-worker/migration/progress` and `/result`. Each
  verifies the signature, looks up the active migration session, and
  forwards `{ level, message }` or the final `{ status, result,
errorMessage }` to the client websocket.
- New `migration-session-registry.ts`
- in-memory map of
  `jobId → { ws, siteIdentifier, siteName, historyId, startedAt }`.
- New `migration-notifier.ts`
- websocket send + audit-log /
  history-status persistence, split from the old ws handler so the
  callback route and initial handler share it.
- `site-migration-ws-handler.ts` no longer imports
  `migrate-static-site-engine` or calls `migrateStaticSite` directly.
  It resolves the site config, creates the `migrationHistory` doc,
  generates a `jobId` via `crypto.randomUUID()`, registers a migration
  session, and submits the job to the worker via
  `submitMigrationJobToIntegrationWorker`.

## Real-time step streaming: fix and DRY up the delivery path

The original `DomainEventPublisher` StageCrewMember (48aca026) worked
under WebdriverIO because WDIO fires `InteractionFinished` /
`TaskFinished` synchronously as each action executes. Under
`@serenity-js/playwright-test` those domain events are buffered by
`PlaywrightEventBuffer` and only announced to the Stage at `onTestEnd`
- so our crew member only ever saw them in one end-of-run batch, which
  is why the audit view only populated when the browser closed.

Replace the whole per-step delivery path with a Playwright-native
`Reporter` (bypasses Serenity's buffer):

- `server/lib/serenity-js/reporters/realtime-step-reporter.ts` -
  listens to `Reporter.onStepEnd`, filters `category === "test.step"`,
  and POSTs each step to `/api/integration-worker/progress` with the
  existing `test-step` envelope (`eventData.timestamp/details.name/
outcome.code`). Registered in `playwright.config.ts` alongside the
  Serenity reporter.

Backend handler, session registry and audit notifier tightened so the
real-time stream behaves correctly:

- `/progress` handler now responds `200 OK` immediately, then processes
  the audit asynchronously. Eliminates the 1-3 s per-request latency
  that was queueing test-step POSTs and bunching them at end-of-run.
- `/progress` for `test-step` now unwraps the
  `{ eventData, finished }` envelope before handing the inner event to
  `parseTestStepEvent`, so the parser sees real `timestamp` /
  `details.name` / `outcome.code` again (the old path was passing the
  envelope, so every event came out with a NaN timestamp and got
  clustered into one write-time window).
- `sendAudit` is now serialised per-session via a promise chain. Each
  background callback runs strictly after the previous one, so
  `session.record` increments monotonically instead of racing (no more
  runs of `rec=8, rec=8, rec=8`).
- `updateRamblersUploadSession` mutates the session object in place
  (`Object.assign`) instead of replacing it in the map, so all captured
  `session` references stay in sync with the incremented record.

## DRY: drop the old WDIO-era event transport entirely

With `RealtimeStepReporter` as the single per-step path, the old
WebSocket-based transport and its wiring are dead weight:

- Deleted:
- `server/lib/serenity-js/screenplay/crew/DomainEventPublisher.ts`
- `server/lib/ramblers/process-test-step-event.ts`
- `server/lib/ramblers/integration-worker-websocket-server.ts`
- `server/lib/websockets/websocket-client.ts`
- Removed `EventType.TEST_STEP_REPORTER` from the shared
  `websocket.model.ts`.
- Dropped the `TEST_STEP_REPORTER` handler from the backend
  `websocket-server.ts` and the `createIntegrationWorkerWebSocketServer`
  call from the worker server.

## Environment enum: move to shared model, prune, enforce

- `Environment` enum moved to
  `projects/ngx-ramblers/src/app/models/environment.model.ts` so the
  Angular app can use it too; deleted the server shim at
  `server/lib/env-config/environment-model.ts`; all 28 server importers
  rewritten to the shared path.
- Added `ADMIN_MONGODB_URI` and `PLAYWRIGHT_HEADLESS` to the enum.
- Every literal `process.env.X` / `env.X` / `secrets.X` access in the
  server now goes through `Environment.X` - deploy script, CLI
  commands, worker routes.
- Frontend `environment-settings.ts` now uses `[Environment.X]`
  computed keys instead of string literals.

## Local CLI ergonomics

- New `--no-headless` flag on `ngx-cli local dev`. Runs the worker
  headed (visible browser) and implies `--no-docker-worker` (Docker
  always headless). Existing `--no-docker-worker` still works for
  fast-feedback headless iteration without an image rebuild.
- `docker-worker` and `headless` threaded through `LocalRunConfig`.
- Force-kills stale processes on backend / frontend / worker ports
  before each start (`SIGTERM` with 4 s grace, then `SIGKILL`);
  tears down any stale worker container.

## Walk export UI

- `autoTitle` added to the `<app-page>` wrapper so the tab view has an
  H1 above the tabs, via the same mechanism used by Bookings / Member
  Bulk Load / Environment Setup.
- Audit sort state persisted in the URL as `?sort=<field>&sort-order=
<asc|desc>`:
- New `StoredValue` entries `AUDIT_MESSAGE`, `AUDIT_TIME`,
  `DURATION`, `STATUS` (kebab-case values in the URL).
- Reuse existing `StoredValue.SORT` + `SORT_ORDER` as the query
  keys and `SortDirection.ASC/DESC` from `sort.model.ts` for the
  values (no duplicate `"asc"`/`"desc"` constants).
- `sortAuditsBy(field: StoredValue)` now flips order on the same
  column and resets to `DESC` on a new column, then persists via
  `updateUrl()`.
- `AUDIT_SORT_FIELD_MAPPING` maps the URL/`StoredValue` kebab-case
  to the internal camelCase field the `sortBy` helper reads.
- Duration column:
- `durationMs` is now computed on every audit unconditionally
  (the `showDetail` gate was stopping the field from being added
  when the checkbox was off, which was why the Duration header sort
  did nothing).
- `dateUtils.formatDuration` no longer short-circuits to "0 secs"
  when `fromTime === 0` - that falsy-zero guard was wrong for
  millisecond-delta inputs and was making every row render "0 secs"
  regardless of the real duration.
- `applyFilter` handles the oldest row (no previous) cleanly
  instead of computing `auditTime - 0`.

## Scripts cleanup

`server/package.json`:
- Dropped `test: "failsafe serenity report"`
- duplicated `serenity`.
- Dropped `lint` / `lintfix` / `lintversion` and the `tslint` dep +
  `server/tslint.json`. tslint has been unmaintained since 2019 and
  none of the hooks or CI workflows invoke them.

`server/test-setenv.sh`: dropped stale `CHROME_BIN` /
`CHROMEDRIVER_PATH` exports.

`server/.env`: local key names renamed to `INTEGRATION_WORKER_*`;
stale Chrome paths removed. (Local values only; remote Fly secrets
re-imported under the new names separately.)

## Dead code removal

- `server/lib/migration/migrate-static-site.ts` (standalone NWK
  script, unreferenced).
- `server/lib/migration/serenity-migration-engine.ts`,
  `serenity-migration-utils.ts`, `serenity-utils.ts` and the
  `screenplay/` scaffolding (parallel implementation, not imported
  by any live route).
- `server/lib/migration/puppeteer-utils.ts` replaced by
  `server/lib/migration/browser-utils.ts` (only compiled into the
  worker image - never called from the website live path).

## Fly secrets

Staged but not deployed on the two apps available to the current
token:
- `ngx-ramblers-upload-worker`: new `INTEGRATION_WORKER_SHARED_SECRET`
  / `INTEGRATION_WORKER_ENCRYPTION_KEY` staged with matching digests
  to the old names.
- `ngx-ramblers`: new `INTEGRATION_WORKER_SHARED_SECRET` /
  `INTEGRATION_WORKER_ENCRYPTION_KEY` / `INTEGRATION_WORKER_URL` /
  `INTEGRATION_WORKER_CALLBACK_BASE_URL` staged.

Staging central DB `config.environments` for `staging`, `ekwg` and
`ekwg-dev-2-staging` renamed the per-env secret keys from
`RAMBLERS_UPLOAD_WORKER_*` to `INTEGRATION_WORKER_*` so future
deploys push the new names consistently.

Old secrets remain Deployed on both apps until the next deploy
activates the new staged names.

### **worker**: switch upload worker to Serenity/JS Playwright stack and cut the website off WebdriverIO/Puppeteer ([#114](https://github.com/nbarrett/ngx-ramblers/issues/114))

Migrate the Ramblers upload worker off the WebdriverIO + Chrome/Chromedriver
runtime onto the officially supported Serenity/JS Playwright image, harden the
end-to-end upload path, and remove every WebdriverIO/Puppeteer usage from the
codebase so the website image no longer ships a browser stack.

## Worker image

- `Dockerfile` `worker` stage now builds from
  `ghcr.io/serenity-js/playwright:v1.58.1-noble` (Node 24 pinned via `n`,
  npm 11, OpenJDK, Playwright browsers pre-installed on Ubuntu 24.04).
  Chrome/Chromedriver/`@puppeteer/browsers` installs are removed - the
  Playwright image already ships the browsers the worker needs.
- `npm run serenity-bdd-update` is pre-run so runtime doesn't re-download
  the Serenity-BDD JAR.
- `.dockerignore` excludes `server/node_modules`, `cli-output/`, `target/`,
  `ts-gen/` and `dist/` so worker images stay lean.

## Website image

- Renamed `--target origin` to `--target website` to reflect its role as
  the pure web tier. Build arg `CHROME_VERSION`, OpenJDK 21 install and
  `serenity-bdd-update` are removed from this stage - the website no
  longer runs Serenity or any browser.
- CI workflow now builds with `--target website` and no longer passes
  `--build-arg CHROME_VERSION`.
- Dockerfile header comment updated to state the website stage ships
  neither a browser nor a JDK.

## Test runner migration (WebdriverIO → Playwright)

- `server/package.json` `serenity-run` now invokes
  `playwright test -c ./playwright.config.ts`. Adds `@playwright/test`,
  `playwright`, `@serenity-js/playwright` and `@serenity-js/playwright-test`.
- New `server/playwright.config.ts` wires the Serenity crew, BDD reporter,
  `ArtifactArchiver` and `Photographer`, using `SerenityFixtures` /
  `SerenityWorkerFixtures`. Honours `PLAYWRIGHT_HEADLESS`,
  `RAMBLERS_FEATURE` and `BASE_URL`.
- Feature specs (`contacts`, `process-command-args`, `walks-and-events-
manager`, `walks-programme`, `walks-upload`) migrated off mocha /
  `actorCalled` statics to the `@serenity-js/playwright-test` `actorCalled`
  fixture; `beforeEach` engagement is replaced with per-scenario
  `actorCalled(...)`.
- `SaveBrowserSource` task replaces the WDIO `@wdio/globals` call with
  `BrowseTheWeb.currentPage().executeScript(...)` and now writes to
  `target/browser-source/` with a fallback page on capture failure.
- New `SetFileInput` task drives file uploads via Playwright's native
  `setInputFiles`, replacing `Enter.theValue(...)` and fixing the
  "Input of type file cannot be filled" failure on the Ramblers upload
  modal.
- Login wait restored to `Wait.upTo(DEFAULT_WAIT_TIMEOUT).until(...)`
  sourcing the 15 s budget from a single shared constant in
  `server/lib/serenity-js/config/serenity-timeouts.ts` (also consumed
  by `playwright.config.ts`). Fixes the 5 s `TimeoutExpiredError` on
  the post-login Create-menu dropdown check.

## Bulk-action selector hardening

- `WalksPageElements` tightens the `Delete` / `Unpublish` / `Publish` /
  `Uncancel` selectors to
  `#vbo-action-form-wrapper input[data-vbo='vbo-action'][value='...']`,
  eliminating ambiguous matches when the VBO action bar renders twice.
- Adds a `loggedInWalksManagerPage` element (body CSS class check) used
  inside `AuthErrorCookieBannerOrCreateMenuDropdown` so post-login
  detection succeeds even before the Create menu renders.

## Live audit streaming (in progress)

- Server spawns the Serenity subprocess with
  `RAMBLERS_UPLOAD_WORKER_CALLBACK_BASE_URL`,
  `_CALLBACK_PROGRESS_PATH`, `_CALLBACK_RESULT_PATH`,
  `_CALLBACK_SECRET` and `_JOB_ID` in the environment (new
  `Environment` enum entries).
- `DomainEventPublisher` posts `TEST_STEP` events directly to the worker
  callback endpoint when that env is present, with diagnostic logging on
  the transport branch actually taken, instead of going through the
  Serenity subprocess websocket (which intermittently dropped with
  close code 1006 and caused audit rows to bunch at end-of-run).
- `processTestStepEvent` now writes step events straight into the active
  local session so the Walk upload audit tab updates while the browser
  is still running.
- Frontend `WebSocketClientService` wraps `onopen` / `onmessage` /
  `onclose` / `onerror` in `NgZone.run(...)` so progress events trigger
  change detection. `walk-export.ts` calls
  `ChangeDetectorRef.detectChanges()` on updates, broadcasts
  `NamedEventType.REFRESH` on completion, and runs a 2 s audit refresh
  loop while the export is in progress.

## Audit parser & timing clean-up

- `ramblers-audit-parser` strips ANSI escape sequences via module-level
  `RegExp`s built from `String.fromCharCode(27)` (no inline control
  characters, passes `no-control-regex` without a disable) and filters
  separator / noise lines such as `Execution Summary`, `Scenarios:`,
  `Real time:`, `Total time:`, `1 passed`,
  `Succeeded with exit code 0 as all scripts passed`, `> playwright`,
  `@playwright/test`, `@serenity-js/playwright` / `-test`,
  `Running 1 test using 1 worker`, the `mkdir -p target/site/archive`
  line and all-`=` separator lines.
- `walk-export.ts` `timing()` no longer has a special SUMMARY case that
  returned the whole-session duration; summary and step rows now both
  show true per-step durations derived from `durationMs` or the
  adjacent-row delta.
- `ramblers-upload-walks.ts` progress line reports
  `PLAYWRIGHT_HEADLESS` and `BASE_URL` instead of the removed
  `CHROME_BIN` / `CHROMEDRIVER_PATH`.

## Report storage (zipped artefacts)

- `ramblers-upload-worker-runner.ts` uploads the Serenity report as a
  single `<keyPrefix>.zip` via `adm-zip`, emitting lifecycle progress
  events for archive start, completion (with file count, byte size and
  elapsed ms) and failure.
- `recordReportLocation` stores the summary message as
  `s3://<bucket>/<keyPrefix>.zip`.
- New `GET /api/aws/report/:bucket/*` route in `aws-routes.ts` streams
  assets from the archive: `aws-controllers.ts` lazy-extracts each
  report under `target/report-cache/<sha1>/`, guards against directory
  traversal, dedupes concurrent extractions via an in-memory promise
  cache, and falls back to a chunked `streamToBuffer` for SDK response
  bodies that aren't already `Buffer`s.
- `walk-export.ts` `openReport()` uses the new route with
  `bucket` + `keyPrefix` + `/_/` + `index.html` as the split marker;
  sessions without a stored bucket are no-ops.

## WebdriverIO / Puppeteer removal across the codebase

- `@serenity-js/webdriverio`, `webdriverio`, `puppeteer` and every
  `@wdio/*` runner/CLI package removed from `server/package.json`; the
  lockfile shrunk by ~4 200 lines.
- Deleted `server/wdio.conf.ts`, `server/lib/serenity-js/features/config/
actors.ts`, `server/lib/migration/puppeteer-utils.ts` and the parallel
  `serenity-migration-engine.ts` / `serenity-migration-utils.ts` /
  `serenity-utils.ts` stack plus their `screenplay/` scaffolding and the
  standalone `migrate-static-site.ts` script (none were imported by any
  live route).
- New `server/lib/migration/browser-utils.ts` provides a Playwright
  `chromium.launch(...)` replacement for the old puppeteer helper.
- `migrate-static-site-engine.ts` now imports `Browser`/`Page` from
  `playwright`, swaps `networkidle2` for `networkidle`, and restructures
  every `page.evaluate(fn, arg1, arg2, ...)` call for Playwright's
  single-argument contract (object-payload pattern).
- `migration-routes.ts` collapses the dual WebdriverIO / Puppeteer
  branches, drops the `USE_SERENITY_FOR_MIGRATION` feature flag and its
  env entry, and uses Playwright-only paths.
- `flickr-provider.ts` is on Playwright: `browser.url`/`waitUntil`/
  `execute`/`getPageSource`/`$` become `page.goto`/`waitForFunction`/
  `evaluate`/`content`/`locator`, and cookie-consent iframe handling is
  reimplemented with `page.frameLocator(...)` rather than
  `switchToFrame`.
- `Environment` enum drops `CHROME_BIN` and `CHROMEDRIVER_PATH`; the
  `ramblers-audit-parser` noise list replaces the webdriverio marker
  with `@serenity-js/playwright`.

## Local dev ergonomics

- `bin/ngx-cli local dev` force-kills stale processes holding the
  backend / frontend / worker ports (`SIGTERM` with a 4 s grace, then
  `SIGKILL`), and tears down any old worker container before each run.
- Docker worker rebuild trigger now watches
  `server/playwright.config.ts`, `server/lib/ramblers`,
  `server/lib/serenity-js` and `server/lib/env-config` recursively, via
  a new `newestMtimeMs` helper, so nested source edits trigger a
  rebuild.
- When `PLAYWRIGHT_HEADLESS=false` is set, the in-process worker path
  is used (headed debugging) instead of the Docker worker, and
  `npm exec playwright install chromium` is ensured with output
  streamed to both stdout and the worker log.
- Backend env injects
  `RAMBLERS_UPLOAD_WORKER_CALLBACK_BASE_URL=http://host.docker.internal:<port>`
  when the Docker worker is used.
- Worker Docker build logs are streamed to the terminal as well as the
  worker log file.

## CI

- `build-push-and-deploy-ngx-ramblers-docker-image.yml` builds a
  separate worker image tagged `worker-${run_number}` whenever
  worker-relevant files changed, deploys it with
  `--image-tag worker-${run_number}`, and now builds the website image
  with `--target website` (no Chrome build arg).
- Manual `build-and-deploy-ramblers-upload-worker.yml` points at the
  `worker` build stage.
- `server/deploy/detect-worker-changes.ts` treats
  `server/playwright.config.ts` as worker-relevant and drops
  `server/wdio.conf.ts` from its watch list.