18-Apr-2026 — Ramblers-upload-worker → Integration-worker #114

build 614 — commit 40e8d62

Rename the Ramblers upload worker to a generic integration worker, cut every in-process browser session out of the website, and replace the buffered Serenity StageCrewMember step-delivery path with a Playwright-native reporter that posts each test.step as it ends. Clean up env-var wiring, settings surface, and scripts along the way.

Rename: ramblers-upload-worker → integration-worker

The worker now handles Ramblers walks upload, Flickr album scraping, arbitrary HTML fetches and full static-site migration - "upload worker" no longer describes it. Rename every piece of the worker plumbing; keep walks-upload-specific files (dispatcher, queue, session registry, audit notifier, execution model) named ramblers-upload-* because they describe one type of integration job.

• Fly app: ngx-ramblers-upload-worker → ngx-ramblers-integration-worker
• Fly config: fly.worker.toml → fly.integration-worker.toml
• Manual deploy workflow: build-and-deploy-ramblers-upload-worker.yml → build-and-deploy-integration-worker.yml
• Docker image tags: ngx-ramblers-worker:N → ngx-ramblers-integration-worker:N; push tag ngx-ramblers:worker-N → ngx-ramblers:integration-worker-N; local tag ngx-worker:local → ngx-integration-worker:local; local container ngx-worker-local → ngx-integration-worker-local
• Env vars: RAMBLERS_UPLOAD_WORKER_* → INTEGRATION_WORKER_*
• Types: RamblersUploadWorker* → IntegrationWorker*
• Files: 12 files in server/lib/ramblers/ (integration-worker-{client,server,routes,runner,callback-client, execution-state,crypto,crypto.spec}) plus server/deploy/deploy-integration-worker.ts and projects/.../integration-worker.model.ts
• Route prefix: /api/ramblers-upload-worker/* → /api/integration-worker/*

Sync browser ops moved to the worker

• POST /api/integration-worker/browser/html-fetch
• launches Chromium, navigates, returns { html, finalUrl, baseHref }. Used by html-from-url, fetchHtmlFromUrl and search-venue-website on the website.
• POST /api/integration-worker/browser/flickr-user-albums
• scrapes a Flickr albums page (TrustArc consent handled via Playwright frameLocator) and extracts FlickrScrapedUserAlbumsData.
• New integration-worker-browser-client.ts on the website: signs with signRamblersUploadBody, POSTs to the worker, falls back to plain fetch if the worker errors out.
• flickr-provider.scrapeUserAlbumsPage becomes a thin wrapper calling scrapeFlickrUserAlbumsViaIntegrationWorker; the ~180-line in-process page scrape + cookie-consent iframe dance moved to the worker.
• migration-routes.ts drops launchPlainBrowser() entirely. The USE_SERENITY_FOR_MIGRATION feature flag and its env entry go with it.

Full site migration moved to the worker (async job + callbacks)

• POST /api/integration-worker/migration/jobs accepts { jobId, siteConfig, persistData, uploadTos3, callback }, runs migrateStaticSite() on the worker, streams progress back via signed callbacks. Refuses a second job while one is running.
• New model types: IntegrationWorkerMigrationJobRequest, IntegrationWorkerMigrationProgressCallback, IntegrationWorkerMigrationResultCallback.
• New backend callback endpoints at /api/integration-worker/migration/progress and /result. Each verifies the signature, looks up the active migration session, and forwards { level, message } or the final { status, result, errorMessage } to the client websocket.
• New migration-session-registry.ts
• in-memory map of jobId → { ws, siteIdentifier, siteName, historyId, startedAt }.
• New migration-notifier.ts
• websocket send + audit-log / history-status persistence, split from the old ws handler so the callback route and initial handler share it.
• site-migration-ws-handler.ts no longer imports migrate-static-site-engine or calls migrateStaticSite directly. It resolves the site config, creates the migrationHistory doc, generates a jobId via crypto.randomUUID(), registers a migration session, and submits the job to the worker via submitMigrationJobToIntegrationWorker.

Real-time step streaming: fix and DRY up the delivery path

The original DomainEventPublisher StageCrewMember (48aca026) worked under WebdriverIO because WDIO fires InteractionFinished / TaskFinished synchronously as each action executes. Under @serenity-js/playwright-test those domain events are buffered by PlaywrightEventBuffer and only announced to the Stage at onTestEnd

• so our crew member only ever saw them in one end-of-run batch, which is why the audit view only populated when the browser closed.

Replace the whole per-step delivery path with a Playwright-native Reporter (bypasses Serenity's buffer):

• server/lib/serenity-js/reporters/realtime-step-reporter.ts - listens to Reporter.onStepEnd, filters category === "test.step", and POSTs each step to /api/integration-worker/progress with the existing test-step envelope (eventData.timestamp/details.name/ outcome.code). Registered in playwright.config.ts alongside the Serenity reporter.

Backend handler, session registry and audit notifier tightened so the real-time stream behaves correctly:

• /progress handler now responds 200 OK immediately, then processes the audit asynchronously. Eliminates the 1-3 s per-request latency that was queueing test-step POSTs and bunching them at end-of-run.
• /progress for test-step now unwraps the { eventData, finished } envelope before handing the inner event to parseTestStepEvent, so the parser sees real timestamp / details.name / outcome.code again (the old path was passing the envelope, so every event came out with a NaN timestamp and got clustered into one write-time window).
• sendAudit is now serialised per-session via a promise chain. Each background callback runs strictly after the previous one, so session.record increments monotonically instead of racing (no more runs of rec=8, rec=8, rec=8).
• updateRamblersUploadSession mutates the session object in place (Object.assign) instead of replacing it in the map, so all captured session references stay in sync with the incremented record.

DRY: drop the old WDIO-era event transport entirely

With RealtimeStepReporter as the single per-step path, the old WebSocket-based transport and its wiring are dead weight:

• Deleted:
• server/lib/serenity-js/screenplay/crew/DomainEventPublisher.ts
• server/lib/ramblers/process-test-step-event.ts
• server/lib/ramblers/integration-worker-websocket-server.ts
• server/lib/websockets/websocket-client.ts
• Removed EventType.TEST_STEP_REPORTER from the shared websocket.model.ts.
• Dropped the TEST_STEP_REPORTER handler from the backend websocket-server.ts and the createIntegrationWorkerWebSocketServer call from the worker server.

Environment enum: move to shared model, prune, enforce

• Environment enum moved to projects/ngx-ramblers/src/app/models/environment.model.ts so the Angular app can use it too; deleted the server shim at server/lib/env-config/environment-model.ts; all 28 server importers rewritten to the shared path.
• Added ADMIN_MONGODB_URI and PLAYWRIGHT_HEADLESS to the enum.
• Every literal process.env.X / env.X / secrets.X access in the server now goes through Environment.X - deploy script, CLI commands, worker routes.
• Frontend environment-settings.ts now uses [Environment.X] computed keys instead of string literals.

Local CLI ergonomics

• New --no-headless flag on ngx-cli local dev. Runs the worker headed (visible browser) and implies --no-docker-worker (Docker always headless). Existing --no-docker-worker still works for fast-feedback headless iteration without an image rebuild.
• docker-worker and headless threaded through LocalRunConfig.
• Force-kills stale processes on backend / frontend / worker ports before each start (SIGTERM with 4 s grace, then SIGKILL); tears down any stale worker container.

Walk export UI

• autoTitle added to the <app-page> wrapper so the tab view has an H1 above the tabs, via the same mechanism used by Bookings / Member Bulk Load / Environment Setup.
• Audit sort state persisted in the URL as `?sort=&sort-order=

<asc|desc>`:

• New StoredValue entries AUDIT_MESSAGE, AUDIT_TIME, DURATION, STATUS (kebab-case values in the URL).
• Reuse existing StoredValue.SORT + SORT_ORDER as the query keys and SortDirection.ASC/DESC from sort.model.ts for the values (no duplicate "asc"/"desc" constants).
• sortAuditsBy(field: StoredValue) now flips order on the same column and resets to DESC on a new column, then persists via updateUrl().
• AUDIT_SORT_FIELD_MAPPING maps the URL/StoredValue kebab-case to the internal camelCase field the sortBy helper reads.
• Duration column:
• durationMs is now computed on every audit unconditionally (the showDetail gate was stopping the field from being added when the checkbox was off, which was why the Duration header sort did nothing).
• dateUtils.formatDuration no longer short-circuits to "0 secs" when fromTime === 0 - that falsy-zero guard was wrong for millisecond-delta inputs and was making every row render "0 secs" regardless of the real duration.
• applyFilter handles the oldest row (no previous) cleanly instead of computing auditTime - 0.

Scripts cleanup

server/package.json:

• Dropped test: "failsafe serenity report"
• duplicated serenity.
• Dropped lint / lintfix / lintversion and the tslint dep + server/tslint.json. tslint has been unmaintained since 2019 and none of the hooks or CI workflows invoke them.

server/test-setenv.sh: dropped stale CHROME_BIN / CHROMEDRIVER_PATH exports.

server/.env: local key names renamed to INTEGRATION_WORKER_*; stale Chrome paths removed. (Local values only; remote Fly secrets re-imported under the new names separately.)

Dead code removal

• server/lib/migration/migrate-static-site.ts (standalone NWK script, unreferenced).
• server/lib/migration/serenity-migration-engine.ts, serenity-migration-utils.ts, serenity-utils.ts and the screenplay/ scaffolding (parallel implementation, not imported by any live route).
• server/lib/migration/puppeteer-utils.ts replaced by server/lib/migration/browser-utils.ts (only compiled into the worker image - never called from the website live path).

Fly secrets

Staged but not deployed on the two apps available to the current token:

• ngx-ramblers-upload-worker: new INTEGRATION_WORKER_SHARED_SECRET / INTEGRATION_WORKER_ENCRYPTION_KEY staged with matching digests to the old names.
• ngx-ramblers: new INTEGRATION_WORKER_SHARED_SECRET / INTEGRATION_WORKER_ENCRYPTION_KEY / INTEGRATION_WORKER_URL / INTEGRATION_WORKER_CALLBACK_BASE_URL staged.

Staging central DB config.environments for staging, ekwg and ekwg-dev-2-staging renamed the per-env secret keys from RAMBLERS_UPLOAD_WORKER_* to INTEGRATION_WORKER_* so future deploys push the new names consistently.

Old secrets remain Deployed on both apps until the next deploy activates the new staged names.

worker: switch upload worker to Serenity/JS Playwright stack and cut the website off WebdriverIO/Puppeteer (#114)

Migrate the Ramblers upload worker off the WebdriverIO + Chrome/Chromedriver runtime onto the officially supported Serenity/JS Playwright image, harden the end-to-end upload path, and remove every WebdriverIO/Puppeteer usage from the codebase so the website image no longer ships a browser stack.

Worker image

• Dockerfile worker stage now builds from ghcr.io/serenity-js/playwright:v1.58.1-noble (Node 24 pinned via n, npm 11, OpenJDK, Playwright browsers pre-installed on Ubuntu 24.04). Chrome/Chromedriver/@puppeteer/browsers installs are removed - the Playwright image already ships the browsers the worker needs.
• npm run serenity-bdd-update is pre-run so runtime doesn't re-download the Serenity-BDD JAR.
• .dockerignore excludes server/node_modules, cli-output/, target/, ts-gen/ and dist/ so worker images stay lean.

Website image

• Renamed --target origin to --target website to reflect its role as the pure web tier. Build arg CHROME_VERSION, OpenJDK 21 install and serenity-bdd-update are removed from this stage - the website no longer runs Serenity or any browser.
• CI workflow now builds with --target website and no longer passes --build-arg CHROME_VERSION.
• Dockerfile header comment updated to state the website stage ships neither a browser nor a JDK.

Test runner migration (WebdriverIO → Playwright)

• server/package.json serenity-run now invokes playwright test -c ./playwright.config.ts. Adds @playwright/test, playwright, @serenity-js/playwright and @serenity-js/playwright-test.
• New server/playwright.config.ts wires the Serenity crew, BDD reporter, ArtifactArchiver and Photographer, using SerenityFixtures / SerenityWorkerFixtures. Honours PLAYWRIGHT_HEADLESS, RAMBLERS_FEATURE and BASE_URL.
• Feature specs (contacts, process-command-args, walks-and-events- manager, walks-programme, walks-upload) migrated off mocha / actorCalled statics to the @serenity-js/playwright-test actorCalled fixture; beforeEach engagement is replaced with per-scenario actorCalled(...).
• SaveBrowserSource task replaces the WDIO @wdio/globals call with BrowseTheWeb.currentPage().executeScript(...) and now writes to target/browser-source/ with a fallback page on capture failure.
• New SetFileInput task drives file uploads via Playwright's native setInputFiles, replacing Enter.theValue(...) and fixing the "Input of type file cannot be filled" failure on the Ramblers upload modal.
• Login wait restored to Wait.upTo(DEFAULT_WAIT_TIMEOUT).until(...) sourcing the 15 s budget from a single shared constant in server/lib/serenity-js/config/serenity-timeouts.ts (also consumed by playwright.config.ts). Fixes the 5 s TimeoutExpiredError on the post-login Create-menu dropdown check.

Bulk-action selector hardening

• WalksPageElements tightens the Delete / Unpublish / Publish / Uncancel selectors to #vbo-action-form-wrapper input[data-vbo='vbo-action'][value='...'], eliminating ambiguous matches when the VBO action bar renders twice.
• Adds a loggedInWalksManagerPage element (body CSS class check) used inside AuthErrorCookieBannerOrCreateMenuDropdown so post-login detection succeeds even before the Create menu renders.

Live audit streaming (in progress)

• Server spawns the Serenity subprocess with RAMBLERS_UPLOAD_WORKER_CALLBACK_BASE_URL, _CALLBACK_PROGRESS_PATH, _CALLBACK_RESULT_PATH, _CALLBACK_SECRET and _JOB_ID in the environment (new Environment enum entries).
• DomainEventPublisher posts TEST_STEP events directly to the worker callback endpoint when that env is present, with diagnostic logging on the transport branch actually taken, instead of going through the Serenity subprocess websocket (which intermittently dropped with close code 1006 and caused audit rows to bunch at end-of-run).
• processTestStepEvent now writes step events straight into the active local session so the Walk upload audit tab updates while the browser is still running.
• Frontend WebSocketClientService wraps onopen / onmessage / onclose / onerror in NgZone.run(...) so progress events trigger change detection. walk-export.ts calls ChangeDetectorRef.detectChanges() on updates, broadcasts NamedEventType.REFRESH on completion, and runs a 2 s audit refresh loop while the export is in progress.

Audit parser & timing clean-up

• ramblers-audit-parser strips ANSI escape sequences via module-level RegExps built from String.fromCharCode(27) (no inline control characters, passes no-control-regex without a disable) and filters separator / noise lines such as Execution Summary, Scenarios:, Real time:, Total time:, 1 passed, Succeeded with exit code 0 as all scripts passed, > playwright, @playwright/test, @serenity-js/playwright / -test, Running 1 test using 1 worker, the mkdir -p target/site/archive line and all-= separator lines.
• walk-export.ts timing() no longer has a special SUMMARY case that returned the whole-session duration; summary and step rows now both show true per-step durations derived from durationMs or the adjacent-row delta.
• ramblers-upload-walks.ts progress line reports PLAYWRIGHT_HEADLESS and BASE_URL instead of the removed CHROME_BIN / CHROMEDRIVER_PATH.

Report storage (zipped artefacts)

• ramblers-upload-worker-runner.ts uploads the Serenity report as a single <keyPrefix>.zip via adm-zip, emitting lifecycle progress events for archive start, completion (with file count, byte size and elapsed ms) and failure.
• recordReportLocation stores the summary message as s3://<bucket>/<keyPrefix>.zip.
• New GET /api/aws/report/:bucket/* route in aws-routes.ts streams assets from the archive: aws-controllers.ts lazy-extracts each report under target/report-cache/<sha1>/, guards against directory traversal, dedupes concurrent extractions via an in-memory promise cache, and falls back to a chunked streamToBuffer for SDK response bodies that aren't already Buffers.
• walk-export.ts openReport() uses the new route with bucket + keyPrefix + /_/ + index.html as the split marker; sessions without a stored bucket are no-ops.

WebdriverIO / Puppeteer removal across the codebase

• @serenity-js/webdriverio, webdriverio, puppeteer and every @wdio/* runner/CLI package removed from server/package.json; the lockfile shrunk by ~4 200 lines.
• Deleted server/wdio.conf.ts, server/lib/serenity-js/features/config/ actors.ts, server/lib/migration/puppeteer-utils.ts and the parallel serenity-migration-engine.ts / serenity-migration-utils.ts / serenity-utils.ts stack plus their screenplay/ scaffolding and the standalone migrate-static-site.ts script (none were imported by any live route).
• New server/lib/migration/browser-utils.ts provides a Playwright chromium.launch(...) replacement for the old puppeteer helper.
• migrate-static-site-engine.ts now imports Browser/Page from playwright, swaps networkidle2 for networkidle, and restructures every page.evaluate(fn, arg1, arg2, ...) call for Playwright's single-argument contract (object-payload pattern).
• migration-routes.ts collapses the dual WebdriverIO / Puppeteer branches, drops the USE_SERENITY_FOR_MIGRATION feature flag and its env entry, and uses Playwright-only paths.
• flickr-provider.ts is on Playwright: browser.url/waitUntil/ execute/getPageSource/$ become page.goto/waitForFunction/ evaluate/content/locator, and cookie-consent iframe handling is reimplemented with page.frameLocator(...) rather than switchToFrame.
• Environment enum drops CHROME_BIN and CHROMEDRIVER_PATH; the ramblers-audit-parser noise list replaces the webdriverio marker with @serenity-js/playwright.

Local dev ergonomics

• bin/ngx-cli local dev force-kills stale processes holding the backend / frontend / worker ports (SIGTERM with a 4 s grace, then SIGKILL), and tears down any old worker container before each run.
• Docker worker rebuild trigger now watches server/playwright.config.ts, server/lib/ramblers, server/lib/serenity-js and server/lib/env-config recursively, via a new newestMtimeMs helper, so nested source edits trigger a rebuild.
• When PLAYWRIGHT_HEADLESS=false is set, the in-process worker path is used (headed debugging) instead of the Docker worker, and npm exec playwright install chromium is ensured with output streamed to both stdout and the worker log.
• Backend env injects RAMBLERS_UPLOAD_WORKER_CALLBACK_BASE_URL=http://host.docker.internal:<port> when the Docker worker is used.
• Worker Docker build logs are streamed to the terminal as well as the worker log file.

CI

• build-push-and-deploy-ngx-ramblers-docker-image.yml builds a separate worker image tagged worker-${run_number} whenever worker-relevant files changed, deploys it with --image-tag worker-${run_number}, and now builds the website image with --target website (no Chrome build arg).
• Manual build-and-deploy-ramblers-upload-worker.yml points at the worker build stage.
• server/deploy/detect-worker-changes.ts treats server/playwright.config.ts as worker-relevant and drops server/wdio.conf.ts from its watch list.