11-Apr-2026 — Incremental S3 backup aligned with MongoDB backup timestamps #187

build 586 — commit 37ea501

Consolidate admin UI, per-object diagnostics, shared-bucket dedupe and manifest analysis

What changed for users

• Backup admin tabs reduced from six to four. The standalone S3 Backup and Backups tabs are gone; their capabilities now live where they logically belong: the consolidated Backup/Restore flow covers S3 object sync automatically, bulk deletion of stored Mongo backup files is a collapsible Manage stored backups panel on the Restore tab, and the S3 snapshot analysis and deletion actions are merged directly onto the matching rows in Backup/Restore History
• One row per backup operation in History. Each session row now shows the matching S3 snapshot's actions inline — View Logs, Analyse S3 snapshot, Delete S3 snapshot — so a Mongo backup and its aligned S3 manifest no longer duplicate each other. Sessions without a matching manifest (e.g. restores, failed S3 phases, Mongo-only backups) still render with just the View Logs button
• Analyse S3 snapshot opens an inline panel with three tables: breakdown by top-level prefix, breakdown by file extension, and the twenty largest individual objects. Surfaces what's actually driving a bucket's size so a ballooning backup is easy to diagnose
• Environment and Collections selectors now carry their own label that reads e.g. "Environments (3 of 14 selected)" and updates live as the selection changes, replacing the previous static "Environment(s)" heading and the redundant "n environments selected" helper text
• History tab gains a single environment filter that applies to both Mongo sessions and (when shown) S3 manifest data. The filter uses the same select-all-capable multi-select component as Backup, and an empty filter shows everything with an explicit "No filter applied — showing sessions for all environments" helper line
• Backup and restore operations stream periodic S3 progress into the session log (every few seconds and at completion), so the admin UI History row reveals how many of how many objects have been processed rather than a silent gap until the phase finishes
• Shared S3 buckets across multiple environments no longer produce duplicate manifest chains or risk destructive in-place restores. When two environments point at the same source bucket, the backup runs once under the alphabetically-first ("primary") site with a note in the session log, and the restore applies the snapshot once to the shared physical bucket
• "In progress" session status pill no longer wraps onto two lines when the Actions column is narrow; the spinner and text stay on one line
• Expanded logs on an in-progress session now show a "Waiting for log output..." placeholder until the first log line arrives, instead of a visually empty panel
• Auto-refresh on the History tab now reloads S3 manifests alongside sessions, so a newly completed backup's Analyse and Delete actions appear on its row without the user having to leave and re-enter the tab
• CLI backup s3 analyse <site> command runs the same analysis (prefix, extension, top-N) from the terminal with optional --timestamp, --depth and --top flags, giving operators a quick way to inspect a snapshot without opening the admin UI

Technical — s3-backup-service

• Add a ProgressCallback type and thread it through performIncrementalBackup, performRestore, startS3Backup and startS3Restore so callers can subscribe to aggregate progress messages without touching the stderr debug log
• Enable the service's debug() logger and emit per-object lines: COPY START, COPY DONE, COPY FAIL, COPY DRY-RUN, SKIPPED, ALREADY-OK, COPY FROM snapshot, COPY FROM earlier, MISS, DELETE-EXTRA — each with the [site] [N/total] prefix, file size and eTag deltas where relevant
• Throttle progress callbacks via PROGRESS_EVERY_MS (3 seconds) plus a guaranteed final message, so the session log gets useful but not spammy updates
• Wrap every CopyObjectCommand in copyObjectWithETagGuard, a recursive async helper that passes CopySourceIfMatch: expectedETag and catches 412 Precondition Failed by re-heading the object, updating the eTag and retrying up to 3 times — prevents a silent integrity mismatch when an object is mutated (e.g. resized) between list and copy
• Record the eTag that was physically copied rather than the one seen at listing time, so the manifest entry is always consistent with the stored bytes
• Extract rawSiteConfigs() and dedupeSiteConfigsByBucket(), make siteConfigs() return one entry per unique source bucket with a new sharedWith?: string[] field populated with the aliased environment names, and teach siteConfigFor() to resolve either the primary or any alias to the primary's config
• Teach manifestByTimestamp() to resolve aliases to the primary site before querying so cross-environment restore through a shared bucket finds the right manifest

Technical — consolidated backup/restore flow

• BackupAndRestoreService.executeBackup and executeRestore now pass (message) => this.addLog(sessionId, message) as the ProgressCallback into startS3Backup / startS3Restore, so aggregate S3 progress flows into the same session log that the admin UI streams
• Cross-environment restore now looks up the source environment from the backup file name (via the existing extractSourceEnvironmentFromBackupName helper) and passes it alongside the target, so a restore triggered through the main Restore tab automatically performs a cross-environment S3 restore when the selected backup came from a different source

Technical — manifest analysis

• New pure module projects/ngx-ramblers/src/app/functions/s3-manifest-analysis.ts exports groupEntriesByPrefix(entries, depth), groupEntriesByExtension(entries) and topLargestEntries(entries, count). Shared verbatim by the CLI analyse command and by the admin UI expand panel, so the three analyses live in exactly one place
• S3ManifestBreakdown interface moved from the functions module to backup-session.model.ts in line with the codebase rule that exported interfaces live in model files

Technical — CLI

• New backup s3 analyse <site> command (alias analyze) with --timestamp, --depth and --top flags. Prints manifest summary, prefix and extension breakdowns, and the top N largest objects. Reuses the shared analysis helpers so behaviour matches the admin UI exactly
• Retired the local PrefixBucket helpers in the CLI command in favour of the shared module

Technical — admin UI

• BackupRestoreTab enum reduced to BACKUP, RESTORE, HISTORY, SETTINGS. The S3_BACKUP and BACKUPS entries are gone, as are their entire tab templates and all the state and methods that only served the standalone S3 backup form (s3BackupAllSites, s3BackupDryRun, s3BackupInProgress, s3BackupResults, s3SelectedSite, s3RestoreResults, startS3Backup(), filteredS3Manifests())
• Restore tab gains a Manage stored backups collapsible button that reveals the Mongo backup bulk selector and delete UI previously living in the Backups tab
• History tab's session row template now includes manifestForSession(session) lookup and conditionally renders Analyse and Delete buttons plus the inline expand panel when a matching manifest exists. Matching is by timestamp parsed from session.sessionId (format backup-{YYYY-MM-DD-HH-mm-ss}-{env}-{db}) against manifest.mongoTimestamp and manifest.site === session.environment
• handleTabChange now loads S3 manifests when entering the History tab (was previously the S3 Backup tab) and the auto-refresh subscription also reloads manifests on every tick so newly completed backups surface their actions without the user leaving and returning to the tab
• EnvironmentSelectComponent gains a label?: string input and an internal labelText() helper that renders the label with a live "(X of Y selected)" suffix in multiple mode. The four existing call sites in the admin component drop their external <label> elements
• CollectionsMultiSelectComponent gains the same label?: string + labelText() treatment, and its two call sites on the Backup and Restore tabs drop their external labels and redundant "n collections selected" helper text
• History tab gains a historyFilterEnvironments: EnvironmentInfo[] + filteredSessions() strict-filter that falls back to all sessions when the filter is empty, with a live helper line explaining the state. The same filter is now reused for the S3 manifest match (the standalone filteredS3Manifests() method was removed when the separate S3 history table was folded into the session rows)
• Session row action column switches to compact icon-only buttons (faFileLines for logs, faChartBar for analyse, faTrash for delete) wrapped in a d-inline-flex gap-1 align-items-center container with text-nowrap so nothing wraps. Each button carries an ngx-bootstrap tooltip describing the action
• .session-status gets white-space: nowrap and display: inline-block so the "In progress" pill never line-wraps inside a narrow Actions column
• Expanded session logs now render a <fa-icon [spin]> + "Waiting for log output..." placeholder when the expanded session is IN_PROGRESS with an empty logs array, instead of a visually empty panel
• Analysis panel's heading block sits in a .manifest-analysis-header with min-height: 5.5rem so both side-by-side breakdown tables start at the same vertical position regardless of description wrapping
• Descriptions under each breakdown heading shortened to one-line form (Bytes per top-level folder., Bytes per file type., Largest individual files in this snapshot.) to further reduce wrap risk
• The Manage stored backups toggle uses btn-sm btn-warning rather than the default Bootstrap link style to stay consistent with the Ramblers palette on the Restore tab

Technical — tests

• New unit tests in s3-backup-service.spec.ts cover collectCopiedKeys (copy-only filtering, empty-backup case) and extend the existing siteConfigs suite to exercise the new dedupe logic
• Pre-existing backup-paths.spec.ts gains tests for extractSourceEnvironmentFromBackupName including multi-segment environment names, path prefix stripping, and the "timestamp missing" fallback
• Full server suite remains green