{"id":"69ecef9a60bb8b34fce600a5","title":"Loading Member Data","path":"how-to/technical-articles/2026-04-21-using-ramblers-salesforce-mock/loading-member-data","contentMarkdown":"# 21-Apr-2026 — Loading member data\n\n_____\n\nTwo ways to populate a tenant: upload a real Insight Hub `ExportAll.xlsx`, or generate synthetic rows from a configurable template. Either replaces the tenant's current dataset — existing members not present in the new batch are soft-removed (their `removed` flag is set so consumers polling with `?since=` can see the removal).\n\n[← Back to the overview](https://ngx-ramblers.org.uk/how-to/technical-articles/2026-04-21-using-ramblers-salesforce-mock)\n\n## Members tab layout\n\nOpen a tenant and stay on the **Members** tab. The top panel shows the first 20 members currently in the dataset. Below it are two sub-tabs:\n\n- **Upload Insight Hub xlsx** — push a real `ExportAll.xlsx` in\n- **Insight Hub export** — pull the current dataset out as a 36-column xlsx\n\nSynthetic generation lives on its own **Generate** tab, covered further down.\n\n![](https://ngx-ramblers.org.uk/api/aws/s3/site-content/7fec8394-2f61-4157-925b-4aae3160e577.png)\n\n*The Members tab shows the first 20 members. Below it, two sub-tabs — Upload Insight Hub xlsx (push data in) and Insight Hub export (pull data out).*\n\n## Upload Insight Hub xlsx\n\nGet an `ExportAll.xlsx` from Insight Hub's bulk download. The mock expects the standard 36-column schema documented in the [#209 field table](https://github.com/nbarrett/ngx-ramblers/issues/209).\n\n- Click **Choose File**, pick your `.xlsx`\n- Hit **Upload**\n\nWhile the file streams, a busy indicator runs and every action button is disabled. When the upload finishes, the page refreshes the Members preview with the first 20 rows of the newly loaded dataset and the tenant header updates to show the new last-import time and row count. There is no JSON summary surfaced in the UI; if you want the parse-and-ingest detail you can hit `POST /admin/api/tenants/:code/upload` directly with curl and inspect the response.\n\nThe detail response (curl path) returns four fields worth knowing about:\n\n| Field | Meaning |\n|---|---|\n| `unmatchedHeaders` | Columns in your file that didn't match any of the 36 expected names. Typos and extras land here. Rename and re-upload if anything important is listed. |\n| `missingHeaders` | Expected columns absent from your file. Not fatal — they come through as undefined on member records. |\n| `warnings` | Rows skipped because they had no `Mem No.` or no `Last Name`. |\n| `softRemoved` | Members that existed before the upload but are absent from this one — flagged with `removed: true`, not hard-deleted, so incremental-sync consumers can see the removal. |\n\nHeader text is matched case-insensitively with whitespace collapsed; the spelling has to be right. The parser reads the first worksheet of the workbook (the writer side, used by export and download, names the worksheet `Full List` to match what consumers like ngx-ramblers' `member-bulk-load.ts` look up by name).\n\n## Insight Hub export — download the current dataset\n\nThe second sub-tab on the Members tab. Click **Download Insight Hub xlsx** and the mock streams the tenant's current dataset back as `InsightHub--.xlsx`, in exactly the 36-column shape Insight Hub itself produces.\n\nUseful for:\n\n- Round-tripping data through a third-party tool that reads `ExportAll.xlsx`\n- Handing a tenant's snapshot to a consumer who hasn't wired up the public API yet\n- Comparing two operators' datasets side by side without API access\n\nNote: granular consent flags (`groupMarketingConsent`, `areaMarketingConsent`, `otherMarketingConsent`) and role assignments are stored on the member record but **are not part of the 36-column Insight Hub schema**, so they don't appear in the downloaded xlsx. They are preserved on the database side and continue to be served by the public API.\n\n## Generate synthetic data\n\nIf you don't have a real spreadsheet — or you want a controlled, reproducible scenario — open the **Generate** tab. The form has six sections.\n\n![](https://ngx-ramblers.org.uk/api/aws/s3/site-content/ccb62af1-5536-4982-ac48-91e2d1dfc7b2.png)\n\n*The full Generate form — size, geography, email pattern, consent distribution (independent or joint), role proportions, and the two submit modes (import directly into the database, or download as Insight Hub xlsx).*\n\n### Size and reproducibility\n\n- **Row count** — 1 to 10,000. Start with a few hundred for quick iteration; 10,000 is enough to stress-test paging.\n- **Seed** (optional) — same seed plus same options gives the same output every time. Leave blank for fresh randomness on each run.\n\n### Geography\n\nPool of towns and postcode prefixes that addresses are drawn from.\n\n| Region | Towns | Postcode areas |\n|---|---|---|\n| Mixed UK (default) | every region's pool combined | every region's pool combined |\n| Kent | Canterbury, Faversham, Maidstone, Tunbridge Wells, … | CT, ME, TN, BR, DA |\n| Staffordshire | Stoke-on-Trent, Newcastle-under-Lyme, Stafford, Lichfield, … | ST, DE, WS |\n| Newcastle upon Tyne and the North East | Newcastle upon Tyne, Gateshead, Sunderland, Durham, … | NE, SR, DH |\n| Hampshire | Winchester, Southampton, Portsmouth, Basingstoke, … | SO, PO, RG, GU, SP |\n\nPicking a single region is useful when you want a tenant whose addresses look believable for a specific group's catchment.\n\n### Email pattern\n\nFour presets, plus custom. Available placeholders: `{firstname}`, `{surname}`, `{nn}` (a small integer disambiguator), `{membershipNumber}`, `{groupCode}`, `{domain}`, `{base}`.\n\n| Preset | Template | Use when |\n|---|---|---|\n| Distinct | `{firstname}.{surname}{nn}@{domain}` | You want addresses that look like real members. The domain needs a catch-all to actually receive mail. |\n| Plus-addressing | `{base}+m{membershipNumber}@{domain}` | You want every test member's mail to land in one shared inbox (e.g. `test+m3300001@…`). |\n| Minimal | `member{membershipNumber}@{domain}` | You want fully synthetic, obviously-fake addresses (no real names involved). |\n| Custom | your own template | You need a different shape — e.g. `{firstname}-{surname}@{domain}`. |\n\nThe default domain is `ngx-ramblers.org.uk`. It's chosen because it's an unreachable address from real Brevo lists — which means accidentally pushing synthetic data into a real outbound campaign won't deliver to anyone.\n\n### Consent distribution\n\nTwo modes for the six consent flags:\n\n**Independent** — each flag is rolled separately against its own probability. Two tables of sliders, grouped by where each flag rides:\n\n- **xlsx + API** — the standard Insight Hub flags: `emailMarketingConsent`, `postDirectMarketing`, `telephoneDirectMarketing`. These round-trip through both the public API and the Insight Hub xlsx download.\n- **API only** — granular flags: `groupMarketingConsent`, `areaMarketingConsent`, `otherMarketingConsent`. Stored on the member record and returned by the public API; not in the 36-column Insight Hub schema, so absent from the downloaded xlsx.\n\n![](https://ngx-ramblers.org.uk/api/aws/s3/site-content/877fd26d-7576-4e5f-b8c8-57db03d85edb.png)\n\n*Switching to Joint mode reveals a probability table over flag combinations. Add rows, set weights, and the running sum tells you when you've reached 100.*\n\n**Joint** — a probability table over flag combinations, weights summing to 100. Useful for correlated scenarios that independent rolls can't express (\"give me a population where members who consent to group email always also consent to area email\"). Add rows with **+ Add row**; the running sum shows live so you can see when you've reached 100.\n\n### Role proportions (API only)\n\nThree flags set on each member's primary `groupMemberships[].roles`: `walkLeader`, `emailSender`, `viewMembershipData`. Defaults are 8% / 5% / 3%. Like the granular consent flags, roles ride only through the public API — they are not part of the Insight Hub schema, so they don't appear in the downloaded xlsx.\n\n### Two ways to take the data away\n\nThe form has two submit buttons:\n\n- **Generate & import** — generate the rows *and* save them into this server's database. Replaces the tenant's current dataset; the public API will now return them. Granular consent and roles survive into the DB because there's no xlsx round-trip on this path.\n- **Generate & download xlsx** — generate the rows *and* stream them as a 36-column Insight Hub xlsx (`ExportAll--.xlsx`) without touching the database. Useful when a downstream consumer reads `ExportAll.xlsx` directly and you just want a synthetic file to feed them.\n\n### Surnames are double-barrelled on collision\n\nA subtlety worth knowing: synthetic data is generated against a fixed pool of first names and surnames. If your row count is high enough that the same `(firstName, lastName)` pair would otherwise repeat, the generator double-barrels the surname (`Smith` → `Smith-Adams`) until uniqueness is restored. This matters because consumers like ngx-ramblers enforce a unique `(lastName, firstName, nameAlias)` index on import; without the double-barrelling, large synthetic batches would be rejected mid-load.\n\n## Next\n\n[**Calling the API →**](https://ngx-ramblers.org.uk/how-to/technical-articles/2026-04-21-using-ramblers-salesforce-mock/calling-the-api)","contentHtml":"

21-Apr-2026 — Loading member data

\n
\n

Two ways to populate a tenant: upload a real Insight Hub ExportAll.xlsx, or generate synthetic rows from a configurable template. Either replaces the tenant's current dataset — existing members not present in the new batch are soft-removed (their removed flag is set so consumers polling with ?since= can see the removal).

\n

← Back to the overview

\n

Members tab layout

\n

Open a tenant and stay on the Members tab. The top panel shows the first 20 members currently in the dataset. Below it are two sub-tabs:

\n
    \n
  • Upload Insight Hub xlsx — push a real ExportAll.xlsx in
    • \n
  • Insight Hub export — pull the current dataset out as a 36-column xlsx
    • \n
\n

Synthetic generation lives on its own Generate tab, covered further down.

\n

\"\"

\n

The Members tab shows the first 20 members. Below it, two sub-tabs — Upload Insight Hub xlsx (push data in) and Insight Hub export (pull data out).

\n

Upload Insight Hub xlsx

\n

Get an ExportAll.xlsx from Insight Hub's bulk download. The mock expects the standard 36-column schema documented in the #209 field table.

\n
    \n
  • Click Choose File, pick your .xlsx
    • \n
  • Hit Upload
    • \n
\n

While the file streams, a busy indicator runs and every action button is disabled. When the upload finishes, the page refreshes the Members preview with the first 20 rows of the newly loaded dataset and the tenant header updates to show the new last-import time and row count. There is no JSON summary surfaced in the UI; if you want the parse-and-ingest detail you can hit POST /admin/api/tenants/:code/upload directly with curl and inspect the response.

\n

The detail response (curl path) returns four fields worth knowing about:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
FieldMeaning
unmatchedHeadersColumns in your file that didn't match any of the 36 expected names. Typos and extras land here. Rename and re-upload if anything important is listed.
missingHeadersExpected columns absent from your file. Not fatal — they come through as undefined on member records.
warningsRows skipped because they had no Mem No. or no Last Name.
softRemovedMembers that existed before the upload but are absent from this one — flagged with removed: true, not hard-deleted, so incremental-sync consumers can see the removal.
\n

Header text is matched case-insensitively with whitespace collapsed; the spelling has to be right. The parser reads the first worksheet of the workbook (the writer side, used by export and download, names the worksheet Full List to match what consumers like ngx-ramblers' member-bulk-load.ts look up by name).

\n

Insight Hub export — download the current dataset

\n

The second sub-tab on the Members tab. Click Download Insight Hub xlsx and the mock streams the tenant's current dataset back as InsightHub-<CODE>-<n>.xlsx, in exactly the 36-column shape Insight Hub itself produces.

\n

Useful for:

\n
    \n
  • Round-tripping data through a third-party tool that reads ExportAll.xlsx
    • \n
  • Handing a tenant's snapshot to a consumer who hasn't wired up the public API yet
    • \n
  • Comparing two operators' datasets side by side without API access
    • \n
\n

Note: granular consent flags (groupMarketingConsent, areaMarketingConsent, otherMarketingConsent) and role assignments are stored on the member record but are not part of the 36-column Insight Hub schema, so they don't appear in the downloaded xlsx. They are preserved on the database side and continue to be served by the public API.

\n

Generate synthetic data

\n

If you don't have a real spreadsheet — or you want a controlled, reproducible scenario — open the Generate tab. The form has six sections.

\n

\"\"

\n

The full Generate form — size, geography, email pattern, consent distribution (independent or joint), role proportions, and the two submit modes (import directly into the database, or download as Insight Hub xlsx).

\n

Size and reproducibility

\n
    \n
  • Row count — 1 to 10,000. Start with a few hundred for quick iteration; 10,000 is enough to stress-test paging.
    • \n
  • Seed (optional) — same seed plus same options gives the same output every time. Leave blank for fresh randomness on each run.
    • \n
\n

Geography

\n

Pool of towns and postcode prefixes that addresses are drawn from.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
RegionTownsPostcode areas
Mixed UK (default)every region's pool combinedevery region's pool combined
KentCanterbury, Faversham, Maidstone, Tunbridge Wells, …CT, ME, TN, BR, DA
StaffordshireStoke-on-Trent, Newcastle-under-Lyme, Stafford, Lichfield, …ST, DE, WS
Newcastle upon Tyne and the North EastNewcastle upon Tyne, Gateshead, Sunderland, Durham, …NE, SR, DH
HampshireWinchester, Southampton, Portsmouth, Basingstoke, …SO, PO, RG, GU, SP
\n

Picking a single region is useful when you want a tenant whose addresses look believable for a specific group's catchment.

\n

Email pattern

\n

Four presets, plus custom. Available placeholders: {firstname}, {surname}, {nn} (a small integer disambiguator), {membershipNumber}, {groupCode}, {domain}, {base}.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
PresetTemplateUse when
Distinct{firstname}.{surname}{nn}@{domain}You want addresses that look like real members. The domain needs a catch-all to actually receive mail.
Plus-addressing{base}+m{membershipNumber}@{domain}You want every test member's mail to land in one shared inbox (e.g. test+m3300001@…).
Minimalmember{membershipNumber}@{domain}You want fully synthetic, obviously-fake addresses (no real names involved).
Customyour own templateYou need a different shape — e.g. {firstname}-{surname}@{domain}.
\n

The default domain is ngx-ramblers.org.uk. It's chosen because it's an unreachable address from real Brevo lists — which means accidentally pushing synthetic data into a real outbound campaign won't deliver to anyone.

\n

Consent distribution

\n

Two modes for the six consent flags:

\n

Independent — each flag is rolled separately against its own probability. Two tables of sliders, grouped by where each flag rides:

\n
    \n
  • xlsx + API — the standard Insight Hub flags: emailMarketingConsent, postDirectMarketing, telephoneDirectMarketing. These round-trip through both the public API and the Insight Hub xlsx download.
    • \n
  • API only — granular flags: groupMarketingConsent, areaMarketingConsent, otherMarketingConsent. Stored on the member record and returned by the public API; not in the 36-column Insight Hub schema, so absent from the downloaded xlsx.
    • \n
\n

\"\"

\n

Switching to Joint mode reveals a probability table over flag combinations. Add rows, set weights, and the running sum tells you when you've reached 100.

\n

Joint — a probability table over flag combinations, weights summing to 100. Useful for correlated scenarios that independent rolls can't express ("give me a population where members who consent to group email always also consent to area email"). Add rows with + Add row; the running sum shows live so you can see when you've reached 100.

\n

Role proportions (API only)

\n

Three flags set on each member's primary groupMemberships[].roles: walkLeader, emailSender, viewMembershipData. Defaults are 8% / 5% / 3%. Like the granular consent flags, roles ride only through the public API — they are not part of the Insight Hub schema, so they don't appear in the downloaded xlsx.

\n

Two ways to take the data away

\n

The form has two submit buttons:

\n
    \n
  • Generate & import — generate the rows and save them into this server's database. Replaces the tenant's current dataset; the public API will now return them. Granular consent and roles survive into the DB because there's no xlsx round-trip on this path.
    • \n
  • Generate & download xlsx — generate the rows and stream them as a 36-column Insight Hub xlsx (ExportAll-<tenant>-<count>.xlsx) without touching the database. Useful when a downstream consumer reads ExportAll.xlsx directly and you just want a synthetic file to feed them.
    • \n
\n

Surnames are double-barrelled on collision

\n

A subtlety worth knowing: synthetic data is generated against a fixed pool of first names and surnames. If your row count is high enough that the same (firstName, lastName) pair would otherwise repeat, the generator double-barrels the surname (Smith → Smith-Adams) until uniqueness is restored. This matters because consumers like ngx-ramblers enforce a unique (lastName, firstName, nameAlias) index on import; without the double-barrelling, large synthetic batches would be rejected mid-load.

\n

Next

\n

Calling the API →

\n"}