{"id":"692b2f5622f526dc2cf758d7","title":"2025 11 29 Codex","path":"how-to/technical-articles/2025-11-29-codex","contentMarkdown":"# Protecting project against the Shai-Hulud Exploit\n\n### Question Originally Asked in local codex session\n\n_I'm the owner of ngx-ramblers and I made [this commit the other day](https://github.com/nbarrett/ngx-ramblers/commit/73a827441618e274bca6bd8be0bde8e8a84a285f) to hopefully protect my codebase against the shai-hulud exploit on 25 Nov. However I am now looking at the [Post-mortem of Shai-Hulud attack on November 24th, 2025](https://posthog.com/blog/nov-24-shai-hulud-attack-post-mortem), and are seeing that there are a few ways that your CI pipeline can be vulnerable to malicious operators opening PRs on your project and running workflows that can harvest secrets on your main pipeline. Can you review the post mortem and then review my project (note there is a client and server portion) and tell me whether there's anything I should do to tighten it up?_\n\n## Security Review Summary (NGX‑Ramblers)\n\n ### 1. Third‑Party `server` Package in Frontend Build (High)\n Your frontend `package.json` brings in the public npm package `server@1.0.41` (`package.json:84`, `package-lock.json:34690`). This is unrelated to your checked‑in Express backend under `/server`, so every `npm ci` executes that third‑party package (and its install scripts), recreating the same supply‑chain attack surface used by Shai‑Hulud.\n\n ```json\n {\n ...\n \"dependencies\": {\n ...\n \"rxjs\": \"7.8.2\",\n \"server\": \"1.0.41\",\n \"turndown\": \"7.2.0\"\n }\n }\n```\n Action: Remove it (or convert to \"server\": \"file:server\" if you meant to re-use the local code) so CI never runs unvetted install hooks.\n\n### 2. Deploy Workflow Runs on Pull Requests with Secrets (High)\n\n .github/workflows/build-push-and-deploy-ngx-ramblers-docker-image.yml triggers on both push and pull_request (lines 3‑7). The deploy job later loads CONFIGS_JSON, Docker, and Fly secrets (lines 65‑109). An attacker with any contributor access could open a PR and immediately exfiltrate secrets—exactly the PostHog scenario.\n```\n on:\n push:\n branches: [ \"main\", \"pre-main\" ]\n pull_request:\n branches: [ \"main\" ]\n ...\n deploy:\n needs: test\n runs-on: ubuntu-latest\n steps:\n ...\n - name: Create configs.json from secrets\n env:\n CONFIGS_JSON: ${{ secrets.CONFIGS_JSON }}\n run: |\n echo \"$CONFIGS_JSON\" > non-vcs/fly-io/configs.json\n```\n Action: Restrict deployments to push events (or wrap the deploy job with if: github.event_name == 'push') so PRs never receive secrets.\n\n### 3. Default GITHUB_TOKEN Permissions (Medium)\n\n Neither workflow sets permissions, so GitHub grants the default token repo write access. If any dependency becomes compromised (e.g., via npm ci), the attacker already holds a write-capable token.\n\n Action: Add explicit permissions blocks per job:\n\n permissions:\n contents: read\n packages: write # only where needed\n\n### 4. Floating Action References and Global Installs (Medium)\n\nBoth workflows use floating references (superfly/flyctl-actions/setup-flyctl@master) and npm install -g ts-node, which pull arbitrary code at runtime.\n```\n - name: Install ts-node\n run: npm install -g ts-node\n\n - name: Install Flyctl\n uses: superfly/flyctl-actions/setup-flyctl@master\n```\n Action: Pin actions by commit SHA and leverage the repo’s checked-in ts-node via npx ts-node … to keep builds within locked dependencies.\n\n### 5. npm ci Executes Third-Party Scripts in PR Jobs (Medium)\n\n Both workflows run npm ci with default script execution (build-push-and-deploy...:23-44, deploy-to-environments.yml:69-74). Shai-Hulud exploited preinstall hooks.\n\n Action: Use npm ci --ignore-scripts for PR/test jobs and explicitly execute the scripts you control afterward (e.g., npx ng test, npm run build).\n\n### 6. Shai-Hulud Scanner Misses Published IoCs (Medium)\n\n`server/deploy/check-shai-hulud.ts` only checks for three persistence artifacts:\n\n`const suspicious: string[] = [\"cloud.json\", \"truffleSecrets.json\", \".github/workflows/discussion.yaml\"];`\n\n The PostHog post-mortem listed additional files (setup_bun.js, bun_environment.js, contents.json, environment.json).\n\n Action: Expand the list:\n```\n const suspicious = [\n \"cloud.json\",\n \"truffleSecrets.json\",\n \".github/workflows/discussion.yaml\",\n \"setup_bun.js\",\n \"bun_environment.js\",\n \"contents.json\",\n \"environment.json\"\n ];\n```\n### 7. Scanner Not Wired Into CI (Low)\n\nThe CLI is defined in server/package.json:\n```\n \"check:shai-hulud\": \"tsx deploy/check-shai-hulud.ts\"\n```\n …but no workflow runs it. A compromised lockfile could therefore merge unnoticed.\n\n Action: Add a lightweight GitHub Actions job:\n```\n - name: Shai-Hulud scan\n run: npm run --prefix server check:shai-hulud -- --skip-audit\n\n Trigger it whenever lockfiles or dependency manifests change.\n```\n\n### 8. Environment Secret Fan-Out (Low)\n\n Workflows hydrate the entire CONFIGS_JSON (all environments) before deploying a single environment. A compromised job gains every environment’s secret set.\n\nAction: Split secrets per environment (via GitHub Environments or separate secrets) so jobs only materialize what they deploy.\n\n### Command Output (Reference)\n```\n$ npm run --prefix server check:shai-hulud -- --path .. --skip-audit\n node:net:1915\n const error = new UVExceptionWithHostPort(rval, 'listen', address, port);\n ...\n Error: listen EPERM: operation not permitted /var/folders/.../tsx-502/40273.pipe\n```\n (The CLI fails locally due to sandbox restrictions but should run normally on GitHub-hosted runners.)\n\n### Next Steps\n\n 1. Remove or relink the npm server dependency, re-lock, and re-run the scanner.\n 2. Update workflows: restrict deploy job to pushes, set explicit permissions, pin all third-party actions/binaries, and run npm ci --ignore-scripts for PR jobs.\n 3. Extend the Shai-Hulud CLI to cover all IoCs and wire it into CI so compromised lockfiles or persistence files block merges.\n\n\n\n• Key Findings\n\n - High – The frontend package.json depends on the published npm package named server (package.json:84, package-lock.json:34690). This is unrelated to your checked-in Express code under /server, so every npm ci pulls a third-party framework with full install scripts. It dramatically enlarges the attack surface for exactly the kind of supply-chain poisoning described in the post-mortem. Unless you truly need\n that npm module, drop it (or swap it for a file:server link).\n - High – The deploy job inside .github/workflows/build-push-and-deploy-ngx-ramblers-docker-image.yml runs for pull_request events as well as push (lines 3‑107). Although forked PRs don’t automatically receive repository secrets, insiders/compromised accounts could open a PR and immediately execute the deployment script with all secrets available, exactly mirroring the Shai‑Hulud entry point. Gate the\n entire deploy job behind if: github.event_name == 'push' (or split it into a push-only workflow) so untrusted PRs can never reach CONFIGS_JSON, Fly tokens, or Docker credentials.\n - Medium – Neither workflow declares a restrictive permissions block, so GITHUB_TOKEN defaults to full repo write on push-triggered runs (.github/workflows/build-push-and-deploy-ngx-ramblers-docker-image.yml:1-11, .github/workflows/deploy-to-environments.yml:1-20). If any step is compromised (for example via a malicious dependency install), the token already has push/delete rights. Explicitly scope\n permissions per job (e.g., contents: read for tests, packages: write only where you push images) to limit blast radius.\n - Medium – superfly/flyctl-actions/setup-flyctl@master is referenced by a floating branch (build-push-and-deploy…:100-107, deploy-to-environments.yml:95-107), and the deploy workflow installs ts-node globally without a version (build-push-and-deploy…:74, deploy-to-environments.yml:85). Both steps bypass your lockfiles and trust whatever code is published at run time, giving attackers a straightforward\n place to plant malicious updates. Pin GitHub Actions to immutable commit SHAs and invoke the checked-in ts-node binary via npx ts-node … instead of npm install -g.\n - Medium – The CI install steps (build-push-and-deploy…:23-44, deploy-to-environments.yml:69-74) run npm ci with default script execution, so every dependency’s preinstall/install hooks run on your runners. Shai-Hulud used exactly that hook to exfiltrate secrets. Consider npm ci --ignore-scripts for the PR/test job and then run the handful of repository-controlled scripts you actually need (e.g., npm run\n tsc, npm run copy-assets) to keep third-party hooks from firing.\n - Medium – The Shai-Hulud scanner you added only looks for cloud.json, truffleSecrets.json, and .github/workflows/discussion.yaml (server/deploy/check-shai-hulud.ts:259-265), but the PostHog IoCs also include setup_bun.js, bun_environment.js, contents.json, and environment.json. Extend the list so the CLI (and any automation built on top) detects every known persistence file mentioned in the post-mortem.\n - Low – The scanner exists as npm run --prefix server check:shai-hulud (server/package.json:26), yet it’s not part of any workflow. Without CI coverage, it relies on manual vigilance and can’t block tainted lockfiles in PRs. Add a lightweight job that runs the checker (with --skip-audit if necessary) whenever package-lock.json or server deps change and fails the build on findings.\n - Low – Secrets for every environment are injected straight into the job workspace via CONFIGS_JSON before the Fly deploy runs (build-push-and-deploy…:65-109, deploy-to-environments.yml:29-110). If an attacker compromises the job, they immediately gain access to all environment credentials, not just the target. Move per-environment tokens into GitHub “environments” with required approvals, or split the\n secret blob so each job only ever materializes the environment it’s deploying.\n\n Open Questions\n\n - The attempt to run npm run --prefix server check:shai-hulud -- --path .. --skip-audit failed locally because tsx tried to open an IPC pipe (EPERM). Please confirm the CLI runs successfully inside your GitHub Actions runner, since that’s where it really matters.\n - Was the npm dependency on \"server\": \"1.0.41\" intentional, or is it a leftover placeholder? Clarifying that will determine whether the safest fix is removal or converting it to a local file reference.\n\n Next Steps\n\n 1. Remove/replace the stray server npm dependency and re-lock.\n 2. Restrict the deploy workflow to push events, add explicit permissions, and pin third-party actions/binaries.\n 3. Update CI install steps to minimize third-party install hooks, and integrate the Shai-Hulud scanner with an expanded IoC list so lockfile compromises are caught automatically.\n\n","contentHtml":"

Protecting project against the Shai-Hulud Exploit

\n

Question Originally Asked in local codex session

\n

I'm the owner of ngx-ramblers and I made this commit the other day to hopefully protect my codebase against the shai-hulud exploit on 25 Nov. However I am now looking at the Post-mortem of Shai-Hulud attack on November 24th, 2025, and are seeing that there are a few ways that your CI pipeline can be vulnerable to malicious operators opening PRs on your project and running workflows that can harvest secrets on your main pipeline. Can you review the post mortem and then review my project (note there is a client and server portion) and tell me whether there's anything I should do to tighten it up?

\n

Security Review Summary (NGX‑Ramblers)

\n

1. Third‑Party server Package in Frontend Build (High)

\n

Your frontend package.json brings in the public npm package server@1.0.41 (package.json:84, package-lock.json:34690). This is unrelated to your checked‑in Express backend under /server, so every npm ci executes that third‑party package (and its install scripts), recreating the same supply‑chain attack surface used by Shai‑Hulud.

\n
{\n  ...\n  "dependencies": {\n    ...\n    "rxjs": "7.8.2",\n    "server": "1.0.41",\n    "turndown": "7.2.0"\n  }\n}\n
\n

Action: Remove it (or convert to "server": "file:server" if you meant to re-use the local code) so CI never runs unvetted install hooks.

\n

2. Deploy Workflow Runs on Pull Requests with Secrets (High)

\n

.github/workflows/build-push-and-deploy-ngx-ramblers-docker-image.yml triggers on both push and pull_request (lines 3‑7). The deploy job later loads CONFIGS_JSON, Docker, and Fly secrets (lines 65‑109). An attacker with any contributor access could open a PR and immediately exfiltrate secrets—exactly the PostHog scenario.

\n
  on:\n    push:\n      branches: [ "main", "pre-main" ]\n    pull_request:\n      branches: [ "main" ]\n  ...\n    deploy:\n      needs: test\n      runs-on: ubuntu-latest\n      steps:\n        ...\n        - name: Create configs.json from secrets\n          env:\n            CONFIGS_JSON: ${{ secrets.CONFIGS_JSON }}\n          run: |\n            echo "$CONFIGS_JSON" > non-vcs/fly-io/configs.json\n
\n

Action: Restrict deployments to push events (or wrap the deploy job with if: github.event_name == 'push') so PRs never receive secrets.

\n

3. Default GITHUB_TOKEN Permissions (Medium)

\n

Neither workflow sets permissions, so GitHub grants the default token repo write access. If any dependency becomes compromised (e.g., via npm ci), the attacker already holds a write-capable token.

\n

Action: Add explicit permissions blocks per job:

\n

permissions:\n contents: read\n packages: write # only where needed

\n

4. Floating Action References and Global Installs (Medium)

\n

Both workflows use floating references (superfly/flyctl-actions/setup-flyctl@master) and npm install -g ts-node, which pull arbitrary code at runtime.

\n
  - name: Install ts-node\n    run: npm install -g ts-node\n\n  - name: Install Flyctl\n    uses: superfly/flyctl-actions/setup-flyctl@master\n
\n

Action: Pin actions by commit SHA and leverage the repo’s checked-in ts-node via npx ts-node … to keep builds within locked dependencies.

\n

5. npm ci Executes Third-Party Scripts in PR Jobs (Medium)

\n

Both workflows run npm ci with default script execution (build-push-and-deploy...:23-44, deploy-to-environments.yml:69-74). Shai-Hulud exploited preinstall hooks.

\n

Action: Use npm ci --ignore-scripts for PR/test jobs and explicitly execute the scripts you control afterward (e.g., npx ng test, npm run build).

\n

6. Shai-Hulud Scanner Misses Published IoCs (Medium)

\n

server/deploy/check-shai-hulud.ts only checks for three persistence artifacts:

\n

const suspicious: string[] = ["cloud.json", "truffleSecrets.json", ".github/workflows/discussion.yaml"];

\n

The PostHog post-mortem listed additional files (setup_bun.js, bun_environment.js, contents.json, environment.json).

\n

Action: Expand the list:

\n
  const suspicious = [\n    "cloud.json",\n    "truffleSecrets.json",\n    ".github/workflows/discussion.yaml",\n    "setup_bun.js",\n    "bun_environment.js",\n    "contents.json",\n    "environment.json"\n  ];\n
\n

7. Scanner Not Wired Into CI (Low)

\n

The CLI is defined in server/package.json:

\n
  "check:shai-hulud": "tsx deploy/check-shai-hulud.ts"\n
\n

…but no workflow runs it. A compromised lockfile could therefore merge unnoticed.

\n

Action: Add a lightweight GitHub Actions job:

\n
  - name: Shai-Hulud scan\n    run: npm run --prefix server check:shai-hulud -- --skip-audit\n\n  Trigger it whenever lockfiles or dependency manifests change.\n
\n

8. Environment Secret Fan-Out (Low)

\n

Workflows hydrate the entire CONFIGS_JSON (all environments) before deploying a single environment. A compromised job gains every environment’s secret set.

\n

Action: Split secrets per environment (via GitHub Environments or separate secrets) so jobs only materialize what they deploy.

\n

Command Output (Reference)

\n
$ npm run --prefix server check:shai-hulud -- --path .. --skip-audit\n  node:net:1915\n        const error = new UVExceptionWithHostPort(rval, 'listen', address, port);\n  ...\n  Error: listen EPERM: operation not permitted /var/folders/.../tsx-502/40273.pipe\n
\n

(The CLI fails locally due to sandbox restrictions but should run normally on GitHub-hosted runners.)

\n

Next Steps

\n
    \n
  1. Remove or relink the npm server dependency, re-lock, and re-run the scanner.
    2. \n
  2. Update workflows: restrict deploy job to pushes, set explicit permissions, pin all third-party actions/binaries, and run npm ci --ignore-scripts for PR jobs.
    3. \n
  3. Extend the Shai-Hulud CLI to cover all IoCs and wire it into CI so compromised lockfiles or persistence files block merges.
    4. \n
\n

• Key Findings

\n\n

Open Questions

\n\n

Next Steps

\n
    \n
  1. Remove/replace the stray server npm dependency and re-lock.
    2. \n
  2. Restrict the deploy workflow to push events, add explicit permissions, and pin third-party actions/binaries.
    3. \n
  3. Update CI install steps to minimize third-party install hooks, and integrate the Shai-Hulud scanner with an expanded IoC list so lockfile compromises are caught automatically.
    4. \n
\n"}