pipebreach logo pipebreach.com
Incident Analysis

TeamPCP Part I: Twenty Days of Silent Access From a Two-Minute PR

Critical GitHub Actions docker-hub supply-chain-compromise Analysis only
March 27, 2026 · 21 min read
TeamPCP Part I, attack chain from Feb 27 Pwn Request to the March 19 coordinated strike

Methodology. Forensic reconstruction from public disclosures by Endor Labs, Wiz Research, safedep.io, Sysdig TRT, Microsoft Security Blog, and Socket Security, cross-referenced where accounts diverge. Code blocks reconstruct documented behavior, not recovered artifacts. All malicious domains and IPs are defanged.

Series: Part I · Part II: Backdooring the AI Credentials Vault →

TeamPCP Cascade: Full Attack Chain
Figure 1. Full Attack Timeline. The breach was a structured, phased campaign rather than an isolated exploit. The stark contrast between the 18 day silent dwell period and the clustered, multi vector execution between March 19 and 24 demonstrates a highly methodical operation.

Two minutes. That’s how long PR #10252 against Aqua Security’s Trivy scanner was open on February 27, 2026. The bot that opened it, hackerbot-claw, was dismissed by a maintainer within minutes. No one escalated it.

Eighteen days later, at 17:43:37 UTC on March 19, TeamPCP launched a coordinated three-vector strike using access they had been quietly holding since that PR. Five software ecosystems. 82 GitHub Actions tags poisoned. Credentials exfiltrated from an estimated 500,000 CI/CD pipelines. A 95-million-monthly-download Python package backdoored four days after that.

The vendor advisories document what happened well. What they leave unanswered is why each step was possible given the one before it. That is what this reconstruction works through, starting on February 27, not March 19.

Author's Note

I started pulling this campaign apart because the public write-ups kept framing the 18-day gap as a mystery. It's not. Once you look at what a non-atomic rotation actually does in an active compromise, the gap is the expected outcome of a procedure that was designed for normal key cycling, not for a scenario where an attacker holds active sessions. I want this reconstruction to be useful beyond the post-mortem: if you run any security tooling in CI, the attack surface described here is your attack surface right now, regardless of whether TeamPCP ever targeted you specifically.


TL;DR

  • A pull_request_target workflow in Trivy’s repo executed fork code with base repository secrets (the Pwn Request pattern, documented since 2021). GITHUB_TOKEN was exfiltrated in minutes.
  • Aqua Security’s incident response rotated credentials non-atomically over several days. During that window, TeamPCP generated replacement tokens using the still-valid aqua-bot service account. They held residual access for 18 days.
  • On March 19, TeamPCP used that access to simultaneously force-push 75 trivy-action tags and 7 setup-trivy tags in 17 minutes, spoofed maintainer commit identities, and scraped Runner.Worker process memory across every victim pipeline that ran the compromised action.
  • Masked secrets (*** in logs) are not protected from code running in the same process. Masking is a display filter; the secret lives in plaintext in the runner’s memory.
  • Part II covers the npm cascade, LiteLLM’s post-build wheel injection, and why AI gateways are a fundamentally different class of supply chain target.

Why security tooling is the highest-value supply chain target

TeamPCP’s target selection follows a consistent logic across every stage of this campaign: they are not looking for the weakest target, they are looking for the most connected one.

Trivy runs inside CI/CD pipelines with access to GITHUB_TOKEN, cloud credentials, and registry tokens because scanning containers requires them. Checkmarx KICS analyzes infrastructure-as-code and sits adjacent to the secrets that IaC deploys. LiteLLM aggregates every AI provider API key an organization uses, by design, because that is the product’s entire value proposition.

Compromising any of these does not yield one organization’s secrets. It yields every secret from every organization that ran the tool during the exposure window. The blast radius scales with adoption, not with any individual target’s defenses.

Escalating Blast Radius Hierarchy
Figure 2. Escalating Blast Radius. Compromise impact follows a strict hierarchy. While a misconfigured bucket or breached developer account limits the fallout to a single organization, compromising a pipeline security tool yields every secret across n adopting organizations simultaneously.

Complete attack timeline

Date (UTC)EventTechniqueDwell
Feb 27hackerbot-claw opens PR #10252 against TrivyPwn Request0
Feb 27GITHUB_TOKEN exfiltrated to recv.hackmoltrepeat[.]comWorkflow secret theft0
Feb 28Trivy repo privatized; 178 releases (v0.27.0–v0.69.1) deletedDestructive access+1d
Feb 28Malicious VS Code extension published to OpenVSXFirst lateral move+1d
Feb 28Aqua Security begins non-atomic credential rotationIR response
~Mar 1Aqua marks incident contained; TeamPCP retains aqua-botRotation gap
Mar 1–18Silent residual access, no anomalous activity visibleDwell+18d
Mar 19 17:4382 tags force-pushed across trivy-action and setup-trivyTag poisoning+0h
Mar 19 17:43v0.69.4 binary to GitHub Releases, GHCR, Docker Hub, ECRBinary backdoor+0h
Mar 19 ~18:00Runner.Worker scrapers fire across victim pipelinesMemory scraping+0h
Mar 1944+ npm packages via CanisterWorm in <60 secondsAutomated worm+0h
Mar 21Checkmarx KICS ast-github-action@2.3.28 compromisedPivot via stolen tokens+2d
Mar 22Docker Hub trivy:0.69.5, 0.69.6, latest backdooredDistribution extension+3d
Mar 2244 repos in aquasec-com org renamed tpcp-docs-*Defacement+3d
Mar 24 10:39LiteLLM 1.82.7 published to PyPIPost-build wheel injection+5d
Mar 24 10:52LiteLLM 1.82.8 published to PyPI.pth system-wide persistence+5d
Mar 24 11:25Both LiteLLM versions quarantined by PyPITakedown

Act I: The Pwn Request, February 27

A known pattern that keeps working

The pull_request_target trigger exists for a legitimate use case: letting PRs from forks interact with the base repository (posting comments, updating check statuses) without exposing base repository secrets to fork code.

That isolation holds under one condition: you must never check out and execute fork code inside a pull_request_target job. The moment you do, the guarantee inverts. Fork code runs with base repository secrets. This specific failure mode has a name, the Pwn Request, documented by Rhys Arkins in 2021, and it has appeared in multiple high-profile compromises since. (ATT&CK: T1195.002, Compromise Software Supply Chain; OWASP CICD-SEC-4: Poisoned Pipeline Execution)

Trivy’s “API Diff Check” workflow had it. The two lines that matter:

# .github/workflows/api-diff.yml — the two lines that broke everything
on:
  pull_request_target:         # runs with BASE repo secrets

jobs:
  api-diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          # This line is the entire vulnerability.
          # It checks out FORK code inside a pull_request_target job.
          # Fork code now runs with GITHUB_TOKEN and every other repo secret.
      - name: Check API diff
        run: ./scripts/api-diff.sh   # attacker-controlled script
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The fix separates privileged and unprivileged work into distinct jobs with no shared memory, only inert text artifacts cross the boundary via the Actions artifact API:

jobs:
  # Job 1 — fork code runs here, zero secret access
  checkout-fork:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          ref: ${{ github.event.pull_request.head.sha }}
      - run: ./scripts/generate-diff-artifact.sh
      - uses: actions/upload-artifact@v4
        with:
          name: diff-output
          path: diff.txt

  # Job 2 — secrets live here, fork code never runs
  post-results:
    needs: checkout-fork
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: diff-output
      - run: gh pr comment ${{ github.event.number }} --body "$(cat diff.txt)"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
pull_request vs pull_request_target Execution Environments
Figure 3. The Pwn Request Architecture. Changing a single word in a YAML file transforms a benign feature into a direct exfiltration vector. Using pull_request keeps untrusted code strictly sandboxed. Appending _target and checking out the fork shatters that boundary, executing unverified code with full access to base repository secrets.

The two-job pattern prevents fork code from reaching secrets. Add a second independent layer: declare permissions: explicitly at the workflow level. This limits what GITHUB_TOKEN can do even if a Pwn Request fires; an exfiltrated token scoped to contents: read cannot delete releases, create app tokens, or push to branches.

# Add at the top level of any workflow — independent of the two-job fix
permissions:
  contents: read         # removes: write, releases, packages, admin
  pull-requests: write   # only what posting a comment actually needs

The implicit default when permissions: is omitted is contents: write for GITHUB_TOKEN in many repo configurations. Declaring it explicitly narrows the blast radius of any future mistake, not just this one.

To find this pattern across your repos right now:

grep -rl "pull_request_target" .github/workflows/ | while read f; do
  grep -q 'pull_request\.head\.sha\|pull_request\.head\.ref' "$f" \
    && echo "VULNERABLE: $f"
done

What happened in the 24 hours after the PR closed

The PR closed in two minutes. The workflow had already fired. The fork’s api-diff.sh ran with write access to Trivy’s GITHUB_TOKEN and exfiltrated it to recv.hackmoltrepeat[.]com.

Closing a PR is not a security control for pull_request_target. The trigger fires when the PR is opened, not when it is merged, dismissed, or closed. By the time a maintainer clicks “Close PR”, the workflow job is either already running or already complete; the exfiltration had finished before the PR was two minutes old. “We review suspicious PRs quickly” is not a defense against this pattern. The only effective control is auditing workflow definitions before the first external PR opens, and enforcing that audit as a branch protection requirement.

Within 24 hours, TeamPCP demonstrated the full scope of that access: the Trivy repository was privatized, all 178 GitHub releases were deleted (v0.27.0 through v0.69.1), and a malicious VS Code extension was published to OpenVSX under a Checkmarx developer account whose credentials were also present in Trivy’s CI environment.

The OpenVSX move is worth pausing on. Checkmarx KICS and Aqua Trivy are both tools in the same broad security tooling ecosystem, and organizations running Aqua’s infrastructure routinely integrate both in shared CI environments. Trivy’s pipeline apparently held publish credentials for a Checkmarx developer account (most likely as part of a joint integration or cross-org workflow). This is the first lateral move of the campaign and it happened silently within 24 hours of the initial compromise: a single GITHUB_TOKEN exfiltration from one repo yielded publishing access to a second organization’s release infrastructure. The blast radius of a CI secret is not bounded by the repo it lives in; it is bounded by every external service that secret can authenticate against.

Aqua Security detected the intrusion and began rotating credentials on February 28.

This is where the story should have ended.


Non-atomic rotation: the gap that handed TeamPCP 18 days

Rotating credentials non-atomically (revoking the old one, creating the new one, spread across days) creates a window that is worse than the original compromise in one specific way: the attacker can observe which credentials are being invalidated and generate replacements before the originals expire.

Non atomic Credential Rotation Timeline
Figure 4. The Credential Rotation Gap. A non atomic rotation creates two dangerously divergent realities. While the incident response team assumed the threat was contained on March 1, the overlapping time window allowed the attacker to generate persistent replacement tokens, securing 18 days of undetected residual access before launching the final strike on March 19.

TeamPCP retained the aqua-bot service account through this window: 18 days of silent access, no anomalous activity, no alerts.

aqua-bot is a GitHub App: a first-party integration account used by Aqua for automated PR checks and release management. This distinction matters for how the retention worked. A GitHub App’s short-lived access tokens (1-hour TTL) are generated on demand from a long-lived App installation. Revoking the App’s tokens does not revoke the installation. If an attacker exfiltrates the App’s private key or generates installation tokens before the App is suspended, they can continue issuing new valid tokens for as long as the installation remains active on the repository. The rotation that Aqua performed rotated the tokens; the installation remained. TeamPCP continued generating fresh tokens through the aqua-bot installation for 18 days. The correct response in a GitHub App compromise is to suspend or uninstall the App immediately, not just rotate its credentials; suspension is instantaneous and terminates all token-generating capability at the installation level.

The correct rotation sequence in an active compromise (the specific change that would have contained this incident):

WRONG — what happened:
  Day 1:  Revoke token_A         # attacker still holds active sessions
  Day 2:  Create token_B         # attacker observes Day 1, generates token_C
  Day 3:  Update systems         # token_C is live and unknown to defenders
  Result: multi-day exploitable overlap

CORRECT — atomic rotation:
  Step 1: Create token_B FIRST, update all systems, verify each accepts it
  Step 2: Enumerate ALL active sessions for token_A via audit log API
  Step 3: Revoke token_A immediately; terminate every session, not just the token
  Step 4: Confirm 401 on token_A within minutes, not days
  Result: zero exploitable overlap

GitHub’s Audit Log API lets you enumerate active tokens during an incident. Most organizations have never used it for this purpose:

gh api /orgs/YOUR_ORG/audit-log \
  --method GET \
  -f phrase="action:org.oauth_access" \
  -f per_page=100 \
  --jq '.[].actor'

Act II: The coordinated strike, March 19 at 17:43:37 UTC

Eighteen days of silence ended at 17:43:37 UTC. Three vectors launched simultaneously. That simultaneity is deliberate: remediating any single vector while the other two are still running does not contain the exposure.

Vector A: 82 tags force-pushed in 17 minutes

Git tags are mutable by default. A tag is a named pointer to a commit. Anyone with push access can move it to any other commit at any time, with no notification to downstream consumers, no diff visible in the GitHub UI, and no change required to the workflow files that reference it. (ATT&CK: T1195.002; OWASP CICD-SEC-3: Dependency Chain Abuse)

TeamPCP moved 75 trivy-action tags and 7 setup-trivy tags in 17 minutes. Every organization with any of these in a workflow began executing TeamPCP’s entrypoint.sh on their next pipeline run, with no notification, no diff, and normal Trivy scan output in the logs.

The fix is one token change per workflow step. A full commit SHA is an immutable content address; it cannot be moved, replaced, or pointed at different code:

# Tag reference — the default everywhere, and wrong:
- uses: aquasecurity/trivy-action@v0.19.0

# SHA pin — immutable, cannot be poisoned:
- uses: aquasecurity/trivy-action@57a97c7e5bb5b2f22af3c94d11f1da7ef84f2ad0

To audit your workflows and get the current SHA for any action version:

# Find every non-SHA reference
grep -r "uses:" .github/workflows/ | grep -v "@[a-f0-9]\{40\}" | grep -v "#"

# Get the current SHA for any action tag
gh api repos/aquasecurity/trivy-action/git/refs/tags/v0.19.0 \
  --jq '.object.sha'

Vector B: Commit identity spoofing

Git does not verify committer identity. The author name and email on a commit are plain text fields set by git config before committing. GitHub displays whatever is in those fields with no indication that it was verified. (ATT&CK: T1036.005, Match Legitimate Name or Location)

TeamPCP crafted commits impersonating two Aqua maintainers: commit 1885610c as DmitriyLewen <dmitriy.lewen@aquasec.com> and 8afa9b9f as Tomochika Hara. Both carried "verified": false, "reason": "unsigned".

Three forensic signals were visible in plain sight, all detectable with standard Git tooling:

1. No GPG signature. Every legitimate commit at these tags was GitHub web-merge signed. The malicious commits had no signature block at all. A single git log --show-signature call would have surfaced this.

2. Impossible chronology. The commits were backdated to 2021, but their parent commits were from March 2026. A commit cannot predate its own parent; git log --format="%H %ai %P" makes this visible immediately.

3. Single-file changes. Every malicious commit modified only entrypoint.sh. Every legitimate commit at the corresponding tags touched multiple files. The pattern is obvious in any git diff-tree output.

None of these triggered automated alerting. A basic CI check running git log --show-signature on any commit modifying an Action’s entrypoint would have caught all three before the workflow ever ran.

Vector C: Runner.Worker memory scraping

When GitHub Actions masks a secret in log output (replacing it with ***), most developers reasonably infer the value is being protected. It isn’t.

Masking is a display filter applied at the log streaming layer. The secret exists as plaintext in the Runner.Worker process’s memory, stored in the JSON structure the runner uses internally: {"value":"ACTUAL_SECRET","isSecret":true}. Any code executing on the same runner can read /proc/<pid>/mem and scan for that pattern. (ATT&CK: T1528, Steal Application Access Token)

Memory scraping logs vs actual process memory
Figure 5. The Illusion of Masked Secrets. The asterisks in GitHub Actions logs provide a false sense of security. While the presentation layer masks the output, the actual process memory at /proc/pid/mem stores these values in plaintext accessible to any code executing on the same runner. The presence of the "isSecret": true flag right next to the plaintext token in the JSON structure highlights that masking is purely visual, not cryptographic.

The 204-line malicious entrypoint.sh escalated to root via passwordless sudo (standard on GitHub’s hosted ubuntu-latest runners), located the Runner.Worker PID, and scanned every readable memory segment for that JSON pattern. What it recovered: every secret defined in the downstream workflow, regardless of masking.

The encrypted bundle (tpcp.tar.gz, AES-256-CBC with RSA-4096 OAEP) was POSTed to scan.aquasecurtiy[.]org (a typosquat of Aqua Security’s own domain, resolving to 45.148.10.212). The fallback: if the C2 was unreachable, the malware created a public repository named tpcp-docs on the victim’s own GitHub account and committed the archive there. Exfiltration to github.com bypasses every outbound IP blocklist. (ATT&CK: T1567.001, Exfiltration to Code Repository)


Structural Mitigations What would have stopped each step per failure described in this post

1. Find every unpinned uses: in your workflows.

Tag references are trust assumptions. This command tells you exactly where you’re exposed right now:

# Every non-SHA action reference across all workflow files
grep -r "uses:" .github/workflows/ \
  | grep -Ev "@[a-f0-9]{40}" \
  | grep -v "# sha:" \
  | awk -F'uses:' '{print $2}' | sort -u

For each result, get the pinnable SHA:

# Resolve a tag to its commit SHA
gh api repos/{OWNER}/{REPO}/git/ref/tags/{TAG} --jq '.object.sha'
# Then pin: uses: aquasecurity/trivy-action@<sha> # v0.30.0

If your org has 20+ repos, run this across all of them: gh repo list ORG --json name --jq '.[].name' | xargs -I{} gh api repos/ORG/{}/contents/.github/workflows --jq '.[].name'. One afternoon of work makes tag poisoning structurally impossible in your pipelines.

2. Audit your rotation runbook now, before you need it under pressure.

The specific change that would have contained this incident: add session invalidation as an explicit step, separate from token revocation. Most engineers assume revoking a token terminates all active sessions. It does not; it stops new authentications. Check your platform’s documentation right now for “session invalidation” or “active session management”. It is almost always a separate API call. Write the exact command into your runbook so it’s not Googled during an incident at 2am.

Correct sequence for active compromise:

1. CREATE new_token           # attacker has no path to this yet
2. DEPLOY new_token           # rotate in all consumers
3. INVALIDATE active sessions # terminate any live sessions under old_token
4. REVOKE old_token           # now safe; no active sessions remain
5. VERIFY no sessions remain  # explicit confirmation step

3. Fail any unsigned commit touching an Action’s entrypoint in CI.

TeamPCP spoofed a maintainer identity on a commit that modified entrypoint.sh. Unsigned commits from previously-signing authors are a detectable signal. Add this check as a required status check on your Actions repos:

# In CI — runs before workflow execution
CHANGED=$(git diff --name-only HEAD~1 HEAD)
if echo "$CHANGED" | grep -qE '(entrypoint|action\.ya?ml|\.github/workflows)'; then
  git log --show-signature -1 HEAD | grep -q "Good signature" \
    || { echo "FAIL: unsigned commit modifying security-critical file"; exit 1; }
fi

This check is free to add and catches identity spoofing on the files attackers care about most.

Platform-level gaps these mitigations cannot fix. The three controls above address what organizations can change. There is a second category (structural gaps in the platforms themselves) that no individual team can remediate unilaterally: GitHub does not notify downstream consumers when a tag they reference in a workflow is force-pushed; there is no native enforcement of SHA pins in uses: references (the linter warns, but does not block); and pull_request_target continues to ship enabled by default with no workflow-level prompt to review checkout behavior. These gaps mean that correct org-level hygiene reduces risk but does not eliminate it; attackers who compromise a trusted action publisher bypass every downstream SHA pin simultaneously.


Next in series: Part II: Backdooring the AI Credentials Vault →

MITRE ATT&CK reference (click to expand)
IDTechniqueWhere
T1195.002Compromise Software Supply ChainPwn Request; tag poisoning
T1528Steal Application Access TokenRunner.Worker memory scraping
T1543.002Systemd Servicesysmon.service on dev machines
T1036.005Match Legitimate NameSpoofed maintainer commits
T1041Exfiltration Over C2 Channeltpcp.tar.gz HTTPS POST
T1567.001Exfiltration to Code Repositorytpcp-docs GitHub fallback
T1027Obfuscated Files or InformationDouble base64 payload encoding

References

  1. Endor Labs: TeamPCP Isn’t Done (Mar 24, 2026)
  2. Wiz Research: Trivy Compromised by TeamPCP (Mar 19, 2026)
  3. safedep.io: Trivy TeamPCP Supply Chain Compromise
  4. Microsoft Security Blog: Detecting the Trivy Supply Chain Compromise (Mar 25, 2026)
  5. Sysdig TRT: TeamPCP Expands to Checkmarx
  6. Socket Security: Trivy Under Attack Again
  7. Legit Security: The Trivy Supply Chain Compromise, Playbooks
  8. GitHub Security Lab: Preventing Pwn Requests (2021)
  9. OWASP: Top 10 CI/CD Security Risks
DM
Daniel Malvaceda

DevSecOps Engineer

Occasionally breaks AI pipelines and supply chains to see what falls out..