TeamPCP Part II: Backdooring the AI Credentials Vault

Part I covers the Pwn Request on Feb 27, the non-atomic rotation that kept the door open for 18 days, and the coordinated three-vector strike on March 19. This part picks up where that ends.

Where we left off

By end of day March 19, TeamPCP had force-pushed 82 GitHub Actions tags across trivy-action and setup-trivy, published a backdoored v0.69.4 binary across GitHub Releases, GHCR, Docker Hub, and ECR, and scraped Runner.Worker process memory across thousands of victim pipelines, recovering GITHUB_TOKEN, AWS_SECRET_ACCESS_KEY, PYPI_API_TOKEN, NPM_TOKEN, and everything else those pipelines had in scope.

The tokens recovered from those runners included credentials for npm package scopes, Checkmarx’s CI systems, and LiteLLM’s PyPI publisher account. What happened between March 19 and March 24 is the part of this campaign with the longest tail for the industry, specifically because of what LiteLLM is, not just what it does.

March 20: CanisterWorm, 44+ npm packages in under 60 seconds

The Runner.Worker scraper had recovered npm publish tokens with write access to several scoped package namespaces. TeamPCP deployed CanisterWorm (deploy.js): an automated tool that, given a valid npm publish token, enumerates every package in the token’s accessible scope via the registry API and publishes a new patch version of each with a credential-stealing postinstall hook injected. (ATT&CK: T1195.002, Compromise Software Supply Chain)

Result in under 60 seconds: 28 packages under @EmilGroup, 16 under @opengov, and additional packages under @teale.io, @airtm, and @pypestream. Every developer or CI pipeline that ran npm install for an affected package executed the credential stealer at install time, no runtime import, no user action.

The postinstall script is the attack surface: npm runs it automatically after installing any package that defines it. Auditing hooks in your dependency tree before installing is the direct mitigation:

# Audit postinstall hooks in your project before installing
npm install --dry-run 2>&1 | grep -i "postinstall\|script"

# Check a specific package's scripts before installing
npm pack <package>@<version> --dry-run --json | jq '.[0].scripts'

CanisterWorm’s own C2 used an ICP canister: tdtqy-oyaaa-aaaae-af2dq-cai.raw.icp0[.]io. ICP is a decentralized compute platform where smart contracts (canisters) run distributed across globally replicated nodes with no single hosting provider. This makes traditional C2 takedown procedures structurally inapplicable: there is no abuse@ email, no server to seize, no registrar to contact. The canister ID is fixed and publicly verifiable, but terminating it requires consensus from the ICP network’s governance system (the Network Nervous System), or voluntary blocking by the boundary node operators who serve as gateways between the internet and the ICP network. The denylisting on March 22 required coordination with Dfinity Foundation and major boundary node operators, a process that took approximately three days from exposure (compared to hours for a traditional hosting provider takedown). This delay matters: CanisterWorm had already completed its run in under 60 seconds on March 20, so the C2 delay did not affect the primary attack. But the infrastructure choice signals deliberate resilience planning, not opportunism.

The Cloudflare Tunnel fallback (plug-tab-protective-relay.trycloudflare[.]com) was a second layer of this same strategy. Cloudflare Tunnels route traffic through Cloudflare’s infrastructure, making the actual origin server invisible. Blocking the subdomain requires a Cloudflare abuse report, not a hosting provider notification. TeamPCP was operating with multi-layer C2 resilience built in from the start.

Persistence on compromised developer machines: ~/.config/systemd/user/pgmon.service masquerading as “PostgreSQL Monitor”, staging payloads at /tmp/pglog, tracking state at /tmp/.pg_state.

March 21–22: Checkmarx KICS and Docker Hub

Using credentials recovered from Trivy’s CI, TeamPCP published checkmarx/ast-github-action@2.3.28 with a malicious setup.sh on March 21 at 12:58 UTC (exposure window closed ~16:50 UTC). The payload architecture was identical to Trivy’s: AES-256-CBC + RSA-4096, tpcp.tar.gz, exfiltration to checkmarx[.]zone (typosquat of Checkmarx’s domain, resolving to 83.142.209.11:443).

Two new Kubernetes artifacts appeared in this wave: DaemonSets named host-provisioner-std and host-provisioner-iran deployed to kube-system. DaemonSets run on every node in the cluster and persist across pod restarts and namespace cleanups. Detecting them requires explicitly checking kube-system, which most security tooling does not scan by default. (ATT&CK: T1611, Escape to Host)

On March 22, aquasec/trivy:0.69.5, 0.69.6, and latest were published to Docker Hub. Any organization pinning Trivy by tag in their container pipeline, not by digest, automatically pulled the malicious image on their next run. Same principle as GitHub Actions tag poisoning, different distribution channel. The fix is identical in structure: replace the mutable tag with an immutable content address:

# Wrong — mutable tag, silently replaced on March 22
docker pull aquasec/trivy:0.69.4
docker pull aquasec/trivy:latest

# Correct — immutable digest, cannot be pointed at different content
docker pull aquasec/trivy@sha256:822dd269ec10459572dfaaefe163dae693c344249a0161953f0d5cdd110bd2a0

# In a Dockerfile or CI step:
# FROM aquasec/trivy@sha256:822dd269ec10459572dfaaefe163dae693c344249a0161953f0d5cdd110bd2a0

# Resolve the current digest for any tag before pinning:
docker buildx imagetools inspect aquasec/trivy:0.69.4 \
  --format '{{json .Manifest}}' | jq -r '.digest'

The SHA-256 digests for the known-compromised images are in the IOC table below. Any pipeline that pulled trivy:0.69.5, 0.69.6, or latest on March 22 should be treated as exposed.

March 24: LiteLLM, the wheel that wasn’t in GitHub

Twenty-two hours after the KICS compromise, at approximately 10:39 UTC on March 24, litellm==1.82.7 was published to PyPI. Version 1.82.8 followed 13 minutes later. Both were quarantined by PyPI at 11:25 UTC.

46-minute exposure window. 95 million monthly downloads. 36% of cloud environments.

Why LiteLLM is a different class of target

LiteLLM is not just another Python package that happened to be compromised. It is an AI API gateway: a reverse proxy that sits between your application and every AI provider you use.

Every AI provider API key lives in LiteLLM’s environment, by design. That is the product’s value proposition: one proxy, standardized API, all providers. Compromising the package compromises all of them simultaneously.

But credential theft is only the beginning of what a compromised LiteLLM proxy running in production can do:

Standard PyPI package compromise:
  — Steal credentials from the developer's machine or CI pipeline

Compromised LiteLLM proxy in production:
  — All of the above, plus:
  — Log every prompt sent to every AI provider (RAG content, system prompts, user data)
  — Log every AI response received (proprietary model outputs, reasoning chains)
  — Modify prompts before they reach the model (infrastructure-level prompt injection)
  — Modify responses before they reach your application
  — Redirect requests to attacker-controlled models
  — All API keys aggregated in one process, one exfil payload

TeamPCP’s payload treated LiteLLM like any other package: standard credential harvester, in and out in 46 minutes. A more patient attacker running the same supply chain compromise could sit silently in the proxy layer and collect everything passing through it, indefinitely, without any payload detectable in process memory.

(OWASP LLM03:2025, Supply Chain Vulnerabilities)

The post-build injection technique

The PyPI publish token was recovered from LiteLLM’s own CI pipeline. LiteLLM used trivy-action to scan its container images for vulnerabilities, a standard DevSecOps practice. During the March 19 exposure window, that pipeline executed the backdoored trivy-action tag, and the Runner.Worker scraper recovered PYPI_API_TOKEN from process memory alongside everything else in scope. TeamPCP did not need to compromise LiteLLM’s GitHub, their build pipeline, or their maintainer accounts. They needed one thing: the PyPI API token.

The recursive structure of this chain is worth naming explicitly: the security scanner that LiteLLM used to protect its software supply chain was the vector through which its supply chain was compromised. This is not irony; it is the logical outcome of the target selection described in Part I. Security tooling runs with privileged access by design. The more conscientiously a team integrates security scanning into CI, the more secrets they concentrate in the scanner’s execution environment.

A Python wheel (.whl) is a ZIP archive. The workflow: download the legitimate 1.82.6 wheel, unzip it, inject malicious code into proxy_server.py, regenerate the RECORD file (which contains SHA-256 hashes of every file in the package) to match the modified content, rezip, publish as 1.82.7.

From pip’s perspective, the package is internally consistent; every file’s hash matches RECORD. pip does not compare distributed artifacts against source commits. That check does not happen automatically anywhere in the standard Python toolchain. (ATT&CK: T1195.002)

The only reliable detection path is comparing the wheel against the GitHub source at the same tag. The script below does it in 30 seconds and catches both the file injection and any unexpected .pth files:

#!/bin/bash
# verify-wheel-vs-source.sh
# pip, poetry, pipenv — none of them do this check automatically.

PACKAGE=$1     # e.g., litellm
VERSION=$2     # e.g., 1.82.6
GITHUB_REPO=$3 # e.g., https://github.com/BerryAI/litellm

WORKDIR=$(mktemp -d)
trap "rm -rf $WORKDIR" EXIT

pip download "${PACKAGE}==${VERSION}" --no-deps -d "$WORKDIR/wheel/" -q
unzip -q "$WORKDIR/wheel/"*.whl -d "$WORKDIR/extracted/"
git clone -q --depth 1 --branch "v${VERSION}" "$GITHUB_REPO" "$WORKDIR/source/" 2>/dev/null

echo "── ${PACKAGE}==${VERSION}: wheel vs source ──"

find "$WORKDIR/extracted/" -name "*.pth" -not -path "*/dist-info/*" \
  | while read f; do
      echo "⚠  Unexpected .pth: $(basename "$f")"
      echo "   $(head -c 200 "$f")"
    done

DIFF=$(diff -rq \
  "$WORKDIR/extracted/${PACKAGE}/" \
  "$WORKDIR/source/${PACKAGE}/" \
  --exclude="*.pyc" --exclude="__pycache__" 2>/dev/null)

[ -z "$DIFF" ] \
  && echo "✓  Wheel matches source" \
  || { echo "✗  DIVERGENCE — possible post-build injection"; echo "$DIFF" | head -40; }

Version 1.82.7: the proxy_server.py injection

SHA-256: 8395c3268d5c5dbae1c7c6d4bb3c318c752ba4608cfcd90eb97ffb94a910eac2

Twelve lines injected at line 128 of litellm/proxy/proxy_server.py, placed between two unrelated legitimate code blocks to minimize visual disruption in a diff. The payload decodes via double base64 at runtime; the dangerous code is only visible after two decode rounds, which most static analysis tools do not apply to string literals.

The attacker’s development history is preserved in the binary artifact. Endor Labs documented three development iterations:

• Iteration 1: Custom RC4 cipher (abandoned; weak and detectable by entropy analysis)
• Iteration 2: Both old and new harvester versions coexisting (transition artifact)
• Iteration 3: RC4 removed; direct exec()/eval() calls replaced by subprocess piping (harder to catch with static analysis rules that grep for exec()

C2 for 1.82.7: checkmarx[.]zone, reusing infrastructure from the KICS attack the previous day.

Version 1.82.8: the .pth escalation (13 minutes later)

SHA-256: d2a0d5f564628773b6af7b9c11f6b86531a875bd2d186d7081ab62748a800ebb SHA-256 of litellm_init.pth: 71e35aef03099cd1f2d6446734273025a163597de93912df321ef118bf135238

The 13-minute gap between 1.82.7 and 1.82.8 means TeamPCP was actively iterating during the attack window. They watched the first version land and pushed a more aggressive variant almost immediately. This iteration speed is a forensic signal in its own right: downloading the legitimate wheel, injecting code, regenerating the RECORD SHA-256 manifest, re-zipping, and publishing to PyPI in 13 minutes is not manual work. It requires automated tooling: a wheel injection pipeline analogous to CanisterWorm’s npm automation, running with live feedback from PyPI’s publication API. TeamPCP was not improvising; they were operating a purpose-built supply chain attack toolkit with real-time deployment capabilities.

The addition is litellm_init.pth. Most developers have never encountered Python .pth files. When Python starts (any process, invoked any way), the interpreter reads every .pth file in site-packages before executing anything else. Lines beginning with import are executed directly. When pip installs a wheel containing a .pth file, that file lands in site-packages and begins executing on every subsequent Python invocation system-wide, regardless of whether LiteLLM is imported, regardless of what the script does, regardless of whether the user knows LiteLLM is installed. (ATT&CK: T1546, Event-triggered Execution)

Every one of these triggers the payload after 1.82.8 is installed:

python3 -c "print('hello')"      python3 manage.py runserver
python3 -m pytest tests/         python3 -m pip install anything
uvicorn app:app                  celery worker --app=tasks
jupyter notebook                 python3 train.py

There is no way to use Python in an affected environment without triggering the payload, short of removing litellm_init.pth from site-packages first.

C2 upgraded to models[.]litellm[.]cloud, crafted to blend into legitimate LiteLLM traffic in proxy logs and network monitoring.

To verify this mechanism safely in an isolated environment:

python3 -m venv /tmp/pth-demo && source /tmp/pth-demo/bin/activate
SITE=$(python3 -c "import site; print(site.getsitepackages()[0])")

echo "import sys; open('/tmp/pth_demo_ran.txt', 'a').write('executed\n')" \
  > "$SITE/demo_test.pth"

python3 -c "print('hello')"    # triggers the .pth side-effect
python3 -m pip list --quiet    # triggers it again
wc -l /tmp/pth_demo_ran.txt   # outputs: 2

rm "$SITE/demo_test.pth" /tmp/pth_demo_ran.txt && deactivate

What the payload actually collected

The Stage 2 credential harvester (332 lines, documented by Endor Labs) swept comprehensively. The scope is worth itemizing because “CI/CD credentials” understates what 36% of cloud environments actually contain:

Identity: SSH private keys (id_rsa, id_ed25519, id_ecdsa, id_dsa), shell histories, /etc/passwd, /etc/shadow

Cloud providers: AWS via IMDS at 169.254.169.254 (role credentials, Secrets Manager, SSM parameters); GCP application_default_credentials.json; Azure ~/.azure/ directory (ATT&CK: T1552.005, Cloud Instance Metadata API)

Kubernetes: In-pod service account tokens; full dump of all accessible namespace secrets via kubectl get secrets -A

Application secrets: .env files recursively across /home, /root, /opt, /srv, /var/www, /app, /data, /var/lib, /tmp (six directories deep); database configs (.pgpass, my.cnf, .mongorc.js); CI/CD config files (terraform.tfvars, terraform.tfstate, .gitlab-ci.yml, Jenkinsfile); TLS/SSL private keys (.pem, .key, .p12, .pfx)

Cryptocurrency: Bitcoin wallet.dat; Ethereum keystore files; Solana validator keypairs

Everything encrypted with AES-256-CBC (session key wrapped under embedded RSA-4096 public key, OAEP padding) and POSTed as tpcp.tar.gz to models[.]litellm[.]cloud. No plaintext on the wire.

Stage 3 persistence: ~/.config/sysmon/sysmon.py with systemd unit ~/.config/systemd/user/sysmon.service (“System Telemetry Service”), polling checkmarx[.]zone/raw every 50 minutes after a 5-minute initial delay. Kill switch: execution halts if youtube.com is unreachable (sandbox indicator). (ATT&CK: T1543.002, Systemd Service)

For systems with Kubernetes access: privileged pods named node-setup-{node_name} in kube-system with hostPID: true, hostNetwork: true, privileged: true, full host filesystem mounted, and tolerations allowing scheduling on every node including control plane. Host-level execution across the entire cluster, surviving namespace cleanups.

Complete IOC list

Threat hunting script

Run this on any machine that executed trivy-action or setup-trivy during March 19–20, checkmarx/ast-github-action@2.3.28 on March 21, or had LiteLLM installed at any point on March 24. Each check runs locally in under 30 seconds; only check 6 requires the gh CLI and network access to GitHub.

Zero indicators means the campaign left no detectable artifacts on this machine; continue rotating credentials per the table below as a precaution. One or more indicators means treat the machine as compromised: rotate all credentials first, then investigate in parallel.

#!/bin/bash
# teampcp-hunt.sh — run on any machine that may have been exposed

ISSUES=0

echo "TeamPCP Threat Hunt"

# 1. Filesystem persistence artifacts
echo ""
echo "[1/6] Persistence files..."
for f in \
    "$HOME/.config/sysmon/sysmon.py" \
    "$HOME/.config/systemd/user/sysmon.service" \
    "$HOME/.config/systemd/user/pgmon.service" \
    "$HOME/.local/share/pgmon/service.py" \
    "/tmp/pglog" "/tmp/.pg_state"
do
    if [ -f "$f" ]; then
        echo "  ✗ FOUND: $f  [sha256: $(sha256sum "$f" | cut -d' ' -f1)]"
        ISSUES=$((ISSUES+1))
    fi
done
[ $ISSUES -eq 0 ] && echo "  ✓ None found"

# 2. Suspicious .pth files
echo ""
echo "[2/6] Python site-packages .pth files..."
PTH_ISSUES=0
while IFS= read -r site_dir; do
    for pth in "$site_dir"/*.pth; do
        [ -f "$pth" ] || continue
        if grep -qE "(exec|subprocess|urllib|base64|eval|__import__)" "$pth" 2>/dev/null; then
            echo "  ✗ SUSPICIOUS: $pth"
            echo "    $(head -c 160 "$pth")"
            PTH_ISSUES=$((PTH_ISSUES+1))
        fi
    done
done < <(python3 -c "import site; print('\n'.join(site.getsitepackages()))" 2>/dev/null)
[ $PTH_ISSUES -eq 0 ] && echo "  ✓ No suspicious .pth files"
ISSUES=$((ISSUES+PTH_ISSUES))

# 3. Compromised LiteLLM version
echo ""
echo "[3/6] LiteLLM version check..."
LITELLM_VER=$(python3 -c "import litellm; print(litellm.__version__)" 2>/dev/null || echo "not installed")
if echo "$LITELLM_VER" | grep -qE "^1\.82\.[78]$"; then
    echo "  ✗ COMPROMISED: litellm==$LITELLM_VER — rotate all AI provider API keys immediately"
    ISSUES=$((ISSUES+1))
else
    echo "  ✓ LiteLLM: $LITELLM_VER"
fi

# 4. Systemd backdoor
echo ""
echo "[4/6] Systemd sysmon service..."
if systemctl --user is-active sysmon &>/dev/null 2>&1; then
    echo "  ✗ ACTIVE: sysmon.service is running"
    ISSUES=$((ISSUES+1))
elif systemctl --user is-enabled sysmon &>/dev/null 2>&1; then
    echo "  ✗ ENABLED: sysmon.service exists (not currently running)"
    ISSUES=$((ISSUES+1))
else
    echo "  ✓ No sysmon service"
fi

# 5. Kubernetes
echo ""
echo "[5/6] Kubernetes..."
if command -v kubectl &>/dev/null && kubectl cluster-info &>/dev/null 2>&1; then
    K8S_HITS=$(kubectl get pods,daemonsets -A --no-headers 2>/dev/null \
        | grep -iE "node-setup-|host-provisioner|sysmon" || true)
    if [ -n "$K8S_HITS" ]; then
        echo "  ✗ SUSPICIOUS resources:"
        echo "$K8S_HITS" | sed 's/^/    /'
        ISSUES=$((ISSUES+1))
    else
        echo "  ✓ No suspicious pods or DaemonSets"
    fi
else
    echo "  – kubectl not available"
fi

# 6. GitHub tpcp-docs repositories
echo ""
echo "[6/6] GitHub exfiltration indicator..."
if command -v gh &>/dev/null && gh auth status &>/dev/null 2>&1; then
    TPCP=$(gh repo list --json name --limit 200 2>/dev/null \
        | python3 -c "import sys,json; [print(r['name']) for r in json.load(sys.stdin) if 'tpcp-docs' in r['name']]" || true)
    if [ -n "$TPCP" ]; then
        echo "  ✗ tpcp-docs repo found — exfiltration confirmed: $TPCP"
        ISSUES=$((ISSUES+1))
    else
        echo "  ✓ No tpcp-docs repositories"
    fi
else
    echo "  – gh CLI not available"
fi

echo ""
echo "══════════════════════════════════════════"
if [ $ISSUES -gt 0 ]; then
    echo "  ✗ $ISSUES indicator(s) — treat as compromised, rotate credentials"
else
    echo "  ✓ No TeamPCP indicators found"
fi
echo "══════════════════════════════════════════"

What to rotate, and when

Absence of confirmed exfiltration evidence is not evidence of absence. The payload operated silently in background threads with encrypted output. Rotate first, investigate in parallel.

Rotate immediately: GitHub tokens, AWS/GCP/Azure credentials, PyPI + npm + Docker Hub publish tokens, all AI provider API keys (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Cohere, Mistral, Vertex AI)

Within 24 hours: SSH keys on CI runners, Kubernetes service account tokens, database credentials in .env files on runners

# Check what versions have been on your systems
pip show litellm | grep Version
docker images --format "{{.Repository}}:{{.Tag}}" | grep litellm
kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | grep litellm

./verify-wheel-vs-source.sh litellm 1.82.6 https://github.com/BerryAI/litellm

# Find all .pth files across all Python environments on this machine
python3 -c "import site; print('\n'.join(site.getsitepackages() + [site.getusersitepackages()]))" \
  | xargs -I{} find {} -name "*.pth" -exec grep -lE 'exec|subprocess|base64|urllib|__import__' {} \;

# Generate pinned hashes from known-good version
pip download "litellm==1.82.6" --no-deps -d /tmp/pins/
pip hash /tmp/pins/litellm-1.82.6-*.whl >> requirements.txt
# requirements.txt now contains:
# litellm==1.82.6 \
#   --hash=sha256:<hash-of-legitimate-wheel>
# pip install -r requirements.txt will reject 1.82.7 — hash mismatch

# Install slsa-verifier
go install github.com/slsa-framework/slsa-verifier/v2/cli/slsa-verifier@latest

# Download wheel and provenance
pip download "litellm==1.82.6" --no-deps -d /tmp/litellm-verify/
# Verify provenance against the package's known source repo
slsa-verifier verify-artifact /tmp/litellm-verify/litellm-1.82.6-*.whl \
  --provenance-path litellm-1.82.6.intoto.jsonl \
  --source-uri github.com/BerryAI/litellm \
  --source-tag v1.82.6

The broader implication for AI infrastructure

TeamPCP’s payload treated LiteLLM like any other package. But the proxy position LiteLLM occupies means a more patient attacker running the same compromise could collect ongoing intelligence (every prompt, every response, every system prompt, every RAG document) without any payload visible in memory, indefinitely, until the package version is updated.

Prompt injection has dominated AI security conversations because it is a model-level threat affecting every deployment. Proxy-layer supply chain compromise achieves everything prompt injection does, plus persistent credential access, plus bidirectional traffic manipulation, before any model-level defense acts, and across every organization using the proxy simultaneously.

The supply chain is the security perimeter for AI systems. TeamPCP just demonstrated what the first move looks like.

References

1. Endor Labs: TeamPCP Isn’t Done (Mar 24, 2026)
2. Wiz Research: TeamPCP Trojanizes LiteLLM (Mar 24, 2026)
3. Wiz Research: Trivy Compromised by TeamPCP (Mar 19, 2026)
4. safedep.io: Trivy TeamPCP Supply Chain Compromise
5. Microsoft Security Blog: Detecting the Trivy Supply Chain Compromise (Mar 25, 2026)
6. Sysdig TRT: TeamPCP Expands to Checkmarx
7. BleepingComputer: LiteLLM PyPI Package Backdoored (Mar 24, 2026)
8. Palo Alto Unit 42: When Security Scanners Become the Weapon
9. Arctic Wolf: TeamPCP Supply Chain Attack Campaign
10. OWASP: LLM Top 10 for Large Language Models 2025
11. PyPI Advisory: PYSEC-2026-2: litellm 1.82.7, 1.82.8