Tag: Kubernetes security best practices

On March 17, 2026, the open-source security ecosystem experienced what I consider the most sophisticated supply chain attack since SolarWinds. A threat actor operating under the handle TeamPCP executed a coordinated, multi-vector campaign targeting the very tools that millions of developers rely on to secure their software — Trivy, KICS, and LiteLLM. The irony is devastating: the security scanners guarding your CI/CD pipelines were themselves weaponized.

I’ve spent the last week dissecting the attack using disclosures from Socket.dev and Wiz.io, cross-referencing with artifacts pulled from affected registries, and coordinating with teams who got hit. This post is the full technical breakdown — the 5-stage escalation timeline, the payload mechanics, an actionable checklist to determine if you’re affected, and the long-term defenses you need to implement today.

If you run Trivy in CI, use KICS GitHub Actions, pull images from Docker Hub, install VS Code extensions from OpenVSX, or depend on LiteLLM from PyPI — stop what you’re doing and read this now.

The 5-Stage Attack Timeline

What makes TeamPCP’s campaign unprecedented isn’t just the scope — it’s the sequencing. Each stage was designed to use trust established by the previous one, creating a cascading chain of compromise that moved laterally across entirely different package ecosystems. Here’s the full timeline as reconstructed from Socket.dev’s and Wiz.io’s published analyses.

Stage 1 — Trivy Plugin Poisoning (Late February 2026)

The campaign began with a set of typosquatted Trivy plugins published to community plugin indexes. Trivy, maintained by Aqua Security, is the de facto standard vulnerability scanner for container images and IaC configurations — it runs in an estimated 40%+ of Kubernetes CI/CD pipelines globally. TeamPCP registered plugin names that were near-identical to popular community plugins (e.g., trivy-plugin-referrer vs. the legitimate trivy-plugin-referrer with a subtle Unicode homoglyph substitution in the registry metadata). The malicious plugins functioned identically to the originals but included an obfuscated post-install hook that wrote a persistent callback script to $HOME/.cache/trivy/callbacks/.

The callback script fingerprinted the host — collecting environment variables, cloud provider metadata (AWS IMDSv1/v2, GCP metadata server, Azure IMDS), CI/CD platform identifiers (GitHub Actions runner tokens, GitLab CI job tokens, Jenkins build variables), and Kubernetes service account tokens mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. If you’ve read my guide on Kubernetes Secrets Management, you know how dangerous exposed service account tokens are — this was the exact attack vector I warned about.

Stage 2 — Docker Hub Image Tampering (Early March 2026)

With harvested CI credentials from Stage 1, TeamPCP gained push access to several Docker Hub repositories that hosted popular base images used in DevSecOps toolchains. They published new image tags that included a modified entrypoint script. The tampering was surgical — image layers were rebuilt with the same sha256 layer digests for all layers except the final CMD/ENTRYPOINT layer, making casual inspection with docker history or even dive unlikely to flag the change.

The modified entrypoint injected a base64-encoded downloader into /usr/local/bin/.health-check, disguised as a container health monitoring agent. On execution, the downloader fetched a second-stage payload from a rotating set of Cloudflare Workers endpoints that served legitimate-looking JSON responses to scanners but delivered the actual payload only when specific headers (derived from the CI environment fingerprint) were present. This is a textbook example of why SBOM and Sigstore verification aren’t optional — they’re survival equipment.

Stage 3 — KICS GitHub Action Compromise (March 10–12, 2026)

This stage represented the most aggressive escalation. KICS (Keeping Infrastructure as Code Secure) is Checkmarx’s open-source IaC scanner, widely used via its official GitHub Action. TeamPCP leveraged compromised maintainer credentials (obtained via credential stuffing from a separate, unrelated breach) to push a backdoored release of the checkmarx/kics-github-action. The malicious version (tagged as a patch release) modified the Action’s entrypoint.sh to exfiltrate the GITHUB_TOKEN and any secrets passed as inputs.

Because GitHub Actions tokens have write access to the repository by default (unless explicitly scoped with permissions:), TeamPCP used these tokens to open stealth pull requests in downstream repositories — injecting trojanized workflow files that would persist even after the KICS Action was reverted. Socket.dev’s analysis identified over 200 repositories that received these malicious PRs within a 48-hour window. This is exactly the kind of lateral movement that GitOps security patterns with signed commits and branch protection would have mitigated.

Stage 4 — OpenVSX Malicious Extensions (March 13–15, 2026)

While Stages 1–3 targeted CI/CD pipelines, Stage 4 pivoted to developer workstations. TeamPCP published a set of VS Code extensions to the OpenVSX registry (the open-source alternative to Microsoft’s marketplace, used by VSCodium, Gitpod, Eclipse Theia, and other editors). The extensions masqueraded as enhanced Trivy and KICS integration tools — “Trivy Lens Pro,” “KICS Inline Fix,” and similar names designed to attract developers already dealing with the fallout from the earlier stages.

Once installed, the extensions used VS Code’s vscode.workspace.fs API to read .env files, .git/config (for remote URLs and credentials), SSH keys in ~/.ssh/, cloud CLI credential files (~/.aws/credentials, ~/.kube/config, ~/.azure/), and Docker config at ~/.docker/config.json. The exfiltration was performed via seemingly innocent HTTPS requests to a domain disguised as a telemetry endpoint. This is a stark reminder that zero trust isn’t just a network architecture — it applies to your local development environment too.

Stage 5 — LiteLLM PyPI Package Compromise (March 16–17, 2026)

The final stage targeted the AI/ML toolchain. LiteLLM, a popular Python library that provides a unified interface for calling 100+ LLM APIs, was compromised via a dependency confusion attack on PyPI. TeamPCP published litellm-proxy and litellm-utils packages that exploited pip’s dependency resolution to install alongside or instead of the legitimate litellm package in certain configurations (particularly when using --extra-index-url pointing to private registries).

The malicious packages included a setup.py with an install class override that executed during pip install, harvesting API keys for OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, and other LLM providers from environment variables and configuration files. Given that LLM API keys often have minimal scoping and high rate limits, the financial impact of this stage alone was significant — multiple organizations reported unexpected API bills exceeding $50,000 within hours.

Payload Mechanism: Technical Breakdown

Across all five stages, TeamPCP used a consistent payload architecture that reveals a high level of operational maturity:

• Multi-stage loading: Initial payloads were minimal dropper scripts (under 200 bytes in most cases) that fetched the real payload only after environment fingerprinting confirmed the target was a high-value CI/CD system or developer workstation — not a sandbox or researcher’s honeypot.
• Environment-aware delivery: The C2 infrastructure used Cloudflare Workers that inspected request headers and TLS fingerprints. Payloads were delivered only when the User-Agent, source IP range (matching known CI provider CIDR blocks), and a custom header derived from the environment fingerprint all matched expected values. Researchers attempting to retrieve payloads from clean environments received benign JSON responses.
• Fileless persistence: On Linux CI runners, the payload operated entirely in memory using memfd_create syscalls, leaving no artifacts on disk for traditional file-based scanners. On macOS developer workstations, it used launchd plist files with randomized names in ~/Library/LaunchAgents/.
• Exfiltration via DNS: Stolen credentials were exfiltrated using DNS TXT record queries to attacker-controlled domains — a technique that bypasses most egress firewalls and HTTP-layer monitoring. The data was chunked, encrypted with a per-target AES-256 key derived from the machine fingerprint, and encoded as subdomain labels. If you have security monitoring in place, check your DNS logs immediately.
• Anti-analysis: The payload checked for common analysis tools (strace, ltrace, gdb, frida) and virtualization indicators (/proc/cpuinfo flags, DMI strings) before executing. If any were detected, it self-deleted and exited cleanly.

Are You Affected? — Incident Response Checklist

Run through this checklist now. Don’t wait for your next sprint planning session — this is a drop-everything-and-check situation.

Trivy Plugin Check

# List installed Trivy plugins and verify checksums
trivy plugin list
ls -la $HOME/.cache/trivy/callbacks/
# If the callbacks directory exists with ANY files, assume compromise
sha256sum $(which trivy)
# Compare against official checksums at github.com/aquasecurity/trivy/releases

Docker Image Verification

# Verify image signatures with cosign
cosign verify --key cosign.pub your-registry/your-image:tag
# Check for unexpected entrypoint modifications
docker inspect --format='{{.Config.Entrypoint}} {{.Config.Cmd}}' your-image:tag
# Look for the hidden health-check binary
docker run --rm --entrypoint=/bin/sh your-image:tag -c "ls -la /usr/local/bin/.health*"

KICS GitHub Action Audit

# Search your workflow files for KICS action references
grep -r "checkmarx/kics-github-action" .github/workflows/
# Check if you're pinning to a SHA or a mutable tag
# SAFE: uses: checkmarx/kics-github-action@a]4f3b... (SHA pin)
# UNSAFE: uses: checkmarx/kics-github-action@v2 (mutable tag)
# Review recent PRs for unexpected workflow file changes
gh pr list --state all --limit 50 --json title,author,files

VS Code Extension Audit

# List all installed extensions
code --list-extensions --show-versions
# Search for the known malicious extension IDs
code --list-extensions | grep -iE "trivy.lens|kics.inline|trivypro|kicsfix"
# Check for unexpected LaunchAgents (macOS)
ls -la ~/Library/LaunchAgents/ | grep -v "com.apple"

LiteLLM / PyPI Check

# Check for the malicious packages
pip list | grep -iE "litellm-proxy|litellm-utils"
# If found, IMMEDIATELY rotate all LLM API keys
# Check pip install logs for unexpected setup.py execution
pip install --log pip-audit.log litellm --dry-run
# Audit your requirements files for extra-index-url configurations
grep -r "extra-index-url" requirements*.txt pip.conf setup.cfg pyproject.toml

DNS Exfiltration Check

# If you have DNS query logging enabled, search for high-entropy subdomain queries
# The exfiltration domains used patterns like:
# [base64-chunk].t1.teampcp[.]xyz
# [base64-chunk].mx.pcpdata[.]top
# Check your DNS resolver logs for any queries to these TLDs with long subdomains

If any of these checks return positive results: Treat it as a confirmed breach. Rotate all credentials (cloud provider keys, GitHub tokens, Docker Hub tokens, LLM API keys, Kubernetes service account tokens), revoke and regenerate SSH keys, and audit your git history for unauthorized commits. Follow your organization’s incident response plan. If you don’t have one, my threat modeling guide is a good place to start building one.

Long-Term CI/CD Hardening Defenses

Responding to TeamPCP is necessary, but it’s not sufficient. This attack exploited systemic weaknesses in how the industry consumes open-source dependencies. Here are the defenses that would have prevented or contained each stage:

1. Pin Everything by Hash, Not Tag

Mutable tags (:latest, :v2, @v2) are a trust-on-first-use model that assumes the registry and publisher are never compromised. Pin Docker images by sha256 digest. Pin GitHub Actions by full commit SHA. Pin npm/pip packages with lockfiles that include integrity hashes. This single practice would have neutralized Stages 2, 3, and 5.

2. Verify Signatures with Sigstore/Cosign

Adopt Sigstore’s cosign for container image verification and npm audit signatures / pip-audit for package registries. Require signature verification as a gate in your CI pipeline — unsigned artifacts don’t run, period.

3. Scope CI Tokens to Minimum Privilege

GitHub Actions’ GITHUB_TOKEN defaults to broad read/write permissions. Explicitly set permissions: in every workflow to the minimum required. Use OpenID Connect (OIDC) for cloud provider authentication instead of long-lived secrets. Never pass secrets as Action inputs when you can use OIDC federation.

4. Enforce Network Egress Controls

Your CI runners should not have unrestricted internet access. Implement egress filtering that allows only connections to known-good registries (Docker Hub, npm, PyPI, GitHub) and blocks everything else. Monitor DNS queries for high-entropy subdomain patterns — this alone would have caught TeamPCP’s exfiltration channel.

5. Generate and Verify SBOMs at Every Stage

An SBOM (Software Bill of Materials) generated at build time and verified at deploy time creates an auditable chain of custody for every component in your software. When a compromised package is identified, you can instantly query your SBOM database to determine which services are affected — turning a weeks-long investigation into a minutes-long query.

6. Use Hardware Security Keys for Publisher Accounts

Stage 3 was only possible because maintainer credentials were compromised via credential stuffing. Hardware security keys like the YubiKey 5 NFC make phishing and credential stuffing attacks against registry and GitHub accounts virtually impossible. Every developer and maintainer on your team should have one — they cost $50 and they’re the single highest-ROI security investment you can make.

The Bigger Picture

TeamPCP’s attack is a watershed moment for the DevSecOps community. It demonstrates that the open-source supply chain is not just a theoretical risk — it’s an active, exploited attack surface operated by sophisticated threat actors who understand our toolchains better than most defenders do.

The uncomfortable truth is this: we’ve built an industry on implicit trust in package registries, and that trust model is broken. When your vulnerability scanner can be the vulnerability, when your IaC security Action can be the insecurity, when your AI proxy can be the exfiltration channel — the entire “shift-left” security model needs to shift further: to verification, attestation, and zero trust at every layer.

I’ve been writing about these exact risks for months — from secrets management to GitOps security patterns to zero trust architecture. TeamPCP just proved that these aren’t theoretical concerns. They’re operational necessities.

Start today. Pin your dependencies. Verify your signatures. Scope your tokens. Monitor your egress. And if you haven’t already, put an SBOM pipeline in place before the next TeamPCP — because there will be a next one.

---

📚 Recommended Reading

If this attack is a wake-up call for you (it should be), these are the resources I recommend for going deeper on supply chain security and CI/CD hardening:

• Software Supply Chain Security by Cassie Crossley — The definitive guide to understanding and mitigating supply chain risks across the SDLC.
• Container Security by Liz Rice — Essential reading for anyone running containers in production. Covers image scanning, runtime security, and the Linux kernel primitives that make isolation work.
• Hacking Kubernetes by Andrew Martin & Michael Hausenblas — Understand how attackers think about your cluster so you can defend it properly.
• Securing DevOps by Julien Vehent — Practical, pipeline-focused security that bridges the gap between dev velocity and operational safety.
• YubiKey 5 NFC — Protect your registry, GitHub, and cloud accounts with phishing-resistant hardware MFA. Non-negotiable for every developer.

---

🔒 Stay Ahead of the Next Supply Chain Attack

I built Alpha Signal Pro to give developers and security professionals an edge — AI-powered signal intelligence that surfaces emerging threats, vulnerability disclosures, and supply chain risk indicators before they hit mainstream news. TeamPCP was flagged in Alpha Signal’s threat feed 72 hours before the first public disclosure.

Get Alpha Signal Pro → — Real-time threat intelligence, curated security signals, and early warning for supply chain attacks targeting your stack.

Related Articles

• Securing Kubernetes Supply Chains with SBOM & Sigstore
• Open Source Security Monitoring for Developers
• Kubernetes Secrets Management: A Security-First Guide
• Backup & Recovery: Enterprise Security for Homelabs

Frequently Asked Questions