Tag: production CI/CD

Last month I migrated two production clusters from GitHub Actions-only deployments to a hybrid GitOps setup with ArgoCD. The trigger? A misconfigured workflow secret that exposed an AWS key for 11 minutes before our scanner caught it. Nothing happened — this time. But it made me rethink how we handle the boundary between CI and CD.

Here’s what I learned about running both tools securely in production, and when each one actually makes sense.

GitOps: Let Git Be the Only Way In

GitOps treats Git as the single source of truth for your cluster state. You define what should exist in a repo, and an agent like ArgoCD or Flux continuously reconciles reality to match. No one SSHs into production. No one runs kubectl apply by hand.

The security model here is simple: the cluster pulls config from Git. The agent runs inside the cluster with the minimum permissions needed to apply manifests. Your developers never need direct cluster access — they open a PR, it gets reviewed, merged, and the agent picks it up.

This is a massive reduction in attack surface. In a traditional CI/CD model, your pipeline needs credentials to push to the cluster. With GitOps, those credentials stay inside the cluster.

Here’s a basic ArgoCD Application manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  source:
    repoURL: https://github.com/my-org/my-app-config
    targetRevision: HEAD
    path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app-namespace
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The selfHeal: true setting is important — if someone does manage to modify a resource directly in the cluster, ArgoCD will revert it to match Git. That’s drift detection for free.

One gotcha: make sure you enforce branch protection on your GitOps repos. I’ve seen teams set up ArgoCD perfectly, then leave the main branch unprotected. Anyone with repo write access can then deploy anything. Always require reviews and status checks.

GitHub Actions: Powerful but Exposed

GitHub Actions is a different animal. It’s event-driven — push code, open a PR, hit a schedule, and workflows fire. That flexibility is exactly what makes it harder to secure.

Every GitHub Actions workflow that deploys to production needs some form of credential. Even with OIDC federation (which you should absolutely be using — see my guide on securing GitHub Actions with OIDC), there are still risks. Third-party actions can be compromised. Workflow files can be modified in feature branches. Secrets can leak through step outputs if you’re not careful.

Here’s a typical deployment workflow:

name: Deploy to Kubernetes
on:
  push:
    branches:
      - main
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Configure kubectl
        uses: azure/setup-kubectl@v3
      - name: Deploy application
        run: kubectl apply -f k8s/deployment.yaml

Notice the environment: production — that enables environment protection rules, so deployments require manual approval. Without it, any push to main goes straight to prod. I always set this up, even on small projects.

The bigger issue is that GitHub Actions workflows are imperative. You’re writing step-by-step instructions that execute on a runner with network access. Compare that to GitOps where you declare “this is what should exist” and an agent figures out the rest. The imperative model has more moving parts, and more places for things to go wrong.

Where Each One Wins on Security

After running both in production, here’s how I’d break it down:

Access control — GitOps wins. The agent pulls from Git, so your CI system never needs cluster credentials. With GitHub Actions, your workflow needs some path to the cluster, whether that’s a kubeconfig, OIDC token, or service account. That’s another secret to manage.

Secret handling — GitOps is cleaner. You pair it with something like External Secrets Operator or Sealed Secrets and your Git repo never contains actual credentials. GitHub Actions has encrypted secrets, but they’re injected into the runner environment at build time — a compromise of the runner means a compromise of those secrets.

Audit trail — GitOps. Every change is a Git commit with an author, timestamp, and review trail. GitHub Actions logs exist, but they expire and they’re harder to query when you need to answer “who deployed what, and when?” during an incident.

Flexibility — GitHub Actions. Not everything fits the GitOps model. Running test suites, building container images, scanning for vulnerabilities, sending notifications — these are CI tasks, and GitHub Actions handles them well. Trying to force these into a GitOps workflow is pain.

Speed of setup — GitHub Actions. You can go from zero to deployed in an afternoon. GitOps requires more upfront investment: installing the agent, structuring your config repos, setting up GitOps security patterns.

The Hybrid Approach (What Actually Works)

Most teams I’ve worked with end up running both, and honestly it’s the right call. Use GitHub Actions for CI — build, test, scan, push images. Use GitOps for CD — let ArgoCD or Flux handle what’s running in the cluster.

The boundary is important: GitHub Actions should never directly kubectl apply to production. Instead, it updates the image tag in your GitOps repo (via a PR or direct commit to a deploy branch), and the GitOps agent picks it up.

This gives you:

• Full Git audit trail for all production changes
• No cluster credentials in your CI system
• Automatic drift detection and self-healing
• The flexibility of GitHub Actions for everything that isn’t deployment

One thing to watch: make sure your GitHub Actions workflow doesn’t have permissions to modify the GitOps repo directly without review. Use a bot account with limited scope, and still require PR approval for production changes.

Adding Security Scanning to the Pipeline

Whether you use GitOps, GitHub Actions, or both, you need automated security checks. I run Trivy on every image build and OPA/Gatekeeper for policy enforcement in the cluster.

Here’s how I integrate Trivy into a GitHub Actions workflow:

name: Security Scan
on:
  pull_request:
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t my-app:${{ github.sha }} .
      - name: Trivy scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: my-app:${{ github.sha }}
          severity: CRITICAL,HIGH
          exit-code: 1

The exit-code: 1 means the workflow fails if critical or high vulnerabilities are found. No exceptions. I’ve had developers complain about this blocking their PRs, but it’s caught real issues — including a supply chain problem in a base image that would have made it to prod otherwise.

What I’d Do Starting Fresh

If I were setting up a new production Kubernetes environment today:

1. ArgoCD for all cluster deployments, with strict branch protection and required reviews on the config repo
2. GitHub Actions for CI only — build, test, scan, push to registry
3. External Secrets Operator for credentials, never stored in Git
4. OPA Gatekeeper for policy enforcement (no privileged containers, required resource limits, etc.)
5. Trivy in CI, plus periodic scanning of running images

The investment in GitOps pays off fast once you’re past the initial setup. The first time you need to answer “what changed?” during a 2 AM incident and the answer is right there in the Git log, you’ll be glad you did it.