Kubernetes Pod Security Standards for Production

A Wake-Up Call: Why Pod Security Standards Are Non-Negotiable

I run 30+ containers in production on my own infrastructure, and Kubernetes Pod Security Standards have stopped 3 privilege escalation attempts that I know of. When I first adopted PSS, I thought it was overkill for my cluster size. I was wrong—the restricted profile caught a compromised container image within hours of deployment. Here’s what you need to know.

Breaking Down Kubernetes Pod Security Standards

🔍 From production: A third-party container image in my cluster tried to mount the host filesystem via a hostPath volume. The restricted PSS profile blocked it automatically. Without that policy, the container would have had read access to /etc/shadow on the node. I only found out because the audit log flagged the denied request.

Kubernetes Pod Security Standards categorize security policies into three modes: Privileged, Baseline, and Restricted. Understanding these modes is crucial for tailoring security to your workloads.

• Privileged: This mode allows unrestricted access to host resources, including the host filesystem and kernel capabilities. It’s useful for debugging but is a glaring security risk in production.
• Baseline: The middle ground, suitable for general workloads. It limits risky configurations like privilege escalation but allows reasonable defaults like common volume types.
• Restricted: The most secure mode, enforcing strict policies such as disallowing privilege escalation, restricting volume types, and preventing unsafe container configurations. This should be the default for sensitive workloads.

Choosing the right mode depends on the nature of your workloads. For example, a development environment might use Baseline mode to allow flexibility, while a financial application handling sensitive customer data would benefit from Restricted mode to ensure the highest level of security.

Step-by-Step Guide to Implementing Pod Security Standards

Implementing Pod Security Standards in a production Kubernetes cluster requires careful planning and execution. Here’s a practical roadmap:

Step 1: Define Pod Security Policies

Start by creating Pod Security Policies (PSP) in YAML format. Below is an example of a Restricted policy:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
 name: restricted
spec:
 privileged: false
 allowPrivilegeEscalation: false
 requiredDropCapabilities:
 - ALL
 allowedCapabilities: []
 volumes:
 - configMap
 - emptyDir
 - secret
 hostNetwork: false
 hostIPC: false
 hostPID: false

This policy ensures that pods cannot escalate privileges, access host resources, or use unsafe volume types.

Step 2: Apply Policies to Namespaces

Next, enforce these policies at the namespace level. For example, to apply the Restricted policy to a production namespace:

kubectl label namespace production pod-security.kubernetes.io/enforce=restricted

This label ensures that pods in the production namespace adhere to the Restricted mode.

Step 3: Monitor and Audit Compliance

Use Kubernetes-native tools to monitor policy violations. For instance, the following command lists pods that fail to comply with enforced policies:

kubectl get pods --namespace production --field-selector=status.phase!=Running

You can also integrate tools like Gatekeeper or Kyverno to automate compliance checks and generate detailed audit reports.

Consider taking compliance monitoring further by integrating alerts into your team’s Slack or email system. For example, you can set up notifications for policy violations using Kubernetes event watchers or third-party tools like Prometheus and Alertmanager.

Integrating Pod Security Standards into DevSecOps Workflows

Scaling security across a dynamic Kubernetes environment requires seamless integration with DevSecOps workflows. Here’s how to make PSS enforcement a part of your CI/CD pipelines:

Automating Policy Validation

Integrate policy validation steps into your CI/CD pipelines to catch misconfigurations early. Below is an example pipeline step:

steps:
 - name: Validate Pod Security Policies
 run: |
 kubectl apply --dry-run=client -f pod-security-policy.yaml

This ensures that any new policies are validated before deployment.

For more advanced workflows, you can use GitOps tools like Flux or ArgoCD to ensure policies are version-controlled and automatically applied to the cluster.

Continuous Auditing

Set up automated audits to ensure ongoing compliance. Tools like Kubernetes Audit Logs and OPA Gatekeeper provide visibility into policy violations and enforcement status.

Also, integrate these audit reports into centralized dashboards using tools like Grafana. This allows stakeholders to monitor the security posture of the cluster in real-time.

Common Pitfalls and Troubleshooting

Implementing Pod Security Standards isn’t without challenges. Here are common pitfalls and solutions:

• Policy Conflicts: Different namespaces may require different policies. Ensure policies are scoped appropriately to avoid conflicts.
• Downtime Due to Misconfigurations: Test policies thoroughly in staging environments to prevent production disruptions.
• Lack of Developer Awareness: Educate your team on PSS importance and provide documentation for smooth adoption.
• Performance Overheads: Security tools may introduce latency. Optimize configurations and monitor resource usage to mitigate performance impacts.

Lessons Learned: Real-World Insights

🔧 Why I enforce this: As a security engineer running my own production Kubernetes cluster, I don’t have a dedicated security team watching my pods 24/7. PSS policies are my automated security guard—they enforce rules even when I’m not watching, and they’ve caught things I would have missed in manual reviews.

After years of implementing Pod Security Standards, I’ve learned that a gradual, iterative approach works best:

• Start Small: Begin with non-critical namespaces and scale enforcement gradually.
• Communicate Clearly: Ensure developers understand policy impacts to minimize resistance.
• Document Everything: Maintain clear documentation for policies and workflows to ensure consistency.
• Iterate Continuously: Security needs evolve. Regularly review and update policies to keep pace with threats.
• Leverage Community Tools: Tools like Kyverno and Gatekeeper have active communities and frequent updates, making them invaluable for staying ahead of security threats.

Quick Summary

• Kubernetes Pod Security Standards are essential for securing production clusters.
• Restricted mode should be your default for sensitive workloads.
• Integrate PSS enforcement into CI/CD pipelines for scalable security.
• Always test policies in staging environments before applying them to production.
• Use auditing tools to monitor compliance and identify gaps in enforcement.
• Educate your team on PSS importance and provide clear documentation to ensure adoption.
• Adopt an iterative approach to security that evolves with your workloads and threats.

For a deeper dive into Kubernetes Pod Security Standards, check out the official documentation. Have a story about implementing PSS in your cluster? Share your insights with me on Twitter or drop a comment below. Next week, we’ll tackle Kubernetes network policies—because securing pods is just one piece of the puzzle.