Tag: Kubernetes production security

I’ve lost count of how many clusters I’ve audited where secrets were stored as plain base64 in etcd — which is encoding, not encryption. After cleaning up secrets sprawl across enterprise clusters for years, I can tell you: most teams don’t realize how exposed they are until it’s too late. Here’s the guide I wish I’d had when I started.

Introduction to Secrets Management in Kubernetes

Most Kubernetes secrets management practices are dangerously insecure. If you’ve been relying on Kubernetes native secrets without additional safeguards, you’re gambling with your sensitive data. Kubernetes makes it easy to store secrets, but convenience often comes at the cost of security.

Secrets management is a cornerstone of secure Kubernetes environments. Whether it’s API keys, database credentials, or TLS certificates, these sensitive pieces of data are the lifeblood of your applications. Unfortunately, Kubernetes native secrets are stored in plaintext within etcd, which means anyone with access to your cluster’s etcd database can potentially read them.

To make matters worse, most teams don’t encrypt their secrets at rest or rotate them regularly. This creates a ticking time bomb for security incidents. Thankfully, tools like HashiCorp Vault and External Secrets provide robust solutions to these challenges, enabling you to adopt a security-first approach to secrets management.

Another key concern is the lack of granular access controls in Kubernetes native secrets. By default, secrets can be accessed by any pod in the namespace unless additional restrictions are applied. This opens the door to accidental or malicious exposure of sensitive data. Teams must implement strict role-based access controls (RBAC) and namespace isolation to mitigate these risks.

Consider a scenario where a developer accidentally deploys an application with overly permissive RBAC rules. If the application is compromised, the attacker could gain access to all secrets in the namespace. This highlights the importance of adopting tools that enforce security best practices automatically.

To get started with secure secrets management, teams should evaluate their current practices and identify gaps. Are secrets encrypted at rest? Are they rotated regularly? Are access logs being monitored? Answering these questions is the first step toward building a solid secrets management strategy.

Vault: A Deep Dive into Secure Secrets Management

🔍 Lesson learned: During a production migration, we discovered that 40% of our Kubernetes secrets hadn’t been rotated in over a year — some contained credentials for services that no longer existed. I now enforce automatic rotation policies from day one. Vault’s lease-based secrets solved this completely for our database credentials.

HashiCorp Vault is the gold standard for secrets management. It’s designed to securely store, access, and manage sensitive data. Unlike Kubernetes native secrets, Vault encrypts secrets at rest and provides fine-grained access controls, audit logging, and dynamic secrets generation.

Vault integrates smoothly with Kubernetes, allowing you to securely inject secrets into your pods without exposing them in plaintext. Here’s how Vault works:

• Encryption: Vault encrypts secrets using AES-256 encryption before storing them.
• Dynamic Secrets: Vault can generate secrets on demand, such as temporary database credentials, reducing the risk of exposure.
• Access Policies: Vault uses policies to control who can access specific secrets.

Setting up Vault for Kubernetes integration involves deploying the Vault agent injector. This agent automatically injects secrets into your pods as environment variables or files. Below is an example configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-app
spec:
 template:
 metadata:
 annotations:
 vault.hashicorp.com/agent-inject: "true"
 vault.hashicorp.com/role: "my-app-role"
 vault.hashicorp.com/agent-inject-secret-config: "secret/data/my-app/config"
 spec:
 containers:
 - name: my-app
 image: my-app:latest

In this example, Vault injects the secret stored at secret/data/my-app/config into the pod. The vault.hashicorp.com/role annotation specifies the Vault role that governs access to the secret.

Another powerful feature of Vault is its ability to generate dynamic secrets. For example, Vault can create temporary database credentials that automatically expire after a specified duration. This reduces the risk of long-lived credentials being compromised. Here’s an example of a dynamic secret policy:

path "database/creds/my-role" {
 capabilities = ["read"]
}

Using this policy, Vault can generate database credentials for the my-role role. These credentials are time-bound and automatically revoked after their lease expires.

Common pitfalls when using Vault include misconfigured policies and insufficient monitoring. Always test your Vault setup in a staging environment before deploying to production. Also, enable audit logging to track access to secrets and identify suspicious activity.

External Secrets: Simplifying Secrets Synchronization

⚠️ Tradeoff: External Secrets Operator adds a sync layer between your secrets store and Kubernetes. That’s another component that can fail — and when it does, pods can’t start. I run it with high availability and aggressive health checks. The operational overhead is real, but it beats manually syncing secrets across 20 namespaces.

While Vault excels at secure storage, managing secrets across multiple environments can still be a challenge. This is where External Secrets comes in. External Secrets is an open-source Kubernetes operator that synchronizes secrets from external secret stores like Vault, AWS Secrets Manager, or Google Secret Manager into Kubernetes secrets.

External Secrets simplifies the process of keeping secrets up-to-date in Kubernetes. It dynamically syncs secrets from your external store, ensuring that your applications always have access to the latest credentials. Here’s an example configuration:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: my-app-secrets
spec:
 refreshInterval: "1h"
 secretStoreRef:
 name: vault-backend
 kind: SecretStore
 target:
 name: my-app-secrets
 creationPolicy: Owner
 data:
 - secretKey: config
 remoteRef:
 key: secret/data/my-app/config

In this example, External Secrets fetches the secret from Vault and creates a Kubernetes secret named my-app-secrets. The refreshInterval ensures that the secret is updated every hour.

Real-world use cases for External Secrets include managing API keys for third-party services or synchronizing database credentials across multiple clusters. By automating secret updates, External Secrets reduces the operational overhead of managing secrets manually.

One challenge with External Secrets is handling failures during synchronization. If the external secret store becomes unavailable, applications may lose access to critical secrets. To mitigate this, configure fallback mechanisms or cache secrets locally.

External Secrets also supports multiple secret stores, making it ideal for organizations with hybrid cloud environments. For example, you can use AWS Secrets Manager for cloud-native applications and Vault for on-premises workloads.

Production-Ready Secrets Management: Lessons Learned

Managing secrets in production requires careful planning and adherence to best practices. Over the years, I’ve seen teams make the same mistakes repeatedly, leading to security incidents that could have been avoided. Here are some key lessons learned:

• Encrypt Secrets: Always encrypt secrets at rest, whether you’re using Vault, External Secrets, or Kubernetes native secrets.
• Rotate Secrets: Regularly rotate secrets to minimize the impact of compromised credentials.
• Audit Access: Implement audit logging to track who accessed which secrets and when.
• Test Failures: Simulate secret injection failures to ensure your applications can handle them gracefully.

One of the most common pitfalls is relying solely on Kubernetes native secrets without additional safeguards. In one case, a team stored database credentials in plaintext Kubernetes secrets, which were later exposed during a cluster compromise. This could have been avoided by using Vault or External Secrets.

Case studies from production environments highlight the importance of a security-first approach. For example, a financial services company reduced their attack surface by migrating from plaintext Kubernetes secrets to Vault, combined with External Secrets for dynamic updates. This not only improved security but also streamlined their DevSecOps workflows.

Another lesson learned is the importance of training and documentation. Teams must understand how secrets management tools work and how to troubleshoot common issues. Invest in training sessions and maintain detailed documentation to help your developers and operators.

Advanced Topics: Secrets Management in Multi-Cluster Environments

As organizations scale, managing secrets across multiple Kubernetes clusters becomes increasingly complex. Multi-cluster environments introduce challenges like secret synchronization, access control, and monitoring. Tools like Vault Enterprise and External Secrets can help address these challenges.

In multi-cluster setups, consider using a centralized secret store like Vault to manage secrets across all clusters. Configure each cluster to authenticate with Vault using Kubernetes Service Accounts. Here’s an example of a Vault Kubernetes authentication configuration:

path "auth/kubernetes/login" {
 capabilities = ["create", "read"]
}

This configuration allows Kubernetes Service Accounts to authenticate with Vault and access secrets based on their assigned policies.

Monitoring is another critical aspect of multi-cluster secrets management. Use tools like Prometheus and Grafana to track secret usage and identify anomalies. Set up alerts for unusual activity, such as excessive secret access requests.

Conclusion: Building a Security-First DevSecOps Culture

This is the exact secrets management stack I run on my own infrastructure — Vault for high-security workloads, External Secrets for dynamic syncing, and encryption at rest as the baseline. Start by auditing what you have: run kubectl get secrets --all-namespaces and check when each was last rotated. That audit alone will tell you where your biggest gaps are.

Here’s what to remember:

• Always encrypt secrets at rest and in transit.
• Use Vault for high-security workloads and External Secrets for dynamic updates.
• Rotate secrets regularly and audit access logs.
• Test your secrets management setup under failure conditions.

Related Reading

• Kubernetes Security Checklist for Production (2026)
• GitOps Security Patterns for Kubernetes
• Kubernetes Pod Security Standards for Production
• Fortifying Kubernetes Supply Chains with SBOM and Sigstore

Want to share your own secrets management horror story or success? Drop a comment or reach out on Twitter—I’d love to hear it. Next week, we’ll dive into Kubernetes RBAC and how to avoid common misconfigurations. Until then, stay secure!

Frequently Asked Questions