Kubernetes Notes

Appearance

How does Kubernetes manage configuration and secrets, and what are best practices for using ConfigMaps and Secrets securely?

Lets break down how Kubernetes manages configuration and sensitive data using ConfigMaps and Secrets, and detail the architectural best practices required to secure them in a production environment.

The Core Concept: Decoupling Configuration ​

Kubernetes follows the cloud-native principle of separating configuration from application code (The Twelve-Factor App). This ensures your container images remain immutable and portable across different environments (Dev, Staging, Production).

To achieve this, Kubernetes provides two primary API resources:

  1. ConfigMaps: For non-confidential data (property files, config flags).
  2. Secrets: For sensitive data (passwords, tokens, keys).

1. Architecture of Configuration & Secrets ​

ConfigMaps ​

A ConfigMap stores data as key-value pairs. Ideally, these represent individual environment variables or entire configuration files (like nginx.conf or game.properties).

  • Storage: They are stored in etcd as plain text.
  • Size Limit: They are designed for small data chunks; the total size is limited to 1 MiB.

Secrets ​

Secrets are structurally similar to ConfigMaps but have specific semantics for security.

  • Encoding vs. Encryption: By default, Secrets are stored as base64-encoded strings in the API. Base64 is encoding, not encryption; it provides no confidentiality against anyone who can access the API or etcd.
  • Node-Level Security:
    • Transmission: A Secret is only sent to a Node if a Pod scheduled on that Node requires it.
    • Storage (Linux): On Linux nodes, Secrets mounted as volumes are stored in tmpfs (RAM-backed filesystems), ensuring they are never written to durable storage on the worker node.
    • Storage (Windows): On Windows nodes, Secrets are written to local disk (as tmpfs is unavailable), requiring reliance on file ACLs and volume-level encryption (e.g., BitLocker) for protection.

2. Consumption Patterns: Volumes vs. Environment Variables ​

You can inject ConfigMaps and Secrets into Pods in two primary ways. From an engineering and security standpoint, Volumes are preferred over Environment Variables.

Method A: Environment Variables ​

You can map keys directly to container environment variables using valueFrom.

  • Pros: Simple for legacy applications.
  • Cons (Operational): Environment variables are static. If you update the ConfigMap/Secret, the running Pod will not see the change until you restart the Pod (triggering a rollout).
  • Cons (Security): Environment variables can unintentionally leak sensitive data through crash dumps, logs, or child processes.

You can mount ConfigMaps and Secrets as files within the container.

  • Automatic Updates: When you update a ConfigMap or Secret in the API, the kubelet eventually syncs the change to the mounted file in the container. This allows for configuration rotation without restarting the Pod.
    • Note on Latency: The update is eventually consistent. The delay depends on the kubelet sync period + cache TTL (typically roughly a minute).
  • Security: You can set strict POSIX permissions (e.g., mode: 0400) on the mounted files.

3. Security Best Practices ​

Securing Secrets is a multi-layered responsibility. Relying on default settings is insufficient for production.

A. Enable Encryption at Rest (Critical) ​

By default, Secrets are stored in plain text in etcd. If an attacker gains access to the etcd files or a backup, they own your cluster.

  • Solution: Configure the EncryptionConfiguration resource in the API Server.
  • Mechanism: This instructs the API server to encrypt resources before writing them to etcd.
  • Provider Choice:
    • Avoid secretbox or aescbc with local keys if possible, as the key lives on the control plane disk.
    • Use KMS v2 (Key Management Service): This is the production standard. It uses an envelope encryption scheme where a remote KMS (like AWS KMS, Google Cloud KMS, or HashiCorp Vault) manages the Key Encryption Key (KEK), while Kubernetes handles the Data Encryption Keys (DEKs).

B. RBAC and Principle of Least Privilege ​

Access to Secrets implies root-equivalent power.

  • Restrict watch and list: Granting list permission on Secrets effectively allows a user to decrypt and read all Secrets in that namespace. Restrict access to get (by specific name) whenever possible.
  • Cluster-Admin Risks: Avoid granting broad access to the kube-system namespace, as it contains tokens that can grant super-user access.

C. Immutable Resources ​

If your configuration should not change during the life of a Pod, mark your ConfigMaps and Secrets as Immutable by setting immutable: true.

  • Benefit: It protects against accidental updates that could break applications.
  • Performance: It significantly reduces load on the API server because Kubelets do not need to "watch" these resources for changes.

D. External Secret Stores (CSI) ​

For high-security environments, avoid storing the "source of truth" for secrets in Kubernetes at all.

  • Secrets Store CSI Driver: Use this to mount secrets directly from an external vault (like Azure Key Vault or HashiCorp Vault) into Pods as volumes. The Secret data exists in the Pod's memory but is managed entirely outside of etcd.

Summary Checklist for Engineers ​

  1. Encryption: Enable KMS v2 provider for encryption at rest in etcd.
  2. Mounts: Prefer mounting Secrets as Volumes rather than Environment Variables to avoid leakage and enable rotation.
  3. Immutability: Set immutable: true for static configuration to improve performance and safety.
  4. RBAC: Audit roles rigorously. Ensure developers cannot list secrets in production namespaces.
  5. GitOps: If using GitOps, use tools like SealedSecrets or Kustomize secretGenerators (which hash names to force rollouts) to manage Secret lifecycles safe.