Secrets Management: Stop Leaks, Not Just Store Them

Secrets management is less about hiding passwords and more about controlling the blast radius of compromised credentials.

Let’s look at a typical application flow that needs to access a database.

graph TD
    A[Application Pod] --> B{Kubernetes API};
    B --> C[etcd];
    C --> B;
    B --> D[Kubelet];
    D --> A;
    A --> E[Database];

Here, the application pod needs credentials to talk to the database. In a naive setup, these credentials might be baked directly into the application’s container image or, more commonly, stored as a Kubernetes Secret object.

A Kubernetes Secret is just a base64 encoded blob of data stored in etcd. While it’s not truly encrypted at rest in etcd by default, it’s not directly accessible from outside the cluster without specific permissions. The application pod then mounts this Secret as a file or injects it as an environment variable.

The problem arises when this application pod is compromised. If an attacker gains access to the pod, they can easily read the mounted secret file or environment variable. Now they have the database credentials. From there, they can pivot to the database, access sensitive data, or even manipulate it. This is the "blast radius" problem: a compromise in one place (the application pod) directly grants access to another critical system (the database).

The goal of modern secrets management is to decouple secrets from the applications that use them and to introduce layers of control and auditing. This means not just storing secrets securely, but also distributing them securely and auditing their access.

Consider a workflow using HashiCorp Vault, a popular secrets management tool.

1. Application requests a dynamic secret: Instead of a static username/password, the application requests a temporary credential from Vault.
2. Vault generates a short-lived credential: Vault, integrated with the database, creates a new, unique credential with a very short TTL (Time To Live) specifically for that application instance.
3. Application uses the credential: The application uses the temporary credential to access the database.
4. Credential expires: Once the TTL is reached, the credential is automatically revoked by Vault.

This dynamic approach drastically reduces the blast radius. If the application pod is compromised, the attacker only gets a credential that will expire in minutes or hours, not one that can be used indefinitely.

Here’s a simplified conceptual diagram:

graph TD
    A[Application Pod] --> B{Vault API};
    B --> C[Vault Server];
    C --> D[Database Auth Method];
    D --> E[Database];
    E --> D;
    D --> C;
    C --> B;
    B --> A;
    A --> F[Database];

The application doesn’t store any long-lived credentials. It only needs a way to authenticate itself to Vault (e.g., using a Kubernetes Service Account token or a specific Vault token).

The core levers you control in a system like this are:

• Secret Storage: Where are the secrets physically stored? (e.g., etcd for Kubernetes Secrets, a dedicated Vault cluster).
• Secret Distribution: How do applications get their secrets? (e.g., mounted files, environment variables, direct API calls).
• Secret Rotation: How often are secrets changed? (e.g., manual rotation, automated rotation policies).
• Access Control: Who or what is allowed to read which secrets? (e.g., Kubernetes RBAC, Vault policies).
• Auditing: Who accessed which secret, when, and from where?

The most surprising true thing about secrets management is that the hardest part isn’t securely storing the secret itself, but managing the lifecycle and access patterns of those secrets across a distributed system.

Let’s say you’re using Vault with its Kubernetes auth method. Your application pod’s ServiceAccount token is used to authenticate to Vault. Vault then verifies this token with the Kubernetes API server. If valid, Vault issues a short-lived Vault token. This Vault token then grants the application access to a specific "path" within Vault where its secrets are stored.

Here’s a snippet of a Vault policy that might grant access:

# Allow access to secrets for pods with the 'app-worker' service account
path "secret/data/myapp/*" {
  capabilities = ["read"]
  # This is a simplified example; in reality, you'd use KV v2 and specific versions
}

And the Kubernetes ServiceAccount definition:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: default

The application would then be configured to use the Kubernetes auth method, providing its ServiceAccount token to Vault. Vault, upon successful authentication, would issue a token that allows it to read from secret/data/myapp/db_credentials.

When an attacker compromises the application pod, they might get hold of the ServiceAccount token. However, this token is only useful for authenticating to Vault. Vault’s policies are designed to limit what that authenticated identity can do. If the attacker tries to access a secret path not allowed by the policy, Vault will deny the request. This is where the "blast radius" is contained.

The one thing most people don’t know is that by default, many Kubernetes Secret objects are only base64 encoded, not encrypted at rest in etcd. If etcd is compromised, all your secrets are exposed in plain text. This is why enabling encryption at rest for etcd is a crucial baseline security measure, but it’s often overlooked.

The next problem you’ll likely encounter is managing the lifecycle of secrets for ephemeral workloads like serverless functions or short-lived batch jobs.