secrets management with external secrets operator

kubernetes secrets are base64-encoded, not encrypted. they sit in etcd, get mounted into pods, and have a habit of leaking into the places secrets should never be: git repositories, helm values files, ci logs, and the clipboard of whoever set up the cluster. teams that take secrets seriously end up with a real secret store - aws systems manager parameter store, secrets manager, vault, gcp secret manager - and then face a new problem. the secret lives in the store, but the workload needs a kubernetes secret to mount.

the gap between those two gets filled with glue: init containers that shell out to the store’s api, sidecars that poll it, ci jobs that copy values from one system into another. each of these is a place where a credential is handled in plaintext and a place where the sync can drift.

the external secrets operator closes that gap. it keeps the source of truth in the external store and continuously reconciles native kubernetes secrets from it. the secret never lives in git, and the cluster always reflects the current value.

how it works

the operator introduces two main custom resources. a SecretStore describes how to connect to a backend - the address, the auth method, the credentials. an ExternalSecret describes which keys to pull from that backend and what the resulting kubernetes secret should look like.

the controller watches ExternalSecret resources. for each one, it authenticates to the referenced store, fetches the requested values, and creates or updates a standard Secret in the same namespace. from the workload’s perspective there is nothing unusual to consume - it mounts an ordinary kubernetes secret. the operator is the only component that talks to the external store.

reconciliation is continuous. the ExternalSecret carries a refreshInterval, and on every tick the controller re-fetches and re-applies. if the value changed in the backend, the kubernetes secret is updated. if nothing changed, nothing happens.

SecretStore is namespace-scoped. ClusterSecretStore is the cluster-wide equivalent, letting ExternalSecret resources in any namespace reference a single shared backend configuration - useful when the connection details and auth are managed centrally by the platform team.

installing the operator

it installs via helm:

helm repo add external-secrets https://charts.external-secrets.io
helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace

the chart installs the controller, a webhook for validation, and a cert controller that manages the webhook’s certificates. it registers the SecretStore, ClusterSecretStore, ExternalSecret, and PushSecret crds. once it’s running, secrets are synced by creating store and external secret resources.

connecting to a backend

the SecretStore is where the provider and authentication live. pointing at aws systems manager parameter store:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: parameter-store
  namespace: production
spec:
  provider:
    aws:
      service: ParameterStore
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets

authentication is the part worth getting right. don’t hand the operator a static iam access key - that is just another long-lived credential to store and rotate, and putting it in the cluster reintroduces the problem the operator exists to solve. use irsa (iam roles for service accounts) instead: annotate the operator’s service account with a role arn, and eks exchanges the projected service account token for short-lived credentials scoped to that role. eks pod identity is the newer equivalent and behaves the same from the operator’s side.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/external-secrets

the role behind that arn should be scoped to exactly the parameters the operator needs - not ssm:* on *. parameter store organises secrets into a path hierarchy, which makes a least-privilege policy straightforward to express:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ssm:GetParameter", "ssm:GetParametersByPath"],
      "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/production/app/*"
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:eu-west-1:123456789012:key/0123abcd-4567-89ef-..."
    }
  ]
}

the kms:Decrypt statement matters. secrets should be stored as SecureString parameters, which encrypt the value with a kms key. without decrypt permission on that key the operator can read the parameter’s metadata but not its value, and the sync fails. scoping the policy to the /production/app/ prefix means a compromised operator in this namespace cannot read another team’s parameters.

for a configuration shared across namespaces, a ClusterSecretStore holds the same provider block once. because it is cluster-scoped, its serviceAccountRef must name both the service account and its namespace.

pulling a secret

with a store configured, an ExternalSecret declares what to sync:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-database
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: parameter-store
    kind: SecretStore
  target:
    name: app-database
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: /production/app/db-username
    - secretKey: password
      remoteRef:
        key: /production/app/db-password

this creates a kubernetes secret named app-database with username and password keys, read from the /production/app/db-username and /production/app/db-password parameters. each remoteRef.key is the full parameter name. every hour the controller re-fetches and reconciles.

creationPolicy: Owner sets the operator as the owner of the resulting secret via an owner reference. when the ExternalSecret is deleted, the managed secret is garbage collected with it. the alternatives are Merge, which writes keys into a secret managed by something else, and Orphan, which leaves the secret behind on deletion.

a parameter can also hold a json document rather than a single value, in which case property extracts one field. to sync every parameter under a path prefix without listing each one, use dataFrom with a find:

spec:
  dataFrom:
    - find:
        path: /production/app/
      rewrite:
        - regexp:
            source: "/production/app/(.*)"
            target: "$1"

this pulls every parameter under /production/app/ in one pass. the rewrite strips the prefix so /production/app/db-username becomes the secret key db-username - without it the key would contain slashes, which kubernetes secret keys don’t allow. it’s convenient, but it couples the secret’s shape to whatever lives under that path: adding a parameter silently adds a key to the cluster. listing keys explicitly with data is more verbose and more predictable.

shaping the output

applications rarely want a secret as loose key-value pairs. they want a connection string, a config file, a specific env var name. the template block builds the output secret from the fetched values:

spec:
  target:
    name: app-database
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@db.production:5432/app?sslmode=require"
  data:
    - secretKey: username
      remoteRef:
        key: /production/app/db-username
    - secretKey: password
      remoteRef:
        key: /production/app/db-password

the fetched username and password become template variables, and the rendered DATABASE_URL is what lands in the secret. this keeps the assembly logic in the cluster declaration rather than baked into the application or a config map. templates can also render whole files - a .pgpass, a json credentials blob, a tls bundle assembled from separate cert and key entries.

pushing secrets the other way

the operator usually pulls, but PushSecret reverses the direction - it takes a kubernetes secret and writes it into the external store. this is useful when a secret is generated inside the cluster and needs to be persisted somewhere durable:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: generated-tls
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRefs:
    - name: parameter-store
      kind: SecretStore
  selector:
    secret:
      name: generated-tls
  data:
    - match:
        secretKey: tls.key
        remoteRef:
          remoteKey: /production/app/tls-key

a common case is a cert-manager-issued certificate that needs to be available to systems outside the cluster, or a password generated by an operator that should be backed up to parameter store so it survives a cluster rebuild. pushing needs a separate grant from reading: the role needs ssm:PutParameter on the target path - and kms:Encrypt on the key, since the pushed parameter should be written as a SecureString. push and pull together let the cluster participate in the secret lifecycle without ever being the only place a secret exists.

rotation

rotation is the whole point of keeping the source of truth external, and it’s where the model earns its keep. parameter store has no built-in rotation, so the new value is written by something else - a rotation lambda, a ci pipeline, an operator. whenever the parameter’s value changes, the external secrets operator picks it up on the next refresh and updates the kubernetes secret. the refreshInterval sets the upper bound on how stale a secret can be - a one-hour interval means the cluster reflects a rotated parameter within an hour.

the catch is that updating the secret does not restart the workload. a pod that read its secret as an environment variable at startup keeps the old value until it restarts - environment variables are a snapshot taken when the container starts. secrets mounted as volumes are different: the kubelet updates the mounted files within a minute or two of the secret changing, so an application that re-reads the file picks up the new value without a restart.

this distinction drives a design choice. for credentials that rotate, mount them as files and have the application reload them, or pair the external secret with a controller like stakater reloader that watches the secret and triggers a rolling restart when it changes. relying on env vars for a rotating secret means the parameter rotates, the kubernetes secret updates, and the running pods carry on with credentials that no longer work.

shortening refreshInterval reduces staleness but increases load on the backend - every interval is a GetParameter call per external secret, multiplied across the cluster. parameter store throttles at a default of 40 requests per second per account, so refreshing thousands of secrets on a thirty-second interval will hit that ceiling and start failing syncs. an interval measured in minutes to hours is usually right, tuned to how aggressively secrets actually rotate.

monitoring

the operator exposes prometheus metrics, and they are the only reliable signal that syncing is healthy. the most important is the sync status per external secret:

# external secrets that are not in a ready/synced state
externalsecret_status_condition{condition="Ready", status="False"} == 1

each ExternalSecret also carries a status condition queryable directly:

kubectl get externalsecret -A

the output shows a STATUS and READY column. a healthy secret reads SecretSynced and True. anything else - SecretSyncedError, an auth failure, a missing key - shows here and should be alerted on.

other useful series are the reconcile error counters and the time since last successful sync. alerting on a sync that hasn’t succeeded within a few multiples of its refresh interval catches a store that has gone unreachable before the stale secret causes an outage.

what to watch out for

silent sync failures. this is the failure mode that hurts. when a fetch fails - the backend is unreachable, auth expired, a key was renamed - the operator logs an error and stops updating that secret. but the existing kubernetes secret is left in place with its last good value. workloads keep running on the stale secret, everything looks fine, and the drift is invisible until the stale credential expires or the value was supposed to have changed. the ExternalSecret status flips to an error state, but nothing forces anyone to look at it. alerting on externalsecret_status_condition{condition="Ready", status="False"} is not optional - it is the only thing standing between a failed sync and a silent production incident.

a renamed key fails open, not closed. if someone renames or deletes a key in the backend that an ExternalSecret references, the sync errors and the old secret value persists. the cluster does not notice the secret is gone; it keeps serving the last value it saw. the source of truth and the cluster have diverged, and only the error condition reveals it.

deletion policy and accidental loss. creationPolicy: Owner ties the secret’s lifecycle to the external secret. deleting the ExternalSecret - or a misconfigured gitops prune - deletes the managed secret out from under running workloads. for secrets that must survive churn, Orphan or deletionPolicy: Retain keeps the secret in place even when the external secret is removed.

the operator is a single point of access. every secret in the cluster flows through one controller with broad credentials to the backend. its service account is effectively able to read every secret it’s configured to sync. it should run with tight rbac, its backend role should be scoped to only the paths it needs, and its own logs should never print secret values - which it doesn’t by default, but custom templates and debug logging can change that.

backend rate limits. a large cluster with hundreds of external secrets all refreshing on a short interval generates a steady stream of api calls. cloud secret stores meter and bill these, and some throttle aggressively. staggering refresh intervals and keeping them as long as the rotation policy allows avoids both the bill and the throttling-induced sync failures.

provider auth expiry. the short-lived credentials that make irsa safe also expire and are renewed continuously. if the operator’s service account annotation, the iam role’s trust policy, or the oidc provider binding is misconfigured, auth fails on renewal and every sync through that store stops at once. this shows up as a wave of secrets going False together - a different signature from a single renamed parameter, and worth a distinct alert.

references

[1] external secrets operator documentation. “introduction.”
external-secrets.io/latest/introduction/overview

[2] external secrets operator documentation. “secretstore.”
external-secrets.io/latest/api/secretstore

[3] external secrets operator documentation. “externalsecret.”
external-secrets.io/latest/api/externalsecret

[4] external secrets operator documentation. “aws parameter store provider.”
external-secrets.io/latest/provider/aws-parameter-store

[5] aws documentation. “aws systems manager parameter store.”
docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html

[6] external secrets operator documentation. “pushsecret.”
external-secrets.io/latest/api/pushsecret

[7] kubernetes documentation. “secrets.”
kubernetes.io/docs/concepts/configuration/secret