Every application eventually needs to hold onto something sensitive — a database password, an API key, a TLS certificate. Kubernetes Secrets exist for exactly this, but I want to be upfront about something a lot of tutorials gloss over: a default Kubernetes Secret is base64-encoded, not encrypted, and base64 is not a security mechanism — it’s trivially reversible. In this guide, I’ll cover how Secrets actually work, how to create and consume them properly, and — critically — how to get real encryption and secure secrets management on AWS EKS.
What a Secret Actually Is
A Secret is a Kubernetes object similar to a ConfigMap, but intended for sensitive data. Structurally, data fields are base64-encoded; functionally, Kubernetes treats Secrets slightly differently from ConfigMaps — they’re not written to disk on nodes unless a pod using them is scheduled there (with tmpfs-backed volumes for mounted secrets), and kubectl get secret -o yaml shows the encoded (not decoded) values by default.
But base64 encoding provides zero confidentiality. Anyone with API access to read the Secret object, or access to the underlying etcd datastore, can trivially decode it:
echo "cGFzc3dvcmQxMjM=" | base64 -d
# password123
This matters enormously for how you think about Secret security — the real protections come from RBAC (who can read Secret objects), encryption at rest (protecting the etcd data store itself), and ideally not storing raw secret values in Kubernetes or Git at all, which I’ll cover below.
Creating Secrets
Imperatively:
kubectl create secret generic db-credentials \
--from-literal=username=appuser \
--from-literal=password='S3cur3P@ssw0rd!' \
-n production
From files:
kubectl create secret generic tls-cert \
--from-file=tls.crt=./server.crt \
--from-file=tls.key=./server.key \
-n production
Declaratively (values must be base64-encoded manually):
echo -n 'appuser' | base64
echo -n 'S3cur3P@ssw0rd!' | base64
apiVersion: v1
kind: Secret
metadata:
name: db-credentials
namespace: production
type: Opaque
data:
username: YXBwdXNlcg==
password: UzNjdXIzUEBzc3cwcmQh
kubectl apply -f db-credentials.yaml
A cleaner declarative alternative — stringData lets you write plaintext in the manifest, and Kubernetes handles the base64 encoding for you on the way in (it’s still stored as base64 in etcd, this is purely a convenience for authoring):
apiVersion: v1
kind: Secret
metadata:
name: db-credentials
namespace: production
type: Opaque
stringData:
username: appuser
password: S3cur3P@ssw0rd!
I want to be clear: writing raw secret values into a YAML file at all — even with stringData — is a bad practice if that file ends up in Git. I’ll cover the proper alternative shortly. This section exists so you understand the mechanics; production usage should skip straight to External Secrets or Sealed Secrets below.
Secret Types
Kubernetes has several built-in Secret types beyond the generic Opaque:
apiVersion: v1
kind: Secret
metadata:
name: tls-cert
type: kubernetes.io/tls
data:
tls.crt: <base64-cert>
tls.key: <base64-key>
kubectl create secret tls tls-cert --cert=server.crt --key=server.key -n production
apiVersion: v1
kind: Secret
metadata:
name: ecr-registry-creds
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: <base64-docker-config>
kubectl create secret docker-registry ecr-registry-creds \
--docker-server=123456789012.dkr.ecr.us-east-1.amazonaws.com \
--docker-username=AWS \
--docker-password=$(aws ecr get-login-password --region us-east-1) \
-n production
Note that ECR passwords from get-login-password expire after 12 hours — for long-running clusters, you either need to refresh this Secret on a schedule (a CronJob is a common pattern) or, better, use IAM Roles for Service Accounts (IRSA) or EKS Pod Identity so pods pull images without needing a stored Docker credential at all, via the node’s IAM role having ECR pull permissions.
Consuming Secrets in Pods
As environment variables:
spec:
containers:
- name: app
image: myapp:1.0
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-credentials
key: password
As mounted files:
spec:
containers:
- name: app
image: myapp:1.0
volumeMounts:
- name: db-creds
mountPath: /etc/secrets/db
readOnly: true
volumes:
- name: db-creds
secret:
secretName: db-credentials
defaultMode: 0400
I generally prefer mounted files over environment variables for secrets specifically: environment variables are more prone to accidental leakage — they show up in kubectl describe pod, in crash dumps, in child process environments, and in some logging frameworks that helpfully dump the full environment on error. A mounted file with restrictive permissions (0400) is a smaller blast radius.
kubectl apply -f app-with-secret.yaml
kubectl exec -it deploy/app -n production -- cat /etc/secrets/db/password
Encryption at Rest (Critical for Production)
By default, Secrets in etcd are only base64-encoded, meaning anyone with direct etcd access (or an etcd backup) can read them in plaintext. On EKS, enable envelope encryption using AWS KMS:
aws kms create-key --description "EKS secrets encryption key"
aws eks associate-encryption-config \
--cluster-name production-cluster \
--encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:us-east-1:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"}}]'
This can also be set at cluster creation time via eksctl or Terraform. Once enabled, Secret data is encrypted with the KMS key before being persisted to etcd — meaningfully raising the bar for anyone who somehow gains etcd or backup access without also having IAM/KMS access.
Verify encryption status:
aws eks describe-cluster --name production-cluster --query 'cluster.encryptionConfig'
RBAC for Secrets — Lock This Down Tightly
Secret read access should be one of the most tightly scoped permissions in your cluster:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
resourceNames: ["db-credentials"]
Scoping by resourceNames restricts access to a specific Secret rather than all Secrets in the namespace — meaningfully better than a blanket get/list/watch on the whole secrets resource type, especially in shared namespaces.
Audit who can read Secrets across your cluster:
kubectl auth can-i list secrets --as=system:serviceaccount:production:some-app -n production
The Real Production Pattern: External Secrets Operator
Storing raw secret values in Kubernetes manifests — even encrypted at rest — still means the source of truth for that secret often lives in a Git repo (in plaintext, if not using something like Sealed Secrets), which is a much bigger attack surface than a proper secrets manager with its own access controls, rotation, and audit logging.
The production-grade pattern on AWS is to keep secrets in AWS Secrets Manager (or Parameter Store) and sync them into Kubernetes Secrets automatically using the External Secrets Operator, so nothing sensitive ever touches Git.
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets --create-namespace
Configure a ClusterSecretStore pointing at AWS Secrets Manager, authenticated via IRSA:
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: aws-secrets-manager
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa
namespace: external-secrets
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-secrets-sa
namespace: external-secrets
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/external-secrets-role
Then define what gets synced:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: db-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: prod/database
property: username
- secretKey: password
remoteRef:
key: prod/database
property: password
kubectl apply -f cluster-secret-store.yaml -f external-secret-db.yaml
kubectl get externalsecret db-credentials -n production
kubectl get secret db-credentials -n production
Now the actual secret value lives only in AWS Secrets Manager, benefiting from its native rotation support, fine-grained IAM policies, and audit trail via CloudTrail — the Kubernetes Secret is just a synced, ephemeral projection of it, refreshed hourly per refreshInterval.
Alternative: Sealed Secrets
If you specifically want to keep secret material committed to Git (encrypted) rather than pulling from an external store, Bitnami’s Sealed Secrets lets you encrypt a Secret client-side into a SealedSecret CRD that only your cluster’s private key can decrypt:
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml
kubeseal --format yaml < db-credentials.yaml > db-credentials-sealed.yaml
db-credentials-sealed.yaml is now safe to commit to Git — it can only be decrypted by the specific cluster’s Sealed Secrets controller, which then materializes it back into a normal Secret object.
Troubleshooting
# Confirm a Secret exists and check its keys (not values)
kubectl get secret db-credentials -n production -o jsonpath='{.data}' | jq 'keys'
# Decode a specific value for debugging (use with caution, avoid pasting into shared terminals/logs)
kubectl get secret db-credentials -n production -o jsonpath='{.data.password}' | base64 -d
# ExternalSecret not syncing
kubectl describe externalsecret db-credentials -n production
kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets --tail=50
Common Mistakes
- Treating base64 encoding as encryption and being surprised anyone with
get secretRBAC access can trivially read plaintext values. - Committing raw Secret manifests (even with
stringData) to Git without Sealed Secrets or an external secrets manager. - Not enabling KMS envelope encryption for etcd on EKS, leaving Secrets effectively in plaintext in etcd backups.
- Using environment variables for highly sensitive secrets when mounted files with restrictive permissions offer a smaller leak surface.
- Forgetting ECR docker-registry secrets expire and need refreshing, or better, switching to IAM-based image pull via node roles or IRSA instead.
Best Practices
- Enable KMS-based envelope encryption for Secrets on every EKS cluster — this should be a non-negotiable baseline, not an afterthought.
- Use External Secrets Operator (or Sealed Secrets if you need Git-committed encrypted material) instead of hand-authoring raw Secret manifests.
- Scope RBAC on Secrets as narrowly as possible — prefer
resourceNamesrestrictions over blanket namespace-wide access. - Prefer mounted-file consumption over environment variables for highly sensitive values.
- Rotate secrets regularly, and lean on your secrets manager’s native rotation support rather than manual rotation processes.
Summary
A Kubernetes Secret is base64-encoded, not encrypted — real protection comes from etcd encryption at rest (via KMS on EKS), tight RBAC scoping, and, ideally, never storing raw secret values in Kubernetes manifests or Git in the first place. The production-grade pattern is syncing from AWS Secrets Manager via the External Secrets Operator, giving you proper rotation, audit trails, and a single source of truth outside your cluster and outside your Git history. Get encryption at rest and RBAC right as a baseline, then build toward External Secrets or Sealed Secrets as your actual secret-management architecture — not raw kubectl create secret commands scattered across runbooks.
