Kubernetes: The Truth No One Wants To Tell About Secret
Do you really need Secret objects?
One may respond “OF COURSE YES, Akintola !!!”. But hold on for a minute friend.
A lot of people argue we should use Kubernetes Secrets for application needs. When I started working with Kubernetes I thought the need for Secret Objects was obvious.
But after a short time period and after working on multiple projects, I started doubting its actual need in a process of deploying applications on production. Let’s elaborate.
What is Secret Object?
I want us to start with a clear understanding of what a Secret object is on Kubernetes. I think the official documentation gives a clear definition.
A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you don’t need to include confidential data in your application code. — Kubernetes Official documentation
Let’s have a look at an example of a Secret YAML file:
apiVersion: v1
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: { ... }
creationTimestamp: 2020-01-22T18:41:56Z
name: mysecret
namespace: default
resourceVersion: "164619"
uid: cfee02d6-c137-11e5-8d73-42010af00002
type: Opaque
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2RmKubernetes mechanisms to keep Secret data safe and secured
Secret Object has got some interesting characteristics that we need to explore.
- One pod cannot access the secrets of another.
- A secret is made available to a node if a pod is scheduled on that node and needs it.
- The maximum permissible size of secrets can only be 1 MB. This helps protect the apiserver memory resources and kubelet from abuse.
- The default View role doesn’t grant access to Secrets.
Those are some of the main security mechanisms Kubernetes implements to provide a baseline of security for secrets.
The Hard Truth About Secrets On Production
Now let’s see from an attacker's point of view, how things look like.
Let’s start with the vulnerabilities that the Kubernetes documentation talks about.
1- Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd).
Actually, secrets data are base64 encoded. But you only need to apply a reverse base64 et you’ll get the plain data.
2- Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd.
3- Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.
To mend those issues, Kubernetes recommends some practices to keep the secret data safe.
What does Kubernetes documentation suggest?
To avoid those traps, the documentation gives some recommendations.
First, you should enable Encryption at Rest for Secrets.
Secondly, you should enable or configure RBAC rules that restrict reading and writing the Secret.
Finally, we should also use mechanisms such as RBAC to limit which principals are allowed to create new Secrets or replace existing ones.
The Problems With Secrets On Production
That said, some issues still exist and things come to a head when we consider them in production. Let’s see.
4- GitOps has taken control over the way we create and maintain infrastructure. So in case, we need to create a secret to store sensitive data, are we going to add it to the main Ops repository?
If we don’t do so, then users (developers, admins, and so on) will have to work with the secrets through imperative commands. If we do then we open the door to different kinds of attacks.
5- Through containers that are in the same pods, we can access any secret inside the pod.
The 4rth issue is the one that catches the most of DevOps's attention: How can we make secret data available on Kubernetes Cluster and at the same time, follow GitOps best practices?
Here are some of the alternatives I’ve found so far and I’m gonna give them a try.
1- Hashicorp Vault:
HashiCorp Vault is an identity-based secrets and encryption management system. Vault provides encryption services that are gated by authentication and authorization methods.
The wonderful thing about Vault is that it’s an open-source tool. You can set up your own instance and manage it yourself. You can also rely on their Cloud Platform.
2- Cloud Providers' Secret Managers
Cloud Providers like AWS, GCP, and Azure provide you with Secret Manager you can use to keep secret data safe and use them on Kubernetes.
GCP has got Google Secret Manager. This tutorial shows how to get started using Google Secret Manager On GKE (Kubernetes For Google).
AWS offers AWS Secret Manager you can take advantage of to manage your secret data.
The same goes for Azure which proposes Key Vault, which can be integrated with Azure Kubernetes Service.
I’m pretty sure there are a lot of other solutions out there which propose the same services and fix better those issues.
Conclusion
There is no such a 100 % secured system we often say but it’s our duties as DevOps, and software engineers to find, build and implement best practices to improve our setup and provide more secure systems to our clients.
Kubernetes Secret is still useful (when it comes to accessing private docker repositories for example) but we have to combine them with more advanced tools and services to keep systems secured.
Thank you for reading me. I’d like to have your thought in the comments section.
Till next time, take care.
Cheers !!!
