.tech Podcast - Building and maintaining External Secrets Operator

Moritz Johner is a Senior Software Engineer at Form3. He has roughly 10 years experience in the tech industry. He comes from a non-conventional background, having studied Media Technology in University. He has worked in a variety of industries, including web agencies, startups and consulting. He is also a Linux Foundation trainer for Kubernetes. He joined Form3 a year ago and works on providing cloud agnostic Kubernetes experience for product teams.

Moritz's journey into open-source began in 2018 when he was working in consulting and had to build a Kubernetes platform that would work across AWS accounts. Most projects did not support assuming a role in another AWS account, so Moritz contributed this functionality to the various OSS projects he was using. Eventually, Moritz provided this same feature to kubernetes-external-secrets which was the de-facto secrets synching solution at the time. He then gradually became more involved in this project and helped take on the work of merging multiple secrets solutions together.

Lucas Severo Alves is a Software Engineer at RedHat, focusing on open-source. He has a background in DevOps, SRE and system administration. His introduction to open-source was also on the kubernetes-external-secrets project that Moritz was working on. He was excited to focus on a single solution for managing secrets across platforms, which is something that he was focused on as part of his role.

Lucas explains that secrets management encompasses everything you do with secrets:

• how you rotate them
• how you give access to a new secret
• how you give access to a new team member
• how you respond to leaked secrets

The word secret itself in technology refers to credentials, passwords, keys, tokens, certificates. Anything that is used as a key to a lock is a secret, either by allowing people to do something or give access to a restricted resource. Everything that you around controlling and protecting the secret is secrets management.

In the context of Kubernetes, secrets management is:

• how you give access to your secrets inside Kubernetes
• how you control access to secrets inside your Kubernetes cluster
• how you isolate access to the secret in the context of multi-tenant applications inside Kubernetes

Moritz explains that although it sounds like pretty simple problem to solve, secrets management is difficult. Secrets are distributed in many places, from developer machines, to CI/CD pipelines and remote environments. Tokens can also have different expiration times and might be dependent on external entities for signing as well.

Secrets management is a huge space and the project Moritz and Lucas work on, external-secrets-operator is simply a tool in this space.

Moritz explains to us that external-secrets-operator is built around the idea of pulling secrets from some secure vault/storage into the cluster. This allows us to pull secrets into the cluster and then consume them with the standard Kubernetes mechanisms such as: mounting a file, reading environment variables or as an ingress resource.

Companies structure their tenants and resource access very differently. Some organisations share one vault across engineering teams, others have one instance per team or multiple instances per team. There are a lot of different possibilities in how you can structure secrets management. External secrets operator was designed with this in mind and helps you map your requirements into Kubernetes. The only prerequisite is that secrets must be stored in a secure place.

Lucas adds that its important to keep secrets synchronised with the vault, which is the single source of truth for these secrets. External secrets operator does this recurrently, allowing you to change secrets in one place. This is extremely useful in the case of leaks, where you might have to rotate secrets very quickly. There are also two highly anticipated features in Beta right now: push secrets and secrets generation.

Moritz explains that external secrets operator supports different cluster topologies. A common solution is to have a central cluster which manages other workload clusters. These clusters often require secrets to be injected to them. The project includes providers for Kubernetes, all the 3 clouds and also HashiCorp Vault. By considering Kubernetes itself a provider, external secrets operator is able to get secrets from one cluster and send it forward to others.

Lucas breaks down how to get started with external secrets operator into a few easy steps:

1. You decide on a secure vault that you trust with your secrets. All secrets that you use in the project consider this vault the single source of truth.
2. An application that needs this secret will need it to be available in its Kubernetes cluster. You install external-secrets-operator with Helm.
3. You create a custom resource with YAML for the external secrets provider, including how to authenticate to it.
4. You create another custom resource for the operator itself, including which secret to get, how it will be named and where to put it.

After this, external secrets operator will take care of fetching and updating the secret it is in charge of and you don't need to worry about these aspects any longer. The only problem that you must solve yourself is where to keep the initial secret that unlocks the vault that you need to pass to the operator. Moritz recommends to use service accounts with Workload Identity, which is available in the major cloud providers, and use that token to authenticate the secrets manager. There are steps on the project's official documentation on the commands and configuration you will need to run.

The oldest project, dating back to 2017, is sealed-secrets. This project was the only solution for a long time. It allows you to encrypt secrets, store them in a Git repository and then decrypt the secrets with a controller inside the cluster. The disadvantage of this tool is that Git repositories can be leaked. Depending on the encryption strength, these files be decrypted. However, you still need to manage the encryption and decryption keys, which are not stored centrally, as you cannot commit them to the repository itself.

Another project is the secrets-store-csi-driver. This project allows you to mount the secret directly inside a pod, without having Kubernetes native secrets.

The main choice you will have to make is what requirements you have when it comes to compliance and company policy. Moritz and Lucas get a lot of questions about which tool to use, but there is no best approach, as each tool will require some tradeoffs. External secrets operator now has over 200 contributors and has endeavoured to bring together multiple projects, making it easy to support new providers.

Moritz explains that the most difficult aspect of being an open-source maintainer is keeping up with the stream of messages and requests on Slack and GitHub. This has taught him to manage his time and prioritise tasks better, dedicating strict time windows to the open-source project. He is also an introvert, so he has grown from a social skills perspective too.

Lucas shares that being a maintainer can mean something different to different people. For example, external secrets operator has a maintainer that only maintains the IBM part of the project and he focuses on testing and client library maintenance. Other maintainers focus on PR review and rotate hosting the community meeting, as well as bringing visibility to the project. Time commitment varies throughout the year, as life sometimes gets in the way, but it's important to coordinate with other maintainers. Both Lucas and Moritz agree that dedicating their time to open-source is very rewarding and worth the effort.

Moritz will be speaking about external secrets operator on the KubeCon EU 2023 stage on April 19th 2023. OpenCollective has made it possible for the project to get funding, making it possible for the maintainers to meet in Amsterdam. The maintainers of external secrets operator also have a dedicated booth at KubeCon EU. Don't hesitate to come and say hello at booth K23 if you are attending KubeCon EU! 👋