DevOps

Integrating Hashicorp Vault with Kubernetes

Owen Pryga

• 8 min read
Share
Integrating Hashicorp Vault with Kubernetes

For a while now one of my largest projects in my homelab is my Kubernetes cluster. I always find it so fascinating working with applications at scale. With my Kubernetes cluster I can learn how to develop applications to horizontally scale, as well as load balancing between all of them. One of my biggest gripes though is secret management, and databases.

I configure my entire Kubernetes stack via GitOps with FluxCD. This allows me to have my entire Kubernetes cluster defined in a series of manifest files within a Git Repo. If my cluster were to die randomly say I messed up some settings, I can rebuild my cluster by just having it pull from the git repo. You can view my entire Kubernetes Configuration on my GitHub. One of the largest challenges that I have is how I can safely and securely get my secrets onto my cluster. Many people either manually add them into the cluster, which wont let you just easily copy the Kubernetes configuration via Git incase something happens. Another way you can approach this problem is to encrypt your secrets like I do via SOPS (Secrets OPerationS).

This is safe for the most part except when you forget to encrypt secrets. I have added in some safeguards. I have added in a pre-commit hook from onedr0p that will look for unencrypted secrets, and prevent you from committing. Another safeguard is adding .sops before my yaml extension so I know which files contains secrets. This is not really the best use case as it still can go through if I am not careful.

Another problem that I have is my ideology that everything on my cluster should be ephemeral, such that I can terminate anything at any time and not worry about losing data. In the past I ran CNPG (Cloud Native Postgres). Its a Kubernetes operator that allowed me to deploy postgres databases via manifest files. While actually using it though I found it extremely temperamental and it always filled up space quickly from logs, so I had to keep adding more storage to my Persistent Volumes, or making scripts to clear out some data. This was just not working for me and I needed to find a better solution.

Killing Many Birds With One Stone

One of the places that I was able to internship with had used Hashicorp Vault to store a lot of their secrets for deployment scripts. I learned really quickly that they had pretty decent integration with the Kubernetes API for authenticating Kubernetes services. I also learned that many people use Hashicorp Vault, I even came across a job listing once where they were looking for a Hashicorp Vault administrator. Crazy to think that Vault can be a whole job in of itself.

One of the bread and butter features of Hashicorp Vault is their dynamic secrets. You can connect it up to a database account to create short lived users. It will also manage the password vault uses so it will automatically rotate the secret of the vault account. With short lived users if your service account gets compromised it will rotate the secret automatically say every hour or so.

With vault I also decide to create a dedicated VM just for a Postgres database which would house my production databases. This allowed me to easily manage and back up my production databases since its all in one place. I created a limited account that just has permissions to assign roles and create accounts and connected it to vault. I can then query the vault API to get Postgres credentials to the database I need.

Configuring Vault

Hashicorp does a great job at ensuring security through all of their applications. When you initially install vault it will encrypt the backend storage with 3 recovery keys. If you ever created vault before you will learn quickly how annoying the recovery keys are. Every time the vault service restarts whether VM restarted or the service restarted the vault gets set in a "sealed" state. You will then need to "unseal" the vault by entering the recovery keys.

The first thing that I configured when I installed vault was auto unsealing via a Azure Key Vault. You can read more about it Here. I set up and created an Azure account, just for it to bill me $0.02 per month, an extremely worth wile investment considering I never need to deal with vault being in a sealed state. It was fairly easy to add in the config just add the following lines to your vault config.

# enable auto-unseal using the azure key vault.
seal "azurekeyvault" {
  client_id      = "YOUR-AZURE-APP-ID"
  client_secret  = "YOUR-AZURE-APP-PASSWORD"
  tenant_id      = "YOUR-AZURE-TENANT-ID"
  vault_name     = "Test-vault-xxxx"
  key_name       = "generated-key"
}

Once you restart vault, and enter in your recovery keys it will automatically migrate to azure key vault.

When it comes to configuring Hashicorp Vault, it really boils down to a few commands:

vault read <mount>/<resource> - Sends GET Request to API for resource
vault write <mount>/<resource> - Sends PUT, POST Request to API to configure resource / create resource
vault delete <mount>/<resource> - Sends DELETE request to API to remove resource
vault list <mount>/<resource> - Sends LIST request to API to list out resources

Now I did boil it down and make them pretty generic but it really is pretty simple. These are also how you create and define policies defining what a policy is allowed to do on what resource. If you want to read a configuration use vault read. If you want to edit a configuration or create new token/resource use vault write. To list out resources use list.

Vault can also be configured via a WebUI, the default port is at 8200 its also defined in the vault.hcl configuration file. Be warned it is a little limiting on some of the API requests though there is a web cli that you can enter more advanced commands. This is the main way I administer vault, and the way I will be going through in this post.

Connecting A Database

Within the vault UI, go to Secrets Engine > Enable New Engine and select database. Once there you can add a new connection these will be what vault uses to create new users and assign permissions. Each connection can be a different database, such as mysql or pgsql. As soon as you add in a new connection vault will prompt you to rotate the secret, feel free to rotate it.

To get a temporary account for the database you will need to create a role. This is where you define what permissions the account will get via creation statements. For postgresql I like to make a role, and when an account gets created assign the role. It will make permissions management so much easier than dealing with postgresql permissions.

Connecting Kubernetes

Like I mentioned above vault has really good integration with Kubernetes. It will access the Kubernetes API, and be able to verify tokens created by Kubernetes services. In essence your applications will be able to request vault resources without having to store a vault token, or worry about rotating the secrets.

In order to enable Kubernetes authentication within vault you will have to go to Access > Authentication Methods > Enable new method. From there select Kubernetes, and within the configuration of the authentication endpoints you will need to set your Kubernetes host, and the Kubernetes client certificate used to authenticate with Kubernetes.

When you have connected you can now add roles. Roles are used to define what permissions and who can log in. When you add a new role you will need to specify what policy should be applied and what service account namespaces can access the role.

Now onto vault policies. The real bread and butter of vault are there policies. These define what a token can or cannot do within vault. For the Kubernetes services they will be accessing various roles within the database secrets engine. The main way to define roles is like the following:

path "database/creds/<role>" {
  capabilities = ["read"]
}

The path specifies the resource that needs access. In this case our secrets engine database is mounted at database/ and we want to give access for our service to have read access a certain role. This will create short lived credentials to our databases. If you wanted to make this policy broader you can just specify the path database/creds though it will give the token access to read all roles. I have this limited down to only the roles it should have access to without creating multiple Kubernetes roles.

Getting Secrets Into Kubernetes

Integrating vault into your applications hosted on Kubernetes brings up a unique challenge. In Kubernetes there are secret resources, they contained base64 encoded key value pairs that can be used in your container definitions like the following:

env:
  - name: API_KEY
    valueFrom:
      secretKeyRef:
        name: env-secrets
        key: apikey

This is great for most applications where secrets don't get rotates. Unfortunately if you want to pull in the new secrets you will need to restart your deployment. Another way vault can work is via the Vault Agent which will authenticate with vault and modify a file that you can mount into your application, though applications can be verry sparse on how they reload and use config files, with many containers not adding that ability.

Fortunately Hashicorp saw this problem and created one of the greatest resources called the Vault Secrets Operator. It is a Kubernetes operator that will authenticate to Vault and create a secret manifest with the updated secrets. The operator will also re-deploy a deployment manifest whenever the secret changes. This is great as when a database account is near expiring it will create a new login, update the Kubernetes secret, and restart the deployment to use the new secret.

I will not be going into deploying the Vault Secrets Operator you can learn more by deploying the helm chart here. You may have different ways to deploy helm charts in your cluster.

Once installed there are just a few manifests you will need to add into your namespace in order to start pulling in secrets. Since we will be using Kubernetes service accounts, you will need:

  1. Service Account
  2. Cluster Role Binding (Access Authorization Endpoint)
  3. Dynamic Secret Config

Service Account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: <name>
  namespace: <namespace>

Cluster Role Binding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: <name>
subjects:
  - kind: ServiceAccount
    name: <name>
    namespace: <namespace>
roleRef:
  kind: ClusterRole
  name: system:auth-delegator
  apiGroup: rbac.authorization.k8s.io

Dynamic Secret

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: <name>
  namespace: <namespace>
spec:
  vaultAuthRef: <name>
  mount: database
  path: creds/<role>
  destination:
    create: true
    name: <secret name>
  rolloutRestartTargets:
    - kind: Deployment
      name: <deployment name>

If you so wish you can also add in additional configuration to the dynamic secret manifest under destination > transformation > templates These would be a jinja template for your key value pairs. You can read more about it on the Vault Secrets Operator Docs

Reflection

This has really become my go-to way of deploying secrets on my Kubernetes cluster, I would like to slowly pivot into using more Hashicorp Vault secrets engine such as kv pair that way I can have all of my secrets live within vault. For now I am using a mix of sops for rarely changing secrets such as API keys, and vault for dynamic secrets for my postgres database. I would really love to start scaling out my Vault instance to see its full potential for what it can do.

I would also love to start using the Vault API some more, and start writing my services to just authenticate with the API directly via a Kubernetes service, that way I may never even have to worry about secrets within my cluster, and just have git repo contain all deployments and other manifests.

If you are a business looking into Vault I highly recommend it, and if you are running a production Kubernetes cluster I feel it is essential. With the Vault Secrets Operator it makes for verry little risk of changes, as you can use it to configure a secret manifest along side the current manifest, and once the values propagate correctly you can migrate your deployments to the new manifest.