Secure CI/CD Practices
Handling Secrets in Azure DevOps Deployment Pipelines and K8s
Avoid leaking secure configuration using CD processes and Kubernetes Secrets
Almost all back end software requires some form of ‘secret’ configuration — that is configuration that the software needs in order to run, but knowledge of the values used needs to be restricted. Examples of this form of configuration could include
- Usernames and passwords, such as database credentials
- Credentials for external services, such as API keys and client secrets
- Digital Certificates and ofther key material for TLS and crypt
- Anything else your production systems need that your devs don’t!
Traditionally most systems have been relatively naïve in their handling of this form of configuration, generally resulting it being stored for all to see in source control, and generally known by far more people than is advisable.
In this article we’ll see how we can combine the variable system in Azure DevOps deployment pipelines with Kubernetes’ built support for secret objects to create a secure solution to this problem with minimal effort.
Before You Begin — The Ops Considerations
Right, before we even start with the ‘techie’ bit of this, we need to consider where, operationally, these secrets are coming from, and who comprises the ‘select few’ that know them. Let’s take, as an example, creating a new service that will be backed by a DBaaS offering such as MongoDB’s Atlas product.
First, someone is going to need to sign up to the service and create a new database instance. At this point they’ll be presented with the credentials needed by the service in order to access the database. So already we have two sets of ‘secret’ credentials; clearly, we don’t want just anyone being able to log in to the Atlas console to create and delete database, and equally we don’t want just any one being able to access the database itself.
We won’t address the first set of credentials in this article, as the software doesn’t need them, this is a purely operational consideration; who has access to create an and manage this ‘aaS’ style offerings. Generally, you want to ensure this is a very small set of senior folks, and that’s its probably the same people who will have access the actual credentials needed by the software services we’ll be covering in the rest of this article.
Obviously the second set of credentials are what we’re discussing here, now we need to think about how we take get them from whatever dashboard or tool was used to generate them, and get them into the pipeline, using the process presented below. It’s at this point we need a manual, operational, process to securely move these items. The simplest option is to ensure that whoever creates the items is responsible for adding them to the pipeline and doesn’t have to disclose them in doing so.
Configuring your DevOps Pipeline
Within a deployment pipeline you can set variables, which can then be accessed by tasks within the pipeline as it executes. These variables can be set per-environment, so that each environment can have unique values for each of the variables, allowing for per-environment configuration whilst maintaining common variable names. For more information in pipeline variables, variable groups, and the built-in variables, please see the DevOps documentation.
For now, we create a per-environment variables to hold our secrets and add the secret to the variable based on our operational process definition, as outlined above. Once set, the padlock can be clicked, which locks down the value of the variable. Once ‘locked’ the value is obscured, so can’t be seen by anyone looking in DevOps, and also can’t be ‘unlocked’ — from here on in only tasks running in the pipeline have access to the value. It’s now secure, and secret!
Deploying secrets into K8s
So now we have our secrets all sorted in DevOps, we need to get them into Kubernetes, without disclosing them along the way. Fortunately, that’s easier than it sounds!
First off, we need to create some YAML which we can use to create our secrets in our cluster. The below snippet gives an example of creating an opaque text secret, but Kubernetes’ documentation covers plenty of other use cases. The key in this case, though, is that the value of the secret isn’t held in the YAML; doing so would undo everything we’re looking to achieve here. Instead we place a token where the value would be, and that token matches the name of the variable we set up in DevOps to hold that actual value, plus a marker so that we can identify it as a token when DevOps runs our pipeline. In this example, we use a couple of hashes (##)
Now we can tweak the deployment step(s) to replace the tokens with the values from our secret variables right before they’re deployed to the cluster. Note that you can’t do this in the build step, as they would mean storing the modified YAML as a build artefact, and again, that would have the values we don’t want disclosed in it. This needs to be done during your deployment, so the file is updated, used to configure the cluster, and then destroyed thus providing a totally secure and opaque workflow.
There are several tokenizers available for DevOps pipelines, personally I like the ‘Replace Tokens’ task, which is part of the ‘Colins ALM Corner’ set of DevOps extensions. It’s easy to configure and does a great job of doing just what we need. Just set it up in the deployment pipeline for each environment, keyed to look for tokens based on the twin hashes we put in the YAML file.
So, now you have a complete YAML within the pipeline, just use the Kubectl step to deploy it to your cluster and create the K8s secret objects ready to be consumed by your pods.
Accessing K8s secrets in your pods
Once created in Kubernetes, consuming the secrets into your pods is as simple as binding them to environment variables, or for files mounting them as volumes, in the usual way.
The Gotchas of secure secrets
So, now you have a solution where your secrets are truly secret, and known only to ‘the select few’; perfect, right?
Well, it will be, right up until something goes wrong and your devs want to poke around in the prod database for a bit to see what’s going down. Back in the day, they did that without even thinking, because everyone knew the credentials, and if they didn’t, they could always ask someone. Now they don’t know, and no one will tell them. So now you’re going to need to think how the bigger picture works. That might mean custom tooling to allow controlled access to the database or going back and looking at the basics like logging and metrics. Either way, you’ll need to resolve this, as you need to be able to support your systems, but we all know we shouldn’t be poking around in production databases anyway!
Speaking of logging, the other gotcha is that secrets are only secrets so long as you don’t tell anyone. Keep an eye out for logs accidentally disclosing secret information, it happens way more than you might think — for example logging the connection string, including credentials, when failing to connect to a database. All your hard work and good intentions undone in a second’s oversight.
The final ‘gotcha’ should be really obvious — if you don’t have wide knowledge, and a hundred ways of finding out what your secrets are (hey, they’re really secret now!) you’ll probably want to ensure you have a secure mechanism for secondary storage of them for that one day when you need them. Again, this is an entirely operational consideration, but it’s really easy to overlook as you start working in more secure ways.