Egg vs chicken — Ansible Vault vs Hashicorp Vault vs Kubernetes Secrets
We all agree that we need to have a centralized system where we should store our encrypted secrets.
We have adopted Hashicorp Vault in my previous company since 2017 while migrating our AWS infra to a Kubernetes cluster.
Hashicorp Vault vs kubernetes Secrets
We got confused between kubernetes Secrets & Hashicorp Vault :
- Which one bootstrap the other ?
- Which one has the highest authority to store secrets ?
This confusion was resolved because of two things:
- The mindset of delivering everything thru CI/CD pipelines.
- Releasing our Kubernetes applications thru Helm charts.
Indeed, you can inject Values while installing helm charts. Some chart values will come from Hashicorp Vault (api responses).
Let’s explore how many CI/CD pipelines we have, then, we will conclude the communication flow among Vault systems
CI/CD — Helm Charts
Briefly, this is the sequence of steps when delivering applications on the top of a Kubernetes Cluster:
- CI/CD system fetches secrets from Hashicorp Vault thru REST calls . Indeed, Vault is an API-based application.
- CI/CD system forwards this values to the command
helm upgrade myrelease private/mychart -f mysecrets.yaml - Helm forwards these values to the right YAML templates (kind: Secret).
- Helm generates the computed manifests and apply changes (
helm installorhelm upgradeby sending them to the Kubernetes api-server ( or delegate to Tiller with helm2 ).
Actually, these steps looks clear. However, Hashicorp Vault itself will be installed by Helm 🤨.
Then, from where does helm fetch secrets before Hashicorp Vault is being installed ?!
CI/CD — Cluster Computing Nodes
Because everything is delivered thru CI/CD pipelines, we don’t release only Kubernetes apps (Helm Charts) thru CI/CD but also the Kubernetes cluster Itself is maintained via CI/CD.
However, we don’t use Helm to provision the cluster. instead, we used Ansible to provision all nodes per its role (master or node ).
Let’s see the sequence of Cluster’s CI/CD executions :
- CI/CD stores Ansible vaultkey (master key) and inject it after checking out playbooks from the source control system (git)
- CI/CD calls
ansible-playbookto provision the Cluster - Idempotently, ansible makes sure that the Cluster up and running.
Note: Ansible handles encryption of data natively without the need to any 3rd party. This feature is coming with the component ansible-vault
CI/CD — Infrastructure
The CI/CD of Cluster assumes that we’ve already some computing resources (VMs) up & running. Then, Ansible installs the Cluster on top of it.
But how do we create these computing resources ?
This is the mission of Infra’s CI/CD. This CI/CD does not create only computing resources, but it also create other types of resources: storage, network, DNS,.. so on and so forth
Indeed, we use Terraform to create any resource :
- CI/CD makes Credentials of Cloud provider available.
- CI/CD executes terraform apply.
- Terraform compares the actual infrastructure with the desired one, detects differences then applies them.
If you are using Kubernetes Managed service (like EKS), your Ansible playbook code will be mitigated if the managed service can be created with Terraform.
Big Picture — Putting all together
NOTES
- Infra Pipeline , Cluster Pipeline,Vault Chart Pipeline then other Charts Pipeline.
- In the Helm CI/CD (3rd pipeline), we delegate Ansible to invoke helm for the sake of idempotency as well as consistency with Cluster CI/CD (2nd pipeline) . Indeed, Ansible provides “helm” module.
- Once the Hashicorp Vault Chart is deployed for the first time, this is the end (step #12) of the first cycle which relies on CI/CD credentials store. The next cycles relies on the Vault as it is now up and running.
- While these orange arrows are our last step (# 12), it is a beginning of a new cycle where all pipelines unify the way of reading credentials.
The rule is simple:
- Before installing Hashicorp Vault, you can rely on the CI/CD system to store some credentials needed for bootstrapping everything until having Hashicorp Vault up & running. With Jenkins, Credentials management component is your friend. With Bamboo, Bamboo Specs Encryption is what you need.
- After installing Hashicorp Vault, all subsequent secrets must be saved on it. Optionally, bootstrap credentials can be migrated from CI/CD store to Hashicorp Vault for consistency.
Bonus:
Udemy course about running EKS on production : https://www.udemy.com/course/aws-eks-kubernetes
Conclusion
The executor has an authority on the executable. The executor is responsible for forwarding secrets to the executable.
Having an extensive hands-on with DevOps tools with architecture mindset gives you a big vision. This vision lets you make the marriage among these tools comfortably.
As consequence, we advised and we are still advising to not neglect any tool: Learn it even if you will not use it now. Indeed, you may need of it one day.
