Egg vs chicken — Ansible Vault vs Hashicorp Vault vs Kubernetes Secrets

We all agree that we need to have a centralized system where we should store our encrypted secrets.

We have adopted Hashicorp Vault in my previous company since 2017 while migrating our AWS infra to a Kubernetes cluster.

Hashicorp Vault vs kubernetes Secrets

We got confused between kubernetes Secrets & Hashicorp Vault :

  • Which one bootstrap the other ?
  • Which one has the highest authority to store secrets ?

This confusion was resolved because of two things:

  • The mindset of delivering everything thru CI/CD pipelines.
  • Releasing our Kubernetes applications thru Helm charts.

Indeed, you can inject Values while installing helm charts. Some chart values will come from Hashicorp Vault (api responses).

CI/CD — Helm Charts

Briefly, this is the sequence of steps when delivering applications on the top of a Kubernetes Cluster:

  • CI/CD system fetches secrets from Hashicorp Vault thru REST calls . Indeed, Vault is an API-based application.
  • CI/CD system forwards this values to the command helm upgrade myrelease private/mychart -f mysecrets.yaml
  • Helm forwards these values to the right YAML templates (kind: Secret).
  • Helm generates the computed manifests and apply changes ( helm install or helm upgrade by sending them to the Kubernetes api-server ( or delegate to Tiller with helm2 ).
CI/CD helm charts

Actually, these steps looks clear. However, Hashicorp Vault itself will be installed by Helm 🤨.

Then, from where does helm fetch secrets before Hashicorp Vault is being installed ?!

CI/CD — Cluster Computing Nodes

Because everything is delivered thru CI/CD pipelines, we don’t release only Kubernetes apps (Helm Charts) thru CI/CD but also the Kubernetes cluster Itself is maintained via CI/CD.

However, we don’t use Helm to provision the cluster. instead, we used Ansible to provision all nodes per its role (master or node ).

Let’s see the sequence of Cluster’s CI/CD executions :

  • CI/CD stores Ansible vaultkey (master key) and inject it after checking out playbooks from the source control system (git)
  • CI/CD calls ansible-playbook to provision the Cluster
  • Idempotently, ansible makes sure that the Cluster up and running.

Note: Ansible handles encryption of data natively without the need to any 3rd party. This feature is coming with the component ansible-vault

CI/CD — Cluster
playbook provisions the whole kubernetes cluster

CI/CD — Infrastructure

The CI/CD of Cluster assumes that we’ve already some computing resources (VMs) up & running. Then, Ansible installs the Cluster on top of it.

This is the mission of Infra’s CI/CD. This CI/CD does not create only computing resources, but it also create other types of resources: storage, network, DNS,.. so on and so forth

Indeed, we use Terraform to create any resource :

  • CI/CD makes Credentials of Cloud provider available.
  • CI/CD executes terraform apply.
  • Terraform compares the actual infrastructure with the desired one, detects differences then applies them.
main.tf — Terraform main file

If you are using Kubernetes Managed service (like EKS), your Ansible playbook code will be mitigated if the managed service can be created with Terraform.

Big Picture — Putting all together

All together 😎

NOTES

- Infra Pipeline , Cluster Pipeline,Vault Chart Pipeline then other Charts Pipeline.

- In the Helm CI/CD (3rd pipeline), we delegate Ansible to invoke helm for the sake of idempotency as well as consistency with Cluster CI/CD (2nd pipeline) . Indeed, Ansible provides “helm” module.

- Once the Hashicorp Vault Chart is deployed for the first time, this is the end (step #12) of the first cycle which relies on CI/CD credentials store. The next cycles relies on the Vault as it is now up and running.

- While these orange arrows are our last step (# 12), it is a beginning of a new cycle where all pipelines unify the way of reading credentials.

The rule is simple:

  • Before installing Hashicorp Vault, you can rely on the CI/CD system to store some credentials needed for bootstrapping everything until having Hashicorp Vault up & running. With Jenkins, Credentials management component is your friend. With Bamboo, Bamboo Specs Encryption is what you need.
  • After installing Hashicorp Vault, all subsequent secrets must be saved on it. Optionally, bootstrap credentials can be migrated from CI/CD store to Hashicorp Vault for consistency.

Bonus:

Udemy course about running EKS on production : https://www.udemy.com/course/aws-eks-kubernetes

Conclusion

The executor has an authority on the executable. The executor is responsible for forwarding secrets to the executable.

Having an extensive hands-on with DevOps tools with architecture mindset gives you a big vision. This vision lets you make the marriage among these tools comfortably.

As consequence, we advised and we are still advising to not neglect any tool: Learn it even if you will not use it now. Indeed, you may need of it one day.

Related:

Software engineer, Cloud Architect, 5/5 AWS|GCP|PSM Certified, Owner of kubernetes.tn