Egg vs chicken — Ansible Vault vs Hashicorp Vault vs Kubernetes Secrets

Image for post
Image for post

We all agree that we need to have a centralized system where we should store our encrypted secrets.

We have adopted Hashicorp Vault in my previous company since 2017 while migrating our AWS infra to a Kubernetes cluster.

Hashicorp Vault vs kubernetes Secrets

  • Which one bootstrap the other ?
  • Which one has the highest authority to store secrets ?

This confusion was resolved because of two things:

  • The mindset of delivering everything thru CI/CD pipelines.
  • Releasing our Kubernetes applications thru Helm charts.

Indeed, you can inject Values while installing helm charts. Some chart values will come from Hashicorp Vault (api responses).

Let’s explore how many CI/CD pipelines we have, then, we will conclude the communication flow among Vault systems

CI/CD — Helm Charts

  • CI/CD system fetches secrets from Hashicorp Vault thru REST calls . Indeed, Vault is an API-based application.
  • CI/CD system forwards this values to the command helm upgrade myrelease private/mychart -f mysecrets.yaml
  • Helm forwards these values to the right YAML templates (kind: Secret).
  • Helm generates the computed manifests and apply changes ( helm install or helm upgrade by sending them to the Kubernetes api-server ( or delegate to Tiller with helm2 ).
Image for post
Image for post
CI/CD helm charts

Actually, these steps looks clear. However, Hashicorp Vault itself will be installed by Helm 🤨.

Then, from where does helm fetch secrets before Hashicorp Vault is being installed ?!

CI/CD — Cluster Computing Nodes

However, we don’t use Helm to provision the cluster. instead, we used Ansible to provision all nodes per its role (master or node ).

Let’s see the sequence of Cluster’s CI/CD executions :

  • CI/CD stores Ansible vaultkey (master key) and inject it after checking out playbooks from the source control system (git)
  • CI/CD calls ansible-playbook to provision the Cluster
  • Idempotently, ansible makes sure that the Cluster up and running.

Note: Ansible handles encryption of data natively without the need to any 3rd party. This feature is coming with the component ansible-vault

Image for post
Image for post
CI/CD — Cluster
Image for post
Image for post
playbook provisions the whole kubernetes cluster

CI/CD — Infrastructure

But how do we create these computing resources ?

This is the mission of Infra’s CI/CD. This CI/CD does not create only computing resources, but it also create other types of resources: storage, network, DNS,.. so on and so forth

Indeed, we use Terraform to create any resource :

  • CI/CD makes Credentials of Cloud provider available.
  • CI/CD executes terraform apply.
  • Terraform compares the actual infrastructure with the desired one, detects differences then applies them.
Image for post
Image for post
Image for post
Image for post
main.tf — Terraform main file

If you are using Kubernetes Managed service (like EKS), your Ansible playbook code will be mitigated if the managed service can be created with Terraform.

Big Picture — Putting all together

Image for post
Image for post
All together 😎

NOTES

- Infra Pipeline , Cluster Pipeline,Vault Chart Pipeline then other Charts Pipeline.

- In the Helm CI/CD (3rd pipeline), we delegate Ansible to invoke helm for the sake of idempotency as well as consistency with Cluster CI/CD (2nd pipeline) . Indeed, Ansible provides “helm” module.

- Once the Hashicorp Vault Chart is deployed for the first time, this is the end (step #12) of the first cycle which relies on CI/CD credentials store. The next cycles relies on the Vault as it is now up and running.

- While these orange arrows are our last step (# 12), it is a beginning of a new cycle where all pipelines unify the way of reading credentials.

The rule is simple:

  • Before installing Hashicorp Vault, you can rely on the CI/CD system to store some credentials needed for bootstrapping everything until having Hashicorp Vault up & running. With Jenkins, Credentials management component is your friend. With Bamboo, Bamboo Specs Encryption is what you need.
  • After installing Hashicorp Vault, all subsequent secrets must be saved on it. Optionally, bootstrap credentials can be migrated from CI/CD store to Hashicorp Vault for consistency.

Bonus:

Conclusion

Having an extensive hands-on with DevOps tools with architecture mindset gives you a big vision. This vision lets you make the marriage among these tools comfortably.

As consequence, we advised and we are still advising to not neglect any tool: Learn it even if you will not use it now. Indeed, you may need of it one day.

Related:

Written by

Software engineer, Cloud Architect, 5/5 AWS|GCP|PSM Certified, Owner of kubernetes.tn

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store