Egg vs chicken — Ansible Vault vs Hashicorp Vault vs Kubernetes Secrets

We all agree that we need to have a centralized system where we should store our encrypted secrets.

We have adopted Hashicorp Vault in my previous company since 2017 while migrating our AWS infra to a Kubernetes cluster.

Hashicorp Vault vs kubernetes Secrets

We got confused between kubernetes Secrets & Hashicorp Vault :

This confusion was resolved because of two things:

Indeed, you can inject Values while installing helm charts. Some chart values will come from Hashicorp Vault (api responses).

Let’s explore how many CI/CD pipelines we have, then, we will conclude the communication flow among Vault systems

CI/CD — Helm Charts

Briefly, this is the sequence of steps when delivering applications on the top of a Kubernetes Cluster:

CI/CD helm charts

Actually, these steps looks clear. However, Hashicorp Vault itself will be installed by Helm 🤨.

Then, from where does helm fetch secrets before Hashicorp Vault is being installed ?!

CI/CD — Cluster Computing Nodes

Because everything is delivered thru CI/CD pipelines, we don’t release only Kubernetes apps (Helm Charts) thru CI/CD but also the Kubernetes cluster Itself is maintained via CI/CD.

However, we don’t use Helm to provision the cluster. instead, we used Ansible to provision all nodes per its role (master or node ).

Let’s see the sequence of Cluster’s CI/CD executions :

Note: Ansible handles encryption of data natively without the need to any 3rd party. This feature is coming with the component ansible-vault

CI/CD — Cluster
playbook provisions the whole kubernetes cluster

CI/CD — Infrastructure

The CI/CD of Cluster assumes that we’ve already some computing resources (VMs) up & running. Then, Ansible installs the Cluster on top of it.

But how do we create these computing resources ?

This is the mission of Infra’s CI/CD. This CI/CD does not create only computing resources, but it also create other types of resources: storage, network, DNS,.. so on and so forth

Indeed, we use Terraform to create any resource :

main.tf — Terraform main file

If you are using Kubernetes Managed service (like EKS), your Ansible playbook code will be mitigated if the managed service can be created with Terraform.

Big Picture — Putting all together

All together 😎

NOTES

- Infra Pipeline , Cluster Pipeline,Vault Chart Pipeline then other Charts Pipeline.

- In the Helm CI/CD (3rd pipeline), we delegate Ansible to invoke helm for the sake of idempotency as well as consistency with Cluster CI/CD (2nd pipeline) . Indeed, Ansible provides “helm” module.

- Once the Hashicorp Vault Chart is deployed for the first time, this is the end (step #12) of the first cycle which relies on CI/CD credentials store. The next cycles relies on the Vault as it is now up and running.

- While these orange arrows are our last step (# 12), it is a beginning of a new cycle where all pipelines unify the way of reading credentials.

The rule is simple:

Bonus:

Udemy course about running EKS on production : https://www.udemy.com/course/aws-eks-kubernetes

Conclusion

The executor has an authority on the executable. The executor is responsible for forwarding secrets to the executable.

Having an extensive hands-on with DevOps tools with architecture mindset gives you a big vision. This vision lets you make the marriage among these tools comfortably.

As consequence, we advised and we are still advising to not neglect any tool: Learn it even if you will not use it now. Indeed, you may need of it one day.

Related:

Software engineer, Cloud Architect, 5/5 AWS|GCP|PSM Certified, Owner of kubernetes.tn