quay.io is down, my Kubernetes cluster cannot pull images

Root Cause

After some investigation, I realized that all pods with images from quay.io are not downloadable.

Workaround

I was lucky that i am running another cluster with the same ingress controller (router) but with the later version.

  1. In the intact cluster, I’ve checked where the pod of ingress-controller is running
k -n kube-system get pod -o wide | grep ingress-controller
# -o wide will show the Node name where the pod is running
ingress-nginx-ingress-controller-b6544fd67-9c8nc 1/1 Running 0 21m 10.0.5.196 ip-10-0-5-230.ap-southeast-1.compute.internal <none> <none>
ssh ip-10-0-5-230$ docker tag quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0 abdennour/nginx-ingress-controller:0.30.0
docker login ...
docker push abdennour/nginx-ingress-controller:0.30.0
rbac:
create: true
controller:
image:
# repository: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
repository: abdennour/nginx-ingress-controller
tag: "0.30.0"
docker-compose run --rm helm3 -n kube-system 
docker-compose run --rm helm3 \
-n kube-system upgrade ingress stable/nginx-ingress \
--version 1.34.2 \
-f cluster-plugins/charts/values/nginx-ingress.yaml

Results

I got some errors:

  • I delete the whole helm release of ingress-controller,
  • and i reinstalled the helm chart again
  • A new ELB has been created, then I updated the DNS record of the wildcard domain (*.company.com) to point to the new ELB DNS.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Abdennour Toumi

Abdennour Toumi

Software engineer, Cloud Architect, 5/5 AWS|GCP|PSM Certified, Owner of kubernetes.tn