Post Mortem of the 7th July 2023 Jenkins Infrastructure Outage

10:30am UTC: After a successful upgrade of the public Kubernetes cluster in Azure to 1.25 (as part of help desk ticket 3582), we realized that the LDAP service was not reachable by the services running inside the cluster (such as accounts.jenkins.io). We quickly identified IP restrictions blocking these requests as the pod originating IP was in a different range than before.

10:55am UTC: The fix (Azure PR 431) is deployed to specify a proper set of IP ranges for the pods. It removed all of the node pools (all the virtual machines where the container was running) and failed to re-create them, causing a full outage of all the services running in this cluster:

11:16am until 13:16pm UTC: The failure to re-create resources led us to spend the 2 next hours creating the cluster from scratch with a fixed network setup.

15:17pm UTC: This pull request is pushed to persist the manual work we did to recreate the cluster including the IP setup.

15:25pm UTC: All services are back to normal

When the cluster was initially created, we selected the 10.244.0.0/16 virtual network IP range (ref. Azure VNets) with a 10.245.0.0/24 sub-network (ref. Azure subnet).

But we ignored that the 10.244.0.0/24 range is the default CIDR for the Kubernetes Pods network in Azure when using the "kubenet" network to support IPv6 instead of the default CNI.

The node pool re-creation failed because we assumed both ranges were able to communicate and tried to deploy an invalid setup.

As soon as we specified a custom Pod CIDR in a distinct range, everything went fine.

When the original cluster was deleted it transitively removed the current Public IPs, as it removed the Nodes Resource Group containing the Public IP.

As per our initial assessment: protect the Public IPs from deletion by adding a Management Lock.

As recommended by other contributors: storing the Public IP in a distinct Resource Group and set up the Kubernetes-managed Load Balancers accordingly (annotation service.beta.kubernetes.io/azure-load-balancer-resource-group).

Improve our network diagrams and documentation to have better access to the representation and potential overlaps when preparing operations.

Avoid changing AKS node pools configurations all at once: we would have caught the issue after the first node pool and could have avoided a full outage (we are working on this topic for the arm64 node pools in PR-3623).