Upgrading our production Kubernetes cluster from 1.27 to 1.33

Tolgee's cloud platform runs on a managed Kubernetes cluster on Azure (AKS). This post describes how we upgraded that cluster from Kubernetes 1.27 to 1.33, the problems we found while doing it, and what the upgrade means for the people who use the platform.

What this means for users

A Kubernetes version upgrade does not, by itself, add any feature that a user of the platform can see or use. The changes between Kubernetes 1.28 and 1.33 are improvements to how the cluster runs workloads: scheduling, resource handling, security defaults, and reliability. They are not changes to the application itself. For someone using Tolgee, the version of Kubernetes underneath is not visible.

The benefit is indirect but real. Running a supported Kubernetes version means the cluster receives security patches, and, just as important, it removes Kubernetes as a blocker for updating the components that sit in front of the application. During this work we found that we could not move our ingress controller (NGINX) or our certificate manager to current, supported versions while the cluster stayed on 1.27. If a security problem is found in one of those components, we need to be able to update it quickly. Staying on a current Kubernetes version keeps that path clear, which protects the availability and security of the service that users depend on.

Why a small task became a large one

Some maintenance work is easy to postpone. A Kubernetes version upgrade is a good example. The cluster keeps running after a version goes out of support, so there is no immediate pressure to act, and the upgrade is quietly moved to "later" again and again. This is a common form of technical debt: it is invisible for a long time, and it does not cause problems until you finally have to deal with it.

The problem with postponing is that the work does not stay the same size. When we finally started, Kubernetes 1.27 was already out of support, and the supported versions between 1.28 and 1.32 were no longer offered in our Azure region. Azure only allowed a direct upgrade to 1.33. Work that could have been a single-version step, repeated a few times over many months, had become one large jump across six minor versions at once. A routine update had turned into a significant operation.

Why the jump could not be split into smaller steps

Kubernetes enforces a version skew rule: a node pool may not be more than three minor versions behind the control plane. We first tried to upgrade only the control plane, as a safe checkpoint before touching any workloads. Azure rejected this, because leaving the nodes on 1.27 while the control plane moved to 1.33 would break that rule. The only supported path was a single managed operation that upgrades the control plane and both node pools together.

Preparation

Before the upgrade we did the following:

• Rehearsed the whole upgrade on a local, throwaway cluster (minikube) running the target Kubernetes version, including every add-on we run.

• Checked that no workload used Kubernetes APIs that were removed between 1.28 and 1.33.

• Set a surge setting on both node pools so a new node is added before an old one is drained.

• Made every change through pull requests, applied by our GitOps tooling, so each step was reviewable and could be undone at the configuration level.

The upgrade itself completed successfully: both node pools now run Kubernetes 1.33. Most of the work, and most of what we learned, was in handling the problems below.

Problems we found

A node that was not really there

Before starting, we compared the number of virtual machines that Azure reported against the number of nodes registered in Kubernetes. They did not match. One virtual machine was running but had never joined the cluster as a node. An upgrade would have stalled on this machine. We reimaged the instance so that it joined correctly, and only then continued.

Disk attachment failures while moving Redis

During a node upgrade, pods are moved off each node before that node is replaced. Our Redis instances use persistent disks, so moving a Redis pod means detaching its disk from the old node and attaching it to a new one. Several of these attachments failed and stayed failed for a long time, with an error that suggested a fault in the storage driver.

The real cause was different. Our node type (Standard_E4as_v5) supports a maximum of eight attached data disks. During the upgrade the scheduler had placed enough disk-backed pods on one node to reach that limit. With no free disk slot on the node, no further disk could attach, and the error only looked like a driver problem. We confirmed this by counting the disks attached to the node, and then moved the affected pods to another node in the same availability zone that still had free slots. Restarting the storage driver, which is what the error message pointed us toward, had no effect and was the wrong path.

A StatefulSet that stopped making progress

Redis runs as a StatefulSet with ordered startup, which means Kubernetes does not create the next pod until the previous one is ready. Because one Redis pod was stuck waiting for its disk, the next Redis pod was never created at all. The set stayed at one of three members, even though two of its nodes were healthy. Freeing the stuck pod allowed the remaining pod to be created automatically.

A short outage from a missing disruption budget

Redis did not have a PodDisruptionBudget. A PodDisruptionBudget tells Kubernetes how many replicas must stay available during voluntary operations such as node drains. Without one, the upgrade removed a second Redis node before the first had recovered. Redis lost its quorum for one to two minutes, and during that time the application could not open new connections to it, which produced a short period of failed requests. We have since added a disruption budget so that only one Redis node can be taken down at a time.

Certificate manager could not be upgraded in one step

No single version of our certificate manager (cert-manager) supported both Kubernetes 1.27 and 1.33. We kept a version that runs on both, carried it through the jump, and upgraded it to a current, supported version afterwards. This is a general point worth noting: some components have their own minimum and maximum Kubernetes versions, and a large jump has to be planned around them.

Degraded connection pools after the upgrade

Some hours after the cluster was healthy again, the application reported intermittent Redis timeouts that affected roughly half of the operations that used Redis. Redis itself was healthy at this point. The cause was on the client side. The application had kept its Redis connection pools open through all the earlier disruption, and some of those connections were left in an unusable state. A rolling restart of the application rebuilt the pools, and the timeouts stopped. It is worth expecting this kind of client-side effect after any infrastructure event that repeatedly interrupts connections.

What this opens up

Because the cluster is now on a current, supported version, the path to future Kubernetes upgrades is open again in Azure. From here we can upgrade one minor version at a time, on a regular schedule. That is far less risky than the jump we just completed, because each step is small, the version skew rule is never a problem, and each add-on only has to support two adjacent versions at a time. Keeping to that schedule is our next step, and it is the main way to avoid ending up in the same position again.

Summary

The main lesson from this work is that infrastructure upgrades do not become cheaper by waiting. They become larger and riskier. A version upgrade that is postponed long enough stops being a routine step and becomes a single large migration, with more moving parts and more ways to fail. Alongside that, a few concrete practices made the difference: rehearse on a throwaway cluster first, give every stateful workload a disruption budget, know the limits of your node type (including how many disks it can attach), and expect client-side effects after any event that interrupts connections. With the cluster now current, we can return to small, regular upgrades, which is where this kind of maintenance is meant to stay.