OpenShift EUS-to-EUS updates: a Control Plane Only runbook

What a Control Plane Only update is

Red Hat denotes even-numbered minor releases of OpenShift Container Platform, such as 4.20 and 4.22, as Extended Update Support (EUS) releases. Many teams run production on EUS releases and move only from one to the next. This article follows the OpenShift Container Platform 4.22 documentation for that move. Check the documentation for your own versions before relying on any detail here.

Minor updates are serialized: a cluster cannot go from 4.20 to 4.22 directly, it must pass through 4.21. A Control Plane Only update, previously called an EUS-to-EUS update, makes that bearable. You pause the machine config pools for your worker and custom nodes, let the control plane update to 4.21 and then 4.22, and unpause the pools at the end. The paused nodes update once, straight to 4.22, so they reboot once instead of twice. It works only between even-numbered minor versions.

Two timing rules follow from Red Hat's process. The path is offered only after updates between all the versions involved are available in stable channels, and Red Hat documents a typical delay of 45 to 90 days after a new minor version's general availability before its update paths reach the stable channels, with EUS channels promoted at the same time. That is an estimate, not a deadline. If the update path is not offered yet, the answer is to wait, not to force it.

Read both release notes, not just the target

The prerequisites start with reading the release notes for both 4.21 and 4.22, and the release notes and life cycles of every layered product and Operator on the cluster. Even though the worker nodes skip past 4.21, the control plane does not, and each hop has its own preparation.

For this pair, the 4.21 and 4.22 update guides each list no Kubernetes API removals, which removes a common blocker. It does not rule out other breaking changes, deprecations or Operator compatibility requirements, which the release notes cover. The 4.21 to 4.22 hop adds a different one on specific platforms: on vSphere clusters, for example, where the managedBootImages parameter has not been configured, the update is blocked until an administrator either acknowledges the new default for updated boot images or explicitly disables the feature for compute nodes. Clusters with in-tree vSphere volumes also need vSphere 7.0u3L+ or 8.0u2+ before the update begins. Finding these in a planning meeting costs minutes. Finding them halfway through a maintenance window costs the window.

Check the cluster is ready to move

OpenShift will not start a minor update unless every cluster Operator reports Upgradeable as True, and the ClusterNotUpgradeable alert tells you when one does not. Resolve critical alerts first as well. The read-only oc adm upgrade recommend command summarizes firing alerts and known update risks for your cluster. Its alert-based checks need token-based authentication: with the installer's certificate-based system:admin kubeconfig the command still runs, but reports the missing token and skips those checks.

Choose only update targets recommended by the OpenShift Update Service. A target shown as a conditional update carries a known risk that applies to your cluster; unless you need that specific version, Red Hat's advice is to wait for a recommended path.

Then check capacity. Nodes are drained one at a time by default, since maxUnavailable defaults to 1, and Red Hat recommends leaving that value alone for the control plane pool. Every node should be available before you begin, the remaining nodes need room for the pods being moved, and PodDisruptionBudgets must allow those pods to move at all. A budget that tolerates no disruption stops a node drain as surely as a failure does. Update the oc CLI to the target version before each hop, and take an etcd backup before you start.

Operators decide how long it takes

The cluster update is the scripted part. The Operators are where Control Plane Only updates usually stall. Before each hop, every Operator installed through Operator Lifecycle Manager (OLM) must be updated to a version compatible with the target release, so that it still has a valid update path when the default catalogs switch to the next minor version. Red Hat provides an Operator Update Information Checker to confirm which cluster versions each Operator version supports.

Layered products are interleaved with the cluster updates. Red Hat's example for OpenShift Data Foundation is: pause the worker pools, update the cluster to 4.21, update ODF to its 4.21 version, update the cluster to 4.22, update ODF to 4.22, and unpause. List every Operator with its current version, its compatible versions and the point in the sequence where it moves, and treat that list as part of the change record.

Clusters running virtual machines

If the cluster runs OpenShift Virtualization, there is an extra step before the first hop. Record the current workloadUpdateMethods setting, then set it to empty. This stops OpenShift Virtualization from migrating or evicting virtual machines while the control plane moves through 4.21.

OpenShift Virtualization then follows the cluster. After the 4.21 hop, it updates to the latest z-stream of its intermediate version, and you wait until the HyperConverged Operator reports Upgradeable again before starting the 4.22 hop. With the default Automatic approval strategy these updates happen on their own; with Manual approval each one must be approved. After the 4.22 hop and the matching OpenShift Virtualization update, restore the recorded workloadUpdateMethods value, check the VM migrations, and only then unpause the pools.

The sequence

The documented sequence, from the CLI or the web console, is short. Update OLM Operators to versions compatible with the target. Confirm every machine config pool is up to date and none is updating. Switch the cluster channel to eus-4.22. Pause every pool except the control plane pool, which cannot be paused. Update to the latest 4.21 release and confirm the update has completed. Update Operators again as needed. Update to 4.22 and confirm. Unpause the pools, and confirm each one reports up to date.

Only the unpause step reboots worker nodes, so it is the step to schedule against the business calendar. If the cluster includes RHEL compute machines, they need the upgrade playbook run as each one enters the NotReady state. Give each step an owner, a check that proves it finished, and a written condition for stopping.

Keep the pause short

Paused pools are not a safe place to wait. While a pool is paused, the Machine Config Operator cannot push configuration to its nodes, and that includes the updated certificate trust bundle after the kube-apiserver-to-kubelet-signer CA is automatically rotated. Pausing does not break anything by itself, but once the API server uses a client certificate signed by the new signer, the paused nodes can reject it, and commands such as oc logs, oc exec, oc debug and oc attach fail. And if pools are left paused after the update, the cluster is not permitted to update to any later minor version, and some maintenance tasks are inhibited.

Until the pools are unpaused, some features and bug fixes in 4.21 and 4.22 are not available on the paused nodes. If a problem appears at 4.21, before the second hop, fixing it may require the paused nodes to complete the update to 4.21 first. That is the path back to a normal two-reboot update, so accept it rather than holding pools paused while you search for another. For large clusters, custom pools can be unpaused in stages, starting with a small canary pool, as long as every pool ends at the same version.

Plan for no cluster rollback

Rolling an OpenShift cluster back to a previous version is not supported. That applies to the cluster version: application-level rollback and documented disaster recovery are separate matters. An etcd backup protects against a cluster that becomes unrecoverable, but Red Hat is explicit that an etcd restore is not a rollback mechanism, is destructive, and is a last resort. If an update fails to complete, the supported route is Red Hat Support.

Everything that would make rollback unnecessary therefore happens before the update: a rehearsal on a non-production cluster that runs the same Operators at the same versions, the Operator and layered product sequence written down, capacity and PodDisruptionBudgets checked, the admin acknowledgments given, and a support case ready to open. The maintenance window should state who can pause the change and on which signals. For application workloads, recovery means the application's own backup and restore, tested beforehand, not a cluster restore.

How we approach it

We treat an EUS-to-EUS update as a planned change with a runbook: version-specific prerequisites checked against the documentation for the exact releases, an Operator compatibility list, a rehearsal, stop conditions agreed in advance, and a pause kept as short as the plan allows. Upgrade runbooks and recovery test evidence are among the documented deliverables of our Kubernetes and OpenShift practice, and controlled upgrades are included in platform support within the agreed coverage.