Cluster API and Kubernetes version support

• Cluster API and Kubernetes version support
  • Version support policies
    • Cluster API release support
      • Skip upgrades
      • Downgrades
      • Cluster API release vs API versions
      • Cluster API release vs contract versions
      • Supported Cluster API - Cluster API provider version Skew
    • Kubernetes versions support
      • Maximum version skew between various Kubernetes components
  • Supported versions matrix by provider or component
    • Core provider (cluster-api-controller)
    • Kubeadm Bootstrap provider (kubeadm-bootstrap-controller)
      • Kubeadm configuration API Support
    • Kubeadm Control Plane provider (kubeadm-control-plane-controller)
      • Bootstrap provider Support
      • Etcd API Support
      • CoreDNS Support
    • Other providers
    • clusterctl
  • Annexes
    • Kubernetes version Support and Cluster API deployment model
    • Kubernetes version specific notes

Version support policies

Cluster API release support

This paragraph documents the general rules defining how we determine Cluster API supported releases.

A Cluster API release correspond to a release in the GitHub repository for this project, and the corresponding images published in the Kubernetes docker registry.

For the sake of this document, the most important artifacts included in a Cluster API release are:

• The Cluster API Core provider image
• The Kubeadm Bootstrap provider image
• The Kubeadm Control Plane provider image
• The clusterctl binary

The Cluster API team will release a new Cluster API version approximately every four months (3 releases each year). See release cycle and release calendars for more details about Cluster API releases management.

The Cluster API team actively supports the latest two minor releases (N, N-1); support in this context means that we:

• Have CI signal with E2E tests, unit tests, CVE scans etc.
• Accept bug fixes, perform golang or dependency bumps, etc.
• Periodically cut patch releases

On top of supporting the N and N-1 releases, the Cluster API team also maintains CI signal for the Cluster API N-2 releases in case we have to do an emergency patch release.

• If there is a need for an emergency patch, e.g. to fix a critical security issue, please bring this up to maintainers and it will be considered on a case-by-case basis.

All considered, each Cluster API minor release is supported for a period of roughly 12 months:

• The first eight months of this timeframe will be considered the standard support period for a minor release.
• The next four months the minor release will be considered in maintenance mode.
• At the end of the four-month maintenance mode period, the minor release will be considered EOL (end of life) and cherry picks to the associated branch are to be closed soon afterwards.

The table below documents support matrix for Cluster API versions (versions older than v1.0 omitted).

Skip upgrades

Cluster API supports at maximum n-3 minor version skip upgrades.

For example, if you are running Cluster API v1.6.x, you can upgrade up to Cluster API v1.9.x skipping intermediate minor versions (v1.6 is v1.9 minus three minor versions).

Downgrades

Cluster API does not support version downgrades.

Cluster API release vs API versions

Each Cluster API release can support one or more API versions.

An API version is determined from the GroupVersion defined in the top-level api/ package of a specific Cluster API release, and it is used in the apiVersion field of Cluster API custom resources.

An API version is considered deprecated when a new API version is published.

API deprecation and removal follow the Kubernetes Deprecation Policy; Cluster API maintainers might decide to support API versions longer than what is defined in the Kubernetes policy.

See 11920 for details about the v1beta1 removal plan. See 11919 for details about the v1alpha3/v1alpha4 removal plan. Note: Cluster API stopped to serve v1alpha3 API types from the v1.5 release and v1alpha4 types starting from the v1.6 release. Those types still exist in Cluster API while we work to a fix (or a workaround) for 10051.

Cluster API release vs contract versions

Each Cluster API contract version defines a set of rules a provider is expected to comply with in order to interact with a specific Cluster API release. Those rules can be in the form of CustomResourceDefinition (CRD) fields and/or expected behaviors to be implemented. See provider contracts

Each Cluster API release supports one contract version, and by convention the supported contract version matches the newest API version in the same Cluster API release.

A contract version might be temporarily compatible with older contract versions to ease transition of providers to a new supported version; compatibility for older contract versions will be dropped when the older contract version is EOL.

See 11920 for details about the v1beta1 removal plan.

Supported Cluster API - Cluster API provider version Skew

When running a Cluster API release, all the provider installed in the same management cluster MUST implement the CustomResourceDefinition (CRD) fields and/or expected behaviors defined by the release’s contract version.

As a corollary, provider’s version number and provider’s API version number are not required to match Cluster API versions.

Kubernetes versions support

This paragraph documents the general rules defining how the Cluster API team determines supported Kubernetes versions for every Cluster API release.

When a new Cluster API release is cut, we will document the Kubernetes version compatibility matrix the release has been tested with in the table below.

Each Cluster API minor release supports (when it’s initially created):

• 4 Kubernetes minor releases for the management cluster (N - N-3)
• 6 Kubernetes minor releases for the workload cluster (N - N-5)

When a new Kubernetes minor release is available, the Cluster API team will try to support it in an upcoming Cluster API patch release, thus extending the support matrix for the latest supported Cluster API minor release to:

• 5 Kubernetes minor releases for the management cluster (N - N-4)
• 7 Kubernetes minor releases for the workload cluster (N - N-6)

For example, Cluster API v1.7.0 would support the following Kubernetes versions:

• v1.26.x to v1.29.x for the management cluster
• v1.24.x to v1.29.x for the workload cluster
• When Kubernetes 1.30 is released, it will be supported in v1.7.x (but not in v1.6.x)

See Kubernetes version Support and Cluster API deployment model to understand how the way you deploy Cluster API might affect the Kubernetes version support matrix for a Cluster.

On a final comment, let’s praise all the contributors keeping care of such a wide support matrix. If someone is looking for opportunities to help with the project, this is definitely an area where additional hands and eyes will be more than welcome and greatly beneficial to the entire community.

Maximum version skew between various Kubernetes components

Standard Kubernetes version Skew Policy defines the maximum version skew supported between various Kubernetes components within a single cluster.

Notably, version skew between various Kubernetes components also define constraints to be observed by Cluster API, Cluster API providers or Cluster API users when performing Kubernetes version upgrades.

In some cases, also Cluster API and/or Cluster API providers are defining additional version skew constraints. For instance:

• If you are using kubeadm as a bootstrapper, you must abide to the kubeadm skew policy.
• If you are using image builder, all the Kubernetes components on a single machine are of the same version
• If your Cluster has a defined topology, with Cluster.spec.topology set and referencing a ClusterClass, Cluster API enforces a single Kubernetes version for all the machines in the cluster.

Supported versions matrix by provider or component

Core provider (cluster-api-controller)

The following table defines the support matrix for the Cluster API core provider. See Cluster API release support and Kubernetes versions support.

See also Kubernetes version specific notes.

Kubeadm Bootstrap provider (kubeadm-bootstrap-controller)

For each version of the Cluster API core provider, there is a corresponding version of the Kubeadm Bootstrap provider.

The Kubeadm Bootstrap provider also follows the same support rules defined in Cluster API release support and Kubernetes versions support.

As a consequence, the support matrix for the Kubeadm Bootstrap provider is the same as the one defined for the Cluster API Core provider.

Kubeadm configuration API Support

When creating new machines, the Kubeadm Bootstrap provider generates kubeadm init/join configuration files using the kubeadm API version recommended for the target Kubernetes version.

Kubeadm Control Plane provider (kubeadm-control-plane-controller)

For each version of the Cluster API core provider, there is a corresponding version of the Kubeadm Control Plane provider.

The Kubeadm Control Plane provider also follows the same support rules defined in Cluster API release support and Kubernetes versions support.

As a consequence, the support matrix for the Kubeadm Control Plane provider is the same as the one defined for the Cluster API Core provider.

Bootstrap provider Support

The Kubeadm Control Plane requires the Kubeadm Bootstrap provider of the same version.

Etcd API Support

The Kubeadm Control Plane provider communicates with the API server and etcd members of every Workload Cluster whose control plane it owns. All the Cluster API Kubeadm Control Plane providers currently supported are using etcd v3 API when communicating with etcd.

CoreDNS Support

Each version of the Kubeadm Control Plane can upgrade up to a max CoreDNS version. Notably, the Max CoreDNS version could change also with patch releases.

See corefile-migration

Other providers

Cluster API has a vibrant ecosystem of awesome providers maintained by independent teams and hosted outside of the Cluster API GitHub repository.

To understand the list of supported version of a specific provider, its own Kubernetes support matrix, supported API versions, supported contract version and specific skip upgrade rules, please see its documentation. Please refer to providers list

In general, if a provider version M says it is compatible with Cluster API version N, then it MUST be compatible with a subset of the Kubernetes versions supported by Cluster API version N.

clusterctl

It is strongly recommended to always use the latest patch version of clusterctl, in order to get all the fixes/latest changes.

In case of upgrades, clusterctl should be upgraded first and then used to upgrade all the other components.

Annexes

Kubernetes version Support and Cluster API deployment model

The most common deployment model for Cluster API assumes Core provider, Kubeadm Bootstrap provider, and Kubeadm Control Plane provider and at least one infrastructure provider running on the Management Cluster, all managing the lifecycle of a set of separate Workload clusters.

In this scenario, the Kubernetes version of the Management and Workload Clusters are allowed to be different. Additionally, Management Clusters and Workload Clusters can be upgraded independently and in any order.

In another deployment model for Cluster API, the Cluster API providers are used not only to managing the lifecycle of separate Workload clusters, but also to manage the lifecycle of the Management cluster itself. This cluster is also referred to as a “self-hosted” cluster.

The Kubernetes version of the “self-hosted” cluster is limited to the Kubernetes version currently supported for the Management clusters.

Kubernetes version specific notes

1.31:

• All providers:
  • It is not possible anymore to continuously apply CRDs that are setting caBundle to an invalid value (in our case Cg==). Instead of setting a dummy value the caBundle field should be dropped (#10972).
• Kubeadm Bootstrap provider:
  • kubeadm dropped the control-plane update-status phase which was used in ExperimentalRetryJoin (#10983).
  • kubeadm introduced the experimental ControlPlaneKubeletLocalMode feature gate which will be automatically enabled by CAPI for upgrades to v1.31 to not cause network disruptions (#10947).

1.29:

• In-tree cloud providers are now switched off by default. Please use DisableCloudProviders and DisableKubeletCloudCredentialProvider feature flags if you still need this functionality. (https://github.com/kubernetes/kubernetes/pull/117503)

1.24:

• Kubeadm Bootstrap provider:
  • kubeadm now sets both the node-role.kubernetes.io/control-plane and node-role.kubernetes.io/master taints on control plane nodes.
  • kubeadm now only sets the node-role.kubernetes.io/control-plane label on control plane nodes (the node-role.kubernetes.io/master label is not set anymore).
• Kubeadm Bootstrap provider and Kubeadm Control Plane provider
  • criSocket without a scheme prefix has been deprecated in the kubelet since a while. kubeadm now shows a warning if no scheme is present and eventually the support for criSocket‘s without prefix will be dropped. Please adjust the criSocket accordingly (e.g. unix:///var/run/containerd/containerd.sock) if you are configuring the criSocket in CABPK or KCP resources.