Experimental Feature: MachineSetPreflightChecks (beta)

The MachineSetPreflightChecks feature can provide additional safety while creating new Machines and remediating existing unhealthy Machines of a MachineSet.

When a MachineSet creates machines under certain circumstances, the operation fails or leads to a new machine that will be deleted and recreated in a short timeframe, leading to unwanted Machine churn. Some of these circumstances include, but not limited to, creating a new Machine when Kubernetes version skew could be violated or joining a Machine when the Control Plane is upgrading leading to failure because of mixed kube-apiserver version or due to the cluster load balancer delays in adapting to the changes.

Enabling MachineSetPreflightChecks provides safety in such circumstances by making sure that a Machine is only created when it is safe to do so.

Feature gate name: MachineSetPreflightChecks

Variable name to enable/disable the feature gate: EXP_MACHINE_SET_PREFLIGHT_CHECKS

Supported PreflightChecks

ControlPlaneIsStable

  • This preflight check ensures that the ControlPlane is currently stable i.e. the ControlPlane is currently neither provisioning, upgrading.
  • For Clusters with a managed topology it also checks if a control plane upgrade is pending.
  • This preflight check is only performed if:
    • The Cluster uses a ControlPlane provider.
    • ControlPlane version is defined (ControlPlane.spec.version is set).

KubernetesVersionSkew

  • This preflight check ensures that the MachineSet and the ControlPlane conform to the Kubernetes version skew.
  • This preflight check is only performed if:
    • The Cluster uses a ControlPlane provider.
    • ControlPlane version is defined (ControlPlane.spec.version is set).
    • MachineSet version is defined (MachineSet.spec.template.spec.version is set).

KubeadmVersionSkew

  • This preflight check ensures that the MachineSet and the ControlPlane conform to the kubeadm version skew.
  • This preflight check is only performed if:
    • The Cluster uses a ControlPlane provider.
    • ControlPlane version is defined (ControlPlane.spec.version is set).
    • MachineSet version is defined (MachineSet.spec.template.spec.version is set).
    • MachineSet uses the Kubeadm Bootstrap provider.

ControlPlaneVersionSkew

  • This preflight check ensures that the MachineSet and the ControlPlane have the same version. The idea behind this check is that it doesn’t make sense to create a Machine with an old version, if we already know based on the control plane version that the Machine has to be replaced soon.
  • This preflight check is only performed if:
    • The Cluster has a managed topology
    • The Cluster uses a ControlPlane provider.
    • ControlPlane version is defined (ControlPlane.spec.version is set).
    • MachineSet version is defined (MachineSet.spec.template.spec.version is set).

Configuring MachineSet PreflightChecks

Per default all preflight checks are enabled for all MachineSets including new and existing MachineSets. The enabled preflight checks can be overwritten with the --machineset-preflight-checks command-line flag.

It is also possible to opt-out of one or all of the preflight checks on a per MachineSet basis by specifying a comma-separated list of the preflight checks via the machineset.cluster.x-k8s.io/skip-preflight-checks annotation on the MachineSet.

Examples:

  • To opt out of all the preflight checks set the machineset.cluster.x-k8s.io/skip-preflight-checks: All annotation.
  • To opt out of the ControlPlaneIsStable preflight check set the machineset.cluster.x-k8s.io/skip-preflight-checks: ControlPlaneIsStable annotation.
  • To opt out of multiple preflight checks set the machineset.cluster.x-k8s.io/skip-preflight-checks: ControlPlaneIsStable,KubernetesVersionSkew annotation.