Contract rules for BootstrapConfig

Bootstrap providers SHOULD implement a BootstrapConfig resource.

The goal of a BootstrapConfig resource is to generates bootstrap data that is used to bootstrap a Kubernetes node. These may be e.g. cloud-init scripts.

The BootstrapConfig resource will be referenced by one of the Cluster API core resources, Machine.

The Machine’s controller will be responsible to coordinate operations of the BootstrapConfig, and the interaction between the Machine’s controller and the BootstrapConfig resource is based on the contract rules defined in this page.

Once contract rules are satisfied by a BootstrapConfig implementation, other implementation details could be addressed according to the specific needs (Cluster API is not prescriptive).

Nevertheless, it is always recommended to take a look at Cluster API controllers, in-tree providers, other providers and use them as a reference implementation (unless custom solutions are required in order to address very specific needs).

In order to facilitate the initial design for each BootstrapConfig resource, a few implementation best practices are explicitly called out in dedicated pages.

Rules (contract version v1beta1)

Note:

• All resources refers to all the provider’s resources “core” Cluster API interacts with; In the context of this page: BootstrapConfig, BootstrapConfigTemplate and corresponding list types

All resources: scope

All resources MUST be namespace-scoped.

All resources: TypeMeta and ObjectMeta field

All resources MUST have the standard Kubernetes TypeMeta and ObjectMeta fields.

All resources: APIVersion field value

In Kubernetes APIVersion is a combination of API group and version. Special consideration MUST applies to both API group and version for all the resources Cluster API interacts with.

All resources: API group

The domain for Cluster API resources is cluster.x-k8s.io, and bootstrap providers under the Kubernetes SIGS org generally use bootstrap.cluster.x-k8s.io as API group.

If your provider uses a different API group, you MUST grant full read/write RBAC permissions for resources in your API group to the Cluster API core controllers. The canonical way to do so is via a ClusterRole resource with the aggregation label cluster.x-k8s.io/aggregate-to-manager: "true".

The following is an example ClusterRole for a FooConfig resource in the bootstrap.foo.com API group:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    name: capi-foo-clusters
    labels:
      cluster.x-k8s.io/aggregate-to-manager: "true"
rules:
- apiGroups:
    - bootstrap.foo.com
  resources:
    - fooconfig
    - fooconfigtemplates
  verbs:
    - create
    - delete
    - get
    - list
    - patch
    - update
    - watch

Note: The write permissions are required because Cluster API manages BootstrapConfig generated from BootstrapConfigTemplates; when using ClusterClass and managed topologies, also BootstrapConfigTemplates are managed directly by Cluster API.

All resources: version

The resource Version defines the stability of the API and its backward compatibility guarantees. Examples include v1alpha1, v1beta1, v1, etc. and are governed by the Kubernetes API Deprecation Policy.

Your provider SHOULD abide by the same policies.

Note: The version of your provider does not need to be in sync with the version of core Cluster API resources. Instead, prefer choosing a version that matches the stability of the provider API and its backward compatibility guarantees.

Additionally:

Providers MUST set cluster.x-k8s.io/<version> label on the BootstrapConfig Custom Resource Definitions.

The label is a map from a Cluster API contract version to your Custom Resource Definition versions. The value is an underscore-delimited (_) list of versions. Each value MUST point to an available version in your CRD Spec.

The label allows Cluster API controllers to perform automatic conversions for object references, the controllers will pick the last available version in the list if multiple versions are found.

To apply the label to CRDs it’s possible to use commonLabels in your kustomize.yaml file, usually in config/crd:

commonLabels:
  cluster.x-k8s.io/v1alpha2: v1alpha1
  cluster.x-k8s.io/v1alpha3: v1alpha2
  cluster.x-k8s.io/v1beta1: v1beta1

An example of this is in the Kubeadm Bootstrap provider.

BootstrapConfig, BootstrapConfigList resource definition

You MUST define a BootstrapConfig resource. The BootstrapConfig resource name must have the format produced by sigs.k8s.io/cluster-api/util/contract.CalculateCRDName(Group, Kind).

Note: Cluster API is using such a naming convention to avoid an expensive CRD lookup operation when looking for labels from the CRD definition of the BootstrapConfig resource.

It is a generally applied convention to use names in the format ${env}Config, where ${env} is a, possibly short, name for the bootstrapper in question. For example KubeadmConfig is an implementation for kubeadm.

// +kubebuilder:object:root=true
// +kubebuilder:resource:path=fooconfig,scope=Namespaced,categories=cluster-api
// +kubebuilder:storageversion
// +kubebuilder:subresource:status

// FooConfig is the Schema for fooconfig.
type FooConfig struct {
    metav1.TypeMeta `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
    Spec FooConfigSpec `json:"spec,omitempty"`
    Status FooConfigStatus `json:"status,omitempty"`
}

type FooConfigSpec struct {
    // See other rules for more details about mandatory/optional fields in BootstrapConfig spec.
    // Other fields SHOULD be added based on the needs of your provider.
}

type FooConfigStatus struct {
    // See other rules for more details about mandatory/optional fields in BootstrapConfig status.
    // Other fields SHOULD be added based on the needs of your provider.
}

For each BootstrapConfig resource, you MUST also add the corresponding list resource. The list resource MUST be named as <BootstrapConfig>List.

// +kubebuilder:object:root=true

// FooConfigList contains a list of fooconfig.
type FooConfigList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []FooConfig `json:"items"`
}

BootstrapConfig: data secret

Each BootstrapConfig MUST store generated bootstrap data into a Kubernetes Secret.

The Secret containing bootstrap data must:

1. Use the API resource’s status.dataSecretName for its name
2. Have the label cluster.x-k8s.io/cluster-name set to the name of the cluster
3. Have a controller owner reference to the API resource
4. Have a single key, value, containing the bootstrap data

Note: because the dataSecretName is part of status, this value must be deterministically recreatable from the data in the Cluster, Machine, and/or bootstrap resource. If the name is randomly generated, it is not always possible to move the resource and its associated secret from one management cluster to another.

When the Secret is created its name MUST surface in the status.dataSecretName field of the BootstrapConfig resource; the Machine controller will surface this info in Machine’s spec.boostrap.dataSecretName when BootstrapConfig: initialization completed.

BootstrapConfig: initialization completed

Each BootstrapConfig MUST report when the bootstrap data secret is fully provisioned (initialization) by setting status.ready in the BootstrapConfig resource.

type FooConfigStatus struct {
    // ready denotes that the foo bootstrap data secret is fully provisioned.
	// NOTE: this field is part of the Cluster API contract and it is used to orchestrate provisioning.
	// The value of this field is never updated after provisioning is completed. Please use conditions
	// to check the operational state of the bootstrap config.
    // +optional
    Ready bool `json:"ready"`
    
    // See other rules for more details about mandatory/optional fields in BootstrapConfig status.
    // Other fields SHOULD be added based on the needs of your provider.
}

Once status.ready the Machine “core” controller will bubble up this info in Machine’s status.bootstrapConfigReady; Also BootstrapConfig’s status.dataSecretName will be surfaced on Machine’s corresponding fields at the same time.

BootstrapConfig: conditions

According to Kubernetes API Conventions, Conditions provide a standard mechanism for higher-level status reporting from a controller.

Providers implementers SHOULD implement status.conditions for their BootstrapConfig resource. In case conditions are implemented, Cluster API condition type MUST be used.

If a condition with type Ready exist, such condition will be mirrored in Machine’s BootstrapConfigReady condition.

Please note that the Ready condition is expected to surface the status of the BootstrapConfig during its own entire lifecycle, including initial provisioning, but not limited to that.

See Cluster API condition proposal for more context.

BootstrapConfig: terminal failures

Each BootstrapConfig SHOULD report when BootstrapConfig’s enter in a state that cannot be recovered (terminal failure) by setting status.failureReason and status.failureMessage in the BootstrapConfig resource.

type FooConfigStatus struct {
    // failureReason will be set in the event that there is a terminal problem reconciling the FooConfig 
    // and will contain a succinct value suitable for machine interpretation.
    //
    // This field should not be set for transitive errors that can be fixed automatically or with manual intervention,
    // but instead indicate that something is fundamentally wrong with the FooConfig and that it cannot be recovered.
    // +optional
    FailureReason *capierrors.ClusterStatusError `json:"failureReason,omitempty"`
    
    // failureMessage will be set in the event that there is a terminal problem reconciling the FooConfig
    // and will contain a more verbose string suitable for logging and human consumption.
    //
    // This field should not be set for transitive errors that can be fixed automatically or with manual intervention,
    // but instead indicate that something is fundamentally wrong with the FooConfig and that it cannot be recovered.
    // +optional
    FailureMessage *string `json:"failureMessage,omitempty"`
    
    // See other rules for more details about mandatory/optional fields in BootstrapConfig status.
    // Other fields SHOULD be added based on the needs of your provider.
}

Once status.failureReason and status.failureMessage are set on the BootstrapConfig resource, the Machine “core” controller will surface those info in the corresponding fields in Machine’s status.

Please note that once failureReason/failureMessage is set in Machine’s status, the only way to recover is to delete and recreate the Machine (it is a terminal failure).

BootstrapConfigTemplate, BootstrapConfigTemplateList resource definition

For a given BootstrapConfig resource, you MUST also add a corresponding BootstrapConfigTemplate resources in order to use it when defining set of machines, e.g. MachineDeployments.

The template resource MUST be named as <BootstrapConfig>Template.

// +kubebuilder:object:root=true
// +kubebuilder:resource:path=fooconfigtemplates,scope=Namespaced,categories=cluster-api
// +kubebuilder:storageversion

// FooConfigTemplate is the Schema for the fooconfigtemplates API.
type FooConfigTemplate struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec FooConfigTemplateSpec `json:"spec,omitempty"`
}

type FooConfigTemplateSpec struct {
    Template FooConfigTemplateResource `json:"template"`
}

type FooConfigTemplateResource struct {
    // Standard object's metadata.
    // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
    // +optional
    ObjectMeta clusterv1.ObjectMeta `json:"metadata,omitempty"`
    Spec FooConfigSpec `json:"spec"`
}

NOTE: in this example BootstrapConfigTemplate’s spec.template.spec embeds FooConfigSpec from BootstrapConfig. This might not always be the best choice depending of if/how BootstrapConfig’s spec fields applies to many machines vs only one.

For each BootstrapConfigTemplate resource, you MUST also add the corresponding list resource. The list resource MUST be named as <BootstrapConfigTemplate>List.

// +kubebuilder:object:root=true

// FooConfigTemplateList contains a list of FooConfigTemplates.
type FooConfigTemplateList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []FooConfigTemplate `json:"items"`
}

BootstrapConfigTemplate: support for SSA dry run

When Cluster API’s topology controller is trying to identify differences between templates defined in a ClusterClass and the current Cluster topology, it is required to run Server Side Apply (SSA) dry run call.

However, in case you immutability checks for your BootstrapConfigTemplate, this can lead the SSA dry run call to errors.

In order to avoid this BootstrapConfigTemplate MUST specifically implement support for SSA dry run calls from the topology controller.

The implementation requires to use controller runtime’s CustomValidator, available in CR versions >= v0.12.3.

This will allow to skip the immutability check only when the topology controller is dry running while preserving the validation behavior for all other cases.

See the DockerMachineTemplate webhook as a reference for a compatible implementation.

Sentinel file

A bootstrap provider’s bootstrap data must create /run/cluster-api/bootstrap-success.complete (or C:\run\cluster-api\bootstrap-success.complete for Windows machines) upon successful bootstrapping of a Kubernetes node. This allows infrastructure providers to detect and act on bootstrap failures.

Taint Nodes at creation

A bootstrap provider can optionally taint worker nodes at creation with node.cluster.x-k8s.io/uninitialized:NoSchedule. This taint is used to prevent workloads to be scheduled on Nodes before the node is initialized by Cluster API. As of today the Node initialization consists of syncing labels from Machines to Nodes. Once the labels have been initially synced the taint is removed from the Node.

Support for running multiple instances

Cluster API does not support running multiples instances of the same provider, which someone can assume an alternative solution to implement multi tenancy; same applies to the clusterctl CLI.

See Support running multiple instances of the same provider for more context.

However, if you want to make it possible for users to run multiples instances of your provider, your controller’s SHOULD:

• support the --namespace flag.
• support the --watch-filter flag.

Please, read carefully the page linked above to fully understand implications and risks related to this option.

Clusterctl support

The clusterctl command is designed to work with all the providers compliant with the rules defined in the clusterctl provider contract.

BootstrapConfig: pausing

Providers SHOULD implement the pause behaviour for every object with a reconciliation loop. This is done by checking if spec.paused is set on the Cluster object and by checking for the cluster.x-k8s.io/paused annotation on the BootstrapConfig object.

If implementing the pause behavior, providers SHOULD surface the paused status of an object using the Paused condition: Status.Conditions[Paused].

Typical BootstrapConfig reconciliation workflow

A bootstrap provider must respond to changes to its BootstrapConfig resources. This process is typically called reconciliation. The provider must watch for new, updated, and deleted resources and respond accordingly.

As a reference you can look at the following workflow to understand how the typical reconciliation workflow is implemented in BootstrapConfig controllers:

Behavior

A bootstrap provider must respond to changes to its bootstrap resources. This process is typically called reconciliation. The provider must watch for new, updated, and deleted resources and respond accordingly.

The following diagram shows the typical logic for a bootstrap provider:

1. If the resource does not have a Machine owner, exit the reconciliation
   1. The Cluster API Machine reconciler populates this based on the value in the Machine‘s spec.bootstrap.configRef field.
2. If the resource has status.failureReason or status.failureMessage set, exit the reconciliation
3. If the Cluster to which this resource belongs cannot be found, exit the reconciliation
4. Deterministically generate the name for the bootstrap data secret
5. Try to retrieve the Secret with the name from the previous step
   1. If it does not exist, generate bootstrap data and create the Secret
6. Set status.dataSecretName to the generated name
7. Set status.ready to true
8. Patch the resource to persist changes