KEP-4322: Cluster Profile API

Implementation History
ALPHA Provisional
Created 2023-09-13
Latest v1.34
Milestones
Alpha v1.28
Related Enhancements
Ownership
Owning SIG
SIG Multicluster
Participating SIGs
SIG Auth
Primary Authors
@qiujian16 @dixudx @mikeshng @corentone

KEP-4322: ClusterProfile API

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

Currently, there is a lack of a standardized approach to define a cluster inventory. However, with the growing number of users managing multiple clusters and deploying applications across them, projects like Open Cluster Management (OCM) , Clusternet , Kubernetes Fleet Manager or Karmada have emerged. This document introduces a proposal for a new universal ClusterProfile API. The objective is to establish a shared interface for cluster inventory, defining a standard for status reporting while allowing for multiple implementations.

Additionally, to manage an inventory of clusters, a platform admin can rely on having the cluster manager output ClusterProfile CRs that point to the clusters. Those CRs are key for multicluster controllers that want to operate on the clusters. However, there isn’t a single way to obtain credentials to reach those clusters. This KEP also provides a standardized way to obtain credentials for clusters when using ClusterProfile and makes it pluggable to allow the diverse ecosystem to support the multitude of ways to obtain credentials. It reuses part of the Kubeconfig external provider semantics (see KEP 541 ) to make implementation easier.

Motivation

A cluster inventory where users can discover Kubernetes clusters, their properties, and their status, is a common component in almost every major multi-cluster management solution. Yet, at this moment, there is not a standard way to access such an inventory; as we see more and more users embrace the cloud-native approach and deploy workloads across multiple clusters in concert with the help of multi-cluster management solutions, we believe that it is critical to devise a common API where applications, toolsets, and human operators can easily discover clusters under management.

By adopting this new ClusterProfile API, consumers no longer need to concern themselves with the implementation details of various projects. Instead, they can leverage a foundational API for multi-cluster management. Examples of consumers includes:

  • multiple cluster workload scheduler: we’ve seen requirements on distributing application/workload to multiple clusters. The scheduling can be based on certain cluster properties, e.g. cloud the cluster resides in, resources a cluster provides, or latency to some external endpoints. A common ClusterProfile API will give schedulers a standard to reason about clusters and help to foster the growth of this area.
  • GitOps tools (ArgoCD, flux etc) are having the requirement to deploy workload to multiple clusters. They either need to build the cluster concept by themselves or understand APIs representing a cluster from each cluster management project. A common ClusterProfile API can provide a thin compatible layer for different projects.
  • Operation tools or customized external consumers: this API gives a common way for different clouds and vendors to define clusters, providing a vendor agnostic integration point for external tools.
  • Cluster manager implementations themselves, for purposes such as grouping clusters into MCS API clustersets.

Furthermore, ClusterInventory is unfinished without an ability to use the clusters and controller writers have been very explicit that credentials are needed. Previous attempts at writing credentials have failed and we believe that a plugin model, also reusing known flows such as KEP 541 , will help solve the “credentials” need for ClusterProfiles.

See also: Credential Plugin introduction slides

Goals

  • Establish a standardized ClusterProfile API to represent clusters.
  • Lay the groundwork for multi-cluster tooling by providing a foundational component.
  • Accommodate multiple implementations to encourage flexibility and adoption.
  • Allow for future extensions and new use cases by leaving room for further development and enhancements. A unified API, such as this one, is most effective when platform extension authors can use it as a foundational tool to create extensions compatible with multiple providers.
  • Allow cluster managers of different types to share a single point inventory.
  • Provide a library for controllers to obtain credentials for a cluster represented by a ClusterProfile
  • Allow cluster managers to provide a method to obtain credentials that doesn’t require to be embedded into the controller code and recompiling.
  • Be a secure mechanism for credential obtention and storage.

Non-Goals

  • Provide a standard reference implementation.
  • Define specific implementation details beyond general API behavior.
  • Offering functionalities related to multi-cluster orchestration.
  • Define the Consumer registration API
  • Define the mechanism for shipping plugins to be used by the controllers and their delivery in the controller image/pod.
  • Design plugin or a library for plugins
  • Mandate Federated workload identity / OIDC frameworks (though they are recommended)

Proposal

the API proposed by this KEP aims to

  • Provide a reliable, consistent, and automated approach for any multi-cluster application (framework, toolset) to discover available clusters and take actions accordingly, in a way similar to service discovery works in a microservice architecture. Through the inventory, the application can query for a list of clusters to access, or watch for an ever-flowing stream of cluster lifecycle events which the application can act upon timely, such as auto-scaling, upgrades, failures, and connectivity issues.
  • Provide a simple, clear interface for human operators to understand clusters under management.
  • Provide a standardized, pluggable mechanism for obtaining credentials to access clusters represented by ClusterProfile objects. The proposed approach is to leverage plugins for retrieving the credentials from an issuer recognized by the target cluster. The controller using ClusterProfile would use a library to run a local executable which would retrieve the credentials for the current controller and a given clusterprofile. Plugins would be exec’ed by the controller so that they don’t need be built-in the binary, allowing flexibility into writing their own access plugins and still leveraging multicluster controllers written by the community. We propose to reuse the exec approach and protocol used for external credentials in Kubeconfig (but not the configuration part of kubeconfig). Finally, in order to retrieve the endpoint for the cluster, we standardize the property names that are used in ClusterProfile.

Terminology

  • Cluster Inventory: A conceptual term referring to a collection of clusters.

  • Member Cluster: A kubernetes cluster that is part of a cluster inventory.

  • Cluster Manager: An entity that creates the ClusterProfile API object per member cluster, and keeps their status up-to-date. Each cluster manager MUST be identified with a unique name. Each ClusterProfile resource SHOULD be managed by only one cluster manager. A cluster manager SHOULD have sufficient permission to access the member cluster to fetch the information so it can update the status of the ClusterProfile API resource.

  • ClusterInventory Consumer: Controllers or tools that leverage the ClusterProfile for the purpose of workload distribution, operation management etc. Their name must be unique for a single inventory. They might need to register themselves with the Cluster Manager which is not defined in this KEP.

User Stories (Optional)

Story 1: Multicluster Workload Distribution

In this scenario, a unified API acts as a foundation for multicluster scheduling decisions across various clusters. This means that the API will provide a single point of contact to retrieve information that can be used to make informed scheduling decisions for workloads across multiple clusters. For instance, workload distribution tools like GitOps or Work API can leverage this API to make informed decisions about which cluster is best suited to handle a particular workload.

Examples of the multicluster scheduling includes:

  • As a user I want to run a workload on an EKS cluster that resides in us-east-1. I want to submit my workload if and only if a cluster satisfies that constraint.
  • As a user I want to run a workload on a cluster that has certain CRDs installed.
  • As a user I want to deploy a workload to my on-prem clusters.
  • As a user I want to run a workload close to the data source or end-users.
  • As a user I want to run a workload to the less busy cluster.
  • As a user I want to deploy workloads to Cluster API managed clusters based on cluster location and platform.

Story 2: Operations and Management

For multi-cluster administrators, the unified API provides a comprehensive view of all the clusters under their management. This includes verifying their memberships, understanding their status, capacity, and healthiness.

The API could also provide insights into the membership of each cluster, such as the number of nodes, their roles, and their current status. This can help administrators manage the clusters more effectively and ensure that they are always operating at optimal levels.

Story 3: Transparent to Consumers

The unified API ensures that consumers have the flexibility to choose different cluster management tools and switch among them as needed. Regardless of the tool they choose, they all use the same API to define clusters.

This means that consumers can switch from one cluster manager to another without having to worry about compatibility issues or learning a new API. This can significantly reduce the learning curve and make it easier for consumers to adopt new tools.

Moreover, the unified API can also provide a consistent user experience across different cluster managers. For example, if a consumer is used to a particular command or function in one tool, they can expect the same command or function to work in the same way in another tool. This can further enhance the usability and adoption of different cluster manager.

Notes/Constraints/Caveats

What’s the relationship between the ClusterProfile API and Cluster Inventory?

The ClusterProfile API represents a single member cluster in a cluster inventory.

What’s the relationship between a cluster inventory and clusterSet?

A cluster inventory may or may not represent a ClusterSet. A cluster inventory is considered a clusterSet if all its member clusters adhere to the namespace sameness principle. Note that a cluster can only be in one ClusterSet while there is not such restriction for a cluster inventory.

How should the API be consumed?

We recommend that all ClusterProfile objects within the same cluster inventory reside on a dedicated Kubernetes cluster (aka. the hub cluster). This approach allows consumers to have a single integration point to access all the information within a cluster inventory. Additionally, a multi-cluster aware controller can be run on the dedicated cluster to offer high-level functionalities over this inventory of clusters.

How should we organize ClusterProfile objects on a hub cluster?

While there are no strict requirements, we recommend making the ClusterProfile API a namespace-scoped object. This approach allows users to leverage Kubernetes’ native namespace-based RBAC if they wish to restrict access to certain clusters within the inventory.

However, if a cluster inventory represents a ClusterSet, all its ClusterProfile objects MUST be part of the same clusterSet and namespace must be used as the grouping mechanism. In addition, the namespace must have a label with the key “clusterset.multicluster.x-k8s.io” and the value as the name of the clusterSet.

Uniqueness of the ClusterProfile object

While there are no strict requirements, we recommend that there is only one ClusterProfile object representing any member cluster on a hub cluster.

However, a ClusterProfile object can only be in one ClusterSet since the namespace sameness property is transitive, therefore it can only be in the namespace of that clusterSet if it is in a ClusterSet.

Risks and Mitigations

Because of its interaction with authentication and credentials, particular attention in the access provider plugin design must be paid to security:

  • Credentials leak: ClusterProfile, Controller configuration and Plugin configuration should never contain sensitive information.
  • Plugin poisoning: supporting access provider plugins in a controller relies on trusting the plugin itself and its path in the filesystem. Particular attention must be provided by the user deploying a controller to make sure the plugins that they install are from a trusted source as they will have access to the controller’s identity. In addition, the path of the plugin may be edited or hijacked by an attacker which would then sit in lieu of the normal plugin, allowing process execution by the controller’s process. This risk is mitigated by the assumption that the pod’s filesystem is private to it and that no lower-privileged (or separate) processes are able to access it.

Another risk is around AuthZ. This design doesn’t cover the distribution of RBAC to multiple clusters and identifying what principal a controller can be identified as. This setup is currently left to the responsibility of the platform admin setting up the different clusters and controllers.

Design Details

We try to summarize the most common properties that should be able to represent a cluster and support the above use case at the minimum scope.

The target consumers of the API are users who manage the clusters and other tools to understand the clusters concept from the API for multi-cluster scheduling, workload distribution and cluster management. The API aims to provide the information for the consumers to answer the below questions:

  • Is the cluster under management?
  • How can I select a cluster?
  • Is the cluster healthy?
  • How to access the cluster?
  • Does the cluster have certain capabilities or properties?
  • Does the cluster have sufficient resources?

Cluster Name

It is required that cluster name is unique for each cluster, and it should also be unique among different providers (cluster manager). It is cluster manager’s responsibility to ensure the name uniqueness.

It’s the responsibility of the cluster manager platform administrator to ensure cluster name uniqueness. The examples below serve more as recommendations than hard requirements, providing guidance on best practices.

Example 1

The metadata.name of the cluster should never be set upon creation. Only metadata.generateName can be set.

Example 2

The metadata.name and metadata.generateName must have prefix which should be the same as the spec.clusterManager.name. Different cluster manager must set a different value of spec.clusterManager.name when the cluster is created.

Spec

Display name

It is a human-readable name of the cluster set by the consumer of the cluster.

Cluster Manager

An immutable field set by a cluster manager when this cluster resources is created by it. Each cluster manager instance should set a different values to this field.

In addition, a predefined label with key “x-k8s.io/cluster-manager” needs to be added by the cluster manager upon creation. The value of the label MUST be the same as the name of the cluster manager. The purpose of this label is to make filter clusters from different cluster managers easier.

Status

Version

Kubernetes version of the cluster

Versions of the kubernetes can let consumers understand the capability of the kubernetes, such as what API is supported.

With recent conversations about kube-apiserver and enabled featureset version, it is possible to incorporate other version relating to the cluster, such as minimum kubelet version, maximum kubelet version, and enabled featureset version.

Properties

Name/value pairs to represent properties of the clusters. It could be a collection of ClusterProperty resources, but could also be info based on other implementations. The name of the cluster property can be predefined name from ClusterProperty resources and is allowed to be customized by different cluster managers.

Conditions

Record cluster’s healthiness condition and easy to extend, conforming to metav1.Condition format.

Predefined condition types:

  • Healthy conditions indicate the cluster is in a good state. (state: True/False/Unknown). Healthiness can have different meanings in different scenarios. We will have multiple condition types to define healthiness:
    • ControlPlaneHealthy is to define whether the controlplane of the cluster is in the healthy state.
      • apiserver/healthz
      • controller-manager/healthz
      • scheduler/healthz
      • etcd/healthz
    • «[UNRESOLVED]» AllNodesHealthy is to define if the nodes in the cluster are in a healthy state. If one node is not healthy, the status of NodeHealthy is set to false with the message indicating details. (todo tolerance, it should be configurable) It would be useful to collect the healthiness of other subsystems in the cluster, e.g. network, dns, storage, or ingress. However, it is not easy to collect that information in a common way with different implementations of network or storage providers. We decide not to include other subsystems healthiness conditions in the initial phase.
  • Joined: indicate the cluster is under management by the cluster manager. The status of the cluster SHOULD be updated by the cluster manager under this condition.

Access Providers

We standardize a new field in ClusterProfile called accessProviders that is stored in the Status of the ClusterProfile.

All the data from this structure is specific to the clusterProfile and does not contain any Controller-specific information. It must be usable by different controller, applications or consumers without requiring changes. It also cannot contain any data considered a secret; and we consider that reachability information is not sensitive.

The definition is as follows:

type AccessProviders struct {
  // +listType=map
  // +listMapKey=name
  accessProviders []AccessConfig
}

// AccessType defines the type of access provider that is used to reach
// the cluster. For example, GCP access (using tokens that are understood
// by GCP's IAM) is designated by the string `google`.
type AccessType string

// AccessConfig gives more details on data that is necessary to reach
// the cluster for this kind of access provider.
type AccessConfig struct {
  Name string
  Cluster *Cluster
}

See Access Provider Plugin Design for full details on how accessProviders is consumed by the access provider plugin mechanism.

Cluster Access

There are multiple methods for a ClusterInventory Consumer to gain access to the cluster represented by a ClusterProfile API. This KEP does not define the exact mechanism for each approach, but it is recommended that ClusterInventory Consumers avoid using a secret if possible. Here are the four approaches:

Pull Model with Work API

The ClusterInventory Consumer can leverage the Work API so that the work API agent on the Member Cluster can pull the necessary objects from the hub cluster where the ClusterInventory Consumer operates to the Member Cluster. This approach allows the Cluster Manager to manage the RBAC for each Consumer on the hub cluster, ensuring that their access is restricted to the corresponding namespaces of the ClusterProfile objects.

The ClusterInventory consumer can utilize identity federation mechanisms, such as Azure Workload Identity Federation or GCP Workload Identity Federation . This allows the ClusterInventory Consumer to use an identity that already has access to the clusters in the Cluster Inventory. While the Cluster Manager can assist in setting up the federation, it is not a mandatory requirement.

The ClusterInventory Consumer can obtain credentials to access the cluster represented by a ClusterProfile object by reading from a secret. In this approach, the Cluster Manager generates secrets containing the necessary credentials within the namespace accessible to the ClusterInventory Consumer. For this to function correctly, Cluster Managers must be aware of the following details about the consumer: their name, whether credentials are required, and the preferred unique namespace for reading credentials as secrets. Those information can be obtained during the “registering” process but this is out of the scope of this KEP.

Secret format
  • The secret MUST reside in the namespace with the label x-k8s.io/cluster-inventory-consumer with the value being the name of the ClusterInventory Consumer.
  • The secret MUST contain the label x-k8s.io/cluster-profile with the value being the name of the ClusterProfile object that the secret is associated with.
  • The secret MAY contain the label x-k8s.io/cluster-profile-namespace with the value being the namespace of the ClusterProfile object that the secret is associated with. If not present, the ClusterProfile is assumed to be in the default namespace.
  • The access information in the secret must contain the following fields
    • Config: This field contains cluster access information compatible with the kubeconfig format .
    • Since a single Kubeconfig supports access to multiple clusters, the Cluster manager MUST ensure that each secret contains access information for only a single consumer.

Push Model via Access Provider Plugins

The ClusterInventory Consumer can obtain credentials to access the cluster represented by a ClusterProfile object by using an access provider plugin. In this approach, the controller using ClusterProfile would use a library to run a local executable which would retrieve the credentials for the current controller and a given clusterprofile. It is expected that plugins would leverage elements local to the controller to help assert the identity of the controller (environment variables, config files, KSA, the local IP, etc.) to retrieve credentials that are valid on the target cluster. This approach reuses the exec protocol defined in KEP 541 , giving the ability to reuse the code in client-go. The cluster’s endpoint and TLS configuration are read from the accessProviders field in the ClusterProfile status.

See Access Provider Plugin Design for the full design details.

Access Provider Plugin Design

The access provider plugin implementation would be done via a library in https://github.com/kubernetes-sigs/cluster-inventory-api . The library would be in golang. The library is provided as the community shared implementation for golang and it is possible that other implementations would be created, and would work with the same plugin mechanism defined here, allowing for reuse of the external providers that cluster managers write.

The expected prototype for a controller is expected to be the following:

func (c *access.Config) BuildConfigFromCP(cp *ClusterProfile) (*rest.Config, error)

The library implementation flow is expected to be as follows:

  1. Build the endpoint details of the cluster by reading properties of the ClusterProfile
  2. Call the external access provider plugin, following the same flow defined in KEP 541 (giving the ability to reuse the code in client-go’s exec package )
  3. If the Cluster includes an extensions entry named client.authentication.k8s.io/exec, pass its extension object through to ExecCredential.Spec.Cluster.Config as plugin configuration.
  4. Build the rest.Config and return it to the caller

External Access Provider Plugin Mechanism

In order to call the plugin, the library execs the plugin defined in the configuration. It passes the Cluster information that was obtained from the ClusterProfile. The library then calls the plugin following the protocol defined in KEP 541 . The library provided in https://github.com/kubernetes-sigs/cluster-inventory-api can leverage the original code that is kept in client-go .

Standardizing the Provider Definition

In order to populate the Cluster object that the exec provider requires, a new field in ClusterProfile called accessProviders is standardized and stored in the Status of the ClusterProfile (see Access Providers ).

Cluster Data

The Cluster structure for the exec defined in KEP 541, implemented in k/client-go assumes the following:

type Cluster struct {
	// LocationOfOrigin indicates where this object came from. It is used for round tripping config post-merge, but never serialized.
	// +k8s:conversion-gen=false
	LocationOfOrigin string `json:"-"`
	// Server is the address of the kubernetes cluster (https://hostname:port).
	Server string `json:"server"`
	// TLSServerName is used to check server certificate. If TLSServerName is empty, the hostname used to contact the server is used.
	// +optional
	TLSServerName string `json:"tls-server-name,omitempty"`
	// InsecureSkipTLSVerify skips the validity check for the server's certificate. This will make your HTTPS connections insecure.
	// +optional
	InsecureSkipTLSVerify bool `json:"insecure-skip-tls-verify,omitempty"`
	// CertificateAuthority is the path to a cert file for the certificate authority.
	// +optional
	CertificateAuthority string `json:"certificate-authority,omitempty"`
	// CertificateAuthorityData contains PEM-encoded certificate authority certificates. Overrides CertificateAuthority
	// +optional
	CertificateAuthorityData []byte `json:"certificate-authority-data,omitempty"`
	// ProxyURL is the URL to the proxy to be used for all requests made by this
	// client. URLs with "http", "https", and "socks5" schemes are supported. If
	// this configuration is not provided or the empty string, the client
	// attempts to construct a proxy configuration from http_proxy and
	// https_proxy environment variables. If these environment variables are not
	// set, the client does not attempt to proxy requests.
	//
	// socks5 proxying does not currently support spdy streaming endpoints (exec,
	// attach, port forward).
	// +optional
	ProxyURL string `json:"proxy-url,omitempty"`
	// DisableCompression allows client to opt-out of response compression for all requests to the server. This is useful
	// to speed up requests (specifically lists) when client-server network bandwidth is ample, by saving time on
	// compression (server-side) and decompression (client-side): https://github.com/kubernetes/kubernetes/issues/112296.
	// +optional
	DisableCompression bool `json:"disable-compression,omitempty"`
	// Extensions holds additional information. This is useful for extenders so that reads and writes don't clobber unknown fields
	// +optional
	Extensions map[string]runtime.Object `json:"extensions,omitempty"`
}

In this structure, not all fields would apply, such as:

  • CertificateAuthority, which points to a file (and a ClusterProfile doesn’t have a filesystem)

And there are fields that require special attention:

  • Extensions, which holds additional, usually cluster-specific information, that might help authenticate with the cluster. (For more information about this field and how it is handled, see the section on Passing Plugin Configuration via Extensions ).
Passing Plugin Configuration via Extensions

Some access providers require cluster-specific, non-secret parameters (for example, a clusterName) in order to obtain credentials. To standardize how this information is conveyed from a ClusterProfile to a plugin, the library follows the existing convention defined by the client authentication API:

Optional: when a plugin needs per-cluster, non-secret config, set an extension entry with name: client.authentication.k8s.io/exec under Cluster.extensions. The library reads only the extension field of that entry and passes it through verbatim to ExecCredential.Spec.Cluster.Config. The content must be non-secret and cluster-specific. Controller- or environment-specific data must not be placed here. Plugins may read values (e.g. clusterName) from ExecCredential.Spec.Cluster.Config.

Reference: client.authentication.k8s.io/v1 Cluster: config sourced from extensions[client.authentication.k8s.io/exec]

Example (embedded in ClusterProfile.status.accessProviders[].cluster):

extensions:
- name: client.authentication.k8s.io/exec
  extension:
    clusterName: spoke-1

In practice, however, there exist certain scenarios where setting the reserved client.authentication.k8s.io/exec extension to pass cluster-specific data might not be appropriate: libraries such as client/go will eventually save the extension data (along with other information, including the CA bundles for a cluster) to an environment variable, KUBERNETES_EXEC_INFO, which exec plugins can read; however:

  • some exec plugins might be expecting inputs from CLI arguments or plugin-specific environment variables directly; they might not read the KUBERNETES_EXEC_INFO environment variable at all, or might make only limited use of the environment variable.
  • it might not be proper to set the KUBERNETES_EXEC_INFO environment variable in the target environment: for example, KUBERNETES_EXEC_INFO includes CA bundles for a cluster and its size might exceed length limitations in the environment.
  • the client.authentication.k8s.io/exec extension keeps the data in the free form, runtime.RawExtension; before it is saved to the KUBERNETES_EXEC_INFO environment variable, client/go will pass it to the ExecConfig.Config struct first, which accepts only runtime.Object data. This brings about possible marshalling/unmarshalling complications, which could be difficult to handle gracefully for the community-provided library proposed in this KEP.

To address the deficiencies above, we further propose that:

  • this KEP reserves a name in the extensions, clusterprofiles.multicluster.x-k8s.io/exec/additional-args, which holds additional CLI arguments that would be supplied to the exec plugin when the ClusterProfile API and community-provided library are used for authentication.

    If an extension under this name is present, the community-provided library will extract the data, and append the additional arguments to the ExecConfig struct (specifically the ExecConfig.Args field) that will be used to prepare the rest.Config output. The arguments will then be used to invoke the exec plugin.

    The additional arguments shall be saved as a string array in the YAML format.

    For simplicity reasons, the community-provided library will not perform any de-duplication on the CLI arguments after the additional arguments are appended.

  • this KEP reserves another name in the extensions, clusterprofiles.multicluster.x-k8s.io/exec/additional-envs, which holds additional environment variables that would be supplied upon calling the exec plugin when the ClusterProfile API and community-provided library are used for authentication.

    If an extension under this name is present, the community-provided library will extract the data, and add the additional variables to the ExecConfig struct (specifically the ExecConfig.Env field) that will be used to prepare the rest.Config output. The variables will then be set when invoking the exec plugin.

    The additional environment variables shall be represented as a string map in the YAML format.

    The community-provided library will de-duplicate the list of environment variables when adding the additional variables; if two entries are present under the same name, the one from the extension will prevail.

Security Concerns

With the addition of newly reserved extensions, understandably there might be situations where users might want to block additional CLI arguments or environment variables from being set due to security reasons. To resolve this, the KEP proposes that the community-provided library implementation must allow users to specify whether additional CLI arguments or environment variables can be set by a ClusterProfile object. By default the reserved extensions should be ignored.

See the Configuring Plugins in the Controller section for more information.

Configuring Plugins in the Controller

Plugins are selected by a string which represents the type of access provider that is used to reach the cluster, for example, “google” for GKE Clusters. This allows the controller to attach a different binary name or path for the binary.

It is expected that the library will have a mapping from its supported type of access provider to the expected binary to call. The library would be fed via a JSON configuration file, specified by the --clusterprofile-provider-file flag. The file maps access provider types to the associated exec configuration and potential flags that should be passed. It cannot contain cluster-specific information (which is not known at that time).

./controller ... --clusterprofile-provider-file "clusterprofile-provider-file.json"

The structure for each Provider in the configuration file:

type Provider struct {
  Name                        string
  ExecConfig                  *clientcmdapi.ExecConfig
  ProfileSourcedCLIArgsPolicy ProfileSourcedCLIArgsPolicy
  ProfileSourcedEnvVarsPolicy ProfileSourcedEnvVarsPolicy
}

Given the plugin is executed directly by the controller, it may expect to have access to the same environment as the controller itself, inclusive of envvars, filesystem and network. It is expected that the identity of the plugin is the same as the controller itself.

The ClusterProfileSourcedCLIArgsPolicy and ClusterProfileSourcedEnvVarsPolicy flags control whether the library will process clusterprofiles.multicluster.x-k8s.io/exec/additional-args and clusterprofiles.multicluster.x-k8s.io/exec/additional-envs extensions, as described earlier. If set to Ignore, additional CLI arguments and/or environment variables cannot be set from the ClusterProfile side.

Plugin Examples

As an example, we provide pseudocode for plugins that could easily be implemented with the protocol. They are ultrasimplified versions of the code and structures to convey the idea and not be an implementation example.

Secret Reader Plugin

This plugin assumes the controller is aware of the list of clusters ahead of time and has created secrets for them in its namespace. It simply reads the token from the secret mapped to the cluster specifically for this controller. Note that namespace comes from the controller config while clusterName is read by the plugin from ExecCredential.Spec.Cluster.Config, which the library populates from the Cluster.extensions entry named client.authentication.k8s.io/exec.

func GetToken(namespace, clusterName string) string {
  // query secrets local to this controller (same cluster, same namespace)
  secret := secrets.Namespace(namespace).Get(clusterName)
  return secret.Data.token
}
GKE with Workload Identity Federation

This plugin uses Workload Identity Federation to call the other clusters that are GKE clusters and therefore understanding google-issued credentials.

func GetToken() string {
  // This library calls looks at the standard envvar called GOOGLE_CREDENTIALS and if not found, calls the Metadata Server IP (169.254.169.254)
  creds := google.GetDefaultCredentials()
  return creds.Token()
}

API Example

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
 name: generated-cluster-name
 labels:
   x-k8s.io/cluster-manager: some-cluster-manager
spec:
  displayName: cluster-us-east
  clusterManager:
    name: some-cluster-manager
status:
  version:
    kubernetes: 1.28.0
  properties:
   - name: clusterset.k8s.io
     value: some-clusterset
   - name: location
     value: apac
  conditions:
   - type: ControlPlaneHealthy
     status: True
     lastTransitionTime: "2023-05-08T07:56:55Z"
     message: ""
   - type: Joined
     status: True
     lastTransitionTime: "2023-05-08T07:58:55Z"
     message: ""

API Examples with Access Providers

Below is an example of a GKE ClusterProfile, which would map to a plugin providing credentials of type google:

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
 name: my-cluster-1
spec:
  displayName: my-cluster-1
  clusterManager:
    name: GKE-Fleet
status:
  version:
    kubernetes: 1.28.0
  properties:
   - name: clusterset.k8s.io
     value: some-clusterset
   - name: location
     value: us-central1
  accessProviders:
  - name: google
    cluster:
      server: https://connectgateway.googleapis.com/v1/projects/123456789/locations/us-central1/gkeMemberships/my-cluster-1

Below are some examples that feature the use of extensions in ClusterProfiles:

  • This example uses the reserved client.authentication.k8s.io/exec extension to pass cluster names to a plugin of the secret reader type:

    apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
  name: my-cluster-1
spec:
  displayName: my-cluster-1
  clusterManager:
    name: inhouse-manager
status:
  accessProviders:
  - name: secretreader
    cluster:
      server: https://<spoke-server>
      certificate-authority-data: <BASE64_CA>
      extensions:
      - name: client.authentication.k8s.io/exec
        extension:
          clusterName: spoke-1

  • This example uses the clusterprofiles.multicluster.x-k8s.io/exec/additional-args extension to pass additional CLI arguments (-audience https://my-on-prem-k8s.example.dev) to the exec plugin when the spire-agent access provider is used, as the cluster’s authentication solution is expecting tokens with this specific audience for security reasons.

    apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
  name: my-on-prem-cluster
spec: ...
status:
  ...
  accessProviders:
  - name: spire-agent
    cluster:
      server: https://my-on-prem-k8s.example.dev
      ...
      extensions:
      - name: "clusterprofiles.multicluster.x-k8s.io/exec/additional-args"
        extension:
        - "-audience"
        - "https://my-on-prem-k8s.example.dev"

  • This example uses the clusterprofiles.multicluster.x-k8s.io/exec/additional-envs extension to pass additional environment variables CLIENT_ID and TENANT_ID to the exec plugin when the kubelogin access provider is used; these entries can help the exec plugin exchange for cluster-specific access tokens.

    apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
 name: my-aks-cluster
spec: ...
status:
  ...
  accessProviders:
  - name: kubelogin
    cluster:
      server: https://braveion-abcxyz.hcp.eastus2.azmk8s.io
      ...
      extensions:
      - name: "clusterprofiles.multicluster.x-k8s.io/exec/additional-envs"
        extension:
          "CLIENT_ID": "my-client-id"
          "TENANT_ID": "my-tenant-id"

Scalability implication

The API should provide summarized metadata of the cluster and relatively “static” cluster status. Dynamic data, e.g. cluster resource usage, should not be included in this API given it will bring heavy traffic to the control plane. A metrics collector system would be better suited in this scenario.

The QPS for each single cluster object is supposed to be less than 1/30 (30s per update on average). It should be achievable since the cluster properties in the status field are not supposed to be changed too frequenly. The Burst of each single cluster object is supposed to be 10 to handle initial join and sudden storm.

Test Plan

  • I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

This KEP proposes and out-of-tree CRD here that is not expected to integrate with any of the Kubernetes CI infrastructure. In addition, it explicitly provides only the CRD definition and generated clients for use by third party implementers, and does not provide a controller or any other binary with business logic to test. Therefore, we only expect unit test to validate the generated client and integration tests for API validation tests.

However, similar to other out-of-tree CRDs that serve third party implementers, such as Gateway API and MCS API, there is rationale for the project to provide conformance tests for implementers to use to confirm they adhere to the restrictions set forth in this KEP that are not otherwise enforced by the CRD definition.

Prerequisite testing updates
Unit tests
  • <package>: <date> - <test coverage>
Integration tests
  • :
e2e tests
  • :

Graduation Criteria

Alpha

  • A CRD definition and generated client.
  • A dummy controller and unit test to validate the CRD and client.
  • An access provider plugin library and protocol definition.
  • Unit tests for the access provider library.

Beta

  • Gather feedback from users during the Alpha stage to identify any issues, limitations, or areas for improvement. Address this feedback by making the necessary changes to the API and iterating on its design and functionality.
  • The API should support the addition of scheduling features, such as:
    • Load balancing: Distribute workloads evenly across clusters based on their current load and capacity.
    • Affinity and anti-affinity rules: Allow users to define rules for placing workloads on specific clusters or ensuring that certain workloads are not placed on the same cluster.
    • Priority-based scheduling: Enable users to assign priorities to workloads, ensuring that higher-priority workloads are scheduled before lower-priority ones.
    • Resource-based scheduling: Schedule workloads based on the availability of specific resources, such as CPU, memory, or storage, in each cluster.
  • The API should expose access information including but not limited to:
    • APIServer endpoint url of the member cluster.
    • Credential with limited access to the member cluster.
  • At least two providers and one consumer using ClusterProfile API.

GA

  • N examples of real-world usage
  • N installs
  • More rigorous forms of testing—e.g., downgrade tests and scalability tests
  • Allowing time for feedback
  • Stability: The API should demonstrate stability in terms of its reliability.
  • Functionality: The API should provide the necessary functionality for multicluster scheduling, including the ability to distribute workloads across clusters, manage cluster memberships, and monitor cluster health and capacity. This should be validated through a series of functional tests and real-world use cases.
  • Integration: Ensure that the API can be easily integrated with popular workload distribution tools, such as GitOps and Work API. This may involve developing plugins or extensions for these tools or providing clear guidelines on how to integrate them with the unified API.
  • Performance and Scalability: Conduct performance and scalability tests to ensure that the API can handle a large number of clusters and workloads without degrading its performance. This may involve stress testing the API with a high volume of requests or simulating large-scale deployments.

Note: Generally we also wait at least two releases between beta and GA/stable, because there’s no opportunity for user feedback, or even bug reports, in back-to-back releases.

Upgrade / Downgrade Strategy

Version Skew Strategy

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?
  • Feature gate (also fill in values in kep.yaml)
    • Feature gate name:
    • Components depending on the feature gate:
  • Other
    • Describe the mechanism:
    • Will enabling / disabling the feature require downtime of the control plane?
    • Will enabling / disabling the feature require downtime or reprovisioning of a node?
Does enabling the feature change any default behavior?
  • No default Kubernetes behavior is currently planned to be based on this feature; it is designed to be used by the separately installed, out-of-tree, multicluster management providers and consumers.
Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
  • Yes, as this feature only describes a CRD, it can most directly be disabled by uninstalling the CRD.
What happens if we reenable the feature if it was previously rolled back?
Are there any tests for feature enablement/disablement?
  • As a dependency only for an out-of-tree component, there will not be e2e tests for feature enablement/disablement of this CRD in core Kubernetes. The e2e test can be provided by multicluster management providers who support this API.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?
What specific metrics should inform a rollback?
Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?
How can someone using this feature know that it is working for their instance?
  • Events
    • Event Reason:
  • API .status
    • Condition name:
    • Other field:
  • Other (treat as last resort)
    • Details:
What are the reasonable SLOs (Service Level Objectives) for the enhancement?
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
  • Metrics
    • Metric name:
    • [Optional] Aggregation method:
    • Components exposing the metric:
  • Other (treat as last resort)
    • Details:
Are there any missing metrics that would be useful to have to improve observability of this feature?

The following metrics would be added into the access provider plugin library to help observability:

  • Number of Credential Obtention, categorized per plugin type, reply state
  • Latency to obtain credentials, categorized per plugin type

Dependencies

Does this feature depend on any specific services running in the cluster?

Scalability

Will enabling / using this feature result in any new API calls?
Will enabling / using this feature result in introducing new API types?
Will enabling / using this feature result in any new calls to the cloud provider?
Will enabling / using this feature result in increasing size or count of the existing API objects?
Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?
Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?
What are other known failure modes?
What steps should be taken if SLOs are not being met to determine the problem?

Implementation History

  • 2025-06-01: KEP-5339 (Plugin for Credentials in ClusterProfile) merged into this KEP. The access provider plugin mechanism, accessProviders API field, and all related design details were consolidated here since the base API and the access-related fields are expected to graduate together.

Drawbacks

Alternatives

Extending Cluster API Cluster resource

We also considered the possibility of extending the existing Cluster API’s Cluster resource to accommodate our needs for describing clusters within a multi-cluster environment. However, this approach was ruled out due to the Cluster API’s primary focus on cluster lifecycle management. Its tight coupling with cluster provisioning processes made it less suitable for scenarios where clusters are either provisioned through different methods or already exist. Furthermore, another distinction is the nature of the information each API conveys: the Cluster API’s Cluster resource outlines the desired state of the cluster. In contrast, the new API is intended to reflect the actual state of the cluster, more similar to the Cluster.status in the Cluster API, but with a broader scope and intended for use in a multi-cluster context. This distinction also extends to the ownership of the resources; the Cluster API’s Cluster is primarily owned by platform administrators focused on provisioning clusters, whereas the new API is designed to be owned by the cluster manager that created the cluster it represents.

ClusterProfile CRD scope

We had extensive discussions in SIG-Multicluster meetings about the appropriate scope for ClusterProfile resources, and ultimately decided that namespace scope would be more flexible than cluster scope while still retaining an adequate UX for simpler usage patterns. As a historical note, a prior attempt at organizing multiple clusters, the ClusterRegistry proposal, had proposed cluster-scoped resources but was met with pushback by potential adopters in part due to a desire to host multiple distinct registry lists on a single control plane, which would be far more straightforward with namespaced resources.

Global hub cluster for multiple clustersets

illustration of global hub for multiple clustersets topology

In this model, a single global hub cluster is used to manage multiple clustersets (a “Prod” clusterset and “Dev” clusterset in this illustration). For this use case, some means of segmenting the ClusterProfile resources into distinct groups for each clusterset is needed, and ideally should facilitate selecting all ClusterProfiles of a given clusterset. Because of this selection-targeting goal, setting clusterset membership within the spec of a ClusterProfile would not be sufficient. While setting a label such as the proposed clusterset.multicluster.x-k8s.io on the ClusterProfile resource (instead of a namespace) could be acceptable, managing multiple cluster-scoped ClusterProfile resources for multiple unrelated clustersets on a single global hub could quickly get cluttered. In addition to grouping clarity, namespace scoping could allow RBAC delegation for separate teams to manage resources for their own clustersets in isolation while still using a shared hub. The group of all clusters registered on the hub (potentially including clusters belonging to different clustersets or clusters not belonging to any clusterset) may represent a single “inventory” or multiple inventories, but such a definition is beyond the scope of this document and is permissible to be an undefined implementation detail.

Global hub cluster per clusterset

illustration of global hub per clusterset topology

In this model, each “inventory” has a 1:1 mapping with a clusterset containing clusters in multiple regions. A cluster-scoped ClusterProfile CRD would be sufficient for this architecture, but it requires a proliferation of hub clusters, which may not be optimal. This model is still implementable with namespace-scoped ClusterProfile CRDs by writing them all to a single namespace, either the default namespace or a specific namespace configured in the cluster manager. The risk of placing resources in the wrong namespace would be somewhat minimal if following the suggested pattern of having ClusterProfile resources be written by a “manager” rather than authored by humans.

Regional hub cluster for multiple clustersets

illustration of regional hub clusters for multiple clustersets topology

In this model, “hub” clusters are limited to a regional scope (potentially for architectural limitations or performance optimizations) and each hub is used to manage clusters only from the local region, but which may belong to separate clustersets. If, as in the pictured example, clustersets still span multiple regions, some out-of-band synchronization mechanism between the regional hubs would likely be needed. This model has similar segmentation needs to the global hub model, just at a smaller scale.

Regional hub clusters per clusterset

illustration of regional hub clusters per clusterset topology

This is creeping pretty far towards excessive cluster proliferation (and cross-region coordination overhead) purely for management needs (as opposed to actually running workloads), and would be more likely to be a reference or testing implementation than an architecture suitable for production scale.

Self-assembling clustersets

illustration of self-assembling clusterset topology

This is the model most suited to a cluster-scoped ClusterProfile resource. In contrast to the prior models discussed, in this approach the ClusterProfile CRD would be written directly to each “member” cluster. ClusterSet membership would either be established through peer-to-peer relationships, or managed by an external control plane. For ClusterSet security and integrity, a two-way handshake of some sort would be needed between the local cluster and each peer or the external control plane to ensure it is properly authorized to serve endpoints for exported services or import services from other clusters. While these approaches could be implemented with a namespace-scoped ClusterProfile CRD in the default or a designated namespace, misuse is most likely in this model, because the resource would be more likely to be authored by a human if using the peer-to-peer model. Due to the complexity and fragility concerns of managing clusterset membership in a peer-to-peer topology, an external control plane would likely be preferable. Assuming the external control plane does not support Kubernetes APIs (if it did, any of the “hub” models could be applied instead), it could still be possible to implement this model with a namespace-scoped ClusterProfile resource, but it is not recommended.

Workload placement across multiple clusters without cross-cluster service networking

In this model, a consumer of the Cluster Inventory API is looking to optimize workload placement to take advantage of excess capacity on existing managed clusters. These workloads may have specific hardware resource needs such as GPUs, but are typically “batch” jobs that do not require multi-cluster service networking to communicate with known services in a specific clusterset. The isolated nature of these jobs could allow them to be scheduled on many known clusters regardless of clusterset membership. A centralized hub which could register clusters in disparate clustersets or no clusterset and return a list of all known clusters from a single API call would be the most efficient for this consumer to query. Namespaced ClusterProfile CRDs on a global hub would be the best fit for this use case.

Workload placement into a specific clusterset

Within a single clusterset, a global workload placement controller may seek to balance capacity across multiple regions in response to demand, cost efficiency, or other factors. Querying a list of all clusters within a single clusterset should be possible to serve this use case, which is amenable to either cluster-scoped or namespaced-scoped ClusterProfile CRDs.

Infrastructure Needed (Optional)