KEP-4330: Compatibility Versions in Kubernetes

• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • Component Flags
    • –emulation-version
    • –min-compatibility-version
  • Skew
  • Changes to Feature Gates
    • Feature Gate Lifecycles
    • Feature gating changes
    • Non-Emulatable Features
      • Alternatives
  • Validation ratcheting
    • CEL Environment Compatibility Versioning
  • StorageVersion Compatibility Versioning
  • API availability
    • API availability in forward-compatible mode
    • Alternatives to API forward compatibility
  • API Field availability
  • Discovery
  • Version introspection
  • User Stories (Optional)
    • Story 1
    • Story 2
    • Story 3
  • Notes/Constraints/Caveats (Optional)
  • Risks and Mitigations
  • Risk: Increased cost in test due to the need to test changes against the e2e tests of multiple release branches
    • Risk: Increased maintenance burden on Kubernetes maintainers
    • Risk: Unintended and out-of-allowance version skew
• Design Details
  • Test Plan
    • Prerequisite testing updates
    • Unit tests
    • Integration tests
    • e2e tests
  • Graduation Criteria
    • Alpha
    • Beta
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
• Production Readiness Review Questionnaire
  • Feature Enablement and Rollback
  • Rollout, Upgrade and Rollback Planning
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Alternatives
• Infrastructure Needed (Optional)

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  • e2e Tests for all Beta API Operations (endpoints)
  • (R) Ensure GA e2e tests meet requirements for Conformance Tests
  • (R) Minimum Two Week Window for GA e2e tests to prove flake free
• (R) Graduation criteria is in place
  • (R) all GA Endpoints must be hit by Conformance Tests
• (R) Production readiness review completed
• (R) Production readiness review approved
• “Implementation History” section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

We intend to introduce version compatibility and emulation options to Kubernetes control plane components to make upgrades safer by increasing the granularity of steps available to cluster administrators.

We will introduce a --emulation-version flag to emulate the capabiliites (APIs, features, …) of a prior Kubernetes release version. When used, the capabilities available will match the emulated version. Any capabilities present in the binary version that were introduced after the emulation version will be unavailable and any capabilities removed after the emulation version will be available. This enables a binary version to emulate the behavior of a previous version with sufficient fidelity that interoperability with other system components can be defined in terms of the emulated version.

We will also introduce a --min-compatibility-version flag to control the minimum version a control plane component is compatible with (in terms of storage versions, validation rules, …). When used, the component tolerates workloads that expect the behavior of the specified minimum compatibility version, component skew ranges extend based on the minimum compatibility version, and rollbacks can be performed back to the specified minimum compatibility version.

Motivation

The notion of more granular steps in Kubernetes upgrades is attractive because it is more rigorous about how we step through a Kubernetes control-plane upgrade, introducing potentially corrupting data (i.e. data only present in N+1 and not in N) only in later stages of the upgrade process.

For example, upgrading from Kubernetes 1.30 to 1.31 while keeping the emulation version at 1.30 would enable a cluster administrator to validate that the new Kubernetes binary version is working as desired before exposing any feature changes introduced in 1.31 to cluster users, and without writing and data to storage at newer API versions.

This extra step increases the granularity of our upgrade sequence so that (1) failures are more easily diagnosed (since we have more granular steps, we have slightly more data sheerly from which point of the upgrade lifecycle we achieved failure) and (2) failures are more easily auto-reverted by upgrade orchestration as we are taking smaller and more incremental steps forward, which means there is less to “undo” on a failure condition.

It also becomes possible to skip binary versions while still performing a stepwise upgrade of Kubernetes control-plane. For example:

• (starting point) binary-version 1.28 (compat-version 1.28)
• upgrade binary-version to 1.31 (emulation-version stays at 1.28 - this is our skip-level binary upgrade)
• keep binary-version 1.31 while upgrading emulation-version to 1.29 (stepwise upgrade of emulation version)
• keep binary-version 1.31 while upgrading emulation-version to 1.30 (stepwise upgrade of emulation version)
• keep binary-version 1.31 while upgrading emulation-version to 1.31 (stepwise upgrade of emulation version)

Benefits to upgrading binary version independent of emulation version:

• A skip-version binary upgrade that transitions through emulation versions has higher probability of bugs in those intermediate binary versions already being fixed.
• During an upgrade, it is possible upgrade the new binary version while the emulation version is fixed (e.g. binaryVersion: 1.28 -> {binaryVersion: 1.29, emulationVersion: 1.28}). This allow differences in the binary (bugs fixed or introduced, performance changes, …) to be introduced into a cluster and verified safe before allowing access to the new APIs and features new version. Once the binary version has been deamed safe, the emulation version can then be upgraded.
• Any upgrade system that successfully detects failures between upgrade steps can report which upgrade step failed. This makes it easier to diagnose the failures, because there are fewer possible causes of the failure. (There’s a huge difference between “A cluster upgrade failed” and “A cluster upgrade failed when only the apiserver’s binary version changed”).
• For upgrades where multiple failures occur, this increases the odds those failures are split across steps. An upgrade system that is able to pause after a step where failures are detected allow for failures at that step to be addressed before proceeding to subsequent steps. These failures can be addressed without the disruption and “noise” from failures in subsequent steps.
• An emulation version rollback can be performed without changing binary version.

A dedicated --min-compatibility-version flag provides direct control of when deprecated features are removed from the API. If the --min-compatibility-version is kept at a fixed version number during a binary version or emulation version upgrade, the cluster admin is guaranteed no features will be removed, reducing the risk of the upgrade step.

Also, --min-compatibility-version can be used to provide a wider skew range between components.

Lastly, a --min-compatibility-version can be set to the binary version for new clusters being used for green field projects, providing immediate access to the latest Kubernetes features without the need to wait an additional release for features to settle in as is typically needed for rollback support.

Goals

• Introduce the metadata necessary to configure features/APIs/storage-versions/validation rules to match the behavior of an older Kubernetes release version
• A Kubernetes binary with emulation version set to N, will pass the conformance and e2e tests from Kubernetes release version N.
• A Kubernetes binary with emulation version set to N does not enable any changes (storage versions, CEL feature, features) that would prevent it from being rolled back to N-1.
• The most recent Kubernetes version supports emulation version being set to the full range of supported versions.
  • In alpha we intend to support:
    • --emulation-version range of binaryMinorVersion..binaryMinorVersion-1
  • In beta, we intend to extend support to:
    • --emulation-version range of binaryMinorVersion..binaryMinorVersion-3
    • --min-compatibility-version to emulationVersion..binaryMinorVersion-3

Non-Goals

• Support --emulation-version for Alpha features. Alpha feature are not designed to be upgradable, so we will not allow alpha features to be enabled when --emulation-version is set.
• --min-compatibility-version will only apply to Beta and GA features. Only Alpha features available in the current binary version will be available for enablement and are allowed to change in behavior across releases in ways that are incompatible with previous versions.
• Changes to Cluster API/kubeadm/KIND/minikube to absorb the compatibility versions will be addressed separate from this KEP

Proposal

Component Flags

–emulation-version

• Defaults to binaryVersion (matching current behavior)
• Must be <= binaryVersion
• Must not be lower than the supported range of minor versions (see graduation criteria for ranges). If below the supported version range the binary will exit and report an invalid flag value error telling the user what versions are allowed.
• Is not a bug-for-bug compatibility model. It specifically controls which APIs, feature gates, configs and flags are enabled to match that of a previous release version.

Adding --emulation-version to kubelet is out of scope for this enhancement. But we do need to define how kubelet skew behaves when the kube-apiserver has --emulation-version set. Our general rule is that we want to define skew using emulation versions when they are in use. So not only must a kubelet version be <= the kube-apiserver binary version, it must also be <= the --emulation-version of the kube-apiserver.

–min-compatibility-version

• Defaults to emulationVersion-1 if emulationVersion.GreaterThan(binaryVersion-3) (matching current behavior in emulation mode), and defaults to emulationVersion if emulationVersion.EqualTo(binaryVersion-3) (because of the max version range we are supporting)
• Must be <= --emulation-version
• Must not be lower than the supported range of minor versions (see graduation criteria for ranges). If below the supported version range the binary will exit and report an invalid flag value error telling the user what versions are allowed.

Skew

The Version skew policy rules be defined in terms of compatibility and emulation versions:

• kube-controller-manager, kube-scheduler, and cloud-controller-manager:

  • Previously: 1.{binaryMinorVersion-1}..{binaryVersion}
  • With this enhancement: {minCompatibilityVersion}..{emulationVersion}

• kube-proxy, kubelet:

  • Previously: 1.{binaryMinorVersion-3}..{binaryVersion}
  • With this enhancement: {minCompatibilityVersion-2}..{emulationVersion}

• kubectl:

  • Previously: 1.{binaryMinorVersion-1}..{binaryVersion+1}
  • With this enhancement: {minCompatibilityVersion}..{emulationVersion+1}

Changes to Feature Gates

Features will track version information, i.e.:

map[Feature]VersionedSpecs{
		featureA: VersionedSpecs{
			{Version: mustParseVersion("1.27"), Default: false, PreRelease: Beta},
			{Version: mustParseVersion("1.28"), Default: true, PreRelease: GA},
		},
		featureB: VersionedSpecs{
			{Version: mustParseVersion("1.28"), Default: false, PreRelease: Alpha},
		},
		featureC: VersionedSpecs{
			{Version: mustParseVersion("1.28"), Default: false, PreRelease: Beta},
		},
		featureD: VersionedSpecs{
			{Version: mustParseVersion("1.26"), Default: false, PreRelease: Alpha},
			{Version: mustParseVersion("1.28"), Default: true, PreRelease: Deprecated},
		}

Features with compatibility implications (like a new API field or relaxing validation to allow a new enum value) could include that in their feature-gate spec:

map[Feature]VersionedSpecs{
		featureA: VersionedSpecs{
			  {Version: mustParseVersion("1.28"), MinCompatibilityVersion: mustParseVersion("1.28"), ...},Alpha},
			  {Version: mustParseVersion("1.29"), MinCompatibilityVersion: mustParseVersion("1.28"), ...},Beta},
			  {Version: mustParseVersion("1.30"), MinCompatibilityVersion: mustParseVersion("1.28"), ...},
		},

And features using a gate to guard a behavior change with compatibility implications that isn’t really going through the feature lifecycle could set the feature version and the min compatibility version to the same thing:

relaxValidationFeatureA: VersionedSpecs{
    {Version: mustParseVersion("1.30"), MinCompatibilityVersion: mustParseVersion("1.30"), Default: true, ...},
},

When a component starts, feature gates VersionedSpecs will be compared against the emulation and compatibility version to determine which features to enable.

Feature Gate Lifecycles

--feature-gates must behave the same as it did for the emulation version. For example, it must be possible to use --feature-gates to disable features that were beta at the emulation version. One important implication of this requirement is that feature gating must be kept in the Kubenetes codebase after a feature has reached GA (or been removed) to support the emulation and compatibility version ranges.

A feature that is promoted once per release would look something like:

map[Feature]VersionedSpecs{
		featureA: VersionedSpecs{
			{Version: mustParseVersion("1.26"), Default: false, PreRelease: Alpha},
			{Version: mustParseVersion("1.27"), Default: true, PreRelease: Beta},
			{Version: mustParseVersion("1.28"), Default: true, PreRelease: GA},
		},
}

The lifecycle of the feature would be:

All feature gating and tracking must remain in code through 1.30 for emulation version support range (see graduation criteria for ranges we plan to support).

For a Beta feature that is removed, e.g.:

map[Feature]VersionedSpecs{
		featureA: VersionedSpecs{
			{Version: mustParseVersion("1.26"), Default: false, PreRelease: Beta},
			{Version: mustParseVersion("1.27"), Default: false, PreRelease: Deprecated},
			{Version: mustParseVersion("1.31"), Default: false, PreRelease: Removed},
		},
}

The steps to remove the Beta feature would be:

(Features that are deleted before reaching Beta do not require emulation version support since we don’t support emulation version for alpha features)

Note that this respects a 1yr deprecation policy.

All feature gating and tracking must remain in code through 1.32 for emulation version support range see (see graduation criteria for ranges we plan to support).

Feature gating changes

In order to preserve the behavior of in-development features across multiple releases, feature implementation history should also be preserved in the code base instead of in place modifications.

Only sigificant and observable changes in feature capabilities should be across releases. We do not want to impose a unreasonable burdon on feature authors. The main criteria to make the decision is: Does this change break the contract with existing users of the feature? i.e. would this change break the workloads of existing feature users if the user does not change the compatibility version?

Here are some common change scenarios and whether the change needs to be preserved or not:

• API change [Yes]
• Change of supported systems [Yes]
• Bug fix [No]
• Performance optimizations [No]
• Unstable metrics change [No]
• Code refactoring [No]

Listed below are some concrete examples of feature changes:

To preserve the behavior, naively the feature implementations can be gated by version number. For example, if FeatureA is partially implemented in 1.28 and additional functionality is added in 1.29, the feature developer is expected to gate the functionality by version. E.g.:

if feature_gate.Enabled(FeatureA) && feature_gate.EmulationVersion() <= "1.28" {implementation 1}
if feature_gate.Enabled(FeatureA) && feature_gate.EmulationVersion() >= "1.29" {implementation 2}

A better way might be to define a featureOptions struct constructed based on the the feature gate, and have the featureOptions control the main code flow, so that the main code is version agnostic. E.g.:

// in kube_features.go
const (
  FeatureA featuregate.Feature = "FeatureA"
  FeatureB featuregate.Feature = "FeatureB"
)

func init() {
	utilfeature.DefaultMutableFeatureGate.AddVersioned(defaultKubernetesFeatureGates)
}

var defaultKubernetesFeatureGates = map[Feature]VersionedSpecs{
		featureA: VersionedSpecs{
			{Version: mustParseVersion("1.27"), Default: false, PreRelease: Alpha},
		},
    featureB: VersionedSpecs{
			{Version: mustParseVersion("1.28"), Default: false, PreRelease: Alpha},
			{Version: mustParseVersion("1.30"), Default: true, PreRelease: Beta},
		},
}

type featureOptions struct {
  AEnabled        bool
  AHasCapabilityZ bool
  BEnabled        bool
  BHandler        func()
}

func newFeatureOptions(featureGate FeatureGate) featureOptions {
  opts := featureOptions{}
  if featureGate.Enabled(FeatureA) {
    opts.AEnabled = true
  }
  if featureGate.EmulationVersion() > "1.29" {
    opts.AHasCapabilityZ = true
  }

  if featureGate.Enabled(FeatureB) {
    opts.BEnabled = true
  }
  if featureGate.EmulationVersion() > "1.28" {
    opts.BHandler = newHandler
  } else {
    opts.BHandler = oldHandler
  }
  return opts
}

// in client.go
func ClientFunction() {
  // ...
  featureOpts := newFeatureOptions(utilfeature.DefaultFeatureGate)
  if featureOpts.AEnabled {
    // ...
    if featureOpts.AHasCapabilityZ {
      // run CapabilityZ
    }
  }
  if featureOpts.BEnabled {
    featureOpts.BHandler()
  }
  // ...
}

Non-Emulatable Features

Generally, if a feature is enabled by default either in Beta or GA, it should be fairly stable already. In very rare cases for some default-off Beta features, a feature could be non-emulatable if the feature implementation history could not be preserved in the code base with reasonable maintenance cost.

For these cases, we should try to emulate these changing features at best effort, and make sure the clients are aware of the risks of enabling default-off Beta features with emulation version.

Alternatives

An alternative would be to keep a exception list of NonEmulatableFeatures, and make the server fail to start if any of the NonEmulatableFeatures are enabled in compatibility mode. This approach has the benefits of predictable outcomes. The downsides are:

1. whether a feature should be added to the list is subjective. It is hard to define a threshold to decide if the difficulty to preserve the implementation history is above this threshold, we could add the feature to the list.
2. this could break automated upgrade processes using the emulation version, depending what features a cluster opts in.
3. there needs proper cleaning up rules to ensure the list would not just keep growing.

Validation ratcheting

All validationg ratcheting needs to account for compatibility version.

If code to support ratcheting is introduced in 1.30, then new values needing the ratcheting may only be written if the compatibility version >= 1.30. Since we require emulation version >= compatibility version, the emulation version must also be 1.30 or greater.

CEL Environment Compatibility Versioning

CEL compatibility versioning is a special case of validation ratcheting.

CEL environments already support a compatibility version . The CEL compatibility version is used to ensure when a kubeneretes contol plane component reads a CEL expression from storage written by a (N+1) newer version (either due to version skew or a rollback), that a compatible CEL environment can still be constructed and the expression can still be evaluated. This is achieved by making any CEL environment changes (language settings, libraries, variables) available for stored expressions one version before they are allowed to be written by new expressions .

The only change we must make for this enhancement is to remove the compatibility version constant and instead always pass in N-1 of the compatibility version introduced by this enhancement as the CEL compatibility version.

StorageVersion Compatibility Versioning

StorageVersions specify what version an apiserver uses to write resources to etcd for each API group. The StorageVersion changes across releases as API groups graduate through stability levels.

During upgrades and downgrades, the storage version is particularly important. To enable upgrades and rollbacks, pre compatibility version, the version selected for storage in etcd in version N must be (en/de)codable for k8s versions N-1 through N+1. With compatibility version, the version selected for storage in etcd for the combination of EmulationVersion and MinCompatibilityVersion must be (en/de)codable for k8s versions MinCompatibilityVersion through EmulationVersion+1.

Thus, to determine the storage version to use at compatibility version N, we will find the set of all supported GVRs for each version in the range of MinCompatibilityVersion and EmulationVersion+1 and intersect them to find a list of all GVRs supported by every binary version in the window. The storage version of each group-resource is the newest (using kube-aware version sorting) version found in that list for that group-resource.

API availability

By default, similar to feature flags, all APIs group-versions declarations will be modified to track which Kubernetes version the API group-versions are introduced (or removed) at.

GA APIs should match the exact set of APIs enabled in GA for the Kubernetes version the emulation version is set to.

All Beta APIs (both off-by-default and on-by-default, if any of those still exist) need to behave exactly as they did for the Kubernetes version the emulation version is set to. I.e. --runtime-config=api/<version> needs to be able to turn on APIs exactly like it did for each Kubernetes version that emulation version can be set to.

Alpha APIs may not be enabled in conjunction with emulation version.

API availability in forward-compatible mode

If we stick to the strict rule of api availability matching the emulation version, we would face some challenging scenarios when emulating the controllers when an API is graduating from Beta to GA and the controller wants to use newer API. For example, to graduate Multiple Service CIDRs to GA, normally the controller code change would look like:

- c.serviceCIDRInformer = networkingv1beta1informers.NewFilteredServiceCIDRInformer(client, 12*time.Hour,
+	c.serviceCIDRInformer = networkingv1informers.NewFilteredServiceCIDRInformer(client, 12*time.Hour,
		cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc},
		func(options *metav1.ListOptions) {
			options.FieldSelector = fields.OneTermEqualSelector("metadata.name", DefaultServiceCIDRName).String()
		})

-	c.serviceCIDRLister = networkingv1beta1listers.NewServiceCIDRLister(c.serviceCIDRInformer.GetIndexer())
+	c.serviceCIDRLister = networkingv1listers.NewServiceCIDRLister(c.serviceCIDRInformer.GetIndexer())
	c.serviceCIDRsSynced = c.serviceCIDRInformer.HasSynced

	return c
  //...

type Controller struct {
	eventRecorder    record.EventRecorder

	serviceCIDRInformer cache.SharedIndexInformer
-	serviceCIDRLister   networkingv1beta1listers.ServiceCIDRLister
+	serviceCIDRLister   networkingv1listers.ServiceCIDRLister
	serviceCIDRsSynced  cache.InformerSynced

To fully emulate the controller for an older version, anywhere v1 api/type is referenced, it would need to switch to the v1beta version if the emulation version is older than the binary version. This would mean a lot of extra work, complicated testing rules, and high maintenance cost even for simple API graduations, while the emulation fidelity is unreliable with the extra complexity.

So instead of truly emulating the feature controllers and API availability at the emulation version, we are allowing apis introduced after the emulation version to be enabled explicitly with --runtime-config=api/<version>. We are also introducing an umbrella --emulation-forward-compatible flag to enable forward compatibility of all APIs in compatibility version mode: if an older version of the API is enabled, newer and more stable versions of the same resource group introduced between the emulation version and binary version would also be enabled. This way the non-emulatable feature controllers would be able to use the newest available API.

For API availability, setting --emulation-forward-compatible=true means:

1. if an API is removed (as indicated by the GVK prerelease lifecycle) after the emulation version, it would still be available at the emulation version (regardless of --emulation-forward-compatible=true/false).
2. if an API is Beta at the emulation version (meaning the Beta API has been introduced and has not been removed by the emulation version), it can be enabled by --runtime-config=api/<version> or it can be on-by-default at the emulation version. If a Beta API is enabled at the emulation version, and it has GAed between the emulation version and the binary version, its GA version(s) would also be enabled at the emulation version. If a newer Beta API is introduced between the emulation version and the binary version, the newer Beta API(s) would also be enabled at the emulation version.
3. if a Beta API is not enabled at the emulation version, its future versions would not be enabled at the emulation version unless explicitly enabled with --runtime-config=api/<version>.
4. If an API has GAed at the emulated version, it would be enabled by default at the emulation version. If a newer stable version of the GA API has been introduced between the emulation version and the binary version, the new GA API(s) would also be enabled at the emulation version along with the old GA API.
5. Alpha APIs may not be enabled in conjunction with emulation version.

Here are some examples for BinaryVersion = 1.33:

For the controller, at the emulation version the controller is still enabled by enabling the Beta API AND the controller feature as before, but under the hood the controller is calling the newer API.

The forward compatibility of API availability should not affect data compatibility because storage version is still controlled by the MinCompatibilityVersion regardless of whether the data are created through future versions of the API endpoints. Webhooks should also work fine if the matching policy is Equivalent.

For the use case of upgrading control plane binary version without changing the emulation version, this would mean api servers should be upgraded first before any other components, because an api server of the old binary version would not be able to serve a controller of the new binary version even though the emulation version is the same as the old binary version.

Alternatives to API forward compatibility

To make API graduation workable for controller code change under compatibility version, we have considered and rejected the following alternative options:

1. if .. else .. statements everywhere is just impractical.
2. have v1Controller and v1beta1Controller code in separate files would mean duplicate maintenance, test work, and developer churn.
3. have some smart wrapper to pick the right version for each API/type reference: it is very hard to design a generic enough wrapper for all cases.
4. convert the data to older version when the newer API is called: this is the essentially same as the enabling the newer API.
5. have special mechanisms for controllers in k/k to call v1 apis, but not expose the v1 apis to clients: clients can spend efforts to duplicate the special mechanisms.

API Field availability

API fields that were introduced after the emulation version will not be pruned. Ideally they would be, but we already show information about unavailable fields in the API today like disabled-by-default features (Alphas mostly) and make no attempt to hide those fields in discovery.

We consider pruning fields based on emulation version useful future work that would benefit multiple aspects of how APIs are served, so while we’re not taking on the effort in this KEP, we are interested in seeing this improved.

Discovery

Discovery will enable the group versions matching the emulation version.

The API fields include will match what is described in the “API Fields” section.

Version introspection

The /version endpoint will be augmented to report emulation version and min compatibility version when this feature is enabled. Note that this changes default behavior by always including a new field in /version responses. E.g.

{
  "major": "1",
  "minor": "32",
  "emulationMajor": "1",
  "emulationMinor": "31",
  "minCompatibilityMajor": "1",
  "minCompatibilityMinor": "30",
  "gitVersion": "v1.30.0",
  "gitCommit": "<something>",
  "gitTreeState": "clean",
  "buildDate": "2024-03-30T06:36:32Z",
  "goVersion": "go1.21.something",
  "compiler": "gc",
  "platform": "linux/arm64"
}

User Stories (Optional)

Story 1

A cluster administrator is running Kubernetes 1.30.12 and wishes to perform a cautious upgrade to 1.31.5 using the smallest upgrade steps possible, validating the health of the cluster between each step.

• For each control plane component, in the recommended order :
  • Upgrades binary to kubernetes-1.31.5 but sets --emulation-version=1.30
  • Verifies that the cluster is healthy
• Next, for each control plane component:
  • Sets --emulation-version=1.31
  • Verifies that the cluster is healthy

Story 2

A cluster administrator is running Kubernetes 1.30.12 and wishes to perform a cautious upgrade to 1.31.5, but after upgrading realizes that a feature the workload depends on had been removed and needs to rollback until the workload can be modified to not depend on the feature.

• For each control plane component, in the recommended order :
  • Cluster admin restarts the component with --emulation-version=1.30 set

This avoids having to rollback the binary version. Once the workload is fixed, the cluster administrator can remove the --emulation-version to roll the cluster forward again.

Story 3

A cluster administrator creating a new Kubernetes cluster for a development of a new project and wishes to make use of the latest available features.

• Cluster admin starts all cluster components with a 1.30 binary version and sets --min-compatibility-version=1.30 as well.

Because the cluster admin has no need to rollback, setting --min-compatibility-version=1.30 can be used to indicate that they do not require any feature availability delay to support a compatibility range and benefit from access to all the latest available features.

Notes/Constraints/Caveats (Optional)

Risks and Mitigations

Risk: Increased cost in test due to the need to test changes against the e2e tests of multiple release branches

TODO: Establish a plan for this for Alpha. Implement for Beta.

Risk: Increased maintenance burden on Kubernetes maintainers

Why we think this is manageable:

• We already author features to be gated. The only change here is include enough information about features so that they can be selectively enabled/disabled based on emulation version.
• We already manually deprecate/remove features. This change will instead leave features in code longer, and require feature gates to track at which verion a feature is deprecated/removed. The total maintenance work is about the same.
• Some maintenance becomes simpler as the additional version data about features makes them easier to reason about and keep track of.

Risk: Unintended and out-of-allowance version skew

From @deads2k: “I see an additional risk of unintended and out-of-allowance version skew between binaries. A kube-apiserver and kube-controller-manager contract is still +/-1 (as far as I see here). Compatibility and emulation versions, especially across three versions, makes it more likely for accidental mismatches.

While a hard shutdown of a process is likely worse than the disease, exposing some sort of externally trackable signal for cluster-admins and describing how to use it could significantly mitigate the problem.”

Possible mitigations:

• Clients send version numbers in request headers. Servers use this to detect out-of-allowance skew. Servers then surface this to cluster administrators through a metric.
• Components register identity leases (apiserver already does this) https://github.com/kubernetes/enhancements/pull/4356 proposes doing it for controller managers. Components include their version information in the identity leases. A separate controller inspects all the leases for skew and surafces it to cluster administrators.

Design Details

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

Unit tests

For Alpha, we will fill this out as we implement.

• <package>: <date> - <test coverage>

Integration tests

For Alpha we will add integration test to ensure --emulation-version behaves expected according to the following grid:

• Other cases we will test are:
  • --emulation-version=<N-2> - fails flag validation, binary exits
  • --emulation-version=<N+1> - fails flag validation, binary exits
  • we only allow data into new API fields once they existed in the previous release, this needs to account for emulation version
  • we only relax validation after the previous release tolerates it, this needs to account for emulation version

e2e tests

For e2e testing, we intend to run e2e tests from the N-1 minor version of kubernetes against the version being tested with –emulation-version set to the N-1 minor versions.

This is a new kind of testing– it requires running the tests from a release branch against the the branch being tested (either master or another release branch).

We intend to have this up and running for Beta

Graduation Criteria

Alpha

• Feature implemented behind a feature flag
• Emulation version support for N-1 minor versions
• Integration tests completed and enabled

Beta

• Initial cross-branch e2e tests completed and enabled
• Emulation version support for N-3 minor versions
• Min compatibility version support for N-3 minor versions
• Clients send version number and servers report out-of-allowance skew to a metric (Leveraging work from KEP-4355 if possible)
• All existing features migrated to versioned feature gate - kubernetes #125031
• Verification machinery added - kubernetes #125032
• Integrate test/featuregate_linter into golangci-lint

Upgrade / Downgrade Strategy

Version Skew Strategy

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

• Feature gate (also fill in values in kep.yaml)
  • Feature gate name: CompatibilityVersions
  • Components depending on the feature gate:
    • kube-apiserver
    • kube-controller-manager
    • kube-scheduler

Does enabling the feature change any default behavior?

Yes, /version will respond with binary{Major,Minor} and minCompatibility{Major,Minor} fields. This addition of fields should be handled by clients in a backward compatible way, and is a relatively safe change.

No other default behavior is changed.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes. Note that when the --emulation-version or --min-compatibility-version flags is set, the feature flag must be turned on (when feature is in Alpha). So to disable the feature, the flag must also be removed.

What happens if we reenable the feature if it was previously rolled back?

Behavior is as expected, --emulation-version or --min-compatibility-version may be set again.

Are there any tests for feature enablement/disablement?

Yes, feature enablement/disablement will be fully tested in Alpha.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

What specific metrics should inform a rollback?

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

How can someone using this feature know that it is working for their instance?

• Events
  • Event Reason:
• API .status
  • Condition name:
  • Other field:
• Other (treat as last resort)
  • Details:

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

• Metrics
  • Metric name:
  • [Optional] Aggregation method:
  • Components exposing the metric:
• Other (treat as last resort)
  • Details:

Are there any missing metrics that would be useful to have to improve observability of this feature?

Dependencies

Does this feature depend on any specific services running in the cluster?

Scalability

Will enabling / using this feature result in any new API calls?

Will enabling / using this feature result in introducing new API types?

Will enabling / using this feature result in any new calls to the cloud provider?

Will enabling / using this feature result in increasing size or count of the existing API objects?

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

What are other known failure modes?

What steps should be taken if SLOs are not being met to determine the problem?

Implementation History

Drawbacks

Alternatives

Infrastructure Needed (Optional)