KEP-4006: Transition from SPDY to Websockets

Implementation History
BETA Implementable
Created 2023-05-15
Latest v1.36
Milestones
Alpha v1.29
Beta v1.31
Stable v1.37
Ownership
Owning SIG
SIG API Machinery
Participating SIGs
SIG CLI
Primary Authors
@seans3

KEP-4006: Transition from SPDY to WebSockets

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

Some Kubernetes clients need to communicate with the API Server using a bi-directional streaming protocol, instead of the standard HTTP request/response mechanism. A streaming protocol provides the ability to read and write arbitrary data messages between the client and server, instead of providing a single response to a client request. For example, the commands kubectl exec, kubectl attach, and kubectl port-forward both benefit from a bi-directional streaming protocol (kubectl cp is build on top of kubectl exec primitives so it utilizes streaming as well). Currently, the bi-directional streaming solution for these kubectl commands is SPDY/3.1. For the communication leg between kubectl and the API Server, this enhancement transitions the bi-directional streaming protocol to WebSockets from SPDY/3.1.

Motivation

The SPDY streaming protocol has been deprecated since 2015, and by now many proxies, gateways, and load-balancers do not support SPDY. Our effort to modernize the streaming protocol between Kubernetes clients and the API Server using WebSockets is necessary to enable the aforementioned intermediaries. WebSockets is a currently supported standardized protocol (https://www.rfc-editor.org/rfc/rfc6455 ) that guarantees compatibility and interoperability with the different components and programming languages. Finally, WebSockets is preferrable to HTTP/2.0 because the updated HTTP standard does not support streaming well. The decision to forego HTTP/2.0 is discussed at greater length in the Alternatives Section .

Goals

  1. Transition the bi-directional streaming protocol from SPDY/3.1 to WebSockets for kubectl exec, kubectl attach, kubectl cp, and kubectl port-forward for the communication leg between kubectl and the API Server.

  2. Extend the WebSockets communication leg from the API Server to Kubelet. After this extension, WebSockets streaming will occur between kubectl and Kubelet (proxied through the API Server).

Non-Goals

  1. We will not make any changes to current WebSocket based browser/javascript clients.

  2. We will not transition the streaming protocol for the communication leg on the Node between the Kubelet and the container runtime. This leg will continue to stream the SPDY protocol.

Proposal

Currently, the bi-directional streaming protocols (either SPDY or WebSockets) are initiated from clients, proxied by the API Server and Kubelet, and terminated at the Container Runtime (e.g. containerd or CRI-O). This enhancement proposes to 1) modify kubectl to request a WebSocket based streaming connection, and to 2) modify the current API Server proxy to translate or tunnel the kubectl WebSockets data stream to a SPDY upstream connection. In this way, the cluster components upstream from the API Server will not initially need to be changed. We intend to extend the communication path for WebSockets streaming from kubectl to Kubelet once the the initial leg is proven to work (i.e. that it goes GA).

User Stories (Optional)

The functionality of this KEP will allow kubectl users to leverage L7 proxies and gateways that support WebSockets but not SPDY. Usually, the setup for these intermediaries is specific to a cloud provider or cluster operator. For example, to use the Anthos Connect Gateway to communicate with (some) Google clusters, users must run gcloud specific commands which update the kubeconfig file to point to the gateway. Afterwards, users can run streaming commands such as kubectl exec ..., and the commands will transparently use the now supported WebSockets protocol.

Notes/Constraints/Caveats (Optional)

N/A

Risks and Mitigations

  • Risk: Security

A possible security vulnerability might occur when a potential upgraded connection is redirected to other API endpoints.

  • Mitigation: Upgraded connections are disallowed from redirecting.

  • Risk: Performance

When transitioning from the SPDY streaming protocol to WebSockets, there may be a performance degradation. In order to implement the WebSocket streaming functionality that SPDY already implements, it is necessary for additional headers to be prepended to the WebSocket data package.

  • Mitigation: Performance testing to ensure the WebSockets implementation is not noticeably slower than the current SPDY streaming implementation.

Design Details

Current SPDY Streaming Architectural Diagram

Current SPDY Streaming Architectural Diagram

Background: Streaming Protocol Basics

kubectl bi-directional streaming connections are created by upgrading an initial HTTP/1.1 request. By adding two headers (Connection: Upgrade, Upgrade: SPDY/3.1), the request can initiate the streaming upgrade. And when the response returns status 101 Switching Protocols signalling success, the connection can then be kept open for subsequent streaming. An example of an upgraded HTTP Request/Response for kubectl exec could look like:

HTTP Request

GET /api/v1/…/pods/nginx/exec?command=<CMD>... HTTP/1.1
Connection: Upgrade
Upgrade: SPDY/3.1
X-Stream-Protocol-Version: v4.channel.k8s.io
X-Stream-Protocol-Version: v3.channel.k8s.io
X-Stream-Protocol-Version: v2.channel.k8s.io
X-Stream-Protocol-Version: v1.channel.k8s.io

HTTP Response

HTTP/1.1 101 Switching Protocols
Connection: Upgrade
Upgrade: SPDY/3.1
X-Stream-Protocol-Version: v4.channel.k8s.io

If the upgrade is successful, one of the requested subprotocol versions is chosen and returned in the response. In this instance, the chosen version of the subprotocol is: v4.channel.k8s.io.

Background: RemoteCommand Subprotocol

Remote Command Subprotocol

Once the connection is upgraded to a bi-directional streaming connection, the client and server can exchange data messages. These messages are interpreted with agreed upon standards which are called subprotocols. The three kubectl commands (exec, attach, and cp) communicate using the RemoteCommand subprotocol. Basically, this subprotocol provides command line functionality from the client to a running container in the cluster. By multiplexing stdin, stdout, stderr, and tty resizing over a streaming connection, this subprotocol supports clients executing and interacting with commands executed on a container in the cluster. An example of kubectl exec running the date command on an nginx pod/container is:

$ kubectl exec nginx -- date
Tue May 16 03:34:04 PM PDT 2023

The RemoteCommand Subprotocol has iterated through four different versions, where the transmitted data has changed. The second version of the subprotocol (v2.channel.k8s.io) included stdin, stdout, and stderr, and the third version (v3.channel.k8s.io) added support for terminal resizing with the tty stream. The fourth and most current version, v4.channel.k8s.io added a structured error stream, which includes the process exit code from the container. All of these streams are multiplexed over a single connnection by prepending a stream identififer byte to the data message. For example, a stdout data message sent over the connection will have the stdout file descriptor (1) prepended to the data message.

Background: API Server and Kubelet UpgradeAwareProxy

In order to route the data streamed between the client and the container, both the API Server and Kubelet must proxy these data messages. Both the API Server and the Kubelet provide this functionality with the UpgradeAwareProxy, which is a reverse proxy that knows how to deal with the connection upgrade handshake.

Proposal: kubectl WebSocket Executor and Fallback Executor

This enhancement proposes adding a WebSocketExecutor to kubectl, implementing the WebSocket client using a new subprotocol version (v5.channel.k8s.io). Additionally, we propose creating a FallbackExecutor to address client/server version skew. The FallbackExecutor first attempts to upgrade the connection with the WebSocketExecutor, then falls back to the legacy SPDYExecutor, if the upgrade is unsuccessful. Note that this mechanism can require two request/response trips instead of one. While the fallback mechanism may require an extra request/response if the initial upgrade is not successful, we believe this possible extra roundtrip is justified for the following reasons:

  1. The upgrade handshake is implemented in low-level SPDY and WebSocket libraries, and it is not easily exposed by these libraries. If it is even possible to modify the upgrade handshake, the added complexity would not be worth the effort.
  2. The streaming is already IO heavy, so another roundtrip will not substantially affect the perceived performance.
  3. As releases increment, the probablity of a WebSocket enabled kubectl communicating with an older non-WebSocket enabled API Server decreases.

Proposal: New RemoteCommand Sub-Protocol Version - v5.channel.k8s.io

The latest RemoteCommand version does not address an important protocol feature–a stream CLOSE signal. In order to communicate to another endpoint that the current stream of sending data is complete, a CLOSE signal is necessary. This problem currently arises when sending data over the STDIN stream, and it is more fully described in the following issue: exec over web sockets protocol is flawed . A new RemoteCommand version (v5.channel.k8s.io) adds this CLOSE signal.

Proposal: API Server RemoteCommand StreamTranslatorProxy

Stream Translator Proxy

Currently, the API Server role within client/container streaming is to proxy the data stream using the UpgradeAwareProxy. This enhancement proposes to modify the SPDY data stream between kubectl and the API Server by conditionally adding a StreamTranslatorProxy at the API Server. If the request is for a WebSocket upgrade with the protocol request for RemoteCommand: v5.channel.k8s.io , the handler will delegate to the StreamTranslatorProxy instead of the UpgradeAwareProxy. This translation proxy terminates the WebSocket connection, and it de-multiplexes the various streams in order to pass the data on to a SPDY connection, which continues upstream (to Kubelet and eventually the container runtime).

Background: PortForward Subprotocol

The following steps articulate the difference betweeen a request for an upgraded streaming connection, and the following subrequests which are made over the upgraded connection.

  1. kubectl port-forward makes a request to the server upgrading to a SPDY streaming connection.
  2. An arbitrary number of subsequent (and possibly concurrent) subrequests can be made over this previously established connection. Example: curl http://localhost:8080/index.html.
  3. Each of these subrequests creates two streams over the connection (a uni-directional error stream and a bi-directional data stream) between the client and the container runtime.
  4. The resources associated with the subrequest are reclaimed once the subrequest is completed.

The PortForward subprotocol is used to implement kubectl port-forward, and it differs from the RemoteCommand subprotocol in how the multiple streams within the single upgraded connection are created. The RemoteCommand subprotocol statically creates streams (e.g STDIN, STDOUT, etc.) when the connection is created. But the PortForward subprotocol dynamically creates two streams (a bi-directional data stream and a unidirectional error stream) for each subsequent portforward subrequest. These streams are removed when the subrequest is complete. The connection, however, continues to exist until it is manually stopped (with a signal). For example, the following portforward command creates the upgraded connection, but without any streams, listening on the local port 8080:

$ kubectl port-forward nginx 8080:80

Once the upgraded, streaming connection is created, portforward subrequests are handled by dynamically creating a data and error stream in another goroutine, forwarding the data to the remote port on the target. In this example, the HTTP subrequest is forwarded over the data stream from the local port 8080 to the nginx container listening on port 80:

$ curl http://localhost:8080/index.html

The nginx HTTP response is returned over the same data stream. Once the subrequest is complete, both streams are closed and removed.

Proposal: New PortForward Tunneling Subprotocol Version - v2.portforward.k8s.io

We propose a new PortForward version v2.portforward.k8s.io, which identifies upgrade requests which require tunneling. PortForward tunneling transparently encodes and decodes SPDY framed messages into (and out of) the payload of a WebSocket message. This tunneling is implemented on the client by substituting a WebSocket connection (which implements the net.Conn interface) into the constructor of a SPDY client. The SPDY client reads and writes its messages into and out of this connection. These SPDY messages are then encoded or decoded into and out of the WebSocket message payload.

Proposal: API Server PortForward – Stream Tunnel Proxy

At the API Server, tunneling is implemented by sending different parameters into the UpgradeAwareProxy. If the new subprotocol version v2.portforward.k8s.io is requested, the UpgradeAwareProxy is called with a new tunnelingResponseWriter. This ResponseWriter contains a tunneling WebSocket connection, which is returned when the connection is hijacked. And this tunneling WebSocket connection encodes and decodes SPDY messages as the downstream connection within the dual concurrent io.Copy proxying goroutines. The upstream connection is the same SPDY connection to the container (through the Kubelet and CRI).

Proposal: Synthetic RBAC CREATE Authorization Check

The transition to WebSockets requires changing the initial streaming upgrade request from a POST to a GET, as the WebSocket protocol specification (RFC 6455 ) mandates that the opening handshake must be an HTTP GET request. This has an unintended security consequence: RBAC policies that only grant the get verb, such as a typical read-only “viewer” role, now unexpectedly allow users to run kubectl exec, attach, and port-forward. To close this privilege escalation vector and restore the principle of least privilege, this proposal introduces a secondary, synthetic authorization check performed within the API Server. When a WebSocket upgrade request is received for the pods/exec, pods/attach, or pods/portforward subresources, the handler will perform an additional check to ensure the user also has the create verb permission for that specific subresource. This new authorization check will be controlled by a feature gate, AuthorizePodWebsocketUpgradeCreatePermission, which will be enabled by default to ensure clusters are secure while allowing operators to temporarily disable it as they update their RBAC policies.

Proposal: Transitioning the API Server-to-Kubelet Connection

The long-term goal of this KEP is to replace SPDY with WebSockets for the entire communication path from the client to the Kubelet. The second phase of this transition, extending WebSocket support to the Kubelet for exec, attach, cp, and port-forward, is controlled by the ExtendWebSocketsToKubelet feature gate. This feature gate depends on the NodeDeclaredFeatures feature gate.

When both feature gates are enabled, the Kubelet advertises its streaming protocol capabilities by adding ExtendWebSocketsToKubelet to the declaredFeatures field in the Node.Status object.

The API server’s handlers for exec, attach, and port-forward are modified to check for this feature on the target node.

  • If the ExtendWebSocketsToKubelet gate is enabled and the node’s declaredFeatures field contains ExtendWebSocketsToKubelet, the API server acts as a simple pass-through proxy, forwarding the client WebSocket connection directly to the Kubelet without any protocol translation or tunneling. This is crucial for distributing the load of translation/tunneling from the API server to the Node, thereby offloading the proxying work from the API server.
  • If the Kubelet does not advertise WebSocket support (or either feature gate is disabled), the API server falls back to the original behavior: it accepts the client’s WebSocket connection and either translates it (exec/attach) or tunnels it (port-forward) to an upstream SPDY connection to the Kubelet.

On the Kubelet side, the HTTP server is updated to handle incoming WebSocket upgrade requests.

  • For exec and attach, it uses a StreamTranslatorProxy to terminate the WebSocket connection and translate the v5.channel.k8s.io subprotocol into a SPDY stream for the final communication leg to the container runtime.
  • For port-forward, it uses a StreamTunnelingProxy to terminate the WebSocket connection, decode the SPDY frames from the WebSocket message payloads, and forward them over a standard SPDY connection to the container runtime.

In both cases, legacy SPDY requests from older API servers are still handled correctly.

Test Plan

[X] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates
Unit tests

The following packages (including current test coverage) will be modified to implement this SDPY to WebSockets migration.

  • k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/portforward: 2024-05-27 - 86.3%
  • k8s.io/kubernetes/staging/src/k8s.io/client-go/tools/remotecommand: 2023-05-31 - 57.3%
  • k8s.io/kubernetes/staging/src/k8s.io/client-go/transport: 2023-05-31 - 57.7%
  • k8s.io/kubernetes/staging/src/k8s.io/apimachinery/pkg/util/httpstream: 2023-05-31 - 76.7%
  • k8s.io/kubernetes/staging/src/k8s.io/apimachinery/pkg/util/proxy: 2023-05-31 - 59.1%
  • k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/util/proxy: 2024-05-27 - 81.5%
  • k8s.io/kubernetes/staging/src/k8s.io/kubectl/pkg/cmd/attach: 2023-06-05 - 43.4%
  • k8s.io/kubernetes/staging/src/k8s.io/kubectl/pkg/cmd/cp: 2023-06-05 - 66.3%
  • k8s.io/kubernetes/staging/src/k8s.io/kubectl/pkg/cmd/exec: 2023-06-05 - 70.0%
  • k8s.io/kubernetes/staging/src/k8s.io/kubectl/pkg/cmd/portforward: 2024-05-27 - 74.7%

An important set of tests for this migration will be loopback tests, which exercise the WebSocket client and the StreamTranslator proxy. These tests create two test servers: a proxy server handling the stream translation, and a fake SPDY server which sends received data from one stream (e.g. stdin) back down another stream (e.g. stdout). These tests send random data from the WebSocket client to the StreamTranslator proxy, which then sends the data to the test SPDY server.

WebSocket client <-> Proxy Server (StreamTranslator) <-> SPDY Server

Once the data is received back at the WebSocket client on the separate stream, it is compared to the data that was sent to ensure the data is the same. These loopback tests have been implemented in a proof-of-concept PR, validating the various streams sent over the WebSocket connection by the client through the StreamTranslator proxy.

Integration tests

PortForward: https://github.com/kubernetes/kubernetes/blob/master/test/integration/apiserver/portforward/portforward_test.go

e2e tests

While there are already numerous current e2e tests for kubectl exec, cp, attach, and port-forward, we will enhance these tests with the permutations of the feature flags for kubectl and the API Server. We will add e2e test coverage for flags and arguments that are not already covered for these commands.

Graduation Criteria

Alpha

v1.29 RemoteCommand Subprotocol (exec, cp, and attach)
  • Implement the alpha version of the RemoteCommand subprotocol, and surface the new kubectl exec, kubectl cp, and kubectl attach behind a kubectl environment variable which is OFF by default.
  • WebSocketExecutor and FallbackExecutor completed and functional behind the kubectl environment variable KUBECTL_REMOTE_COMMAND_WEBSOCKETS which is OFF by default.
  • StreamTranslatorProxy successfully integrated into the UpgradeAwareProxy behind an API Server feature flag which is off by default.
  • Initial exec, cp, and attach unit tests completed and enabled.
  • Existing exec, cp, and attach integration tests continue to work.
  • Existing exec, cp, and attach e2e tests continue to work.
v1.30 PortForward Subprotocol (port-forward)
  • Implement the alpha version of the PortForward subprotocol, and surface the new kubectl port-forward behind the kubectl environment variable KUBECTL_PORT_FORWARD_WEBSOCKETS which is OFF by default.
  • FallbackDialer is completed and functional behind the kubectl environment variable KUBECTL_PORT_FORWARD which is OFF by default. The FallbackDialer executes legacy SPDY port-forward if the server does not support the new WebSockets functionality.
  • PortForward StreamTunnelingProxy successfully added and integrated, living behind the API Server feature flag PortForwardWebsockets which is OFF by default.

Beta

v1.30 RemoteCommand Subprotocol (exec, cp, and attach)
  • kubectl environment variable KUBECTL_REMOTE_COMMAND_WEBSOCKETS is ON by default.
  • API Server feature flag TranslateStreamCloseWebsocketRequests is ON by default.
  • Additional exec, cp, and attach unit tests completed and enabled.
  • Additional exec, cp, and attach integration tests completed and enabled.
  • Additional exec, cp, and attach e2e tests completed and enabled.
v1.31 PortForward Subprotocol (port-forward)
  • kubectl port-forward is behind the kubectl environment variable KUBECTL_PORT_FORWARD_WEBSOCKETS which is ON by default.
  • FallbackDialer is completed and functional behind the kubectl environment variable KUBECTL_PORT_FORWARD which is ON by default. The FallbackDialer executes legacy SPDY port-forward if the server does not support the new WebSockets functionality.
  • PortForward StreamTunnelingProxy successfully added and integrated, living behind the API Server feature flag PortForwardWebsockets which is ON by default.
  • Additional port-forward unit tests completed and enabled.
  • Additional port-forward integration tests completed and enabled.
  • Additional port-forward e2e tests completed and enabled.
v1.35 Synthetic RBAC CREATE Authorization Check
  • Force synthetic RBAC CREATE authorization check for WebSocket upgrades on the following subresources: pods/exec, pods/attach, and pods/portforward. This additional check will be gated by the API Server AuthorizePodWebsocketUpgradeCreatePermission feature flag, which defaults to TRUE.
v1.36 Extend WebSockets to Kubelet
  • ExtendWebSocketsToKubelet feature gate is ON by default (Beta), controlling the extension for both RemoteCommand (exec/attach) and PortForward.
  • API Server uses the Kubelet’s declaredFeatures field in the Node.Status to determine if it can proxy WebSocket requests directly to the Kubelet for all streaming commands.
  • Kubelet handles incoming WebSocket requests for exec/attach (translation) and port-forward (tunneling), converting them to SPDY for the container runtime.
  • Unit and integration tests for the new API Server and Kubelet logic are completed and enabled.

GA

  • kubectl environment variables and API Server feature gates are locked to on by default.
  • Deprecate kubectl environment variables and API Server feature gates for future removal.
  • Add WebSocket support for HTTPS proxies.
  • Conformance tests for RemoteCommand completed and enabled.
  • Conformance tests for RemoteCommand have been stable and non-flaky for two weeks.
  • Conformance tests for PortForward completed and enabled.
  • Conformance tests for PortForward have been stable and non-flaky for two weeks.
  • Achieve stable (GA) status for the extension of the WebSockets communication leg from the API Server to Kubelet.

Upgrade / Downgrade Strategy

Upgrade requires both the kubectl environment variable and API Server feature flags to be enabled. Downgrade requires one of the kubectl environment variable or API Server feature flags to be disabled.

Version Skew Strategy

This feature needs to take into account the following version skew scenarios:

RemoteCommand Subprotocol

  1. A newer WebSockets enabled kubectl communicating with an older API Server that does not support the newer StreamTranslator proxy.

In this case, the initial upgrade request for WebSockets/RemoteCommand will fail, because the WebSockets upgrade request v5.channel.k8s.io will be proxied to the current container runtime which only supports up to version v4.channel.k8s.io. The FallbackExecutor will follow up with a subsequent legacy upgrade request for SDPY/RemoteCommand. The streaming functionality in this case will work exactly as it has for the last several years.

  1. A legacy non-WebSockets enabled kubectl communicating with a newer API Server that supports the newer StreamTranslator proxy.

The legacy kubectl will successfully request an upgrade for SPDY/RemoteCommand - V4, just as it has for the last several years.

PortForward Subprotocol

  1. A newer WebSockets enabled kubectl communicating with an older API Server that does not support the newer PortForward StreamTunneling proxy.

In this case, the initial upgrade request for PortForward WebSockets will fail, because the WebSockets upgrade request v2.portforward.k8s.io will be proxied to the current container runtime which only supports version v1.portforward.k8s.io. Upon receiving this upgrade failure, the portforward client will fallback to the legacy SPDY v1.portforward.k8s.io. In this fallback case, the PortForward streaming functionality in this case will work exactly as it has for the last several years.

  1. A legacy non-WebSockets enabled kubectl communicating with a newer API Server that supports the newer PortForward StreamTunneling proxy.

The kubectl port-forward will successfully request an upgrade for legacy SPDY/PortForward - V1, just as it has for the last several years.

Version Skew within the Control Plane and Nodes

The phased rollout of this feature creates three distinct API server behaviors and three Kubelet behaviors. The system is designed to handle version skew gracefully across all combinations.

API Server Versions:

  1. Legacy SPDY-Only (prior to k8s 1.29): Does not support WebSocket streaming.
  2. Phase 1 (WS @ API Server) (k8s 1.29+, enabled by default since 1.30 for RemoteCommand and 1.31 for PortForward): Supports WebSocket streaming from the client but always translates/tunnels this to an upstream SPDY connection for the Kubelet.
  3. Phase 2 (WS @ Kubelet) (k8s 1.36+, beta): Supports WebSocket streaming and can proxy it directly to a capable Kubelet, falling back to Phase 1 behavior if the Kubelet does not support WebSockets.

Kubelet Versions:

  1. SPDY-Only (prior to k8s 1.36): The legacy Kubelet which only accepts SPDY streams.
  2. WS-Capable during development/rollout (k8s 1.36+, beta): The newer Kubelet which accepts both SPDY and WebSocket streams, advertising its capabilities during the period of time there are supported Kubelet versions which do not have the feature or which can disable the feature.
  3. WS-Capable after all supported versions have this feature (stable): The newer Kubelet which accepts both SPDY and WebSocket streams, no longer needing to advertise its capabilities because all supported kubelet versions have the feature locked enabled.

Interaction Scenarios:

  • Any kubectl vs. Legacy SPDY-Only API Server: kubectl’s initial WebSocket upgrade request is rejected, and it automatically falls back to using SPDY. Streaming works via legacy SPDY.

  • Phase 1 API Server vs. SPDY-Only or WS-Capable Kubelet: The API server accepts the client’s WebSocket stream, performs the translation/tunneling itself, and sends a SPDY stream to the Kubelet. Since both Kubelet versions accept SPDY, streaming works.

  • Phase 2 API Server vs. SPDY-Only Kubelet: The API server checks the Kubelet’s declaredFeatures field, does not find ExtendWebSocketsToKubelet, and gracefully falls back to the Phase 1 behavior (translating/tunneling locally). Streaming works via API server fallback.

  • Phase 2 API Server vs. WS-Capable Kubelet: The API server checks the declaredFeatures field, finds ExtendWebSocketsToKubelet, and acts as a pass-through proxy for the WebSocket stream directly to the Kubelet. Streaming works via the optimal end-to-end WebSocket path.

This design ensures that clusters can be upgraded one component at a time without breaking streaming functionality.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?
  • Feature gates (also fill in values in kep.yaml)

The feature is controlled by a combination of client-side environment variables and server-side feature gates.

Client-side (kubectl):

  • KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true: Enables WebSocket usage for exec, attach, and cp commands.
  • KUBECTL_PORT_FORWARD_WEBSOCKETS=true: Enables WebSocket usage for the port-forward command.

Server-side (Feature Gates):

  • TranslateStreamCloseWebsocketRequests: (Component: kube-apiserver) Enables the API server to handle the WebSocket-based v5.channel.k8s.io subprotocol for remote commands.
  • PortForwardWebsockets: (Component: kube-apiserver) Enables the API server to handle WebSocket-based tunneling for port forwarding.
  • AuthorizePodWebsocketUpgradeCreatePermission: (Component: kube-apiserver) Enforces a synthetic CREATE authorization check on WebSocket upgrade requests to maintain least privilege.
  • ExtendWebSocketsToKubelet: (Components: kube-apiserver, kubelet) Enables the end-to-end WebSocket communication path to the Kubelet, allowing the API server to proxy streams directly instead of translating/tunneling them. (Note: Depends on the NodeDeclaredFeatures gate).
Does enabling the feature change any default behavior?

For each of the two streaming subprotocols: RemoteCommand (such as /exec and /attach APIs) and PortForward (for /portforward), enabling the respective feature gate on the API Server will allow the streaming mechanism to be WebSockets instead of SPDY for communication between kubectl and the API Server. Additionally, the kubectl client must also have the KUBECTL_REMOTE_COMMAND_WEBSOCKETS environment variable set to ON for exec, cp, and attach commands. While the KUBECTL_PORT_FORWARD_WEBSOCKETS environment variable must be set to ON for port-forward command. These modifications, however, will be transparent to the user unless the kubectl/API Server communication is communicating through an intermediary such as a proxy (which is the whole reason for the feature). The API Server feature flag AuthorizePodWebsocketUpgradeCreatePermission forces a synthetic, secondary RBAC check for the CREATE verb permission on WebSocket upgrade requests. When this feature gate is TRUE, the additional permission check will apply to endpoints pods/exec, pods/attach, and pods/portforward. Enabling the ExtendWebSocketsToKubelet feature gate on the API Server will change the default behavior for exec, attach, and port-forward by attempting to proxy WebSocket requests directly to Kubelets that advertise support for it, thus offloading protocol translation and tunneling from the API Server to the Kubelet.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

The features can be disabled for a single user by setting the kubectl environment variable associated with the feature to OFF. Or the features can be turned off for all kubectl users communicating with a cluster by turning off the feature flags for the API Server. A cluster operator can temporarily disable the more stringent permissions for subresources pods/exec, pods/attach, and pods/portforward by setting the AuthorizePodWebsocketUpgradeCreatePermission feature flag to FALSE. Disabling the ExtendWebSocketsToKubelet feature gate in the API Server causes a reversion to the previous behavior where translating and tunneling occur in the API Server, even if the Kubelet supports this functionality. Additionally, disabling the ExtentWebSocketsToKublet in the Kubelet ensures new code implementing the translation and tunneling in the Kubelet does not run.

What happens if we reenable the feature if it was previously rolled back?

The feature does not depend on state, and can be disabled/enabled at will.

Are there any tests for feature enablement/disablement?
  • There will be unit tests for the kubectl environment variable KUBECTL_REMOTE_COMMAND_WEBSOCKETS.
  • There are unit tests for the kubectl environment variable KUBECTL_PORT_FORWARD_WEBSOCKETS.
  • There will be unit tests in the API Server which exercise the feature gate within the UpgradeAwareProxy, which conditionally delegates to the StreamTranslator proxy (depending on the feature gate and the upgrade parameters).
  • There are unit tests in the API Server which exercise the feature gate within the UpgradeAwareProxy, which conditionally delegates to the StreamTunneling proxy for the PortForward subprotocol.
  • There will be unit tests in the API Server to verify the feature gate forcing more stringent RBAC checks for pods/exec, pods/attach, and pods/portforward.
  • There will be unit tests in the API Server and Kubelet to verify the feature gate ExtendWebSocketsToKubelet.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

This feature does not impact already running workloads, as it only affects the initiation of new kubectl exec/attach/port-forward commands.

The design is resilient to most rollout and rollback failures due to its discovery and fallback mechanisms. However, failures can still occur:

  • Partial API Server Rollout: During a rolling update, a kubectl client may connect to an API server that does not yet support WebSockets. In this case, the client’s connection upgrade will be rejected, and it will automatically fall back to using the legacy SPDY protocol for that request. This is seamless to the user.

  • Partial Node Rollout (Kubelet Extension): When ExtendWebSocketsToKubelet is enabled, a rolling update of nodes can lead to a mix of capable and incapable Kubelets. This is handled gracefully by the API server, which checks the Node.Status.declaredFeatures for each request. It will use the WebSocket-to-Kubelet path for updated nodes and revert to the SPDY translation path for older nodes.

  • Misconfiguration: A rollout could fail if the ExtendWebSocketsToKubelet gate is enabled on a Kubelet, but a network policy between the API server and the Kubelet blocks the WebSocket protocol’s port and upgrade headers. This would cause streaming commands to those specific nodes to fail. This would also fail in the case of legacy SPDY streaming, so it would not be a change.

  • Implementation Bugs: As with any new feature, a bug in the WebSocket implementation in the API server or Kubelet could cause failures for streaming commands that use the new code paths. A rollback of the specific feature gate (ExtendWebSocketsToKubelet) would be the primary mitigation strategy.

What specific metrics should inform a rollback?

The most straightforward signal indicating a problem for the feature is a high rate of failures for kubectl exec, cp, attach, and port-forward commands.

For the Kubelet extension specifically, a key rollback signal would be a significant increase in streaming connection failures or latency that is isolated to nodes that have the ExtendWebSocketsToKubelet feature enabled, while nodes that still use SPDY connections from the API server continue to function normally. This can be monitored via the metrics proposed in the Service Level Indicators section.

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

This feature is stateless (it does not persist any new API objects or fields), so a formal stateful upgrade->downgrade->upgrade test path is not applicable.

However, the resilience of the system during rolling upgrades and rollbacks has been validated through comprehensive version skew testing. These tests cover all permutations of client, API server, and Kubelet versions to ensure that the fallback and feature-discovery mechanisms work as intended. For example:

  • A newer kubectl client correctly falls back to SPDY when communicating with an older API server that does not support WebSockets.
  • A newer API server correctly reverts to translating/tunneling SPDY when communicating with an older Kubelet that does not support the WebSocket extension.

This ensures that streaming functionality is maintained across the cluster during the entire upgrade or rollback process.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No.

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

  • An operator can detect if the WebSocket functionality is enabled by checking either the num_ws_remote_command_v5_total[success] metric for RemoteCommand or the num_ws_port_forward_v2_total[success] metric for PortForward.

  • To determine if the Kubelet extension is active for a specific node, an operator can inspect the status.declaredFeatures field of the Node object for the presence of ExtendWebSocketsToKubelet. Additionally, proposed new metrics will differentiate between API server connections that are proxied directly to the Kubelet versus those that are translated to SPDY.

How can someone using this feature know that it is working for their instance?

To confirm this feature is working for a given instance, both its configuration and runtime behavior can be observed:

  • Configuration Verification:

    • API Server Feature Gates: Inspect the API server’s startup flags (--feature-gates) to confirm that relevant feature gates (e.g., TranslateStreamCloseWebsocketRequests, PortForwardWebsockets, ExtendWebSocketsToKubelet) are enabled.
    • Kubelet Feature Gates: Confirm the kubelet is started with --feature-gates=ExtendWebSocketsToKubelet=true (or without ExtendWebSocketsToKubelet=false if it is enabled by default).
    • Node Declared Features: For the Kubelet extension, verify that individual nodes are advertising support: kubectl get node <node-name> -o jsonpath='{.status.declaredFeatures}' should include ExtendWebSocketsToKubelet.

  • Runtime Verification:

    • Client-side (kubectl -v=7 logs):
      • For RemoteCommand (exec, attach, cp), look for ...websocket.go:137] The subprotocol is v5.channel.k8s.io.
      • For PortForward, look for ...websocket-dialer.go:91] negotiated protocol: v2.portforward.k8s.io.
    • Server-side (API Server and Kubelet logs):
      • API Server logs: When ExtendWebSocketsToKubelet is active, API server logs will indicate it is proxying the WebSocket connection directly to the Kubelet, rather than translating it.
      • Kubelet logs: Kubelet logs on the target node will show it receiving and processing a WebSocket connection for the streaming request.
What are the reasonable SLOs (Service Level Objectives) for the enhancement?

At a high level, we are aiming for an SLO which is the same as the current SPDY streaming SLO. Turning on WebSockets streaming between the kubectl client and the API Server should not noticeably degrade streaming performance compared to the current SPDY implementation. But since we do not have numbers for our current SPDY streaming implementation, these SLO’s are necessarily educated estimates.

  • 99.9% of initial HTTP connections eventually succeed in upgrading to a streaming connection (i.e. 101 Switching Protocols as the HTTP response).

  • 99.9% of streaming connections complete without writing to the error channel. Each of the streaming subprotocols, RemoteCommand and PortForward, create a separate error channel to communicate problems while streaming. But a normally successful streaming command should NOT need to use these channels.

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
  • Metrics

    • Metric name: num_ws_remote_command_v5_total with type dimension containing enum values success and failure. Counts the total number of times the API Server witnessed a WebSocket/V5 RemoteCommand connection upgrade attempt (either success or failure).

    • Components exposing the metric: API Server

    • Metric name: num_ws_port_forward_v2_total with type dimension containing enum values success and failure. Counts the total number of times the API Server witnessed a WebSocket/V2 PortForward connection upgrade attempt (either success or failure).

    • Components exposing the metric: API Server

    • Metric name: apiserver_streaming_active_connections (Gauge)

      • Help: “Gauge of active streaming connections, to distinguish between connections proxied directly to the Kubelet vs. those that require protocol translation or tunneling at the API server.”
      • Dimensions: subresource (exec, attach, portforward), proxy_type (proxied, translated_or_tunneled)
      • Component exposing the metric: kube-apiserver

    • Metric name: kubelet_streaming_websocket_requests_total (Counter)

      • Help: “Counter of WebSocket streaming upgrade requests handled by the Kubelet.”
      • Dimensions: subresource (exec, attach, portforward), result (success, failure)
      • Component exposing the metric: kubelet
Are there any missing metrics that would be useful to have to improve observability of this feature?

Yes. To properly observe the Kubelet extension, the following metrics are needed:

  • An API server metric (e.g., apiserver_streaming_connections) is needed to differentiate between WebSocket connections that are proxied directly to the Kubelet versus those that are translated to SPDY. This is critical for understanding whether the feature is active and for debugging rollout.
  • A Kubelet metric (e.g., kubelet_streaming_websocket_requests_total) is needed to monitor the rate and success of incoming WebSocket requests directly on the node. This provides visibility into the Kubelet’s performance as a streaming server, which is currently not available.

Dependencies

  • Gorilla/WebSockets library: The new WebSockets streaming functionality imports the Gorilla/WebSockets library.
  • NodeDeclaredFeatures (KEP-5328 ): The ExtendWebSocketsToKubelet feature gate depends on the NodeDeclaredFeatures feature gate.
Does this feature depend on any specific services running in the cluster?

No.

Scalability

Will enabling / using this feature result in any new API calls?

The proposed design envisions a fallback mechanism when a new kubectl communicates with an older API Server. The client will initially request an upgrade to WebSockets, but it will fallback to the legacy SPDY if it is not supported. In this version skew scenario where the client implements the new functionality but the server does not, there is an extra request/response. Since bi-directional streaming already is very IO intensive, this extra request/response should not be significant. Additionally, as releases are incremented, the probability of the version skew will continually decrease.

The newer WebSockets streaming mechanism will also include heartbeat messages, which will require network IO. But this heartbeat mechanism should contain no more messages than the current SPDY heartbeat mechanism.

Will enabling / using this feature result in introducing new API types?

No.

Will enabling / using this feature result in any new calls to the cloud provider?

No.

Will enabling / using this feature result in increasing size or count of the existing API objects?

No.

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

No.

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

Enabling the WebSocket transition to the API server does not in itself increase resource usage.

However, enabling the ExtendWebSocketsToKubelet feature will shift the resource load (CPU and memory) of protocol translation/tunneling from the central API servers to the individual Kubelets on the nodes. This is the explicit goal of this part of the enhancement. While it does increase resource usage on each Kubelet handling a WebSocket stream, it is part of a deliberate strategy to distribute the load and improve the overall scalability of the control plane.

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

kubectl exec, cp, attach, and port-forward commands spawn container runtime processes, so there is the danger of node resource exhaustion. This feature, however, does not change the current legacy mechanism for how these container runtime processes execute or communicate, except for the communication leg between kubectl and the API Server. There should be no more risk of node resource exhaustion than already exists.

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?
  • This feature fails when the API server is unavailable. The WebSocket streaming is proxied through the API server. If the API server is unavailable, streaming functionality does not work (this applies to the legacy SPDY streaming as well, so there is no detectable difference in this scenario).
What are other known failure modes?

  • Failure Mode: Proxy or Gateway that does not support SPDY or WebSockets. SPDY had been deprecated for since 2015, and not all proxies support WebSockets. If the intermediary does not support either streaming protocol it will not be possible to run the kubectl streaming commands.

    • Detection: The kubectl streaming command will return a connection error. The error returned to the client will be from the proxy or gateway; not from the API Server (since the communication never reaches the API Server).
    • Mitigations: None
    • Diagnostics: N/A
    • Testing: N/A

  • Failure Mode: Overloaded API Server from too many concurrent streaming commands.

    • Detection: The streaming commands will show increased latency and timeout errors. The streaming commands could also hang after initiation on the client.
    • Mitigations: The mitigations for this phenomenon are described in detail in the Risks and Mitigations section. But a simple way for a user to mitigate this problem would be to reduce the number of concurrent streaming command from clients.
    • Diagnostics: API Server /healthz and /metrics monitoring.
    • Testing: Manual stress tests of kubectl streaming commands through shell scripts which initiate numerous concurrent streaming commmands.

  • Failure Mode: Increased network latency between the client and the API Server causes problems with streaming commands.

    • Detection: The client streaming commands will lag, timeout, and possibly block.
    • Mitigations: Mitigating this problem would require fixing the problematic network connection.
    • Diagnostics: Adding extra logging from the kubectl streaming commands will show the timings for individual streaming request/responses. For example: $ kubectl exec -v=7 ... will produce logging which shows response times, like: ... Response Status: 101 Switching Protocols in 20 milliseconds Also, API Server /healthz and /metrics monitoring will produces output demonstrating increased response times and increased timeouts.
    • Testing: Manual stress tests which simulate a bad network connection by randomly dropping streamed packets.

  • Failure Mode: API Server that does not support the new WebSockets functionality or the the feature flags are not enabled. If kubectl supports new WebSockets functionality, but the API Server does not, we automatically fall back to legacy SPDY streaming.

    • Detection: The fallback functionality is automatic. But if the user wanted to see if this fallback was occurring, the user could add the -v=7 flag to the kubectl command. The output logging would display the initial WebSockets connection upgrade attempt and failure, and the subsequent SPDY completion of the command.
    • Mitigations: The mitigation (fallback to legacy SPDY) is automatic.
    • Diagnostics: We have suggested metrics to measure the number of fallbacks. the metrics num_ws_remote_command_v5_total[failure] as well as num_ws_port_forward_v2_total[failure] will measure the number of fallbacks.
    • Testing: We have implemented tests for both the FallbackWebSocketExecutor (for RemoteCommand), and the FallbackDialer (for PortForward).

  • Failure Mode: Kubelet advertises WebSocket support but fails to handle streams.

    • Detection: The kubectl command will fail with a generic streaming error. API server logs will show a successful WebSocket upgrade request being proxied to the Kubelet, but the apiserver_streaming_connections metric with proxy_type="proxied" will show failures or short-lived connections. Kubelet logs on the target node will show errors in the WebSocket handling or translation/tunneling logic.
    • Mitigations: An operator can disable the ExtendWebSocketsToKubelet feature gate on the failing node(s) and restart the kubelet service. This will cause the node to stop advertising the feature, and the API server will revert to translating streams for that node, restoring functionality while the issue is investigated.
    • Diagnostics: Kubelet logs (with increased verbosity if necessary) on the failing node will be the primary source for debugging. They will contain errors related to WebSocket handshake, subprotocol translation, or SPDY forwarding to the container runtime.
    • Testing: Unit and integration tests for the Kubelet’s WebSocket server and proxying logic cover the expected behavior. e2e tests will be added to validate the end-to-end flow with the feature gate enabled.
What steps should be taken if SLOs are not being met to determine the problem?
  • Step 1: Turn off the kubectl feature gate, and check the SLO afterwards.

For kubectl exec, kubectl cp, and kubectl attach:

$ unset KUBECTL_REMOTE_COMMAND_WEBSOCKETS
# Run simple exec command with higher verbosity to see relevant logging.
# Look for successful connection upgrade with response 101 Switching Protocols.
# Example: $ kubectl exec -v=7 <TARGET> -- date
#
$ kubectl exec -v=7 nginx -- date
...
I0206 02:22:40.500200 3666799 round_trippers.go:463] POST https://127.0.0.1:37243/api/v1/namespaces/default/pods/nginx/exec?command=date&container=nginx&stderr=true&stdout=true
...
I0206 02:22:40.521184 3666799 round_trippers.go:574] Response Status: 101 Switching Protocols in 20 milliseconds
...
Tue Feb  6 02:22:40 UTC 2024

For kubectl port-forward:

$ unset KUBECTL_PORT_FORWARD_WEBSOCKETS
# Run simple port-forward command against webserving pod/deployment/service
# with higher verbosity to see relevant logging. Look for successful connection
# upgrade with response 101 Switching Protocols.
# Example: $ kubectl port-forward -v=7 <TARGET> <LOCAL_PORT>:<REMOTE_PORT>
#
$ kubectl port-forward -v=7 nginx 8080:80
...
I0206 02:28:04.352912 3669354 round_trippers.go:463] POST https://127.0.0.1:37243/api/v1/namespaces/default/pods/nginx/portforward
...
I0206 02:28:04.372175 3669354 round_trippers.go:574] Response Status: 101 Switching Protocols in 19 milliseconds
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
I0206 02:28:04.372540 3669354 portforward.go:309] Before listener.Accept()...
I0206 02:28:04.372586 3669354 portforward.go:309] Before listener.Accept()...
...

# In another terminal, print out index.html served from TARGET after request forwarded.
# Example: $ curl http://localhost:<LOCAL_PORT>/index.html
#
$ curl http://localhost:8080/index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
  • Step 2: For a more thorough possible fix, restart the API server with the cluster feature gate turned off. Check the SLO after restart.

While there are many ways to start a Kubernetes cluster, I will detail how to modify the cluster feature flags for this WebSockets functionality using kubeadm or kind and cluster configuration files.

Example:

$ kind create cluster --config cluster-config.yaml --name <NAME> --image <IMAGE>

or

$ kubeadm init --config cluster-config.yaml

where the cluster-config.yaml looks like:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  # Enables/Disables WebSockets for RemoteCommand (e.g. exec, cp, attach)
  "TranslateStreamCloseWebsocketRequests": false
  # Enables/Disables WebSockets for PortForwar (e.g. port-forward)
  "PortForwardWebsockets": true
...

  • Step 3: If investigation points to an issue with the Kubelet extension (e.g., failures are isolated to nodes with the feature enabled), disable the ExtendWebSocketsToKubelet feature gate on the API servers. This will cause the API server to revert to performing the protocol translation for all nodes, effectively pausing the extension of WebSocket communication to the Kubelet. The method for configuring API server feature gates varies by provider; for example, on a managed provider like GKE, this would be done via a cluster update command. The goal is to pass the following flag to the API server:

    --feature-gates=ExtendWebSocketsToKubelet=false

Implementation History

  • First Kubernetes release with initial version of KEP available: v1.29
  • RemoteCommand over WebSockets shipped as alpha: v1.29
  • RemoteCommand over WebSockets shipped as beta: v1.30
  • First Kubernetes release where PortForward over WebSockets described in KEP: v1.30
  • PortForward over WebSockets shipped as alpha: v1.30
  • PortForward over WebSockets shipped as beta: v1.31
  • WebSocket HTTPS Proxy functionality shipped: v1.33
  • Synthetic RBAC CREATE authz check for WebSocket upgrade requests: v1.35
  • Extend WebSocket communication to the Kubelet for RemoteCommand and PortForward shipped as beta: v1.36

Drawbacks

The main motivation for taking the risk to change the streaming protocol from SPDY to WebSockets is to support proxies or gateways in between Kubernetes clients and the API Server. If we do not believe it is worth it to support these intermediaries with a modern bi-directional streaming protocol, then we should re-consider this effort.

Alternatives

The only currently supported bi-directional streaming protocol is WebSockets. When HTTP/2.0 was initially proposed, many believed it would provide streaming functionality; this belief appears to have been misplaced. Ironically, HTTP/2.0 is based on SPDY. But the upgraded HTTP/2.0 standard did not surface streaming functionality. For example, HTTP/2.0 specifically does not support Upgrade requests to create a streamable connection. In the golang standard library, HTTP/2.0 requests with the Upgrade header return an error code.

Extend WebSocket Streaming to the Kubelet: Try WebSockets and Fallback to SPDY

Instead of the Kubelet advertising its capabilities, the API server could try to use WebSockets and fallback to SPDY, similar to how kubectl interacts with the API server.

  • Pros:

    • No Feature Declaration: The Kubelet would not need to advertise its capabilities, simplifying the Kubelet’s implementation.
    • Familiar Pattern: This approach reuses the “try and fallback” logic already used by kubectl.

  • Cons:

    • Extra Network Call: A failed WebSocket connection attempt would be required before falling back to SPDY, adding latency to every streaming command for older Kubelets.
    • Implementation Simplicity: The current API server code is already designed to check for features and proxy accordingly. Adding a new “try and fallback” dialing mechanism would be more complex than reusing the existing logic.

Report Protocol Support Per-Subresource

Instead of the single ExtendWebSocketsToKubelet feature gate for all streaming subresources, we could report protocol support on a per-subresource basis (e.g., for exec/attach and port-forward individually).

  • Pros:

    • Granular Control: Allows for a phased rollout and targeted testing of each subresource’s migration to WebSockets.
    • Isolation: An issue with one subresource’s WebSocket implementation would not impact the others.

  • Cons:

    • Increased Complexity: The Kubelet would need to report a more complex data structure, and the API server logic to check for it would be more involved.
    • Complex Configuration: This could lead to a more complex configuration for administrators.
    • Cognitive Overhead: It increases the mental model for developers and users to understand which subresource uses which protocol.

Use metadata.annotations

Instead of using the status.declaredFeatures field, the Kubelet’s WebSocket capabilities could be advertised using an annotation on the Node object.

  • Pros:

    • Simpler Implementation: Uses the existing, well-understood annotation mechanism.
    • No NodeDeclaredFeatures Dependency: Avoids a formal dependency on the NodeDeclaredFeatures feature gate.
    • Flexibility: Key-value nature allows for more expressive feature declarations.

  • Cons:

    • Not Standardized: NodeDeclaredFeatures is the purpose-built, standard way to declare node features (see KEP-5328 for more reasoning behind using a unified mechanism).
    • Less Secure: The RBAC permissions for modifying annotations are typically broader than for the more protected status subresource.
    • No Centralized Validation: Prone to errors due to the lack of a centralized validation mechanism for feature names.
    • Poor Integration: Other components, like the scheduler, would need custom logic to interpret the annotation.

Infrastructure Needed (Optional)

N/A