KEP-2718: Client Executable Proxy

Implementation History
ALPHA Withdrawn
Created 2021-05-11
Latest v1.27
Milestones
Alpha v1.27
Beta TBD
Stable TBD
Related Enhancements
Ownership
Owning SIG
SIG Auth
Participating SIGs
SIG CLISIG API Machinery
Primary Authors
@nckturner

KEP-2718: Client Executable Proxy

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

Client authentication in Kubernetes can be achieved via one of a handful of mechanisms: by passing a bearer token directly to kubectl using the --token flag, by inserting a bearer token into the kubeconfig, by inserting a client certificate and key into the kubeconfig, or by defining an exec-based token plugin in the kubeconfig which supports calling an arbitrary binary which is expected to return a bearer token and/or client certificate/key. The client exec proxy adds an extension point for additional authentication mechanisms that are difficult to set up with the current architecture. Client request signing, and an external TLS certificate authenticator are two use cases that would benefit from this proposal.

Motivation

The motivation for this KEP stems from the inability of the existing authentication mechanisms to satisfy the author’s requirement to implement a Kubernetes client that implements AWS’s request signing process, known as Signature Version 4 (sig-v4) [1] [2] . In order to implement sig-v4, a canonical version of the request must be created, combined with any required additional metadata, signed by the secret key, and the signature appended to the request in a header or query string parameter. In order to achieve this, the request signer must have access to the entire request, so the existing exec-based authentication is unsatisfactory. A generic solution is desirable to satisfy other future request signing or request modifying feature requests.

The solution that exists today is to use the existing proxy configuration in client-go. The includes configuring a proxy via the environment variables, HTTP_PROXY or HTTPS_PROXY, or using the explicit proxy URL supported by kubeconfig [3] . The problem with this solution is that the operator experience of setting up and managing an external proxy is poor. For example, to implement a request signing proxy for kubelet, one has to manage the proxy process with an init system of some kind, and secure the proxy endpoint, both of which require significant, non-trivial configuration outside of Kubernetes.

Another use case is using an external TLS certificate signer [4] . This would allow usage of TLS client certificates within client-go, by delegating digital key operations to external processes, for example, an HSM.

Goals

The goals of this KEP include:

  1. Defining a specification which enables Kubernetes clients to do arbitrary request modification before sending the request to the API server, via an extensible mechanism to support the following use cases:
    • Client request signing.
    • Using hardware protected keys for request signing or mTLS.
    • Adding or manipulating arbitrary headers to the request.
  2. Defining a specification which would provide the ability for the client-side proxy server to be executed and communicated with according to configuration solely contained in a kubeconfig file.
  3. Defining a specification which is interoperable with an existing client-side proxy setup, e.g. a proxy configured with the HTTP_PROXY environment variable.
  4. Defining a specification for a client-side proxy server that is easy to secure without taking additional setup steps, by:
    • Using a unix domain socket for the communication between the client and the proxy, secured with file permissions and TLS.

Non-Goals

The following are non-goals of this KEP:

  1. Solving use cases outside of the above-mentioned client authentication workflows.
  2. Implementing a specific flavor of request signing (like SigV4), or adding support for any hardware security module protocol directly in the Kubernetes client libraries.
  3. Defining any new mechanisms for validating request signatures server-side. This is because they can be validated by a front-proxy, which is already a feature of Kubernetes.

Proposal

This proposal is to implement an extension in Kubernetes client libraries to forward requests to a proxy process for request modification. The proxy will receive an HTTP request and will have the opportunity to sign the request, attach headers, use hardware protected keys for mTLS, and/or modify the request. The final request will be constructed and forwarded to the apiserver.

It is already possible, by setting the appropriate HTTP_PROXY, HTTPS_PROXY environment variables, to configure a Kubernetes client to send requests to a proxy. While this may work in some situations, it is not ideal for a generic request signing solution across all client configurations. Firstly, http proxies are already commonly integrated in client networking configurations, so for these clients setting up an additional proxy configuration might not be possible, or at least not straightforward. Secondly, this approach requires a long running external proxy process which introduces operational complexity. Thirdly, communicating with a proxy for request signing requires a secure communication channel between the client and the proxy, which means in a case where multiple users share a client computer (for example, a shared bastion), configuring and securing a local proxy to prevent signing requests as a different user becomes more complicated.

Risks and Mitigations

The risks of this proposal are similar to the existing exec credential provider. There are risks associated with execing a binary, for example in a scenario where users are downloading binaries from untrustworthy sources to use in their cluster. However, this is not a departure from existing security posture of the exec credential feature present in k8s.io/client-go today.

Design Details

Proxy Proposal

The proxy exec plugin is a long-running proxy process which is executed by the client on first use. When executed, the exec proxy plugin will bind and listen on a socket, and will a output URL on stdout, over which the client would proxy all API server bound requests. The proxy binary location is defined in the ExecConfig section of a kubeconfig user.

The proxy plugin process will be executed from client-go the first time it is needed, on transport setup. If the client fails to connect to the socket because no one is listening, client-go will reattempt to exec the plugin. Client-go will support both plugins binaries which are themselves the proxy process, and additionally plugin binaries which complete, but are responsible for starting a proxy process and returning a URL to the proxy.

The client will send its request to the proxy plugin process, and the proxy will forward the request to the API server, rather than returning the modified request to the client to be sent (see alternatives considered). The downside of this approach is that the proxy must duplicate the some of the transport functionality of client packages, like respecting existing proxy configuration, but allows for more flexibility for proxy plugin developers.

Authentication to Proxy

Options:

  1. The client will authenticate to its proxy using mTLS. On exec, the proxy will pass back client key and certificate authority data for the client to use for authentication when establishing a connection.
  2. An alternative method for authentication to the proxy would be file permission based. The GID owner of the client process could passed to the proxy over stdin, which the proxy should give read/write permissions on the socket to. However, this could be less portable if we need to use a non UDS communication mechanism for windows, or if we support TCP/IP in the future.
  3. A third mechanism for authentication for windows based systems would be using named pipes, which support the Win32 access control model.

An option will be chosen during implemenation of alpha.

Caching

The connection from the client to the proxy can be cached in the client based on the cluster details, the existing HTTP_PROXY configuration, the exec-proxy URL, and the certificates used for mTLS.

Proxy Shutdown

In the case where the client dies unexpectedly, due to a crash or SIGKILL, the proxy should also die. To handle this, the proxy will attempt to take a file lock that the parent holds and quit if it ever successfully takes the lock. Before exec, the client should take the lock. The lock file path will be provided to the proxy in the ExecCredentialSpec.

Alterntatively, the proxy could quit after a certain amount of idle time. For example, proxies that are idle for more than 5 minutes will be required to quit. Idle quit is simpler, but the lock approach handles the kubectl use case better, so it is preferred.

API

The communication over stdin and stdout with the exec’d binary will be done with using the existing ExecCredentialSpec and ExecCredentialStatus objects.

// ExecCredentialSpec holds request and runtime specific information provided by
// the transport.
type ExecCredentialSpec struct {
	// Cluster contains information to allow an exec plugin to communicate with the
	// kubernetes cluster being authenticated to. Note that Cluster is non-nil only
	// when provideClusterInfo is set to true in the exec provider config (i.e.,
	// ExecConfig.ProvideClusterInfo).
	// +optional
	Cluster *Cluster `json:"cluster,omitempty"`

	// Interactive declares whether stdin has been passed to this exec plugin.
	Interactive bool `json:"interactive"`

	// New:

	// LockFilePath specifies a lock file to be taken by the client
	// +optional
	LockFilePath string `json:"lockFilePath,omitempty"`
}

// ExecCredentialStatus holds credentials for the transport to use.
//
// Token and ClientKeyData are sensitive fields. This data should only be
// transmitted in-memory between client and exec plugin process. The Exec plugin
// itself should at least be protected via file permissions.
type ExecCredentialStatus struct {
	// ExpirationTimestamp indicates a time when the provided credentials expire.
	// +optional
	ExpirationTimestamp *metav1.Time `json:"expirationTimestamp,omitempty"`
	// Token is a bearer token used by the client for request authentication.
	Token string `json:"token,omitempty" datapolicy:"token"`
	// PEM-encoded client TLS certificates (including intermediates, if any).
	ClientCertificateData string `json:"clientCertificateData,omitempty"`
	// PEM-encoded private key for the above certificate.
	ClientKeyData string `json:"clientKeyData,omitempty" datapolicy:"security-key"`

	// New:

	// ProxyConfig is the config returned by the exec-proxy
	// +optional
	ProxyConfig *ProxyConfig `json:"proxyConfig,omitempty"`
}


// ProxyConfig configures the client to proxy requests via the returned address.
type ProxyConfig struct {
	// URL is the location of the exec proxy listener.
	URL string `json: "url"`

	CertificateAuthorityData []byte `json:"certificate-authority-data"`
}

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates
Unit tests
  • k8s.io/kubernetes/vendor/k8s.io/client-go/tools/auth/exec: 2022-10-12 - 75
  • k8s.io/kubernetes/vendor/k8s.io/client-go/transport: 2022-10-12 - 57.6
Integration tests
  • Authentication between client and proxy:
  • Connection from client to apiserver through proxy:
  • Proxy use with kubectl (possible test location: command-line integration test suite ):
  • UDS connections from the client to the proxy will be tested:
e2e tests
  • Communication from client to apiserver through proxy in a real cluster. The e2e tests will require a proxy implementation and apiserver, etcd, and a client:

Graduation Criteria

Alpha

  • Implementation of UDS connections to proxy.

Alpha -> Beta Graduation

  • Implementation of UDS (or equivalent) on windows.
  • Full reference proxy implementation with TLS offload. This will be built using code from client-go that provides a harness to offload to a Go crypto.Signer.
  • As part of client-go, it needs to work on macOS, windows and linux.
  • Tests that show SPDY based APIs like pods/exec still work.
  • Tests that exercise webhooks from API server because they share the same auth code.
  • Gather feedback from developers, SIG Auth, SIG CLI.
  • Tests are in Testgrid and linked in KEP.

Upgrade / Downgrade Strategy

For clients which are not currently using an exec plugin, a modification to the kubeconfig must be made and a proxy binary must be located as specified in the kubeconfig. For clients already using an exec plugin without proxy support, the exec plugin would need to upgraded. For clients moving from one version of the proxy to another, the proxy binary must be upgraded.

Version Skew Strategy

The proxy binary may be required to be updated before the client is updated in situations where the API over stdio between them changes.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

This section must be completed when targeting alpha to a release.

  • How can this feature be enabled / disabled in a live cluster?

    • Feature gate (also fill in values in kep.yaml)
      • Feature gate name: ClientExecProxy
      • Components depending on the feature gate: kubelet, kube-controller-manager, kube-scheduler (for any that need to use the proxy–this will be environment dependent).
      • For kubectl, an env var will be used to enable when its an alpha feature.
    • Other
      • Describe the mechanism:
      • Will enabling / disabling the feature require downtime of the control plane?
      • Will enabling / disabling the feature require downtime or reprovisioning of a node?

  • Does enabling the feature change any default behavior? Any change of default behavior may be surprising to users or break existing automations, so be extremely careful here.

    No.

  • Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? Also set disable-supported to true or false in kep.yaml. Describe the consequences on existing workloads (e.g., if this is a runtime feature, can it break the existing applications?).

    Yes, the feature can be disabled. In the case where it is being used by long running clients to talk to the API server, the client would have to be reconfigured to stop using the proxy for connections. Depending on how the client is configured, this may require a restart.

  • What happens if we reenable the feature if it was previously rolled back?

    The feature can be reenabled after being disabled without consequence.

  • Are there any tests for feature enablement/disablement? The e2e framework does not currently support enabling or disabling feature gates. However, unit tests in each component dealing with managing data, created with and without the feature, are necessary. At the very least, think about conversion tests if API types are being modified.

    No.

Rollout, Upgrade and Rollback Planning

This section must be completed when targeting beta graduation to a release.

  • How can a rollout fail? Can it impact already running workloads? Try to be as paranoid as possible - e.g., what if some components will restart mid-rollout?

  • What specific metrics should inform a rollback?

  • Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? Describe manual testing that was done and the outcomes. Longer term, we may want to require automated upgrade/rollback tests, but we are missing a bunch of machinery and tooling and can’t do that now.

  • Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? Even if applying deprecation policies, they may still surprise some users.

Monitoring Requirements

This section must be completed when targeting beta graduation to a release.

  • How can an operator determine if the feature is in use by workloads? Ideally, this should be a metric. Operations against the Kubernetes API (e.g., checking if there are objects with field X set) may be a last resort. Avoid logs or events for this purpose.

  • What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

    • Metrics
      • Metric name: rest_client_exec_plugin_with_proxy_call_total
      • Components exposing the metric: client-go (kube-controller-manager, kube-scheduler, kubelet, kube-apiserver)
    • Other (treat as last resort)
      • Details:

  • What are the reasonable SLOs (Service Level Objectives) for the above SLIs? At a high level, this usually will be in the form of “high percentile of SLI per day <= X”. It’s impossible to provide comprehensive guidance, but at the very high level (needs more precise definitions) those may be things like:

    • per-day percentage of API calls finishing with 5XX errors <= 1%
    • 99% percentile over day of absolute value from (job creation time minus expected job creation time) for cron job <= 10%
    • 99,9% of /health requests per day finish with 200 code

  • Are there any missing metrics that would be useful to have to improve observability of this feature? Describe the metrics themselves and the reasons why they weren’t added (e.g., cost, implementation difficulties, etc.).

Dependencies

This section must be completed when targeting beta graduation to a release.

  • Does this feature depend on any specific services running in the cluster? Think about both cluster-level services (e.g. metrics-server) as well as node-level agents (e.g. specific version of CRI). Focus on external or optional services that are needed. For example, if this feature depends on a cloud provider API, or upon an external software-defined storage or network control plane.

    For each of these, fill in the following—thinking about running existing user workloads and creating new ones, as well as about cluster-level services (e.g. DNS):

    • [Dependency name]
      • Usage description:
        • Impact of its outage on the feature:
        • Impact of its degraded performance or high-error rates on the feature:

Scalability

For alpha, this section is encouraged: reviewers should consider these questions and attempt to answer them.

For beta, this section is required: reviewers must answer these questions.

For GA, this section is required: approvers should be able to confirm the previous answers based on experience in the field.

  • Will enabling / using this feature result in any new API calls? Describe them, providing:

    • API call type (e.g. PATCH pods)
    • estimated throughput
    • originating component(s) (e.g. Kubelet, Feature-X-controller) focusing mostly on:
    • components listing and/or watching resources they didn’t before
    • API calls that may be triggered by changes of some Kubernetes resources (e.g. update of object X triggers new updates of object Y)
    • periodic API calls to reconcile state (e.g. periodic fetching state, heartbeats, leader election, etc.)

  • Will enabling / using this feature result in introducing new API types? Describe them, providing:

    • API type
    • Supported number of objects per cluster
    • Supported number of objects per namespace (for namespace-scoped objects)

  • Will enabling / using this feature result in any new calls to the cloud provider?

  • Will enabling / using this feature result in increasing size or count of the existing API objects? Describe them, providing:

    • API type(s):
    • Estimated increase in size: (e.g., new annotation of size 32B)
    • Estimated amount of new objects: (e.g., new Object X for every existing Pod)

  • Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs ? Think about adding additional work or introducing new steps in between (e.g. need to do X to start a container), etc. Please describe the details.

  • Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components? Things to keep in mind include: additional in-memory state, additional non-trivial computations, excessive access to disks (including increased log volume), significant amount of data sent and/or received over network, etc. This through this both in small and large cases, again with respect to the supported limits .

Troubleshooting

The Troubleshooting section currently serves the Playbook role. We may consider splitting it into a dedicated Playbook document (potentially with some monitoring details). For now, we leave it here.

This section must be completed when targeting beta graduation to a release.

  • How does this feature react if the API server and/or etcd is unavailable?

The proxy’s behavior when the API server is unavailable will be up to the proxy implementor.

  • What are other known failure modes? For each of them, fill in the following information by copying the below template:

    • [Failure mode brief description]
      • Detection: How can it be detected via metrics? Stated another way: how can an operator troubleshoot without logging into a master or worker node?
      • Mitigations: What can be done to stop the bleeding, especially for already running user workloads?
      • Diagnostics: What are the useful log messages and their required logging levels that could help debug the issue? Not required until feature graduated to beta.
      • Testing: Are there any tests for failure mode? If not, describe why.

  • What steps should be taken if SLOs are not being met to determine the problem?

Implementation History

  • 2021-05-07 KEP PR created, first reviews received.
  • 2022-01-27 PR updated, more feedback received.
  • 2022-06-22 PR updated to use ExecCredentialSpec/ExecCredentialStatus.

Drawbacks

It introduces a new feature into client libraries which comes with a maintenance burden. It also introduces a new extension point which might make client installs which choose to use this feature slightly more complicated, in that the user must install an additional binary.

Alternatives

Alternative Proposal: Request Replacement

In this proposal, the the proxy would be replaced by a localhost/UDS server. The client would package the HTTP request into the body of a GRPC or HTTP request and send to the server, expecting a response back immediately with the modified request as a payload. The client would then extract the request, replace the original request with the modified request, and send it as it normally would. This would be done in the client transport. Otherwise, the proposal is largely the same–the client uses configuration in the kubeconfig to start the signing process, and the client receives a unix domain socket or localhost address from the server process on exec. The server process listens at the unix domain socket address, and the client connects to it. The client attempts to exec the server binary if it is unable to connect using the provided address.

This proposal was not preferred to the chosen proposal because, after discussions at SIG-Auth, the overall feeling was that defining the request response protocol between the client and the signing process would be more cumbersome, and be less suited to a generic solution that accommodated any proxy related use case. It is also not possible to implement TLS offload with this approach.