KEP-4412: Projected service account tokens for Kubelet image credential providers

• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • User Stories (Optional)
    • Story 1
    • Story 2
    • Story 3
  • Notes/Constraints/Caveats (Optional)
  • Risks and Mitigations
• Design Details
  • Credential Provider Configuration
    • Kubernetes API Server (KAS) Changes
  • Credential Provider Request API
  • Caching Credentials
    • Existing behavior
    • New behavior when the serviceAccountTokenAudience field is set
  • How will this work with the Ensure secret pull images KEP?
  • Test Plan
    • Prerequisite testing updates
    • Unit tests
    • Integration tests
    • e2e tests
  • Graduation Criteria
    • Alpha
    • Beta
    • GA
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
• Production Readiness Review Questionnaire
  • Feature Enablement and Rollback
  • Rollout, Upgrade and Rollback Planning
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Possible future work
• Alternatives
• Infrastructure Needed (Optional)

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  • e2e Tests for all Beta API Operations (endpoints)
  • (R) Ensure GA e2e tests meet requirements for Conformance Tests
  • (R) Minimum Two Week Window for GA e2e tests to prove flake free
• (R) Graduation criteria is in place
  • (R) all GA Endpoints must be hit by Conformance Tests
• (R) Production readiness review completed
• (R) Production readiness review approved
• “Implementation History” section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

As Kubernetes has matured, we have taken strides towards limiting the need for long lived credentials that are stored in the Kubernetes API. The canonical example of this change is the move of Kubernetes Service Account (KSA) tokens from ones that were long lived, persisted in the Kubernetes API, and never rotated to tokens that are ephemeral, short lived, and automatically rotated. By hardening these tokens and stabilizing their schema to match the semantics of OIDC ID tokens, we have enabled external consumers to validate these tokens for use cases beyond Kubernetes. For example, it is possible to use a KSA token that is bound to a specific pod to fetch secrets from an external cloud provider vault without the need for any long lived secrets in the cluster.

This KEP aims to make it possible for a similar ‘secret-less’ flow to be created for image pulls, which happen before the pod is running. Today, admins are limited to image pull secrets that are stored directly in the Kubernetes API and thus are long lived and hard to rotate, or secrets that are managed at the Kubelet level via a Kubelet credential provider (meaning that any pod running on that node can access those images). A pod should instead be able to use its own identity to pull images, i.e. we should enable image pull authorization to be tied to a particular workload while avoiding the need for long lived, persisted secrets.

Motivation

By moving to image pulls that are based on KSA tokens, we seek to:

• Reduce secret management overhead for developers and admins
• Reduce security risks associated with long lived secrets

Goals

• Allow workloads to pull images based on their own runtime identity without long lived / persisted secrets
• Avoid needing a kubelet/node based identity to pull images

Non-Goals

Proposal

We will expand the on-disk kubelet credential provider configuration to allow an optional tokenAttributes field to be configured. When this field is not set, no KSA token will be sent to the plugin. When it is set, the Kubelet will provision a token with the given audience bound to the current pod and its service account. This KSA token along with required annotations on the KSA defined in configuration will be sent to the credential provider plugin via its standard input (along with the image information that is already sent today). The KSA annotations to be sent will be configurable in the kubelet credential provider configuration.

User Stories (Optional)

Story 1

As a Kubernetes administrator, I need to securely manage image pulls for my multi-tenant cluster so that I can prevent unauthorized access to private container images and reduce the security risks associated with static secrets.

Story 2

As a developer deploying applications in Kubernetes, I want my pods to use dynamic service account tokens to pull images securely, ensuring that each pod only accesses images that it is explicitly permitted to use, thereby maintaining proper segmentation of access between services.

Story 3

As a security engineer, I need to eliminate the use of static secrets for image pulls and leverage workload identity to enhance security through ephemeral, tightly scoped tokens. This change reduces the attack surface and the risk of credential leaks, aligning with best practices for security.

Notes/Constraints/Caveats (Optional)

Risks and Mitigations

To keep this KEP small both in terms of implementation and scope, the design is written to meet the following criteria:

• No changes to any Kubernetes REST APIs
• No changes to the CRI API
• No changes to how the Kubelet interacts with the CRI API
• No changes to any registry
• Minimal changes to Kubelet
• Minimal changes to Kubelet credential providers
• Minimal changes to Kube API Server

Design Details

Credential Provider Configuration

Adding a new field TokenAttributes to the CredentialProvider struct in the kubelet configuration.

1. ServiceAccountTokenAudience is the intended audience for the credentials. If set, the kubelet will generate a service account token for the audience and pass it to the plugin.
   1. Only a single audience can be configured.
   2. The credential provider can be set up multiple times for each audience.
      1. The name in the credential provider config must match the name of the provider executable as seen by the kubelet. To configure multiple instances of the same provider with different audiences, the name of the provider executable must be different and this can be done by creating a symlink to the same executable with a different name.
   3. By setting this field, the credential provider is opting into using service account tokens for image pull.
2. RequireServiceAccount indicates whether the plugin requires the pod to have a service account. If set to true, kubelet will only invoke the plugin if the pod has a service account. If set to false, kubelet will invoke the plugin even if the pod does not have a service account and will not include a token in the CredentialProviderRequest in that scenario. This is useful for plugins that are used to pull images for pods without service accounts (e.g., static pods).
3. RequiredServiceAccountAnnotationKeys is the list of annotation keys that the plugin is interested in and that are required to be present in the service account. The keys defined in this list will be extracted from the corresponding service account and passed to the plugin as part of CredentialProviderRequest. If any of the keys defined in this list are not present in the service account, kubelet will not invoke the plugin and will return an error. This field is optional and may be empty. Plugins may use this field to extract additional information required to fetch credentials or allow workloads to opt in to using service account tokens for image pull. If non-empty, RequireServiceAccount must be set to true.
4. OptionalServiceAccountAnnotationKeys is the list of annotation keys that the plugin is interested in and that are optional to be present in the service account. The keys defined in this list will be extracted from the corresponding service account and passed to the plugin as part of CredentialProviderRequest. The plugin is responsible for validating the existence of annotations and their values. This field is optional and may be empty. Plugins may use this field to extract additional information required to fetch credentials.

Regarding required and optional service account annotations keys, there are existing integrations like Azure and GCP that use the extra metadata to link the KSA name to a cloud identity, and this field will allow them to continue to do so. None of the existing integrations use this metadata as any form of security decision, it is simply to aid in doing token exchanges with the KSA token. 1. Service account annotations are significantly more stable (and map naturally to the service account being the partitioning dimension). 2. These fields makes the provider opt into getting the annotations it wants, avoids sending arbitrary annotation contents down to the plugin (including stuff like client-side apply annotations) and shrinks the set of annotations that could invalidate the cache. 3. These fields can’t be set without setting the ServiceAccountTokenAudience field.

The configuration will not support a plugin to function in 2 modes simultaneously (one with service account tokens and one without). If a plugin needs to function in both modes, it will need to be configured twice with different names. This can be done to facilitate a gradual migration to the new mode for the plugin on a per-image or per-registry basis.

--- a/pkg/kubelet/apis/config/types.go
+++ b/pkg/kubelet/apis/config/types.go
@@ -653,10 +653,19 @@ type CredentialProvider struct {
        // +optional
        Env []ExecEnvVar
 
+       // tokenAttributes is the configuration for the service account token that will be passed to the plugin.
+       // The credential provider opts in to using service account tokens for image pull by setting this field.
+       // When this field is set, kubelet will generate a service account token bound to the pod for which the
+       // image is being pulled and pass to the plugin as part of CredentialProviderRequest along with other
+       // attributes required by the plugin.
+       //
+       // The service account metadata and token attributes will be used as a dimension to cache
+       // the credentials in kubelet. The cache key is generated by combining the service account metadata
+       // (namespace, name, UID, and annotations key+value for the keys defined in
+       // serviceAccountTokenAttribute.requiredServiceAccountAnnotationKeys and serviceAccountTokenAttribute.optionalServiceAccountAnnotationKeys).
+       // The pod metadata (namespace, name, UID) that are in the service account token are not used as a dimension
+       // to cache the credentials in kubelet. This means workloads that are using the same service account
+       // could end up using the same credentials for image pull. For plugins that don't want this behavior, or
+       // plugins that operate in pass-through mode; i.e., they return the service account token as-is, they
+       // can set the credentialProviderResponse.cacheDuration to 0. This will disable the caching of
+       // credentials in kubelet and the plugin will be invoked for every image pull. This does result in
+       // token generation overhead for every image pull, but it is the only way to ensure that the
+       // credentials are not shared across pods (even if they are using the same service account).
+       // +optional
+       TokenAttributes *ServiceAccountTokenAttributes
+}
+
+    const (
+        // TokenServiceAccountTokenCacheType means the kubelet will cache returned credentials
+        // on a per-token basis. This should be set if the returned credential's lifetime is limited
+        // to the input service account token's lifetime.
+        // For example, this must be used when returning the input service account token directly as a pull credential.
+        TokenServiceAccountTokenCacheType ServiceAccountTokenCacheType = "Token"
+        // ServiceAccountServiceAccountTokenCacheType means the kubelet will cache returned credentials
+        // on a per-serviceaccount basis. This should be set if the plugin's credential retrieval logic
+        // depends only on the service account and not on pod-specific claims.
+        // Use this when the returned credential is valid for all pods using the same service account.
+        ServiceAccountServiceAccountTokenCacheType ServiceAccountTokenCacheType = "ServiceAccount"
+    )
+
+    // ServiceAccountTokenAttributes is the configuration for the service account token that will be passed to the plugin.
+    type ServiceAccountTokenAttributes struct {
+       // serviceAccountTokenAudience is the intended audience for the projected service account token.
+       // +required
+       ServiceAccountTokenAudience string
+       // cacheType indicates the type of cache key use for caching the credentials returned by the plugin
+       // when the service account token is used.
+       // The most conservative option is to set this to "Token", which means the kubelet will cache returned credentials
+       // on a per-token basis. This should be set if the returned credential's lifetime is limited to the service account
+       // token's lifetime.
+       // If the plugin's credential retrieval logic depends only on the service account and not on pod-specific claims,
+       // then the plugin can set this to "ServiceAccount". In this case, the kubelet will cache returned credentials
+       // on a per-serviceaccount basis. Use this when the returned credential is valid for all pods using the same service account.
+       // +required
+       CacheType ServiceAccountTokenCacheType
+
+       // requireServiceAccount indicates whether the plugin requires the pod to have a service account.
+       // If set to true, kubelet will only invoke the plugin if the pod has a service account.
+       // If set to false, kubelet will invoke the plugin even if the pod does not have a service account
+       // and will not include a token in the CredentialProviderRequest in that scenario. This is useful for plugins that
+       // are used to pull images for pods without service accounts (e.g., static pods).
+       // +required
+       RequireServiceAccount *bool
+
+       // requiredServiceAccountAnnotationKeys is the list of annotation keys that the plugin is interested in
+       // and that are required to be present in the service account.
+       // The keys defined in this list will be extracted from the corresponding service account and passed
+       // to the plugin as part of the CredentialProviderRequest. If any of the keys defined in this list
+       // are not present in the service account, kubelet will not invoke the plugin and will return an error.
+       // This field is optional and may be empty. Plugins may use this field to extract
+       // additional information required to fetch credentials or allow workloads to opt in to
+       // using service account tokens for image pull.
+       // If non-empty, requireServiceAccount must be set to true.
+       // +optional
+       RequiredServiceAccountAnnotationKeys []string
+
+       // optionalServiceAccountAnnotationKeys is the list of annotation keys that the plugin is interested in
+       // and that are optional to be present in the service account.
+       // The keys defined in this list will be extracted from the corresponding service account and passed
+       // to the plugin as part of the CredentialProviderRequest. The plugin is responsible for validating
+       // the existence of annotations and their values.
+       // This field is optional and may be empty. Plugins may use this field to extract
+       // additional information required to fetch credentials.
+       // +optional
+       OptionalServiceAccountAnnotationKeys []string
 }

Example credential provider configuration:

apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
  - name: acr-credential-provider
    matchImages:
      - "*.registry.io/*"
    defaultCacheDuration: "10m"
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    tokenAttributes:
      serviceAccountTokenAudience: my-audience
      cacheType: Token
      # requireServiceAccount is set to true, so the plugin will only be invoked if the pod has a service account
      requireServiceAccount: true
      # requiredServiceAccountAnnotationKeys is the list of annotations that the plugin is interested in
      # the annotation key and corresponding value in the service account will be passed to the plugin. If
      # any of the keys defined in this list are not present in the service account, kubelet will not invoke
      # the plugin and will return an error.
      requiredServiceAccountAnnotationKeys:
      - domain.io/identity-id
      - domain.io/identity-type
      # optionalServiceAccountAnnotationKeys is the list of annotations that the plugin is interested in
      # the annotation key and corresponding value in the service account will be passed to the plugin. The
      # plugin is responsible for validating the existence of annotations and their values.
      optionalServiceAccountAnnotationKeys:
      - domain.io/some-optional-annotation
      - domain.io/annotation-that-does-not-exist

Kubernetes API Server (KAS) Changes

Today we only verify the SA in KAS and allow the kubelet to generate a token for any audience. We want to start verifying the audience as well, so we need to be explicit here about what audiences are allowed. The other sources of audiences for the SA token can be observed via the Kubernetes API but this one can’t, so we need a dynamic configuration for it.

KAS will be updated to allow for dynamically configuring service accounts and audiences for which the kubelet is allowed to generate a service account token for as part of the node audience restriction feature. This will be done by a synthetic authz check. The resulting subject access review (SAR) in KAS will look like this:

Verb: request-serviceaccounts-token-audience
Resource: $audience
APIGroup: $request.apiGroup (which is the service account API group)
Name: $request.name (which is the service account name)
Namespace: $request.namespace (which is the service account namespace)

The node audience restriction will be enforced by the NodeRestriction admission controller and kubelet will only be allowed to generate a service account token for the audience configured in the credential provider configuration if it is present in the list of audiences configured in KAS.

• Allow any audience for any service account

rules:
- verbs: ["request-serviceaccounts-token-audience"]
  apiGroups: [""]
  # wildcard for audiences
  resources: ["*"]
  # unrestricted resourceNames allows all service account names

• Allow any audience for a specific service account mysa

rules:
- verbs: ["request-serviceaccounts-token-audience"]
  apiGroups: [""]
  # wildcard for audiences
  resources: ["*"]
  resourceNames: ["mysa"]

• Allow a specific audience myaudience for any service account

rules:
- verbs: ["request-serviceaccounts-token-audience"]
  apiGroups: [""]
  resources: ["myaudience"]
  # unrestricted resourceNames allows all service account names

• Allow a specific audience myaudience for a specific service account mysa

rules:
- verbs: ["request-serviceaccounts-token-audience"]
  apiGroups: [""]
  resources: ["myaudience"]
  resourceNames: ["mysa"]

• Allow API server audience for all service accounts

rules:
- verbs: ["request-serviceaccounts-token-audience"]
  apiGroups: [""]
  resources: [""]
  # unrestricted resourceNames allows all service account names

Credential Provider Request API

Add ServiceAccountToken and ServiceAccountAnnotations fields to CredentialProviderRequest.

1. ServiceAccountToken is the service account token bound to the pod for which the image is being pulled. If the ServiceAccountTokenAudience field is configured in the kubelet’s credential provider configuration, the token will be sent to the plugin.
2. ServiceAccountAnnotations is a map of annotations on the KSA for which the image is being pulled. Only annotations defined in the RequiredServiceAccountAnnotationKeys and OptionalServiceAccountAnnotationKeys field in the credential provider configuration will be passed to the plugin. If the RequiredServiceAccountAnnotationKeys and OptionalServiceAccountAnnotationKeys fields are not set in the configuration, this field will be empty.

--- a/staging/src/k8s.io/kubelet/pkg/apis/credentialprovider/types.go
+++ b/staging/src/k8s.io/kubelet/pkg/apis/credentialprovider/types.go
@@ -32,6 +32,16 @@ type CredentialProviderRequest struct {
        // credential provider plugin request. Plugins may optionally parse the image
        // to extract any information required to fetch credentials.
        Image string
+
+       // serviceAccountToken is the service account token bound to the pod for which
+       // the image is being pulled. This token is only sent to the plugin if the
+       // tokenAttributes.serviceAccountTokenAudience field is configured in the kubelet's credential provider configuration.
+       ServiceAccountToken string
+
+       // serviceAccountAnnotations is a map of annotations on the service account bound to the
+       // pod for which the image is being pulled. The list of annotations in the service account
+       // that need to be passed to the plugin is configured in the kubelet's credential provider
+       // configuration.
+       ServiceAccountAnnotations map[string]string
 }

Example Kubernetes Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: my-namespace
  annotations:
    domain.io/identity-id: "12345"
    domain.io/identity-type: "user"
    # this annotation will not be passed to the plugin because it's not in tokenAttributes.requiredServiceAccountAnnotationKeys or
    # tokenAttributes.optionalServiceAccountAnnotationKeys field in the credential provider configuration
    domain.io/annotation-that-will-not-be-passed: "value"

Example credential provider request:

{
  "image": "my-image",
  "serviceAccountToken": "<service-account-token>",
  "serviceAccountAnnotations": {
    "domain.io/identity-id": "12345",
    "domain.io/identity-type": "user"
    // In the tokenAttributes.optionalServiceAccountAnnotationKeys field in the credential provider configuration above we have configured
    // domain.io/annotation-that-does-not-exist which is not present in the service account annotations. Because of this, this annotation
    // will not be passed to the plugin. The plugin is responsible for validating the existence of annotations and their values that's 
    // defined in the optionalServiceAccountAnnotationKeys field in the credential provider configuration.
  }
}

Caching Credentials

The cache key generation will be updated to support new caching modes while preserving the existing behavior when the audience field is not set.

Existing behavior

If the serviceAccountTokenAudience field is not set in the provider configuration for the plugin, the key generation should fall back to the current behavior: using only the image registry URL or other existing identifiers.

New behavior when the serviceAccountTokenAudience field is set

Based on the PluginCacheKeyType definitions (Image, Registry, Global) and cacheType field in the credential provider configuration, the cache key will be generated as follows:

1. TokenServiceAccountTokenCacheType (Token)
   • The cache key will be generated based on (Image, Registry, Global) plugin cache key type and the hash of the service account token.
2. ServiceAccountServiceAccountTokenCacheType (ServiceAccount)
   • The cache key will be generated based on (Image, Registry, Global) plugin cache key type, service account metadata (namespace, name, UID), and the service account annotations that are passed to the plugin in the CredentialProviderRequest.ServiceAccountAnnotations field.

How will this work with the Ensure secret pull images KEP?

2535-ensure-secret-pull-images is a KEP that aims to give admin the ability to ensure pods that use an image are authorized to access the image. As of the beta implementation in kubernetes/kubernetes#132771 , this feature now works correctly with service account token-based credential providers.

The implementation tracks service account coordinates (namespace, name, and UID) when a credential provider that uses service account tokens provides registry credentials for an image. This enables the following behavior:

1. Different KSA for same image: Pods using different service accounts will trigger a fresh image pull from the registry since they have different service account coordinates.
2. Same KSA for same image: Pods using the exact same service account (matching namespace, name, and UID) will be allowed to reuse the previously pulled image without triggering a new registry pull.
3. Service account lifecycle management: If administrators want to revoke access to previously pulled images for a service account, they can delete and recreate the service account. This changes the UID, which invalidates any cached image credentials associated with the old service account coordinates.

The implementation only tracks namespace, name, and UID (not annotations or audience) for the ensure secret pull images cache key, as discussed in this PR comment . This design choice aligns with the intent of the ensure secret pull images feature: if an image was pulled to a node via a service-account-based credential, all future pods that use that exact service account should be able to access the already-pulled image, even if the current state of the service account or credential provider config had since changed in a way that meant a pull re-attempt would fail.

This approach is consistent with how pull secrets are handled - the system remembers which secrets worked to pull the image in the past, and any pod allowed to use one of those secrets gets access to the already pulled image, even if the content of the secret had since changed.

This integration ensures that image access authorization is properly tied to service account identity while maintaining the security boundaries that the Ensure secret pull images KEP intended to enforce.

Test Plan

• I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

N/A

Unit tests

The unit test coverage is in k8s.io/kubernetes/pkg/credentialprovider package:

• k8s.io/kubernetes/pkg/credentialprovider: 09/11/2024 - 53.7

Unit tests will be added to cover:

1. Config validation
2. Credential provider logic when token attributes are not set
3. Credential provider logic when token attributes are set
4. Cache key generation

Integration tests

This kubelet feature is fully tested with unit and e2e tests.

For the node audience restriction changes in KAS, integration tests were added as part of the implementation in v1.32 release .

• test/integration/auth/node_test.go
• triage history

e2e tests

There is an existing e2e test for kubelet credential providers using gcp credential provider.

• test/e2e_node/image_credential_provider.go: https://testgrid.k8s.io/sig-node-kubelet#kubelet-credential-provider

As part of alpha implementation, the e2e test has been updated to cover the new credential provider configuration and the new behavior of the kubelet when the TokenAttributes field is set.

We created a symlink to the existing gcp credential provider executable with a different name to use for testing service account token for credential provider. The credential provider has been updated to validate the following when plugin is run in service account token mode:

1. Check the required annotations are sent as part of the CredentialProviderRequest.ServiceAccountAnnotations field.
2. Check the service account token is sent as part of the CredentialProviderRequest.ServiceAccountToken field.
3. Extract the claims from the service account token and validate the audience claim matches the ServiceAccountTokenAudience field in the kubelet’s credential provider configuration.

Graduation Criteria

Alpha

• Feature implemented behind a feature flag
• Unit tests for config validation
• Unit tests for current credential provider logic unchanged when token attributes are not set
• Unit tests for credential provider logic when token attributes are set
• Initial e2e tests completed and enabled
• ServiceAccountNodeAudienceRestriction feature gate implemented in KAS as a beta feature
  • Audience validation is enabled by default for service account tokens requested by the kubelet

Beta

• Make the feature compatible with the Ensure secret pull images KEP .
• ServiceAccountNodeAudienceRestriction feature gate is beta in KAS and enabled by default. This feature needs to be beta/enabled by default at least one release before this KEP goes to beta. This is critical to support downgrade use cases.
• Caching KSA tokens per pod-sa to prevent generating tokens during hot loop/multiple containers with images.
• Some indication of whether the credentials are SA or SA+pod-scoped
  • whether that’s indicated in the config or in the plugin-returned content, and what the default is if unspecified (defaulting to pod is less performance, defaulting to SA risks incorrect cross-pod caching)

GA

• Gather feedback
  • Cloudsmith has developed a new credential provider plugin that authenticates with cloudsmith registries using service account tokens. Blog , Demo video , Feedback on slack .

Upgrade / Downgrade Strategy

This feature is feature gated so explicit opt-in is required on upgrade and explicit opt-out is required on downgrade.

Version Skew Strategy

To migrate to the new approach,

1. KAS will need to be updated to the new version to configure the audiences for which the kubelet is allowed to generate service account tokens for image pulls via ClusterRole or Role with the request-serviceaccounts-token-audience verb.
2. Kubelet on the node needs to be updated to enable the feature flag and configure the credential provider to use service account tokens for image pull via the TokenAttributes field in the kubelet credential provider configuration.
   1. The credential provider plugin will need to be updated to use the service account token for the audience configured in the kubelet credential provider configuration.

Migration of the workloads to the new approach can be done per image or per registry basis by configuring the credential provider multiple times with different names for the same plugin (w or w/o service account tokens, different audiences).

When things can fail:

If the kubelet is updated to enable the feature flag and the credential provider is configured with the TokenAttributes field set, but the KAS is not updated to the version that detects dynamic configuration of allowed audiences via ClusterRole or Role with the request-serviceaccounts-token-audience verb, to allow the kubelet to generate service account tokens for the audience configured in the kubelet credential provider configuration, the image pull will fail. The old KAS will reject the token request from the kubelet.

Today we don’t do any validation on the audience value that the kubelet requested -> we only check that the SA is in use on some pod scheduled to the kubelet. As part of this KEP, we’re introducing a new feature gate ServiceAccountNodeAudienceRestriction in KAS to validate the audience value that the kubelet requests is either part of any API spec or is part of the audiences configured in the ClusterRole or Role with the request-serviceaccounts-token-audience verb. This feature gate will be beta/enabled by default at least one release before this KEP goes to beta.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

• Feature gate (also fill in values in kep.yaml)

  • Feature gate name: KubeletServiceAccountTokenForCredentialProviders
  • Components depending on the feature gate: kubelet

• Feature gate (also fill in values in kep.yaml)

  • Feature gate name: ServiceAccountNodeAudienceRestriction
  • Components depending on the feature gate: kube-apiserver

The purpose of the two feature gates is different, which is why they weren’t named similarly.

The KubeletServiceAccountTokenForCredentialProviders feature gate is used to enable the kubelet to use service account tokens for image pull in the kubelet credential provider.

The ServiceAccountNodeAudienceRestriction feature gate is used to enable the kube-apiserver to validate the audience of the service account token requested by the kubelet. The feature gate in the Kubernetes API Server (KAS) was introduced to strictly enforce which audiences the kubelet can request tokens for. Before this change, the kubelet could request a token with any audience. With the feature gate enabled, the API server starts validating the requested audience.

The KAS feature gate doesn’t need to be enabled for the kubelet feature to work. It graduated to beta in v1.32 and is enabled by default. The two are unrelated in functionality, but the KAS gate was necessary to ensure strict enforcement of the allowed audiences the kubelet can request tokens for.

If the KAS feature gate is not enabled, there will be no validation of the audience requested by the kubelet, and the kubelet will be able to request tokens for any audience. This is not recommended.

Does enabling the feature change any default behavior?

No. After enabling the feature flag in the kubelet, the credential provider still needs to opt into using service account tokens for image pull by setting the TokenAttributes field in the kubelet credential provider configuration.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes. The feature flag needs to be disabled and the credential provider configuration for the provider that is using service account tokens for image pull needs to be updated to not use the TokenAttributes field or the provider needs to be removed.

Kubelet needs to be restarted to invalidate the in-memory cache after removing the provider or updating the configuration.

Steps to disable the feature:

1. Update the kubelet credential provider configuration to remove providers that are using service account tokens for image pull.
2. Disable the feature flag in the kubelet.
3. Restart the kubelet.

These steps need to be performed on all nodes in the cluster. After restarting the kubelet on all nodes, remove the allowed audiences for which the kubelet is allowed to generate service account tokens for image pulls in KAS by removing the previous ClusterRole or Role with the request-serviceaccounts-token-audience verb, along with the corresponding ClusterRoleBinding or RoleBinding that binds the role to the kubelet.

What happens if we reenable the feature if it was previously rolled back?

No impact. The credential provider will continue to use the configuration set in the kubelet credential provider configuration.

Are there any tests for feature enablement/disablement?

Feature enablement/disablement unit/integration tests will be added.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

Feature is enabled but exec plugin does not properly fetch and return credentials to the kubelet. Impact is that kubelet cannot authenticate and pull credentials from those registries.

What specific metrics should inform a rollback?

High error rates from kubelet_credential_provider_plugin_error and long durations from kubelet_credential_provider_plugin_duration.

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

No, upgrade->downgrade->upgrade were not tested. Manual validation will be done prior to promoting this feature to beta in v1.34.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No.

Monitoring Requirements

New metrics:

• kubelet_credential_provider_config_info indicates the hash of the kubelet credential provider configuration file. This metric can be used by operators to determine if the kubelet credential provider configuration has changed.

How can an operator determine if the feature is in use by workloads?

Operators can use kubelet_credential_provider_config_info metric to determine if the kubelet credential provider configuration has changed. If the hash of the configuration file changes, it indicates that the kubelet credential provider configuration has been updated.

How can someone using this feature know that it is working for their instance?

Users can observe events for successful image pulls that use the service account token for image pull.

• Events
  • Event Reason: " Successfully pulled image “xxx” in 11.877s (11.877s including waiting). Image size: xxx bytes."

For registries or images configured to be pulled using a credential provider with a service account, a successful image pull seems to be the only way to confirm that it’s working. If the credential provider is misbehaving, the kubelet will not be able to authenticate to the registry and pull images, which will result in image pull errors.

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

On failure to fetch credentials from an exec plugin, the kubelet will retry after some period and invoke the plugin again. The kubelet will retry whenever it attempts to pull an image, but until then, kubelet will not be able to authenticate to the registry and pull images. The SLO for successfully invoking exec plugins should be based on the SLO for successfully pulling images for the container registry in question.

The SLOs defined in Pod startup latency SLI/SLO details don’t apply to this feature because image pull SLI is explicitly excluded from the pod startup latency SLI/SLO. However, if the kubelet is unable to pull images due to misconfiguration of the credential provider plugin, it will result in pod startup failures.

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

• Metrics
  • Metric name: kubelet_credential_provider_plugin_errors indicates number of errors from the credential provider plugin
  • Metric name: kubelet_credential_provider_plugin_duration indicates the duration of the execution in seconds for credential provider plugin
  • Components exposing the metric: kubelet

The metrics above will indicate if the credential provider plugin configured to use service account tokens for image pull is functioning properly.

Are there any missing metrics that would be useful to have to improve observability of this feature?

TBA.

Dependencies

Does this feature depend on any specific services running in the cluster?

This feature depends on the existence of a credential provider plugin binary on the host and a configuration file for the plugin to be read by the kubelet.

Scalability

Will enabling / using this feature result in any new API calls?

Yes. Kubelet will request a KSA token from KAS for pod + KSA for every provider (audience) that is configured to use service account tokens for image pull. The load is small because the token generation happens only once during pod startup.

Will enabling / using this feature result in introducing new API types?

No.

Will enabling / using this feature result in any new calls to the cloud provider?

This depends on the credential provider plugin implementation on how it fetches the credentials using the service account token. The plugin will now be called for every unique KSA + (image/registry) combination that is being pulled if not already cached unlike today where the cache is based on the image/registry URL. This could result in more calls to the cloud provider from the plugin to exchange the KSA token for the credentials.

Will enabling / using this feature result in increasing size or count of the existing API objects?

No.

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

SLIs/SLOs for image pull times might be impacted with the credential plugin using KSA token for fetching credentials. The credentials are granular in scope (per service account) like image pull secrets, but the plugin must dynamically fetch the credentials using the KSA token and these credentials could have shorter TTLs resulting in more frequent fetching of credentials.

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

As part of the ServiceAccountNodeAudienceRestriction feature, KAS will need to watch PersistentVolumeClaims, PersistentVolumes and CSIDrivers to determine the audiences that the kubelet is allowed to generate service account tokens for. These new informers (which are feature gated) will result in additional resource usage in the KAS.

• Node authorizer is already watching persistent volumes via informers today.
• CSIDriver objects are expected to be ~few and ~slow-moving, so the impact is expected to be minimal.
• PersistentVolumeClaims are expected to be more numerous and more dynamic, so there could be more impact here.

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

No.

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

If the API server is unavailable, kubelet will not be able to fetch service account tokens for image pull. The kubelet will retry fetching the token after some period, but until then, kubelet will not be able to authenticate to the registry and pull images that rely on the credential provider plugin using service account tokens for image pull.

What are other known failure modes?

What steps should be taken if SLOs are not being met to determine the problem?

• check logs of kubelet
• check service availability of container registries used by the cluster

Implementation History

1.33: Alpha release 1.34: Beta release

Drawbacks

Possible future work

Passthrough mode where KSA tokens are sent directly to the CRI. This would allow the CRI to use the KSA token to pull images directly from the registry.

For example, openshift uses imagePullSecrets with the PSAT to pull images directly from the registry. If we support passthrough mode with this feature, kubelet can pass the PSAT in the docker config credentials (username: “”, password: “”) to the CRI. The CRI can then use the PSAT to pull images directly from the registry. The kubelet can determine which registries it’s ok to pass the PSAT to the CRI.

Alternatives

always passthrough SA token to registry and rely on registry proxies to do sts

Infrastructure Needed (Optional)

N/A