KEP-5958: Client Opt-out for managedFields in API Response

• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • User Stories
    • Story 1
    • Story 2
  • Risks and Mitigations
• Design Details
  • Accept Parameter
  • Implementation Details
    • In-tree Controller Migration
    • Client Configuration
  • Test Plan
    • Prerequisite testing updates
    • Unit Tests
    • Integration Tests
    • e2e Tests
  • Graduation Criteria
    • Alpha
    • Beta
    • GA
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
    • Client Support and Version Skew
• Production Readiness Questionnaire
  • Feature Enablement and Rollback
  • Rollout, Upgrade and Rollback Planning
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Alternatives
• Infrastructure Needed (Optional)

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  • e2e Tests for all Beta API Operations (endpoints)
  • (R) Ensure GA e2e tests meet requirements for Conformance Tests
  • (R) Minimum Two Week Window for GA e2e tests to prove flake free
• (R) Graduation criteria is in place
  • (R) all GA Endpoints must be hit by Conformance Tests within one minor version of promotion to GA
• (R) Production readiness review completed
• (R) Production readiness review approved
• “Implementation History” section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

This enhancement allows Kubernetes clients to opt-out of receiving metadata.managedFields in API responses (GET, LIST, WATCH, PUT, POST, and PATCH) via an HTTP Accept parameter. This reduces network bandwidth, API Server CPU serialization costs, and client-side memory allocations and Garbage Collection overhead.

Motivation

metadata.managedFields is used by the API server for Server-Side Apply (SSA) conflict resolution. However, the vast majority of Kubernetes clients do not actively process this data. Many core components, such as kube-controller-manager and kube-scheduler, currently use client-side transforms to drop managed fields to save memory.

As documented in the Server-Side Apply KEP , managedFields can represent up to 60% of the total size of an object. The actual overhead varies with the number of field managers and the fields they touch — objects managed by multiple controllers, webhooks, and users in production clusters carry proportionally larger managedFields. Reducing this overhead is important for supporting larger resource sizes as a scalability dimension .

Relying on client-side transforms still incurs significant system-wide costs:

• API Server CPU: The API server still performs expensive serialization of managedFields, even when the client will immediately discard it.
• Network Overhead: managedFields payloads significantly increase transfer time during LIST and WATCH operations. This contributes to request timeouts and limits the API server’s ability to handle large resources (see #134375 ).
• Client-side GC: Clients must allocate structural objects (string headers, maps, and slice backing arrays) for managedFields before discarding them, adding unnecessary memory pressure and GC overhead.

Goals

• Reduce API server serialization CPU by eliminating the work of encoding managedFields for clients that do not need it. The savings are proportional to the managedFields share of each object, which varies by the number of field managers.
• Reduce network transfer sizes between the API server and clients proportionally to the managedFields overhead (up to 60% per the SSA KEP ), unblocking support for larger resource sizes and reducing LIST/WATCH timeout risk.
• Reduce client-side memory allocations and GC overhead by eliminating the need to allocate and immediately discard managedFields data structures.
• Migrate in-tree clients (kube-controller-manager, kube-scheduler) from client-side informer transforms to server-side opt-out, moving the cost savings from the client to the entire request path.

Non-Goals

• General-purpose field selection or opting out of other fields (though the API design is intended to be extensible to support this in the future without a redesign).
• Opting out of fields in the request body of write operations. This KEP drops fields from API responses (including responses to write operations like PUT/POST/PATCH), not from request bodies. kube-controller-manager and kube-scheduler already send objects without managedFields on PUT (due to informer transforms). This is safe because the field manager falls back to the stored object’s managedFields when the request has none.

Proposal

This KEP proposes using an Accept header parameter (drop=metadata.managedFields) to allow clients to opt-out of receiving metadata.managedFields in API responses. When the API server receives a request with this parameter, it uses an alternate serializer that skips managedFields during encoding. This is implemented as a new serializer mode with a distinct runtime.Identifier, which allows the watch cache’s cachingObject to naturally cache both the full and stripped serialized forms as separate entries without any changes to the watch cache itself.

User Stories

Story 1

As a cluster operator or developer of a high-traffic controller (like the scheduler), I want to avoid the overhead of receiving managedFields because I don’t use them for my reconciliation logic. This will save CPU on both the API server and my controller, and reduce network bandwidth.

Story 2

As a developer using client-go, I want to be able to easily opt-out of managedFields for certain informers where the data is redundant.

Risks and Mitigations

• Silent Data Loss in Clients: If a client opts out but later attempts an operation that requires managedFields (e.g., using managedfields.ExtractInto), ExtractInto will silently return empty results (it already returns nil when no matching entry is found).
  • Mitigation: Document clearly that clients using the extract-modify-apply workflow must not opt out of managedFields. This is an explicit opt-in by the client, so the responsibility lies with the client author.
• Watch Cache Memory Overhead: Mixed opt-in and opt-out watchers will cause the API server to maintain two serialized versions of each object in the cachingObject transient cache.
  • Mitigation: The cachingObject is transient and created per dispatch event. The cost is limited to one additional serialization per unique format. If all watchers opt out, the single cached version is smaller and cheaper to produce.

Design Details

Accept Parameter

We propose using an Accept header parameter to signal the request to exclude managedFields.

Example: Accept: application/json; drop=metadata.managedFields

This follows Kubernetes API conventions where Accept parameters are used for structural transformations (e.g., as=PartialObjectMetadata, as=Table).

While this KEP is strictly scoped to metadata.managedFields, the drop parameter is designed to be extendable to other fields in the future using + as a separator (e.g., drop=metadata.managedFields+metadata.annotations). Unknown drop targets are silently ignored for forward compatibility.

Implementation Details

1. API Server Serializer: Add an ExcludeManagedFields option to the JSON, Protobuf, and CBOR serializers. When this option is set, the serializer strips metadata.managedFields from the Go object before encoding and exposes this variant as a distinct codec on runtime.SerializerInfo with its own Identifier(). The content type negotiation layer selects the appropriate serializer based on the drop parameter in the Accept header. Stripping happens at the Go-object level before encoding, so it is serializer-agnostic and applies uniformly across all formats. Protobuf support is critical since kube-controller-manager and kube-scheduler use Protobuf by default. CRDs are also in scope — the apiextensions-apiserver constructs its own SerializerInfo for custom resources and will need to wire in the ExcludeManagedFields variant alongside its existing serializers.
2. Watch Cache: No changes are needed to the watch cache or cachingObject. The cachingObject’s serializationsCache is keyed by runtime.Identifier. Since the stripped serializer has a different Identifier than the full serializer, the cache naturally maintains both forms as separate entries. This means that until all watchers migrate to dropping managedFields, each watch event will be serialized twice (once with and once without managedFields). Benchmarking shows this dual-serialization adds roughly 62% more time and 83% more memory per event, but this overhead is constant regardless of watcher count and is offset by the smaller payload sizes.
3. Discoverability: The capability should be discoverable via the supported media types in the OpenAPI schema.
4. Client-side Mitigation: managedfields.ExtractInto already returns nil (no error) when no matching managed fields entry is found, which is a safe no-op. No changes to ExtractInto error handling are needed. Clients that opt out should be aware that ExtractInto will silently return empty results.

In-tree Controller Migration

As part of this KEP, we will migrate kube-controller-manager and kube-scheduler to use server-side opt-out instead of their current client-side informer transforms:

• Both components currently use TransformFunc on their informers to nil out managedFields after deserialization. This saves client memory but does not reduce serialization CPU or network transfer costs.
• With this KEP, both components will be updated to send the drop=metadata.managedFields parameter in the Accept header, moving the field stripping to the API server’s serializer. This eliminates the serialization and network costs in addition to the existing client memory savings.
• The migration will be gated behind the ManagedFieldsOptOutClient feature gate. When enabled, the informer transforms for managedFields stripping become redundant and will be removed.

Client Configuration

drop=metadata.managedFields is an extension of the existing Accept-header content negotiation mechanism (the same family as as=PartialObjectMetadata and as=Table). The exact client-go surface for opting in (e.g., a client-level default vs. a per-informer option) will be decided during implementation.

The broader question of per-client vs. per-request configuration is a general property of the Accept header in client-go (a single client uses one Accept header for all requests) and is out of scope for this KEP. Because the opt-out is carried in the Accept header, the design is forward-compatible with finer-grained configuration if it is added later.

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code reproducible.

Prerequisite testing updates

None.

Unit Tests

• Test API server encoders (JSON, Protobuf, and CBOR) with and without the managedFields exclusion flag.
• Test cachingObject serialization cache hits and misses with the exclusion flag.
• Test that an unrecognized drop parameter value is ignored and the full object (including managedFields) is returned. This confirms the version-skew/downgrade behavior, where an older API server that does not understand the parameter returns the full object.
• Test the client-side defensive stripping fallback: when the client requested the drop but the response still contains managedFields, the client strips it.

Integration Tests

• Verify that requests with the Accept parameter correctly return objects without managedFields across all logical verbs: GET, LIST, WATCH, CREATE (POST), UPDATE (PUT), PATCH, DELETE, and DELETECOLLECTION (the latter two are covered even though no behavior change is needed for them).
• Verify that standard requests (without the parameter) still return managedFields.
• Verify that a request with an unrecognized drop value returns the full object (forward/backward compatibility).
• Verify mixed watch scenarios with both opt-in and opt-out clients.

e2e Tests

• Ensure core components (e.g., scheduler) can successfully opt-out and function correctly.

Graduation Criteria

Both gates start at Alpha (disabled by default) and are independent. The server-side gate exposes a new kube-apiserver code path that any client can trigger via the Accept header, so its risk can’t be discounted just because in-tree clients don’t use it — it bakes at Alpha before being enabled by default at Beta.

Alpha

• ManagedFieldsOptOut (server-side, in kube-apiserver): introduced at Alpha, disabled by default. Recognizes the drop=metadata.managedFields Accept parameter for JSON, Protobuf, and CBOR across GET, LIST, WATCH, PUT, POST, and PATCH.
• ManagedFieldsOptOutClient (client-side, in client-go): introduced at Alpha, disabled by default. Controls whether in-tree clients (kube-scheduler, kube-controller-manager) send the Accept header to drop managedFields.

Beta

• ManagedFieldsOptOut (server-side) promoted to Beta and enabled by default.
• ManagedFieldsOptOutClient (client-side) promoted to Beta and enabled by default; in-tree controllers use the opt-out by default.
• Performance benchmarks confirming savings in API server and clients.
• User-facing documentation published in kubernetes/website .

GA

• In-tree controllers have run with the opt-out enabled by default for at least 2 releases with no related regressions.
• Both feature gates locked to their default values (removed 3 releases later).

Upgrade / Downgrade Strategy

• Upgrade: New API servers will recognize the Accept parameter. Old clients will continue to work as before (not sending the parameter, thus receiving managedFields).
• Downgrade: If a client starts using the parameter and the API server is downgraded to a version that doesn’t recognize it, the API server will ignore the unknown parameter and return the full object with managedFields. The client will receive more data than requested but should be able to handle it (ignoring the field if it doesn’t need it).

Version Skew Strategy

Supported. API servers that do not recognize the drop parameter will simply ignore it and return the full object, which is a safe default.

Client Support and Version Skew

To ensure consistent behavior and support clients running against older API servers:

• client-go Modification: client-go will be modified to support field dropping by adding a configuration option to request dropping specific fields. When enabled, it will automatically add the drop=metadata.managedFields parameter to the Accept header.
• Informer Configuration: Informers will inherit this configuration from the client they use. If a client is configured to drop managedFields, the informer will receive objects without managedFields.
• Client-side Stripping (Defensive): If a client requests drop=metadata.managedFields but talks to an older API server that does not support the feature, the server will return the full object with managedFields. To provide a consistent experience and avoid memory overhead in the client, client-go will be updated to strip managedFields from the response client-side if the client requested it but the server failed to drop it. This ensures that clients requesting the drop never see managedFields in the returned objects, regardless of server version.

Production Readiness Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

• Feature gate (also fill in values in kep.yaml)
  • Feature gate name: ManagedFieldsOptOut and ManagedFieldsOptOutClient
  • Components depending on the feature gate: kube-apiserver (for ManagedFieldsOptOut), kube-scheduler, kube-controller-manager (for ManagedFieldsOptOutClient)

Does enabling the feature change any default behavior?

No. The API server behavior only changes when clients explicitly opt-out using the Accept header parameter. Enabling the client-side feature gate will cause those specific clients to start omitting the data from their informers and responses.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes. Disabling ManagedFieldsOptOut will cause the API server to ignore the drop parameter and always return managedFields. Disabling ManagedFieldsOptOutClient will cause clients to stop sending the Accept header. Feature gates are typically disabled by setting the flag to false and restarting the component.

What happens if we reenable the feature if it was previously rolled back?

Clients requesting the opt-out will start receiving objects without managedFields again. There are no known side effects of reenabling the feature.

What happens if the feature is enabled or disabled in the middle of a rollout?

Clients requesting the opt-out might inconsistently receive managedFields depending on which API server instance they hit. This is safe as clients are expected to handle the presence of managedFields.

Are there any tests for feature enablement/disablement?

Yes, integration tests will cover behavior with the feature gate on and off.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

A bug in the new serializer path could cause errors or panics when serving a request that uses the drop parameter. The API server’s WithPanicRecovery filter (plus Go’s per-request panic recovery) converts such a panic into a 500 for that request only — it does not crash the process or affect other clients. Requests without the drop parameter take the unchanged path. Disabling the ManagedFieldsOptOut gate removes the new path entirely.

What specific metrics should inform a rollback?

Monitor kube_apiserver_request_duration_seconds and network egress. A sudden increase in 5xx errors or unexpected latency spikes should inform a rollback.

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

To be tested during implementation.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No.

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

Per-request metrics for Accept parameters are not currently available. We may consider adding a metric or logging to track the usage of this specific parameter if required.

How can someone using this feature know that it is working for their instance?

• Other (treat as last resort)
  • Details: Clients can verify that metadata.managedFields is absent in responses when the Accept parameter is used.

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

The feature should not introduce any measurable latency increase for API requests. Serialization of objects without managedFields should be faster than serialization with them.

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

• Metrics
  • Metric name: kube_apiserver_request_duration_seconds
  • Components exposing the metric: kube-apiserver

Are there any missing metrics that would be useful to have to improve observability of this feature?

A metric counting requests that use the drop=metadata.managedFields parameter would help measure adoption during Alpha/Beta graduation. Its long-term value is limited, so it may be a temporary adoption signal rather than a permanent metric.

Dependencies

Does this feature depend on any specific services running in the cluster?

No.

Scalability

Will enabling / disabling this feature involve terminating any processes?

No.

Will enabling / disabling this feature take a long time?

No.

Will the feature increase the size of any objects?

No, it decreases the size of returned objects.

Will the feature increase the memory or CPU usage of any component?

It may slightly increase transient memory in kube-apiserver if there is a highly mixed population of opt-in and opt-out watchers (due to the extra entries in the cachingObject cache). However, the reduction in serialization work and smaller payload sizes are expected to yield an overall decrease in CPU and memory usage.

Note on Benchmarking: Standard scalability tests often use minimal pods that do not reflect realistic managedFields usage. To properly measure the impact, we plan to create micro-benchmarks in the API server (similar to the prototype explored in the PoC) using mock pods with realistic managedFields (simulating multiple managers). For full cluster scalability tests, we will work on defining ’exemplary pods’ that reflect realistic production usage.

Troubleshooting

How can an operator determine if the feature is broken?

If clients that have opted out suddenly start receiving managedFields despite the header, or if the API server returns errors for requests containing the Accept parameter.

Implementation History

Drawbacks

• Slight complexity increase in API server encoding logic.
• Potential for two cached serialized versions of the same object in the watch cache.
• Adds to the technical performance debt by introducing another permutation of object representation (e.g., Protobuf without managedFields). This increases the matrix of combinations (format + options) that must be tracked, tested, and optimized in the API server over time.

Alternatives

• ListOptions/GetOptions flag: e.g., excludeManagedFields=true. Rejected because Accept parameters are the standard way in Kubernetes to control object representation.
• General-purpose field selector: A much larger undertaking that has been discussed for years. This KEP provides a targeted solution for a high-impact field while leaving room for future generalization. Note that a general solution would likely require a different implementation approach—such as walking the serialized form (Protobuf or CBOR) to dynamically emit the filtered response—rather than modifying Go structs before serialization as done in this targeted KEP.

Infrastructure Needed (Optional)

None.