KEP-5825: CRI List Streaming

• Release Signoff Checklist
• Summary
• Motivation
  • Message Size Analysis
  • Goals
  • Non-Goals
• Proposal
  • User Stories
  • Notes/Constraints/Caveats
  • Risks and Mitigations
• Design Details
  • API Changes
    • StreamContainers
    • StreamPodSandboxes
    • Other Stream Operations
    • Behavior Matrix
  • Test Plan
    • Prerequisite testing updates
    • Unit tests
    • Integration tests
    • e2e tests
  • Graduation Criteria
    • Alpha
    • Beta
    • GA
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
• Production Readiness Review Questionnaire
  • Feature Enablement and Rollback
  • Rollout, Upgrade and Rollback Planning
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Alternatives
  • Increase gRPC message limit
  • Improve garbage collection
  • Token-based pagination
  • Offset-based pagination
• References
  • Standards and Documentation
  • Related Issues and PRs

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  • e2e Tests for all Beta API Operations (endpoints)
  • (R) Ensure GA e2e tests meet requirements for Conformance Tests
  • (R) Minimum Two Week Window for GA e2e tests to prove flake free
• (R) Graduation criteria is in place
  • (R) all GA Endpoints must be hit by Conformance Tests within one minor version of promotion to GA
• (R) Production readiness review completed
• (R) Production readiness review approved
• “Implementation History” section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

This checklist tracks beta in v1.38.

Summary

This KEP proposes adding server-side streaming RPCs to CRI’s List* operations. Currently, these APIs return all results in a single unary response, causing failures on nodes with 10k+ containers when responses exceed the 16 MB gRPC message limit. The proposed solution introduces new streaming RPCs (e.g., StreamContainers, StreamPodSandboxes) that stream results one item at a time, bypassing the per-message size limit.

Motivation

CRI’s List* RPCs currently return all results in a single unary response without any way to break up large result sets. On nodes with many containers, responses exceeding the gRPC limit (currently 16 MB) cause complete RPC failures.

Message Size Analysis

When the 16 MB limit is exceeded:

Goals

• Add streaming support to List* operations via new streaming RPCs
• Maintain backward compatibility with existing kubelet/runtime combinations
• Eliminate the gRPC message size limit as a bottleneck for list operations

Non-Goals

• Changing the default gRPC message size limit
• Modifying container/pod garbage collection behavior
• Implementing server-side filtering beyond existing ContainerFilter/PodSandboxFilter
• Modifying the existing unary List* RPCs

Proposal

Add new server-side streaming RPCs alongside the existing unary List* RPCs. Each new RPC (e.g., StreamContainers, StreamPodSandboxes) accepts the same filter as its unary counterpart and streams results back one item per message.

User Stories

Story 1: High CronJob Churn

As a cluster operator running hundreds of CronJobs, I want kubelet to successfully list containers even when thousands of completed job containers exist, so that new pods can be scheduled without hitting gRPC limits.

Story 2: CI/CD Node

As a platform engineer managing CI nodes that run many short-lived containers, I want to avoid kubelet failures when container counts spike, so that CI pipelines remain reliable.

Notes/Constraints/Caveats

1. Streaming does not reduce kubelet memory usage: Because kubelet uses a wrapper function that collects all streamed items into a single list before processing, the full result set is still held in memory. Streaming solely addresses the gRPC per-message size limit, not memory pressure.

Risks and Mitigations

1. Consistency during modifications: Containers are created/deleted frequently during streaming. It is the runtime’s responsibility to maintain consistency and ensure no duplicate containers appear across the streamed results.

2. Kubelet atomic view: Kubelet is designed to see an atomic view of all pods.

   Mitigation: Kubelet creates a wrapper function that receives all streamed items and presents the aggregated result as a single list to the caller.

3. Stream interruption: If the runtime crashes or the stream is interrupted mid-transfer, partial results are discarded.

   Mitigation: Kubelet discards any partial results received so far and retries the entire streaming call. This is no worse than the unary RPC failure case.

Design Details

API Changes

New server-side streaming RPCs are added to the RuntimeService and ImageService. The existing unary RPCs remain unchanged for backward compatibility.

StreamContainers

Service definition:

service RuntimeService {
    // Existing unary RPC (unchanged)
    rpc ListContainers(ListContainersRequest) returns (ListContainersResponse) {}

    // New server-side streaming RPC
    rpc StreamContainers(StreamContainersRequest) returns (stream StreamContainersResponse) {}
}

StreamContainersRequest:

message StreamContainersRequest {
    ContainerFilter filter = 1;
}

StreamContainersResponse:

message StreamContainersResponse {
    Container container = 1;
}

The runtime sends one StreamContainersResponse message per container over the stream. How the runtime iterates over containers is an implementation detail; the simplest approach is to build the full list (as the unary RPC already does) and iterate over it. When all containers have been sent, the runtime returns nil from the gRPC handler, which closes the stream and signals io.EOF to the kubelet.

StreamPodSandboxes

Service definition:

service RuntimeService {
    // Existing unary RPC (unchanged)
    rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSandboxResponse) {}

    // New server-side streaming RPC
    rpc StreamPodSandboxes(StreamPodSandboxesRequest) returns (stream StreamPodSandboxesResponse) {}
}

StreamPodSandboxesRequest:

message StreamPodSandboxesRequest {
    PodSandboxFilter filter = 1;
}

StreamPodSandboxesResponse:

message StreamPodSandboxesResponse {
    PodSandbox pod_sandbox = 1;
}

Other Stream Operations

The same streaming pattern will be applied to the following existing unary RPCs:

Each streaming RPC follows the same pattern: a request message containing the same fields as the existing unary request, and a streamed response where each message contains a single item of the corresponding resource type.

ListMetricDescriptors is excluded because it returns a fixed set of metric descriptors whose total size will not approach the gRPC message limit.

Behavior Matrix

Test Plan

• I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

Unit tests

• Test kubelet streaming wrapper: verify it correctly aggregates all items from a mock stream into a single list.
• Test fallback behavior: verify kubelet falls back to the unary RPC when the streaming RPC returns UNIMPLEMENTED.
• Test stream error handling: verify kubelet discards partial results and returns an error when the stream fails mid-transfer.
• Test the streaming gauge for each service: streaming, cached fallback, and absence when the feature gate is disabled.

Integration tests

kubelet does not have integration tests.

e2e tests

• End-to-end streaming on nodes with high container counts.
• Using CRI-proxy, verify that:
  • kubelet calls the streaming RPC when the feature gate is enabled.
  • kubelet falls back to the unary RPC when the runtime does not implement the streaming RPC.
  • kubelet correctly aggregates all streamed items into a complete list.
• Cover the behavior matrix scenarios (old/new kubelet x old/new runtime).
• Verify the streaming gauge for both services during streaming and fallback.
• Compare pod startup latency with streaming enabled and disabled under the same load, including high container counts.

Graduation Criteria

Alpha

• Feature gate: CRIListStreaming
• Streaming RPCs added to CRI proto
• Kubelet implements streaming with fallback to unary RPCs
• Reference implementation in containerd or CRI-O
• Basic unit tests

Beta

• Feature gate enabled by default
• Both containerd and CRI-O implement streaming RPCs
• critest coverage for streaming RPCs added in cri-tools and passing for both containerd and CRI-O
• Streaming and fallback e2e tests passing in Kubernetes CI with supporting versions of both runtimes
• Manual upgrade->downgrade->upgrade and feature gate rollback results recorded
• kubelet_cri_streaming_enabled metric available
• Existing kubelet CRI metrics (kubelet_runtime_operations_errors_total, kubelet_runtime_operations_duration_seconds) cover streaming transparently
• Documentation updated

GA

• All supported runtimes implement streaming RPCs
• Feature gate locked to enabled
• Existing unary RPCs remain available (no removal, it can be a different KEP)

Upgrade / Downgrade Strategy

Upgrade path:

1. Upgrade runtime and kubelet (order-agnostic)

Downgrade path:

1. Downgrade runtime and kubelet (order-agnostic)

Upgrade and downgrade operations are order-agnostic because kubelet automatically falls back to unary RPCs when the runtime does not support streaming. The streaming RPCs are additive and do not modify existing RPCs.

Version Skew Strategy

The fallback mechanism handles all version skew scenarios:

• New kubelet + Old runtime: Kubelet calls the streaming RPC. The runtime returns UNIMPLEMENTED. Kubelet falls back to the unary RPC and retrieves the full list.
• Old kubelet + New runtime: Kubelet uses the existing unary RPC. The streaming RPCs exist on the runtime but are never called.
• New kubelet + New runtime: Kubelet uses the streaming RPCs.

Because the streaming RPCs are entirely new RPCs (not modifications to existing ones), there is no risk of field-level incompatibility. The version skew is handled purely through gRPC status codes.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

• Feature gate
  • Feature gate name: CRIListStreaming
  • Components depending on the feature gate: kubelet

Does enabling the feature change any default behavior?

No. When enabled, kubelet will use streaming RPCs if the runtime supports them, falling back to unary RPCs otherwise. The end result (complete container/pod list) is identical.

Can the feature be disabled once it has been enabled?

Yes. Disabling the feature gate causes kubelet to use only the existing unary RPCs.

What happens if we reenable the feature if it was previously rolled back?

Streaming resumes if the runtime supports it. No state is persisted.

Are there any tests for feature enablement/disablement?

Manual enable->disable->reenable testing is described in the upgrade and rollback procedure below.

The planned coverage includes:

• Unit tests confirm that when CRIListStreaming is disabled, kubelet uses only unary RPCs and never attempts streaming calls.
• CRI-proxy e2e tests confirm that when the feature gate is enabled, kubelet calls the streaming RPCs.
• CRI-proxy e2e tests confirm that when the runtime returns UNIMPLEMENTED, kubelet falls back to unary RPCs automatically.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

Rollout/rollback cannot impact running workloads. Streaming only affects how container/pod lists are retrieved, not container lifecycle.

What specific metrics should inform a rollback?

kubelet_runtime_operations_errors_total{operation_type="list_containers"} (and equivalent list_podsandbox, list_images, etc.)

A sustained increase in CRI list operation errors may indicate runtime instability with streaming RPCs. This existing metric already captures mid-stream failures because the error propagates through the CRI client to the instrumented service layer.

kubelet_runtime_operations_duration_seconds{operation_type="list_containers"} (and equivalent)

An unexpected increase in list operation latency after enabling the feature gate may indicate streaming overhead.

Regressions in kubelet_pod_start_sli_duration_seconds after enabling streaming should also inform a rollback.

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

Yes. Manual upgrade, rollback, and upgrade->downgrade->upgrade verification was completed using the following procedure:

1. Start with kubelet v1.37, CRIListStreaming=false, and a streaming-capable runtime. Keep a long-running pod while creating and deleting short-lived pods. Keep list sizes below the unary message limit to allow rollback.
2. Upgrade kubelet to the v1.38 build under test with the gate enabled. Confirm both service gauges report streaming after list calls.
3. Disable the gate and restart kubelet. Confirm unary calls through CRI-proxy and absence of the gauge. Reenable the gate and restart to confirm streaming.
4. Downgrade to v1.37 with the gate disabled, then upgrade to v1.38 with it enabled.
5. Use CRI-proxy to return UNIMPLEMENTED and verify fallback. Restore streaming support, verify fallback remains cached, then restart kubelet to retry streaming.

At each step, verify complete lists, no restarts of existing containers, and successful new pod startup without CRI error or latency regressions.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No.

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

Not applicable. It’s not a workload-level feature.

How can someone using this feature know that it is working for their instance?

• kubelet_cri_streaming_enabled is 1 for both services after list calls.
• kubelet_runtime_operations_errors_total for list operations (list_containers, list_podsandbox, etc.) is not incrementing: CRI list calls are completing successfully, whether via streaming or unary RPCs.
• kubelet_runtime_operations_duration_seconds for list operations is not increasing: streaming is not adding latency overhead.

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

The existing pod startup latency SLO applies: in a default Kubernetes installation, the 99th percentile per cluster-day is at most 5 seconds for schedulable stateless pods, excluding image pulls and init container execution. Streaming must not regress this SLO.

Operators can use the existing kubelet-side metrics to monitor streaming health:

• kubelet_runtime_operations_errors_total for list operations near zero: CRI list calls completing successfully.
• kubelet_runtime_operations_duration_seconds for list operations stable: no latency regression from streaming.

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

• Metrics
  • Metric name: kubelet_cri_streaming_enabled
  • Type: Gauge; stability: Alpha; label: service (runtime or image).
  • Reports streaming selection (1) or cached unary fallback after UNIMPLEMENTED (0), reset on kubelet restart. Monitor per node or by service.
  • Available only with CRIListStreaming enabled, after list calls. Missing series do not indicate successful streaming.
• Metrics
  • Metric name: kubelet_pod_start_sli_duration_seconds
  • Tracks kubelet pod startup latency excluding image pulls and init containers.
• Metrics
  • Metric name: kubelet_runtime_operations_errors_total
  • Tracks CRI operation errors by operation type. Mid-stream failures propagate as errors and are captured by this existing metric.
• Metrics
  • Metric name: kubelet_runtime_operations_duration_seconds
  • Tracks CRI operation duration by operation type. Covers both streaming and unary paths transparently.
• Runtime Metrics
  • Metric name: crio_operations_latency_seconds_total (CRI-O)
  • Tracks CRI RPC latency.

Are there any missing metrics that would be useful to have to improve observability of this feature?

The kubelet_cri_streaming_enabled gauge addresses the observability gap. Dedicated failure and fallback counters would require instrumentation in shared cri-client, which should not register kubelet-specific metrics. Use existing CRI error metrics with the gauge to identify the affected service’s current streaming or unary path.

Dependencies

Does this feature depend on any specific services running in the cluster?

No. This feature only requires a CRI-compatible runtime (containerd, CRI-O) with streaming RPC support.

Scalability

Will enabling / using this feature result in any new API calls?

No new Kubernetes API calls. The streaming RPCs replace the unary CRI calls with a single streaming call per list operation.

Will enabling / using this feature result in introducing new API types?

New streaming RPC methods and their request/response messages are added to the CRI proto. No new Kubernetes API types.

Will enabling / using this feature result in any new calls to the cloud provider?

No.

Will enabling / using this feature result in increasing size or count of the existing API objects?

No.

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

Streaming adds gRPC framing overhead that could increase CRI list latency and delay pod startup. The test plan includes latency comparisons to detect this.

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

No.

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

No.

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

Not applicable. This feature is kubelet-to-runtime communication only.

What are other known failure modes?

What steps should be taken if SLOs are not being met?

If kubelet_runtime_operations_errors_total for list operations is incrementing, check the streaming gauge and runtime logs. Verify runtime support and restart kubelet after a runtime upgrade to clear cached fallback.

For pod startup regressions, compare CRI latency and errors with the baseline before enablement and consider disabling the gate.

Disabling the feature gate will cause kubelet to fall back to unary RPCs. Note that if the node has enough resources to exceed the 16 MB gRPC message limit, the unary RPCs will also fail.

Implementation History

• 2026-01-21: KEP created
• 2026-02-11: KEP refactored from pagination to server-side streaming
• 2026-03-19: Alpha implementation merged (kubernetes/kubernetes#136987)
• 2026-03-21: Data race fix merged (kubernetes/kubernetes#137949)
• 2026-04-13: CRI-O streaming implementation merged (cri-o/cri-o#9761)
• 2026-06-03: KEP updated for beta graduation in v1.37
• 2026-06-05: Beta target deferred to v1.38 pending containerd support and CI coverage
• 2026-07-23: Containerd streaming implementation merged (containerd/containerd#13187)

Drawbacks

1. Implementation burden: All CRI implementations (containerd, CRI-O, etc.) must implement the new streaming RPCs.

2. Proto surface area: New RPCs and message types are added to the CRI proto, increasing the API surface.

3. Stream lifecycle management: Runtimes must manage stream lifecycle correctly, including proper stream termination and error handling.

Alternatives

Increase gRPC message limit

Pros:

• Simple, no API changes required
• Already done twice (4MB->8MB->16MB)

Cons:

• Only delays the problem

Improve garbage collection

Pros:

• Addresses root cause (too many dead containers)
• No API changes needed

Cons:

• May conflict with forensics/debugging needs
• Doesn’t help legitimate high-container-count workloads

Token-based pagination

Pros:

• Follows established patterns (AIP-158)
• No new RPCs needed, only new fields on existing messages

Cons:

• Requires token management (generation, validation)
• Requires stable ordering semantics for consistency
• Multiple independent round trips instead of a single stream

Offset-based pagination

Pros:

• Simpler token (just an integer offset)
• Easier to implement

Cons:

• Vulnerable to inconsistency when items are added/deleted
• Less aligned with AIP-158 recommendations

References

Standards and Documentation

• gRPC Server-side Streaming
• kubernetes/cri-api
• CRI in Kubernetes Blog
• CRI API Version Skew Policy

Related Issues and PRs

• #5825 - Enhancement tracking issue (beta target: v1.38)
• #136987 - Alpha kubelet streaming implementation
• CRI-O#9761 - CRI-O streaming implementation
• containerd#13187 - Containerd streaming implementation
• #139500 - Proposed kubelet streaming status metric
• #63858 - Original gRPC message limit bug (2018)
• #63977 - Increased CRI limit from 4MB to 8MB
• #64672 - Increased CRI limit from 8MB to 16MB
• #90340 - Feature request for pagination
• #107190 - Discussion on CRI feature implementation
• #131407 - High CronJob count hitting gRPC limit (2025)
• #134750 - 16MB limit exceeded with ~18.5k containers (2025)