KEP-6030: Dynamic Resize of Memory-Backed Volumes

Implementation History
BETA Implementable
Created 2026-04-27
Latest 1.38
Milestones
Alpha 1.37
Beta 1.38
Stable TBD
Related Enhancements
Ownership
Owning SIG
SIG Node
Participating SIGs
SIG Storage
Primary Authors
@natasha41575

KEP-6030: Dynamic Resize of Memory-Backed Volumes

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

This KEP proposes a mechanism to dynamically resize emptyDir volumes with medium: Memory (tmpfs) without requiring Pod recreation or Pod restart, and while maintaining the files already written to the volume’s filesystem. It leverages the InPlacePodVerticalScaling (IPPVS) infrastructure to manage the transition from the desired state to the actual state via an allocation and actuation loop.

Motivation

Currently, increasing memory-backed storage (tmpfs emptyDir volumes) requires Pod recreation, which is disruptive for high-performance caches or in-memory databases that are otherwise capable of resizing their memory footprints dynamically.

This enhancement builds directly on the foundation laid by In-Place Pod Vertical Scaling (IPPVS) (KEP-1287). Just as IPPVS allows scaling container CPU and memory resources without Pod disruption, high-performance workloads frequently require scaling their scratch space or in-memory caches simultaneously. Since memory-backed emptyDir volumes consume the node’s memory (and can be bounded by the Pod’s memory limits), integrating their resize lifecycle with the IPPVS allocation and actuation phases ensures consistent resource accounting and prevents node exhaustion.

Goals

  • Allow updating emptyDir.sizeLimit for existing Pods when medium is Memory.
  • Reflect resize progress in Pod.Status using an allocation/actuation model similar to IPPVS.
  • Actuate the resize via tmpfs remount without Pod disruption and while maintaining the files already written to the volume’s filesystem.
  • Support dynamic resizing of memory-backed volumes on cgroup v2 nodes only (where memory charging and charge migration for tmpfs are properly supported by the kernel).

Non-Goals

  • Resizing non-memory-backed emptyDir volumes (disk-backed).
  • Resizing other volume types (such as PVCs, CSI ephemeral volumes etc.) through this mechanism.
  • Supporting this feature on nodes running with cgroup v1 (due to limitations in memory accounting and tracking for shared tmpfs volumes).
  • Supporting this feature on Windows nodes (in-place memory-backed volume resize is Linux-only, just like InPlacePodVerticalScaling).

Proposal

We propose extending the InPlacePodVerticalScaling pattern to memory-backed emptyDir volumes.

User Stories

Story 1

A user runs an in-memory database in a Pod and needs to increase the available cache size. The user updates the emptyDir.sizeLimit in the Pod spec. The Kubelet remounts the tmpfs volume with the new size limit without restarting the container, allowing the database to expand its cache immediately.

Story 2

An autoscaling controller dynamically scales the memory limits of workload worker pods (e.g., for machine learning or data processing) to handle varying demands. These workers utilize memory-backed emptyDir volumes for shared memory and caching, typically sizing the volume to a fixed proportion of the container’s total memory limit.

While the controller can use InPlacePodVerticalScaling to scale up a container’s memory limit without restarting the pod, the inability to dynamically resize the associated memory-backed volume limits the worker’s ability to utilize the expanded capacity for shared memory tasks without a disruptive restart. This feature allows the controller to update both the container memory limits and the emptyDir.sizeLimit simultaneously, providing seamless vertical scaling for distributed workloads.

Risks and Mitigations

OOM-kills

The user is allowed to set the volume sizeLimit higher than the pod or container memory limits. Attempting to add such a restriction is considered out of scope for this KEP (see Restricting sizeLimit to not exceed memory limits). While setting sizeLimit higher than the pod memory limit is permitted by the API, the Kubelet caps the actual mounted tmpfs capacity to $\min(\text{NodeAllocatableMemory}, \text{PodMemoryLimit}, \text{VolumeSizeLimit})$ and emits a Warning VolumeSizeExceedsPodMemoryLimit event during admission.

However, users remain responsible for ensuring that the volume sizeLimit is set with sufficient headroom below the pod and container memory limits. Because tmpfs consumption is accounted against the container and pod memory cgroups, allowing an emptyDir volume to fill up to the pod or container limit leaves no execution memory for processes, which will result in container OOM kills.

Design Details

API Changes

State Mapping and Status Tracking

To support dynamic resizing, the Pod.Spec.Volumes[].EmptyDir.SizeLimit field is made mutable when modified via the /resize subresource for existing Pods (when medium is Memory). This represents the Desired State.

This enhancement builds upon the resource state tracking model established in In-Place Pod Vertical Scaling (KEP-1287), extending the lifecycle progression to memory-backed volumes:

  • Desired State: Represented by the mutable Pod.Spec.Volumes[].EmptyDir.SizeLimit.
  • Allocated State: Checkpointed and maintained by the Kubelet’s AllocationManager for local resource accounting, but not exposed in Pod.Status.
    • Definition: Represents the volume capacity acknowledged and admitted by the Kubelet on the node.
    • Initial Admission: Admitted and checkpointed locally by the Kubelet once the initial resource admission phase completes.
    • Admission during Resize: Upon admitting a resize update, the AllocationManager updates the Kubelet’s local checkpoint with the newly acknowledged capacity.
    • Scope: Maintained only for memory-backed emptyDir volumes.
  • Actuated State: Checkpointed and maintained locally by KuberuntimeManager to track the volume capacity passed to the volume plugin to actuate.
    • Definition: Represents the target size that the Kubelet successfully actuated via ResizeEphemeralVolume.
    • Scope: Not reported in the API.
  • Actual State: Represents the actual mounted filesystem capacity of the tmpfs volume on the node.
    • Definition: Represents the actual mounted capacity reported by the mount/remount call.
    • Scope: Unlike container resources, actual volume capacity is not exposed in Pod.Status. Users can inspect the actual mount size on the node or within the container via df command on the volume mount point.

API Validation and Restrictions

API validation logic ensures that Pod.Spec.Volumes[].EmptyDir.SizeLimit is only mutable when:

  • The volume is memory-backed (meaning the medium field is set to Memory).
  • Updates occur through the resize subresource.

Only modifications to the sizeLimit of existing memory-backed volumes are permitted:

  • The addition or removal of a memory-backed volume in a pod spec is forbidden.
  • Adding or removing a sizeLimit to an existing memory-backed volume is supported.

Resize Restart Policy

There are existing resize restart policies that control when a container is restarted during a resize operation, which are described in more detail in the InPlacePodVerticalScaling KEP.

We do not introduce an analogous policy nor honor the existing restart policies when resizing memory-backed emptyDir volumes. The resizing of memory-backed emptyDir volumes is fundamentally non-disruptive to running processes and does not require application-level awareness beyond standard handling of filesystem capacity. If a volume resize is accompanied by a container or pod resource resize, the container’s existing RestartPolicy governs container restarts.

Atomic Resize Principle

The atomic resize principle is preserved. The Kubelet treats updates to volume sizeLimit and container or pod-level resources (both requests and limits) as a single atomic transaction. This applies to requests that modify multiple volumes, multiple containers, pod-level resources, or any combination thereof.

The AllocationManager performs an All-or-Nothing Admission check, rejecting the entire update if the resize feasibility checks do not pass. The volume sizeLimit is also checkpointed atomically along with container and pod resources.

  • The volume sizeLimit is not taken into consideration during the resource-fit checks, but will fail the admission check atomically if the container/pod resize fails to be admitted.
  • On cgroups v1, volume resize is not supported, so any resize request containing a sizeLimit update will be atomically rejected as Infeasible during the admission phase.

The Pod’s ResizeStatus (tracked via PodResizeInProgress condition) only transitions to completed once all container and volume states reach their respective desired values. If any component of the update fails, the status remains in a PodResizeInProgress state (with an error) to reflect the incomplete transition. This ensures resource integrity by preventing “partial resizes” that could leave a volume expanded within a container that fails to scale its memory budget.

See Non-atomic handling of volume and resource resizes for the alternative that was considered and rejected.

Component Interaction & Design Flow

The following diagram illustrates the proposed end-to-end flow.

graph TD
    User["User / Controller"]
    API["API Server"]
    Kubelet["Kubelet Pod Worker"]
    Alloc["AllocationManager"]
    KRT["KuberuntimeManager"]
    VM["Volume Manager"]
    CRI["CRI"]
    Kernel["Linux Kernel tmpfs"]

    User -->|1. Updates Spec| API
    API -->|2. Syncs Pod| Kubelet
    Kubelet -->|3. Requests Allocation| Alloc
    Alloc -->|4. Checkpoints State Locally| Alloc
    Kubelet -->|"5. Updates Status (Container Resources)"| API
    KRT -->|6. Reads Checkpointed Allocation| Alloc
    KRT -->|7. Evaluates Resize Direction| KRT
    KRT -->|"8a. Remounts Volume (Downsize)"| VM
    KRT -->|8b. Updates Cgroups| CRI
    KRT -->|"8c. Remounts Volume (Upsize)"| VM
    VM -->|9. Executes Remount System Call| Kernel
    KRT -->|"10. Clears PodResizeInProgress Condition"| API

The following steps elaborate on this flow:

1. User/Control Plane

  • The user or an external controller updates Pod.Spec.Volumes[].EmptyDir.SizeLimit.
  • The API server preserves volume updates during the dropNonResizeUpdates phase.

2. Kubelet Pod Workers & Allocation

  • The Kubelet detects the specification change in HandlePodUpdates.
  • The AllocationManager validates the new resize request. The changing sizeLimit does not affect the feasibility of the resize on cgroup v2, as pod admission takes only pod and container requests into consideration. If the node is running with cgroup v1, the AllocationManager will reject the resize request as Infeasible.
  • If allocation succeeds, the AllocationManager updates the Kubelet’s internal allocated state. The AllocatedSizeLimit is checkpointed locally (along with container and pod requests/limits) to ensure that memory-backed volume size limits survive Kubelet restarts.

3. Coordination & Actuation (KuberuntimeManager)

  • The KuberuntimeManager reads the AllocatedSizeLimit from the local checkpointed state maintained by AllocationManager.
  • It acts as the central orchestrator for the actuation phase to enforce strict ordering. See Resource Coordination: Volume and Cgroup Ordering of Updates for the detailed rationale behind this ordering.
  • The KuberuntimeManager checks the direction of the volume resize (upsize vs. downsize) by comparing the newly allocated size with the checkpointed Actuated State (the target size that the Kubelet last attempted to apply, which is maintained in Kuberuntime’s internal checkpoint). Comparing against the actuated state rather than the live kernel state avoids unnecessary remount requests caused by kernel rounding differences.
  • The KuberuntimeManager directly calls a new interface on the Volume Manager (see Volume Manager interface) to perform the remount at the correct time, bypassing the async reconciler loop. The Volume Manager executes the remount: mount -o remount,size=<limit> -t tmpfs tmpfs <path>. Once this actuation completes, the KuberuntimeManager checkpoints this sizeLimit in its own Actuated State. This Actuated State represents what the Kubelet tries to actuate, which may diverge slightly from what is read directly from the kernel (Actual State).
  • Note: Bypassing the async reconciler loop is both safe and inconsequential. The emptyDir reconciler today never remounts because RequiresRemount is hardcoded to always return false; this means that the KuberuntimeManager is the only component that makes this change. We considered alternatives such as relying on the emptyDir reconciler to detect changes asynchronously or moving emptyDir handling entirely to KuberuntimeManager, but they are ruled out. See Alternatives for more details.

4. Observation & Feedback Loop

  • Upon successful remount and cgroup update, the KuberuntimeManager checkpoints the new target values as the current ‘actuated resources’ (the baseline for future resizes).
  • Once the actual size matches the allocated size checkpoint (and container resource updates are complete), the Kubelet clears the PodResizeInProgress condition.

Resource Coordination: Volume and Cgroup Ordering of Updates

Background: The Enforcement Divergence

Memory-backed volumes consume the same physical RAM as the container’s processes. While all usage is enforced by the container and pod cgroups, it is governed by two different enforcement thresholds:

  • tmpfs size option: An internal quota for the volume. When a write exceeds the volume’s sizeLimit, the kernel returns a disk full (ENOSPC) error. This is a graceful I/O exception.
  • cgroup memory.max: The external limit for the entire container. Since tmpfs usage counts toward the cgroup total, hitting this limit triggers an OOM (Out of Memory) Kill, resulting in container termination.
Memory Attribution & cgroup Limits

Because memory-backed volumes are backed by tmpfs, their memory usage is charged directly to the cgroup of the process that writes the data, rather than to the volume itself.

  • Container-level charging: If Container A writes to the shared volume, the memory is initially charged against Container A’s cgroup memory limit. If Container B writes to the shared volume, it is charged against Container B’s cgroup limit.
    • cgroup v2 Page Charge Migration: In clusters running cgroup v2, if Container A writes files to a shared tmpfs volume and Container B later accesses or page-faults those same files, the kernel’s active memory controller can dynamically migrate or split the page cache attribution between Container A’s and Container B’s cgroups under memory pressure. This behavior is a key enhancement of cgroup v2 but makes individual container-level memory metrics unpredictable, as a container’s memory usage can rise or fall based on access to shared cached files.
  • Pod-level charging: Since container cgroups are nested under the parent Pod cgroup, all files written to the memory-backed volume always count against the Pod-level cgroup memory limit, regardless of any dynamic charge migration between containers.

This dynamic page migration has implications for how we validate the safety of resource down-sizing. See Shrinkage Safety for details on how this is taken into account.

Sharing Volumes Between Multiple Containers

When a single memory-backed emptyDir volume is mounted and shared by multiple containers under the same Pod:

  • Volume Quota (ENOSPC): The volume-level sizeLimit acts as a hard boundary on the aggregate space consumed. If the sum of bytes written across all sharing containers exceeds this limit, standard disk-full (ENOSPC) errors are returned to subsequent writes. The containers are not OOM-killed at this boundary.
  • cgroup Enforcement (OOM-Kill): If a container’s write to the shared volume pushes its total memory usage (processes + files written) past its cgroup memory limit, the container’s processes are OOM-killed by the kernel—even if the shared volume usage is well below its sizeLimit.
  • Persistence of Orphaned Space: If a container is terminated or OOM-killed, the files it wrote remain in host memory (tmpfs survives container lifecycles). This memory continues to consume headroom in the parent Pod cgroup.
Init Container Lifecycles & Handoff

Init containers are commonly used to pre-populate a memory-backed volume (e.g., pre-loading templates, caches, or datasets) before primary application containers start:

  1. Init Execution: During execution, files written by the Init container count against the Init container’s individual limit.
  2. Termination: When the Init container exits and its cgroup is cleaned up, the files remain in the memory volume. The memory usage of these files continues to count against the overall Pod cgroup memory limit.
  3. App Container Handoff: When the primary application container starts, it can read the pre-populated data. Under standard memory accounting rules, simply reading pre-existing files does not charge that memory to the app container’s individual limit. The app container can use the cache without it counting toward its own limit (it only consumes Pod-level cgroup headroom).
Priority of Enforcement

The volume sizeLimit acts as a safety “inner boundary” to ensure volume growth does not starve the container’s processes of execution RAM. Ideally, sizeLimit is lower than the container’s memory limit.

When mounting or resizing a memory-backed volume, the Kubelet today computes the actual mounted tmpfs capacity as: $$\text{capacity} = \min(\text{NodeAllocatableMemory}, \text{PodMemoryLimit}, \text{VolumeSizeLimit})$$

  • If sizeLimit is omitted (nil or 0), the volume size defaults to the total pod memory limit (or node allocatable if pod limits are unset).
  • If sizeLimit is specified and exceeds the total pod memory limit (e.g. 256Mi on a 200Mi pod), the mounted tmpfs volume capacity is capped to the total pod memory limit (200Mi). The Kubelet emits a Warning VolumeSizeExceedsPodMemoryLimit event during admission.

Order of Actuation

To prevent race-condition OOMs during a resize, the order of operations is driven primarily by the direction of the Volume sizeLimit change to ensure the container maintains the maximum possible “slack” during transition.

  • When Increasing Volume sizeLimit:
    1. Adjust Cgroup First: Adjust memory.max to ensure the container envelope is prepared for potential volume expansion.
    2. Remount Volume Second: Increase the tmpfs quota.

(Rationale: Ensures the physical budget is expanded before the internal quota allows more usage.)

  • When Decreasing Volume sizeLimit:
    1. Remount Volume First: Shrink the tmpfs quota to reclaim space.
    2. Adjust Cgroup Second: Adjust memory.max.

(Rationale: Ensures the volume’s claim on memory is reduced before the cgroup envelope is tightened around it.)

In the case of multiple volumes, all volumes with a decreasing sizeLimit are downsized first, followed by the cgroup memory limit update, followed by all volumes with an increasing sizeLimit upsized.

Asynchronous Updates or Opposite Directions

It is possible that the resize requests for the container memory limits and the volume sizeLimit occur in separate updates or in opposite directions. Both scenarios are permitted. In these cases, the actuation order is less critical because the opposing nature of the updates means the sequence does not materially increase or decrease the risk of OOM kills. For implementation simplicity and predictability, actuation follows the standard order described above.

The user remains responsible for ensuring the container memory limit provides sufficient headroom for both volume data and process execution to prevent OOM kills.

Actuation Priority Matrix

The following table illustrates the finalized actuation priority matrix that the Kubelet follows, based on the direction of the volume’s sizeLimit change:

ScenarioVolume sizeLimitContainer memory.max1st Action2nd ActionRationale
ExpansionIncreasing ⬆️Increasing ⬆️CgroupVolumeExpand envelope before expanding the internal quota.
ContractionDecreasing ⬇️Decreasing ⬇️VolumeCgroupShrink the internal quota before tightening the envelope.
Volume OnlyIncreasing / DecreasingUnchangedVolumeN/ANo cgroup update necessary.
Opposite (Vol Up)Increasing ⬆️Decreasing ⬇️CgroupVolumeIf this configuration results in an OOM kill, it does so regardless of ordering; follows “Vol Up” sequence for consistency.
Opposite (Vol Down)Decreasing ⬇️Increasing ⬆️VolumeCgroupIf this configuration results in an OOM kill it does so regardless of ordering; follows “Vol Down” sequence for consistency.
Cgroup OnlyUnchangedIncreasing / DecreasingCgroupN/AStandard container resize; no volume remount required.

Role of the Allocation Manager and Kuberuntime Manager

The AllocationManager’s role is focused on maintaining the source of truth for allocated resources (checkpointing) and ensuring atomic admission of resource updates. The volume manager should only ever see the allocated size limits, rather than the desired size limits.

The KuberuntimeManager serves as the coordination layer that understands the nested relationship between volume size and cgroup limits and enforces the strict ordering of updates.

Volume Manager interface

To support direct resizing from the KuberuntimeManager while maintaining separation of concerns, this proposal extends the Volume Manager and Volume Plugin interfaces:

  • VolumeManager Extension: The VolumeManager interface is extended with ResizeEphemeralVolume(pod *v1.Pod, volumeName string) error. This allows the KuberuntimeManager to trigger volume operations synchronously at the correct time in its sync loop.
  • ResizableEphemeralVolumePlugin Interface: A new optional interface ResizableEphemeralVolumePlugin is introduced in pkg/volume/plugins.go. Volume plugins that support online, synchronous resizing (in this case, the emptyDir plugin for memory-backed volumes) can implement this interface: ResizeEphemeralVolume(spec *volume.Spec, pod *v1.Pod) error. The VolumeManager checks if the plugin for the volume implements this interface and delegates the call to it.

This design allows KuberuntimeManager to act as the central orchestrator for resource updates (ensuring strict ordering with cgroups) while leaving the actual filesystem manipulation logic encapsulated within the volume plugins.

We considered alternatives such as relying on the Volume Reconciler to detect changes asynchronously or moving emptyDir handling entirely to KuberuntimeManager, but they are ruled out. See Alternatives for more details.

The Role of the Container Runtime

On startup, the container runtime helps to bind the Kubelet’s host path into the container’s mount namespace, but subsequent remount requests do not require intervention from the container runtime, nor does the container runtime need to be notified of changing volume sizes.

Shrinkage Safety

In the existing InPlacePodVerticalScaling feature, a best-effort safety logic ensures that container memory limits are not decreased below usage. We analogously validate the safety of volume size decreases.

Existing Cgroup Validation

The Kubelet already contains validation logic (in validateMemoryResizeAction) to ensure that proposed cgroup limit reductions do not fall below active memory usage. We will leverage this check:

  • Container-level Validation: Ensures the proposed container memory limit is not decreased below its current container cgroup usage. Note that this check is insufficient if dynamic kernel-level page charge migration has temporarily shifted memory charges to another container (see Memory Attribution & cgroup Limits).
  • Pod-level Validation: For pods with enforced Pod-level cgroup limits (e.g., pods with pod-level limits specified, Guaranteed pods, or Burstable pods where all containers specify limits), the Kubelet validates the proposed aggregate Pod limit against the Pod-level cgroup usage. This acts as a reliable backstop since it compares against the aggregate memory footprint regardless of charge migration.

Volume-level safety

If a user attempts to shrink a volume below its current usage, /bin/mount fails at the kernel level:

  • On modern Linux kernels (>= 5.2, util-linux >= 2.36) using the fsconfig tmpfs mount API, /bin/mount fails with exit status 32, returning fsconfig() failed: tmpfs: Too small a size for current use..
  • On older Linux kernels (< 5.2) using legacy mount(2) with MS_REMOUNT, the syscall fails with EINVAL or ENOSPC, surfaced by /bin/mount as mount point not mounted or bad option.

In all Linux kernel versions, the kernel rejects the downsize operation without modifying the mount or corrupting in-use pages.

The Kubelet does not perform string parsing on mount error output to distinguish shrinkage failures from other mount errors. Any /bin/mount remount failure is returned directly by ResizeEphemeralVolume to KuberuntimeManager, which exposes it to the user:

  • In the PodResizeInProgress condition (Status: True, Reason: Error, with the raw mount error message).
  • In a Warning ResizeError event with the podResourceSummary JSON payload describing the observed resize state and error.

The volume remains mounted at its previous capacity, running containers are unaffected (no container restarts), and the Kubelet periodically retries the resize operation on subsequent sync iterations until it succeeds, is cancelled, or the Pod terminates.

emptyDir Size Limit Monitoring and Eviction

The Kubelet runs a local storage capacity monitoring system (part of the eviction manager) to ensure that the space usage of emptyDir volumes does not exceed their volume size limits, and triggers evictions when necessary.

For memory-backed emptyDir volumes (medium: Memory), the eviction manager skips size limit eviction when InPlacePodVerticalScalingMemoryBackedVolumes is enabled. Because memory-backed volumes are backed by tmpfs, their size limit is enforced directly by the Linux kernel, returning ENOSPC when the volume size limit is reached. Eviction manager enforcement is not needed.

Interaction with Ephemeral Storage: Resource Accounting and Eviction

Memory-backed emptyDir volumes are functionally distinct from disk-backed ephemeral storage. This KEP maintains the existing isolation between these resource pools to ensure that dynamic resizing does not interfere with node-level storage management.

While this KEP only deals with memory-backed volumes, we call out the following behavior to avoid confusion:

  • Resource Accounting: Usage of medium: Memory volumes is tracked via the cgroups, rather than the ephemeral-storage (disk) resource. Resizing the sizeLimit has no effect on the ephemeral-storage capacity or quotas assigned to the node.
  • Exclusion from Disk Monitoring: The Kubelet’s eviction manager monitors disk-backed ephemeral storage usage using periodic filesystem scans. Because memory-backed volumes are mounted as tmpfs, they are naturally excluded from these root-partition disk-usage calculations.
  • Eviction Logic: Scaling a memory-backed volume up or down never triggers DiskPressure evictions. Pressure resulting from these volumes is handled via the OOM killer (at the container level) or MemoryPressure eviction thresholds (at the node level). Pod-level emptyDirLimitEviction is bypassed for memory-backed volumes because tmpfs capacity is enforced directly by the kernel via ENOSPC, and memory usage is governed by container/pod cgroups.

Interaction with Secrets, Projected Volumes, and DownwardAPI

Projected volumes, secrets, and downwardAPI are typically implemented as tmpfs mounts to ensure data is never persisted to physical storage. These share the same underlying kernel infrastructure as memory-backed emptyDir volumes.

While projected volumes are out of scope for this KEP, we call out their behavior here to avoid confusion:

  • Implicit Memory Consumption: Memory used by projected volumes is accounted for in the container or Pod-level memory cgroup usage. The AllocationManager does not explicitly track or reserve space for these volumes, as they do not have user-defined resource requests or sizeLimits.
  • Resize Validation Boundaries: When validating a resize request, the AllocationManager ensures the new container and pod requests are feasible based on node capacity. It does not account for the “hidden” overhead of projected volumes; users must ensure that their container memory limit provides sufficient headroom for both the expanded emptyDir and any projected volumes.
  • Actuation Isolation: Dynamic resizing of an emptyDir volume is a targeted remount operation on a specific mount point. It does not interfere with the file descriptors or mount states of existing Secret or ConfigMap volumes.
  • Shared Fate in OOM Scenarios: Because all tmpfs mounts share a memory budget, an over-provisioned emptyDir expansion can push the cgroup usage to its limit. If a process attempts to access a large Secret or write to the emptyDir when the cgroup is exhausted, the kernel triggers an OOM kill based on the cgroup threshold, not the individual volume quota.

ConfigMaps

At the time of writing, ConfigMaps are implemented using disk-backed volumes rather than tmpfs. This means that they are currently independent of the memory-backed volumes implementation. There is a pending action item to move it (see the code comment here), at which point they fall into the same category as Secrets and Projected Volumes described above.

Instrumentation

kubelet_volume_requested_resizes_total

This is a new metric that tracks the total number of resize attempts observed by the Kubelet, counted at the volume level. A single pod update changing multiple volumes will increment the counter once for each modified volume.

Labels:

  • medium: The storage medium of the volume (e.g. memory).
  • operation: Whether the resize is an increase, decrease, addition, or removal of the size limit. Possible values: increase, decrease, add, or remove.

This metric is recorded as a counter.

kubelet_pod_resize_duration_milliseconds

This metric already exists in the Kubelet to track the duration of doPodResizeAction, and now also includes memory-backed volume resizes since that function is responsible for actuating both container cgroups and memory-backed volumes.

This metric is recorded as a histogram.

kubelet_pod_infeasible_resizes_total

This already-existing counter metric tracks the total number of resize requests rejected as infeasible. Dynamic memory-backed volume resize increments this counter when a resize request is admitted on a node running cgroups v1.

This proposal adds a new possible label value cgroups_v1_unsupported to the reason_detail label.

  • reason_detail: The reason the resize was marked infeasible (e.g. cgroups_v1_unsupported).

Resize Lifecycle Events

The Kubelet emits lifecycle events for in-place pod resizes (ResizeStarted, ResizeCompleted, ResizeError). The event payload (podResourceSummary) is extended to include memory-backed volumes alongside containers:

{
  "containers": [...],
  "volumes": [{"name": "mem-vol", "emptyDir": {"medium": "Memory", "sizeLimit": "128Mi"}}],
  "generation": 2
}

Test Plan

[X] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

None.

Unit tests

Unit tests in the following packages are added or extended to cover the new logic:

  • pkg/apis/core/validation: Test ValidatePodResize to verify that sizeLimit of memory-backed volumes is mutable on resize, and that adding/removing/renaming volumes or modifying size limits of non-memory volumes is forbidden.
  • pkg/registry/core/pod: Test dropNonResizeUpdates to verify that volume size limit updates are preserved on resize when the feature gate is enabled, and reverted when disabled.
  • pkg/kubelet/allocation: Test allocation manager to verify that sizeLimit of memory-backed volumes is checkpointed as allocated resources, and that cgroups v1 nodes reject resize requests as Infeasible.
  • pkg/kubelet/kuberuntime: Test computeVolumeResizeAction to ensure it detects size differences, doPodResizeAction to verify the strict ordering of cgroup and volume updates, and IsPodResizeInProgress to check volume resize progress.
  • pkg/kubelet/volumemanager: Test ResizeEphemeralVolume to ensure correct delegation to supporting plugins.
  • pkg/volume/emptydir: Test ResizeEphemeralVolume (remount execution on Linux and stubs for unsupported OSes).
  • pkg/kubelet/eviction: Test emptyDirLimitEviction to ensure that memory-backed emptyDir volumes are skipped.
Integration tests

Integration tests in test/integration/pods/pods_test.go (TestNodeDeclaredFeatureAdmission) verify API server admission behavior when InPlacePodVerticalScalingMemoryBackedVolumes is declared or missing on the destination node.

e2e tests

E2E tests in the following suites verify end-to-end functionality:

  • test/e2e/common/node/pod_resize.go: Verifies online upsize and downsize of memory-backed emptyDir volumes alone and concurrently with container resource updates, asserting mount capacity inside the container and verifying that PodResizeInProgress clears.
  • test/e2e/node/pod_resize.go: Serial E2E tests verifying deferred resize behavior for memory-backed volumes under resource pressure.
  • test/e2e_node/pod_resize_criproxy_linux_test.go: Node E2E tests using CRI proxy interception to verify actuation ordering and error handling/retry loops.

For Beta, we plan to add failure on shrink tests for both pod-level and container-level.

Graduation Criteria

Alpha

  • The feature is implemented behind the InPlacePodVerticalScalingMemoryBackedVolumes feature gate.
  • Unit test coverage is complete.
  • E2E tests implemented in CI verify successful upsize and downsize operations in a live cluster.

Beta

  • Gather feedback from users.
  • Metrics are defined and implemented.
  • Kubelet emits a warning event when sizeLimit exceeds pod-level memory limits.
  • The plan evaluates whether integration tests are required.
  • Edge cases around “shrinkage safety” are resolved.

GA

  • The feature is stable in Beta and enabled by default for at least one release.
  • Critical bugs discovered during the Beta phase are resolved.

Upgrade / Downgrade Strategy

Upgrade

Upgrading is non-disruptive. Existing pods continue to run with their original volume sizes. To utilize the feature on existing pods, users send patch requests to the resize subresource.

Downgrade

If the control plane rolls back to a previous version where the feature is disabled:

  • The API server stops accepting updates to sizeLimit via the resize subresource.
  • Pods that are already resized retain their updated sizeLimit in etcd.
  • If these pods are restarted on an older Kubelet, the old Kubelet still reads the updated sizeLimit from the spec during the initial mount phase, starting the pod with the updated size limit.

Version Skew Strategy

Kubelet vs API Server

If the API Server supports the feature but the target Kubelet does not, the NodeDeclaredFeatures framework rejects the resize request during admission.

Adding or removing sizeLimits is not supported in Alpha, but is planned for Beta.

If N-1 Kubelet supports an Alpha version of the feature, but the API server is upgraded to support the Beta version of the feature, the API server will permit add / remove of the sizeLimit field on memory-backed volumes. Due to Kubelet implementation details in N-1, add operations will succeed, but remove operations will be silently ignored until the Kubelet is upgraded.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?
  • Feature gate (also fill in values in kep.yaml)
    • Feature gate name: InPlacePodVerticalScalingMemoryBackedVolumes
    • Components depending on the feature gate: kube-apiserver, kubelet

Enabling or disabling this feature requires setting the InPlacePodVerticalScalingMemoryBackedVolumes feature gate flag on the respective components. This operation requires a restart of the control plane components (kube-apiserver) and the kubelet on each node. It cannot be enabled or disabled dynamically without component restarts.

Does enabling the feature change any default behavior?

Yes. Enabling this feature disables the eviction manager for all emptyDir volumes with medium: Memory.

Pods today (before this feature) already get ENOSPC on write when workloads attempt to surpass the sizeLimit for memory-backed emptyDir volumes. Since the size limit is hard-enforced by the kernel, the eviction manager for memory-backed volumes does not occur for normal operations, so this change in behavior should not impact many users, if any.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes. The feature can be disabled by turning off the feature gate and restarting the components.

Consequences of disablement:

  • If a volume was already resized while the feature was enabled, the actual mounted tmpfs volume retains its new size on the node (we do not automatically shrink it back).
  • The API server rejects further mutations to emptyDir.sizeLimit via the resize subresource.
  • For Pods that were already resized, the Pod spec retains the updated sizeLimit value, but it becomes immutable again.
What happens if we reenable the feature if it was previously rolled back?

If the feature is re-enabled:

  • The API server once again permits mutations to emptyDir.sizeLimit via the resize subresource.
  • The Kubelet resumes its reconciliation logic. If a Pod’s actual mounted volume size differs from the target allocated capacity checkpointed locally (e.g., if a resize is pending or if the spec was updated but not actuated before rollback), the Kubelet proceeds to actuate the resize to match the desired state.
Are there any tests for feature enablement/disablement?

Yes.

  • API Validation Tests: Unit tests in pkg/apis/core/validation verify that mutating emptyDir.sizeLimit is forbidden when the feature gate is disabled and permitted when it is enabled.
  • Kubelet Tests: Unit tests in the Kubelet (specifically in KuberuntimeManager and the eviction manager) verify that actuation and enhanced monitoring logic are skipped or fall back to default behavior when the feature gate is disabled.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

Rollout Failure Scenarios and Impact:

  • Node-Declared Feature Gating (Skew Prevention): If a client attempts to mutate emptyDir.sizeLimit on a Pod scheduled on a Kubelet that does not support the feature (or does not have the feature gate enabled), the API Server automatically rejects the patch via Node Declared Features. This prevents skew issues where a spec is modified but cannot be actuated by the Kubelet.
  • Kubelet Crash or Restart Mid-Actuation: If the Kubelet restarts or crashes after writing the local AllocationManager checkpoint but before executing the dynamic tmpfs remount syscall, the Kubelet’s startup reconciliation loop automatically discovers the discrepancy between the checkpointed state (allocated limit) and the actual host mount parameters, and it will safely re-attempt and complete the remount.
  • Actuation Failure: If the mounter’s remount syscall fails, the volume stays at its previous size limit. The actual mounted volume capacity remains out-of-sync with the desired spec (surfaced via the PodResizeInProgress condition with Reason: Error), but the running workload continues to execute safely at the old size.

Rollback Failure Scenarios and Impact:

  • Pending Resize State Mismatch (Disablement Skew): If the feature gate is disabled while a Pod has an active, pending resize operation (i.e., the Pod spec has been updated to a new sizeLimit but the Kubelet has not yet completed the mount/remount actuation):
    • Once the feature gate is disabled, the Kubelet stops processing dynamic volume updates. The actual volume size remains stuck at the old limit, creating a permanent discrepancy between the desired state in the Pod spec and the actual state on the node.
    • The PodResizeInProgress condition will no longer reflect the discrepancy and may be cleared despite the desired and actual volume size mismatch, as the Kubelet will no longer check the volume size as part of its reconciliation loop.
  • Workload Safety: Disabling the feature gate does not automatically shrink already-resized volumes back to their original limits.
What specific metrics should inform a rollback?

Operators should monitor the following signals to identify anomalies that might require a rollback:

  • API Server Metrics:
    • A high rate of client request errors (4xx or 5xx response codes) on the Pod resize subresource endpoint, tracked via the apiserver_request_total{subresource="resize", verb="PATCH"} counter.
  • Kubelet Volume Operation Metrics:
    • An increase in failure rates for volume mount or filesystem resize operations, tracked via the storage_operation_duration_seconds_count{volume_plugin="kubernetes.io/empty-dir", operation_name=~"volume_mount|volume_fs_resize", status=~"fail-.*"} metric.
    • A failure of actual reported capacity to transition to the newly desired capacity, tracked via the kubelet_volume_stats_capacity_bytes{volume_plugin="kubernetes.io/empty-dir"} gauge.
Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

The following manual verification plan will validate the behavior during development and testing:

  • Upgrade Path Test:
    1. Deploy a cluster with the InPlacePodVerticalScalingMemoryBackedVolumes feature gate disabled.
    2. Create a Pod with a memory-backed emptyDir volume configured with a static sizeLimit: 128Mi.
    3. Enable the feature gate and restart the control plane and Kubelet components.
    4. Submit a dynamic resize request targeting 256Mi. Verify that the Kubelet successfully remounts the tmpfs filesystem, the host mount reflects 256Mi via df, and the PodResizeInProgress condition clears.
  • Downgrade/Rollback Path Test:
    1. With the feature gate active, create a Pod with a memory-backed volume of 128Mi and dynamically resize it to 256Mi.
    2. Write 200Mi of files to the volume to confirm active capacity utilization.
    3. Disable the feature gate and restart the API servers and Kubelets.
    4. Verify that the running Pod’s memory volume is not disrupted, the files remain fully readable/writeable, and the mounted directory keeps its 256Mi capacity.
    5. Attempt a new dynamic volume resize update and verify that the API server returns a validation error rejecting the request.
  • Re-upgrade/Re-enablement Path Test:
    1. Re-enable the feature gate and restart the Kubelets and API servers.
    2. Verify that the API server resumes accepting mutations on the subresource and that the Kubelet resumes full dynamic actuation.
Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No. This feature is strictly additive and introduces new dynamic capabilities for existing fields and volume types. No existing APIs, fields, flags, or default volume behaviors are deprecated or removed as part of this enhancement. Legacy static workloads remain fully compatible.

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

The Kubelet exposes the Prometheus metric kubelet_volume_requested_resizes_total with label medium="memory". Operators can query this metric to track the total count of memory-backed volume resize requests processed across the cluster.

How can someone using this feature know that it is working for their instance?
  • Events
    • Event Reason: ResizeStarted (emitted when kubelet accepts a resize request, along with the new requested size)
    • Event Reason: ResizeCompleted (emitted when kubelet successfully resizes the volume)
    • Event Reason: ResizeError (emitted when kubelet encounters an error attempting to remount the volume, e.g., kernel rejection on shrinkage below current usage)
    • Event Reason: ResizeInfeasible (emitted when a resize request cannot be admitted, e.g., on nodes running cgroups v1)
  • API .status
    • Condition name: PodResizeInProgress (transitions to True during active resize and cleared upon successful completion)
  • Other (treat as last resort)
    • Details: Users can run df inside their container on the volume mount point to inspect the actual kernel-enforced mount size.
What are the reasonable SLOs (Service Level Objectives) for the enhancement?
  • Resize requests should succeed (kubelet_pod_resize_duration_milliseconds{error="true"} should be low)
  • Volume resize operations should complete quickly (kubelet_pod_resize_duration_milliseconds{error="false"} < 3000ms for 99% of requests)
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
  • Metrics
    • Metric name: kubelet_pod_resize_duration_milliseconds
      • Components exposing the metric: Kubelet
    • Metric name: kubelet_volume_requested_resizes_total
      • Components exposing the metric: Kubelet
    • Metric name: kubelet_pod_infeasible_resizes_total
      • Components exposing the metric: Kubelet
Are there any missing metrics that would be useful to have to improve observability of this feature?

None.

Dependencies

Does this feature depend on any specific services running in the cluster?

This feature depends on the NodeDeclaredFeatures feature.

Scalability

Will enabling / using this feature result in any new API calls?

Yes.

  • API call type: Standard PATCH /status calls during Pod status reconciliation to reflect the transition of the PodResizeInProgress condition.
  • Estimated throughput: Extremely low. Resizing memory-backed volumes is an infrequent, on-demand event triggered manually by cluster administrators or periodically by vertical autoscalers. It is not a standard high-frequency workload operation.
  • Originating component(s): Kubelet (via PodStatus updates).
  • Listing / Watching: No new listing or watching of resources is introduced.
Will enabling / using this feature result in introducing new API types?

No. No new API types are introduced. The feature purely extends existing fields and makes the existing Pod.Spec.Volumes[].EmptyDir.SizeLimit mutable for memory-backed volumes.

Will enabling / using this feature result in any new calls to the cloud provider?

No. Volume resizing is strictly local and does not require any external cloud provider storage controller interactions.

Will enabling / using this feature result in increasing size or count of the existing API objects?

No.

  • API Type(s): None.
  • Estimated increase in size: None. No new fields are added to Pod.Spec or Pod.Status. The existing sizeLimit field under EmptyDirVolumeSource is simply made mutable.
  • Estimated amount of new objects: None.
Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

No.

  • Resizing memory-backed emptyDir volumes is performed synchronously in the Kubelet pod sync loop via lightweight Linux system calls (mount -o remount).
  • The duration of the remount operation is on the order of microseconds/milliseconds and does not block pod creation, container startup, or container termination.
  • There is zero impact on scheduling latency, pod startup latency, or any other standard Kubernetes SLIs/SLOs.
Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

No.

  • Kubelet: The memory and CPU overhead for tracking the actuated volume state is negligible. The local Kubelet checkpoint footprint increases by a negligible amount (a few bytes) to store the internal allocated state.
  • kube-apiserver / etcd: The etcd storage footprint increases negligibly. No additional CPU or memory overhead is introduced.
  • Node RAM/CPU: Physical memory consumption remains governed entirely by the container/pod memory cgroups and standard kernel-level tmpfs quota enforcement.
Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

No.

  • Resizing acts on existing mount points by modifying their options. No new mount points, file descriptors, or socket connections are created.
  • The inode limit on tmpfs mounts remains governed by kernel defaults or specific sizeLimit attributes, ensuring no new paths for resource exhaustion are introduced.

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

If the API server or etcd is unavailable, users will not be able to submit new resize requests. Existing requests will be processed by the Kubelet, but the pod status will not be updated with the resize status.

Once the API server or etcd is available again, new requests will be accepted and processed normally.

Note: Like In-Place Pod Vertical Scaling in general, dynamic resize of memory-backed volumes is not supported for static (mirror) Pods, as static pods cannot be mutated via the /resize subresource.

What are other known failure modes?

  • Volume shrinkage below current in-use memory:

    • Detection: The PodResizeInProgress condition in pod status is set to Status: True, Reason: Error with a message from the mount failure (e.g., tmpfs: Too small a size for current use.), and a Warning ResizeError event is emitted on the pod.
    • Mitigations: Running containers are unaffected (0 container restarts) and the volume remains mounted at its previous capacity. The workload can delete files to reduce usage, or the user/autoscaler can update the pod spec with a higher sizeLimit or cancel the resize.
    • Diagnostics: Kubelet logs at v=2 with Resizing emptyDir volume and mount error details.
    • Testing: Covered by unit tests in pkg/volume/emptydir and node E2E tests verifying retry behavior and condition reporting on shrink failures.

  • Resize requested on a node running cgroups v1:

    • Detection: The resize request is marked Infeasible in Pod status (PodResizePending: Infeasible), the metric kubelet_pod_infeasible_resizes_total is incremented, and a Warning ResizeInfeasible event is emitted on the Pod.
    • Mitigations: Migrate the node to cgroups v2 (cgroups v1 has been deprecated in Kubernetes since 1.28). Users or autoscalers can revert the resize request to clear the condition.
    • Diagnostics: Kubelet logs a warning during admission: in-place volume resize is not supported on cgroups v1.
    • Testing: Covered by unit tests in pkg/kubelet/allocation (TestDisallowResizeForMemoryBackedVolumesOnCgroupV1).
What steps should be taken if SLOs are not being met to determine the problem?
  • Inspect the PodResizeInProgress condition and Pod warning events (ResizeError, VolumeSizeExceedsPodMemoryLimit).
  • Check Kubelet logs for volume remount failures or container runtime cgroup update errors.
  • Query Kubelet metrics: kubelet_volume_requested_resizes_total, kubelet_pod_resize_duration_milliseconds, and kubelet_pod_infeasible_resizes_total.

Implementation History

  • 2026-04-30: Initial alpha KEP submitted.
  • 2026-08-25: Correct the alpha KEP based on what was actually implemented.
  • 2026-09-01: Update KEP for Beta graduation in 1.38.

Drawbacks

No drawbacks have been identified.

Alternatives

Volume Reconciler detects changes (Async approach)

  • Description: Relying on the Volume Manager’s periodic reconciler to detect changes in emptyDir.sizeLimit (via RequiresRemount) and trigger the remount asynchronously.
  • Why ruled out: It makes coordination with cgroup updates extremely difficult. Cgroup updates are handled synchronously in kuberuntime’s SyncPod flow. Attempting to coordinate strict ordering (upsize: cgroups first; downsize: volume first) between a synchronous flow and an asynchronous polling loop introduces complex state management and potential race conditions.

Moving emptyDir handling entirely to KuberuntimeManager

  • Description: Handling creation and resizing of memory-backed emptyDir volumes directly in KuberuntimeManager.
  • Why ruled out:
    • Violation of Concerns: KuberuntimeManager manages containers and cgroups via CRI, not volumes.
    • Architectural Fragmentation: Special-cases emptyDir and breaks the unified volume lifecycle model used for all other volumes.
    • Metrics Breakage: Bypassing Volume Manager breaks centralized metrics collection, requiring duplicated filesystem monitoring logic in KuberuntimeManager to feed the stats API.

Restricting sizeLimit to not exceed memory limits

  • Description: Not allowing users to adjust sizeLimit to exceed pod or container level memory limits as part of this KEP.
  • Why ruled out: This was deemed out of scope for this KEP since this behavior is allowed during pod creation today. Enforcing this restriction only during resize updates would introduce inconsistency in API behavior.

Non-atomic handling of volume and resource resizes

  • Description: Treating updates to volume sizeLimit and container/pod resource limits as separate, non-atomic transactions.
  • Why ruled out: This introduces a risk of partial resizes. If a user updates both expecting the volume to maintain a certain proportion of the container’s memory limit, a failure or delay in applying the cgroup limit while the volume is expanded could lead to unexpected OOM kills if the application attempts to fill the volume.