KEP-5365: ImageVolume with an image digest

• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • User Stories (Optional)
    • Story 1
    • Story 2
    • Story 3
  • Notes/Constraints/Caveats (Optional)
  • Risks and Mitigations
• Design Details
  • API Changes
  • Test Plan
    • Prerequisite testing updates
    • Unit tests
    • Integration tests
    • e2e tests
  • Graduation Criteria
    • Alpha
    • Beta
    • GA
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
• Production Readiness Review Questionnaire
  • Feature Enablement and Rollback
  • Rollout, Upgrade and Rollback Planning
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Alternatives
• Infrastructure Needed (Optional)

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
  • e2e Tests for all Beta API Operations (endpoints)
  • (R) Ensure GA e2e tests meet requirements for Conformance Tests
  • (R) Minimum Two Week Window for GA e2e tests to prove flake free
• (R) Graduation criteria is in place
  • (R) all GA Endpoints must be hit by Conformance Tests
• (R) Production readiness review completed
• (R) Production readiness review approved
• “Implementation History” section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

The ImageVolume feature, implemented in KEP-4639 , introduces a new way to define volumes for pods using a container image as the source. The image can be specified by its name and tag, or by its digest, alongside a pull policy, just like a container image is being specified in a pod spec.

However, a difference between ImageVolume and a “standard” container image is that the image digest for container images is advertised in the pod’s status as part of the containerStatuses field. As opposed to that, the digest of the image used for an ImageVolume is not advertised in the pod’s status. This KEP aims to close this gap, by adding a new field to the pod’s status that will contain the digest of the image.

Motivation

A container image digest is a unique identifier for a specific version of a container image. It is a hash of the image’s content, which serves as a unique fingerprint for that image, ensuring that it is the exact version that originally used by the container. This is unlike a tag, which can be reused or changed over time.

Just like container digests are advertised in the pod’s status for containers, the same should be done for ImageVolumes with very similar use-cases, like reporting the image used through monitoring or debug tools, supporting git-ops workflows that rely on the image digest, and more (see Goals section below).

A special use-case for this KEP is using live-migration in Kubevirt. Kubevirt runs VMs inside pods, and supports to move a VM instance between nodes. Behind the scenes, this is done by creating a similar pod (with the same images), then transferring the VM state. An image digest would ensure that the same image is used on both nodes, which is crucial for live-migrations to succeed.

Goals

• Add a new field to the pod’s status that will contain the digest of the image used for ImageVolumes.

Non-Goals

• This KEP does not aim to change the way ImageVolumes are defined in the pod spec. The image digest will be added to the pod’s status, but the pod spec will remain unchanged.

Proposal

User Stories (Optional)

Story 1

As a user, I want to monitor which images are used by my pods, so I can ensure that the correct versions are running. This is important for ensuring the images are updated, not containing CVEs, etc.

Story 2

As a user, I want to use git-ops workflows that rely on the image digest, so I can ensure that the correct versions are running.

Story 3

As a user, I want to use live-migration in Kubevirt relying on ImageVolumes, so I can move my VM instances between nodes without downtime.

Notes/Constraints/Caveats (Optional)

Risks and Mitigations

Design Details

API Changes

This KEP proposes to add an ImageRef to the VolumeMountStatus struct (which is part of the pod’s status):

// VolumeMountStatus shows status of volume mounts.
type VolumeMountStatus struct {
	// Name corresponds to the name of the original VolumeMount.
	Name string `json:"name" protobuf:"bytes,1,opt,name=name"`
	// MountPath corresponds to the original VolumeMount.
	MountPath string `json:"mountPath" protobuf:"bytes,2,opt,name=mountPath"`
	// ReadOnly corresponds to the original VolumeMount.
	// +optional
	ReadOnly bool `json:"readOnly,omitempty" protobuf:"varint,3,opt,name=readOnly"`
	// RecursiveReadOnly must be set to Disabled, Enabled, or unspecified (for non-readonly mounts).
	// An IfPossible value in the original VolumeMount must be translated to Disabled or Enabled,
	// depending on the mount result.
	// +featureGate=RecursiveReadOnlyMounts
	// +optional
	RecursiveReadOnly *RecursiveReadOnlyMode `json:"recursiveReadOnly,omitempty" protobuf:"bytes,4,opt,name=recursiveReadOnly,casttype=RecursiveReadOnlyMode"`
	// ImageRef is the digest of the image used for this volume.
	// It should have a value that's similar to the pod's status.containerStatuses[i].imageID.
	// If the volume source is not an ImageVolume, this field will be empty.
	ImageRef *string `json:"imageRef,omitempty" protobuf:"bytes,5,opt,name=imageRef"`
}

Note that the ImageRef field is a pointer and should be omitted whenever the volume source is not an ImageVolume. In addition, during admission, it should be disallowed to populate the ImageRef field for non-ImageVolume sources.

In addition, the KEP proposes to add an ImageRef field to the ImageSpec cri-api struct :

// ImageSpec is an internal representation of an image.
message ImageSpec {
    // Container's Image field (e.g. imageID or imageDigest). Might not contain the image's digest.
    string image = 1;
    // Unstructured key-value map holding arbitrary metadata.
    // ImageSpec Annotations can be used to help the runtime target specific
    // images in multi-arch images.
    map<string, string> annotations = 2;
    // The container image reference specified by the user (e.g. image[:tag] or digest).
    // Only set if available within the RPC context.
    string user_specified_image = 18;
    // Runtime handler to use for pulling the image.
    // If the runtime handler is unknown, the request should be rejected.
    // An empty string would select the default runtime handler.
    string runtime_handler = 19;
    // The digest of the image used for this volume.
    string image_ref = 20;
}

Test Plan

Since we’re proposing a CRI-API change, we’ll need to test this feature with both containerd and CRI-O runtimes.

[X] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

This feature is pretty straightforward to test. Basically what needs to be validated is that whenever the feature is enabled, the image digest is added to the pod’s status when an ImageVolume is used.

Unit tests

Code is not implemented yet, so this is only an initial assessment of the unit test coverage. In general, we expect to have unit tests somewhere around:

• pkg/kubelet/status/: 2025-06-18 - coverage: 86.8% of statements
• pkg/volume/image/: 2025-06-18 - coverage: 20.0% of statements
• pkg/kubelet: 2025-06-18 - coverage: 70.2% of statements
• pkg/api/pod: 2025-06-18 - coverage: 80.5% of statements

Integration tests

No new integration tests for kubelet are planned.

e2e tests

Not sure there’s a need for e2e tests for this feature. In general we need to validate that the image digest is added to the pod’s status when an ImageVolume is used.

• test name : SIG … , triage search

Graduation Criteria

Alpha

• Feature implemented behind a feature flag
• Initial e2e tests completed and enabled

Beta

• Feature is supported by both containerd and CRI-O runtimes

GA

• Add a conformance test that ensures the cluster will properly set the field when expected and not set when not expected.

Upgrade / Downgrade Strategy

Upgrade: Feature gate is off by default in Alpha.

Downgrade: The image digest field will be omitted from the pod’s status.

Version Skew Strategy

The CRI would have to support the new ImageRef field in the VolumeMountStatus struct that had been added to the CRI-API. If the CRI does not support this field, the kubelet would not be able to populate it in the pod’s status, which means that the image digest would not be available in the pod’s status.

This feature affects only kubelet and the CRI, no other components are affected.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

• Feature gate (also fill in values in kep.yaml)
  • Feature gate name: ImageVolumeWithDigest.
  • Components depending on the feature gate: kubelet.
• Other
  • Describe the mechanism: Kubelet needs to be restarted when enabling/disabling this feature.
  • Will enabling / disabling the feature require downtime of the control plane? No.
  • Will enabling / disabling the feature require downtime or reprovisioning of a node? Yes.

Does enabling the feature change any default behavior?

By default, the image ID should be added to the pod’s status when the feature is enabled and an ImageVolume is used. This should not break any existing flows / behaviors.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

The feature gate can be turned off. Kubelet would be needed to be restarted for the image digest to be removed from the pod’s status.

What happens if we reenable the feature if it was previously rolled back?

After re-enabling the feature, any new pod that would be created would include the image digest in its status. Vice-versa for disabling the feature.

Are there any tests for feature enablement/disablement?

After the feature gate is enabled, the image digest should be added to the pod’s status when an ImageVolume is used. Vice-versa for disabling the feature. Eventually, when it is GAed, the digest will always be added to the pod’s status when an ImageVolume is used.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

The worst case scenario is that some pods would have the image digest in their status, while others would not. This is not expected to cause any issues or break existing behavior.

What specific metrics should inform a rollback?

Users can monitor failures around updating the pod’s status, or failures around creating pods with ImageVolumes. This could be a good indication that the feature is not working as expected.

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

A unit test will be added to verify that the kubelet correctly transitions between enabled and disabled states of the feature gate, ensuring safe handling of PodStatus data across upgrades and rollbacks.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No.

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

By checking the pod’s status for the ImageRef field in the VolumeMountStatus struct for pods that use ImageVolumes. We can consider using more metrics to track the usage of this feature if people think it is useful.

How can someone using this feature know that it is working for their instance?

• Events
  • Event Reason:
• API .status
  • Condition name:
  • Other field: pod.status.volumeMounts[i].ImageRef
• Other (treat as last resort)
  • Details:

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

kubelet should not fail to report pod status at any time - feature on or off. In general, pod status updates should not fail.

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

find the kube apiserver metric for number of requests failing for pods/status.

• Metrics
  • Metric name:
  • [Optional] Aggregation method:
  • Components exposing the metric:
• Other (treat as last resort)
  • Details:

Are there any missing metrics that would be useful to have to improve observability of this feature?

No.

Dependencies

This feature depends on the ImageVolume feature, which is implemented in KEP-4639.

Does this feature depend on any specific services running in the cluster?

• CRI
  • The CRI must support the new ImageRef field in the VolumeMountStatus struct that had been added to the CRI-API.
  • This is not yet supported by CRI-O or containerd, but both runtimes are expected to support it in the future.
• Kubelet
  • Should be able to handle the new ImageRef field in the VolumeMountStatus struct and populate it in the pod’s status.
• kube-apiserver
  • The API server has to be available in order to update the pod’s status with the new ImageRef field.

Scalability

Scalability issues are not expected.

Will enabling / using this feature result in any new API calls?

No, just for another field in a cri-api call.

Will enabling / using this feature result in introducing new API types?

Yes, a new image digest field will be added to the VolumeMountStatus struct in the pod’s status.

Will enabling / using this feature result in any new calls to the cloud provider?

No.

Will enabling / using this feature result in increasing size or count of the existing API objects?

Additional field.

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

No.

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

No.

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

No.

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

If the API server is unavailable, the kubelet will not be able to update the pod’s status with the image digest.

What are other known failure modes?

This feature depends on the CRI to function correctly and populate the ImageRef field in the VolumeMountStatus struct. If the CRI does not support this field, the kubelet will not be able to populate it in the pod’s status and it will be omitted.

Theoretically, if there’s a bug in the CRI, a wrong image digest could be reported in the pod’s status.

What steps should be taken if SLOs are not being met to determine the problem?

If the SLOs are not being met, the kubelet should be checked for errors around updating the pod’s status. The feature gate can be disabled to roll-back.

Implementation History

Drawbacks

Alternatives

Infrastructure Needed (Optional)