KEP-4438: Restarting sidecar containers during Pod termination

Implementation History
ALPHA Implementable
Created 2024-01-25
Updated 2025-01-28
Latest v1.33
Milestones
Alpha v1.33
Beta v1.34
Stable v1.35
Related Enhancements
Ownership
Owning SIG
SIG Node
Participating SIGs
Scheduler
Primary Authors
@SergeyKanzhelev @matthyx

KEP-4438: Restarting sidecar containers during Pod termination

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

Sidecar containers should be restarted if they exit prematurely when a Pod is terminated to ensure that they are running while the containers that should terminate prior to its termination are still running. This feature was originally planned for beta release in the initial KEP, but after further analysis, we decided to postpone and introduce a separate feature gate for it.

Motivation

The reason for this KEP is that restarting sidecar containers during Pod termination requires fundamental changes to the Pod lifecycle, and such changes cannot be introduced in beta (enabled by default) without risking disruption to users.

On the other hand, the code for sidecar containers is already well tested, and we are confident that many use cases will benefit from them even without the restart during Pod termination.

For this reason, we are introducing a separate feature gate for the sidecar containers KEP to decouple the two features and allow users to use sidecar containers without the refactoring required for the restart during Pod termination.

Goals

The following behaviors should be maintained during pod termination:

  • sidecar containers restarting
  • liveness, readiness and startup probing
  • container lifecycle hooks running
  • service account token rotation
  • secret and configmap volume updates

Also, container termination should be non-blocking, which will fix issue #121398 a container cannot restart when there is any terminating container in the same pod.

Non-Goals

Proposal

The proposal is to introduce a new feature gate for the sidecar containers KEP to decouple the sidecar feature from the restart during Pod termination feature and allow users to use sidecar containers without the refactoring required for the restart during Pod termination.

Please refer to the original KEP for the details of the sidecar containers feature: https://git.k8s.io/enhancements/keps/sig-node/753-sidecar-containers

User Stories (Optional)

Story 1

Story 2

Notes/Constraints/Caveats (Optional)

Risks and Mitigations

The main risk of this proposal is that the lifecycle of sidecar containers will be changed, which could potentially break existing behavior. To mitigate this risk, we will introduce several e2e tests and peer review to ensure that the changes are safe.

There is no security impact of this proposal.

The UX impact of this proposal is minimal, as it adds an expected behavior to the lifecycle of sidecar containers.

Design Details

Test Plan

[X] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

The test plan is currently in progress and will be reflected here as we make progress: https://docs.google.com/document/d/14En0jesyZFrDG8Epn1mYcWB0dSZqCLv-HwtIdMuov0o/edit?usp=sharing

Prerequisite testing updates
Unit tests
  • pkg/kubelet/kuberuntime: 2024/02/06 - 66.8%
Integration tests
  • :
e2e tests
Existing tests
New tests

Probes:

  • Readiness probes are still running while in preStop
  • Readiness status is beings updated for the container and the Pod while in preStop
  • Liveness probes are NOT running for regular containers while the Pod is terminating
  • SIDECAR: Liveness probes DO run for sidecar containers while the Pod is terminating
  • SIDECAR: sidecar container will be restarted when liveness probe failed during Pod termination

Not fully started containers:

  • preStop will not be executed for the container that hasn’t started yet
  • preStop will be called on the container even if postStart is still running
  • postStart hook CONTINUE EXECUTE even if container started termination
  • postStart hook will stop once pod passed it’s graceful termination period

Pod with some containers terminated:

  • BUGFIX, SIDECAR: Container can be restarted when there are terminating containers in the Pod A container cannot restart when there is any terminating container in the same pod · Issue #121398

Re-terminating the Pod:

  • When the Pod is terminating, another call to terminate the pod with the smaller grace period will override the grace period to terminate Pod faster
  • When the Pod is terminating, another call to terminate the pod with the greater grace period will override the grace period to allow longer termination
  • BUGFIX: Service account token gets invalidated while terminating pod is re-deleted · Issue #122568

Pre-stop vs. SIGTERM traps:

  • Same as existing and above tests, need to validate that the container that traps the SIGTERM behaves the same way as with preStop:
  • Respect the grace period
  • Liveness probes are not running
  • Readiness probes are running

Test what is available for during preStop:

  • BUGFIX: While the Pod is terminating, service account tokens are rotated Kubelet stops rotating service account tokens when pod is terminating, breaking preStop hooks · Issue #116481
  • BUGFIX: Service account token is valid if the terminating Pod was deleted again Service account token gets invalidated while terminating pod is re-deleted · Issue #122568

Eviction and OOM kills:

  • preStop is called when Pod is evicted
  • preStop is NOT called when Container is OOMkilled

Graduation Criteria

Alpha

  • Allow sidecar containers to restart during the shutdown of the Pod.
  • Enable livenessProbe, readinessProbe and startupProbe during the shutdown of the Pod for sidecar containers that restart.
  • Enable postStart and preStop hooks during the shutdown of the Pod for sidecar containers that restart.

Beta

TBD

GA

TBD

Upgrade / Downgrade Strategy

This feature only concerns the kubelet, so the upgrade and downgrade strategy is limited to the kubelet. Moreover, the Pod spec is not altered, so no changes are required for existing workloads to make use of the feature. Likewise, no changes are required for these workloads to revert to previous behavior.

Version Skew Strategy

There is no version skew strategy for this feature. The kubelet is the only component that needs to be updated to make use of this feature.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?
  • Feature gate (also fill in values in kep.yaml)
    • Feature gate name: RestartContainerDuringTermination
    • Components depending on the feature gate:
      • kubelet
  • Other
    • Describe the mechanism:
    • Will enabling / disabling the feature require downtime of the control plane?
    • Will enabling / disabling the feature require downtime or reprovisioning of a node?
Does enabling the feature change any default behavior?

Enabling the feature will change the behavior of the kubelet when terminating a Pod with sidecar containers. Sidecar containers that exit prematurely will be restarted during the termination of the Pod to ensure they are running until the main containers that should terminate prior to the sidecar containers are still running.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes, the feature can be disabled once it has been enabled. There is no alteration to the Pod spec, so existing workloads will be terminated according to the current behavior, after the kubelet is restarted with the feature gate disabled.

What happens if we reenable the feature if it was previously rolled back?

No side effect, the feature can be switched on or off.

Are there any tests for feature enablement/disablement?

Yes, unit tests will be added to ensure the feature can be enabled and disabled. The KEP will be updated with the details of the tests as they are added.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?
What specific metrics should inform a rollback?
Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?
How can someone using this feature know that it is working for their instance?
  • Events
    • Event Reason:
  • API .status
    • Condition name:
    • Other field:
  • Other (treat as last resort)
    • Details:
What are the reasonable SLOs (Service Level Objectives) for the enhancement?
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
  • Metrics
    • Metric name:
    • [Optional] Aggregation method:
    • Components exposing the metric:
  • Other (treat as last resort)
    • Details:
Are there any missing metrics that would be useful to have to improve observability of this feature?

Dependencies

Does this feature depend on any specific services running in the cluster?

Scalability

Will enabling / using this feature result in any new API calls?
Will enabling / using this feature result in introducing new API types?
Will enabling / using this feature result in any new calls to the cloud provider?
Will enabling / using this feature result in increasing size or count of the existing API objects?
Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?
Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?
What are other known failure modes?
What steps should be taken if SLOs are not being met to determine the problem?

Implementation History

  • 2024-01-30: Summary and Motivation sections merged
  • 2024-02-08: Proposal section merged, KEP marked as implementable
  • v1.33: Alpha release

Drawbacks

The main drawback of this KEP is that it introduces a new feature gate for the sidecars, which can be confusing for users. However, we believe that the current behavior of sidecar containers is already useful and that the restart during Pod termination feature is not critical for many use cases. This is why this feature is introduced as a separate feature gate, so that KEP-753 can reach GA faster.

Alternatives

The alternative would be to introduce KEP-753 with the restart during Pod termination feature. However, this would have required a significant refactoring of the kubelet and the Pod lifecycle, which would introduce a risk of disruption to users. This is why we decided to introduce a separate feature gate for the sidecar containers feature.

Infrastructure Needed (Optional)