KEP-2258: Node log query

Release Signoff Checklist
Summary
Motivation
- Goals
- Non-Goals
Proposal
Design Details
Production Readiness Review Questionnaire
Implementation History
Drawbacks
Alternatives
Infrastructure Needed (Optional)

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

(R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
(R) KEP approvers have approved the KEP status as implementable
(R) Design details are appropriately documented
(R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
- e2e Tests for all Beta API Operations (endpoints)
- (R) Ensure GA e2e tests meet requirements for Conformance Tests
- (R) Minimum Two Week Window for GA e2e tests to prove flake free
(R) Graduation criteria is in place
- (R) all GA Endpoints must be hit by Conformance Tests within one minor version of promotion to GA
(R) Production readiness review completed
(R) Production readiness review approved
“Implementation History” section is up-to-date for milestone
User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

A Kubernetes cluster administrator has to log in to the relavant control-plane or worker nodes to view the logs of the API server, kubelet etc. Or they would have to implement a client side reader. A simpler and more elegant method would be to allow them to use a kubelet API or kubectl plugin to also view these logs similar to using it for other interactions with the cluster. Given the sensitive nature of the information in node logs, this feature will only be available to cluster administrators.

Motivation

Troubleshooting issues with control-plane and worker nodes typically requires a cluster administrator to SSH into the nodes for debugging. While certain issues will require being on the node, issues with the kube-proxy or kubelet, to name a couple, could be solved by perusing their logs. However this too requires the administrator to SSH access into the nodes. Having a way for them to view the logs using a kubelet API or kubectl plugin will significantly simplify their troubleshooting.

Goals

Provide a cluster administrator with a streaming view of logs using a kubelet API without them having to implement a client side reader or logging into the node. This would work for:

Services on Linux worker and control plane nodes:
- That have systemd / journald support.
- That have services that log to /var/log/
Windows worker nodes (all supported variants) that log to C:\var\log, and Application logs.

Non-Goals

Providing support for non-systemd Linux distributions.
Reporting logs for nodes that have config or connection issues with the cluster.
Getting logs from services that do not use /var/log/.

Proposal

Implement client for logs endpoint (OS agnostic)

Implement a client for the /proxy/logs/ kubelet endpoint viewer.

Linux distributions with systemd / journald

Supplement the the /proxy/logs/ endpoint viewer on the kubelet with a thin shim over the journal directory that shells out to journalctl.

Linux distributions without systemd / journald

Running the new “kubectl node-logs” command against services on nodes that do not use systemd / journald should return “OS not supported”. However getting logs from /var/log/ should work on all systems.

Windows

Reuse the kubelet API for querying the Linux journal for invoking the Get-WinEvent cmdlet in a PowerShell.

User Stories

Consider a scenario where pods / containers are refusing to come up on certain nodes. As mentioned in the motivation section, troubleshooting this scenario involves the cluster administrator to SSH into nodes to scan the logs. Allowing them to use the kubelet API to do the same as they would to debug issues with a pod / container would greatly simply their debug workflow. This also opens up opportunities for tooling and simplifying automated log gathering. The feature can also be used to debug issues with Kubernetes services especially in Windows nodes that run as native Windows services and not as DaemonSets or Deployments.

Here are some example of how a cluster administrator would use this feature:

# Fetch kubelet logs from a node named node-1.example
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"

# Fetch kubelet logs from a node named node-1.example that have the word "error"
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet&pattern=error"

# Display foo.log from a node name node-1.example
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=/foo.log

Risks and Mitigations

Large log files and events

If the log that is attempted to be viewed is very large (GBs) there is potential for the node performance to be degraded. To mitigate this we only allow returning of messages that can be retrieved within 30 seconds.

Wider access to all node level service logs

The cluster administrator can now view all logs in /var/log/, systemd/journald services and Windows services. Given that the cluster administrator can log into the nodes and view the same information this should not be an issue. However there is potential for scenarios where the cluster administrator does not have access to the infrastructure. This again would benefit from real world usage feedback.

Design Details

kubelet

The kubelet already has a /var/log/ endpoint viewer that is lacking a client. Given its existence we can supplement that with a wafer thin shim that shells out to journalctl. This allows us to extend the endpoint for getting logs from the system journal on Linux systems that support systemd. To enable filtering of logs, we can reuse the existing filters supported by journalctl.

On the Windows side viewing of logs from services that use C:\var\log will be supported by the existing endpoint. For Windows services that log to the the Application logs,we can leverage the Get-WinEvent cmdlet that supports getting logs from all these sources. The cmdlet has filtering options that can be leveraged to filter the logs in the same manner we do with the journal logs.

Please note that filtering will not be available for logs in /var/log/ or C:\var\log\.

The complete list of options that can be used are:

Option	Description
`boot`	boot show messages from a specific system boot
`pattern`	pattern filters log entries by the provided PERL-compatible regular expression
`query`	query specifies services(s) or files from which to return logs (required)
`sinceTime`	an RFC3339 timestamp from which to show logs (inclusive)
`untilTime`	an RFC3339 timestamp until which to show logs (inclusive)
`tailLines`	specify how many lines from the end of the log to retrieve; the default is to fetch the whole log

The feature now enables the cluster administrator to interrogate all services. This could be prevented by having a whitelist of allowed services. But this comes with severe disadvantages as there could be nodes (especially with Windows) that have other services to support networking and monitoring. These services are variable and will depend on how the nodes have been configured. Here are some examples:

The /var/log/ endpoint is enabled using the enableSystemLogHandler kubelet configuration options. To gain access to this new feature, this option and a newly introduced enableSystemLogQuery needs to be enabled. In addition when introducing this feature it will be hidden behind a NodeLogQuery feature gate in the kubelet that needs to be explicitly enabled. So you need to enable both options to get access to this new feature. Disabling enableSystemLogQuery will disable the new feature irrespective of the NodeLogQuery feature gate.

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

Unit tests

Add unit tests to kubelet and kubectl that exercise the new arguments that have been added.

Given that a new kubelet package is introduced as part of this feature there is no existing test coverage to link to.

Integration tests

Given that we need the kubelet running locally to test this feature, integration tests will not be possible for this feature.

e2e tests

Tests have been added that query the kubelet service logs on Linux nodes and Microsoft-Windows-Security-SPP logs on Windows nodes with various options.

These tests are part of the kubelet node e2e tests that are run as a daily periodic job:

https://testgrid.k8s.io/sig-windows-master-release#capz-master-windows-alpha-nodelogquery

This job runs tests against both Windows and Linux nodes.

Graduation Criteria

Alpha

Feature implemented behind a feature flag (NodeLogQuery)
Initial e2e tests completed and enabled
Feature introduced in v1.27 behind the NodeLogQuery kubelet feature gate and enableSystemLogQuery kubelet option

Alpha -> Beta Graduation

The plan is to graduate the feature to beta in the v1.30 time frame. So far we have not received any negative feedback from cluster administrators and developers who have enabled the feature.

A kubectl plugin has been released and added to the Krew index for querying the logs more elegantly instead of using raw API calls.

Beta -> GA Graduation

The plan is to graduate the feature to stable (GA) in the v1.36 time frame at which point any major issues should have been surfaced and addressed during the alpha and beta phases.

GA Requirements met:

Feature has been stable in Beta for at least 2 releases (v1.30, v1.31)
No outstanding critical bugs or regressions reported
Comprehensive e2e test coverage for all supported platforms (Linux and Windows)
User feedback incorporated from Beta usage
Documentation complete and reviewed
Performance validated under production workloads
Upgrade/downgrade paths tested and documented
API stability confirmed (no breaking changes planned)
All GA e2e tests meet requirements for Conformance Tests
Minimum two week window for GA e2e tests to prove flake free

Upgrade / Downgrade Strategy

Upgrade:

When upgrading to a version with this feature enabled, no changes are required to maintain previous behavior if the feature gate is not explicitly enabled.
To make use of the enhancement after upgrade, enable the NodeLogQuery feature gate and enableSystemLogQuery kubelet configuration option.
The feature is backward compatible - nodes without the feature can coexist with nodes that have it enabled.

Downgrade:

When downgrading from a version with this feature to an older version, the feature will simply become unavailable.
No data migration or cleanup is required as the feature does not persist any state.
Existing workloads are not affected by enabling or disabling this feature.

Testing has been performed covering upgrade/downgrade scenarios as documented in the “Rollout, Upgrade and Rollback Planning” section.

Version Skew Strategy

This feature is kubelet-only and does not involve coordination between control plane components and nodes in terms of feature functionality.

Kubelet version skew:

If the API call is made against a kubelet that does not support the new feature (older version), a 404 will be returned.
Newer kubelet with the feature enabled can coexist with older kubelet without the feature in the same cluster.
The feature does not affect pod scheduling, networking, or any other cluster operations.

Control plane version skew:

This feature does not introduce any new APIs at the API server level.
The feature uses the existing kubelet proxy endpoint (/api/v1/nodes/{node}/proxy/logs/).
No changes are required to kube-apiserver, kube-controller-manager, or kube-scheduler.

kubectl version skew:

The feature can be accessed using kubectl get --raw with any kubectl version.
A dedicated kubectl plugin is available for improved user experience but is not required.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

Feature gate (also fill in values in kep.yaml)
- Feature gate name: NodeLogQuery
- Components depending on the feature gate: kubelet
Other
- Describe the mechanism: In addition to the feature gate, the enableSystemLogQuery kubelet configuration option must be enabled.
- Will enabling / disabling the feature require downtime of the control plane? No
- Will enabling / disabling the feature require downtime or reprovisioning of a node? Yes, requires kubelet restart

Does enabling the feature change any default behavior?

No. The feature only adds a new capability to query node logs. It does not change any existing behavior.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

For versions v1.35 and earlier (Beta): Yes. The feature can be disabled by setting the NodeLogQuery feature gate to false in the kubelet configuration and restarting the kubelet. No other changes are necessary to disable the feature.

For versions v1.36 and later (GA): No. The NodeLogQuery feature gate is locked to enabled and cannot be disabled. (LockToDefault: true). The feature is considered stable and is always available.

The feature can be effectively disabled by setting the enableSystemLogQuery kubelet configuration option to false and restarting the kubelet. This will disable the log query functionality while keeping the feature gate enabled.

What happens if we reenable the feature if it was previously rolled back?

There will be no adverse effects of enabling the feature gate after it was disabled. The feature will become available again after kubelet restart.

Are there any tests for feature enablement/disablement?

Feature Gate Tests: Yes. E2E tests verify the NodeLogQuery feature gate can be enabled and disabled. These tests are part of the sig-windows e2e tests .

Configuration Option Tests: Yes. Unit tests in pkg/kubelet/apis/config/validation/validation_test.go verify the enableSystemLogQuery kubelet configuration option validation behavior.

Additionally, the feature has been manually tested for both enablement mechanisms as documented in the “Rollout, Upgrade and Rollback Planning” section, including:

Both mechanisms disabled (feature should not work)
Feature gate enabled with config option disabled (feature should not work)
Feature gate enabled with config option enabled (feature should work)

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

A rollout can fail on enabling the feature if there is a bug in the node log query code which can cause the kubelet to crash. However this has not been observed in practice or in the end to end tests. When the kubelet comes up successfully on enabling the feature, it will have no impact on workloads. There should be no impact on rolling back this feature.

What specific metrics should inform a rollback?

A kubelet crash on enabling just this feature would be an indicator that a rollback is required. So far no CPU or memory spikes have been observed on enabling this feature but that could be another indicator.

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

Yes. The following manual tests were done:

Brought up a 1.30-alpha cluster without the kubelet feature gate and kubelet option. Enabled it the feature and ensured that the feature worked. Disabled the feature and ensured that the log proxy endpoint worked as before.
Brought up a 1.29 cluster and enabled the feature. Upgraded the kubelet to 1.30-alpha and ensured that the feature continued to work. Downgraded the kubelet to 1.29 and ensured that the feature continued to work. Upgraded the kubelet again to 1.30 and ensured that the feature worked.
Brought up a 1.29 cluster and enabled the feature. Upgraded the kubelet to 1.30-alpha and ensured that the feature continued to work. Disabled the feature and downgraded the kubelet to 1.29 and ensured that the log proxy endpoint worked as before. Upgraded the kubelet to 1.30-alpha and ensured that the log proxy endpoint worked as before. Enabled the feature again and ensured it worked as advertised.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

While this feature does not affect any workloads an operator can determine if this feature is enabled by checking the kubelet logs for “feature gates: {map[NodeLogQuery:true]}”.

How can someone using this feature know that it is working for their instance?

Other (treat as last resort)
- Details: The cluster administrator can confirm that this feature works by querying the kubelet log proxy endpoint. Example: “kubectl get –raw “/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet”

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

This feature provides on-demand log retrieval and does not have continuous operation requirements. Reasonable SLOs include:

99% of log query requests should complete successfully when the kubelet is healthy
Log queries should return within 30 seconds (enforced by implementation timeout)
The feature should not impact kubelet’s primary responsibilities for pod management

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

Metrics
- Metric name: kubelet_http_requests_total (existing kubelet metric, filtered by endpoint)
- Aggregation method: Rate of requests to the /logs/ endpoint, error rate
- Components exposing the metric: kubelet
Other (treat as last resort)
- Details: Kubelet logs will contain error messages if log query operations fail. Operators can monitor for errors related to log query operations in kubelet logs.

Are there any missing metrics that would be useful to have to improve observability of this feature?

Currently, the feature relies on existing kubelet HTTP metrics. Additional metrics that could be useful include:

A dedicated metric for node log query request duration
A metric for the size of logs returned
A metric for rate-limited or rejected requests

These metrics were not added to minimize the footprint of the feature.

Dependencies

Does this feature depend on any specific services running in the cluster?

kubelet
- Usage description: The feature is implemented as part of the kubelet and requires kubelet to be running.
  - Impact of its outage on the feature: If kubelet is not running on the node this feature will not work.
  - Impact of its degraded performance or high-error rates on the feature: If the kubelet is degraded this feature will also be degraded i.e. the node logs will not be returned.

Scalability

Will enabling / using this feature result in any new API calls?

Will enabling / using this feature result in introducing new API types?

The feature does not introduce a new API from an API server perspective but the existing kubelet proxy/log endpoint will have new features built into it.

Will enabling / using this feature result in any new calls to the cloud provider?

Will enabling / using this feature result in increasing size or count of the existing API objects?

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

In the case of large logs, there is potential for an increase in RAM and CPU usage on the node when an attempt is made to stream them. However, so far no CPU or memory spikes have been reported from the field.

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

The feature only responds to explicit API requests and does not consume resources in the background, has a built-in timeout of 30 seconds which prevents long-running queries from exhausting resources. However, if a large number of concurrent requests are made to the log query endpoint, there is potential risk for resource exhaustion. The kubelet and API server have existing safeguards to handle concurrent requests, but it is important to monitor resource usage when this feature is enabled.

Concurrent requests could potentially exhaust the following resources:

File descriptors: Each query may open log files or pipes (for journalctl/Get-WinEvent)
PIDs: Each query spawns a subprocess (journalctl on Linux, PowerShell on Windows)
Memory: Large logs being streamed could consume memory
CPU: Log processing and regex pattern matching

Example attack scenario:

A malicious or misconfigured client making 10,000 concurrent requests to /api/v1/nodes/{node}/proxy/logs/?query=kubelet could:

Spawn 10,000 subprocesses (exhausting PIDs)
Open 10,000+ file descriptors (pipes/sockets)
Consume significant CPU for concurrent log processing

Mitigations:

RBAC protection: The feature requires cluster administrator privileges (nodes/proxy resource access). Only trusted users or roles can access this endpoint.
Authentication/authorization overhead: Every request must pass through API server authentication and authorization before reaching the kubelet, providing natural rate limiting through API server request handling.
Built-in timeout: Each query is limited to 30 seconds, preventing indefinite resource consumption per request.
Kubelet request limits: The kubelet has existing safeguards for handling concurrent HTTP requests through its HTTP server configuration.

Recommendations for production deployments:

Monitor kubelet resource usage (CPU, memory, file descriptors, PIDs) when this feature is enabled
Consider implementing additional rate limiting at the API server level for the /proxy/logs/ endpoint if needed
Audit and restrict RBAC permissions to the nodes/proxy resource to only necessary service accounts and users
Set appropriate resource limits on the kubelet process itself

Considerations for future improvements:

Implement explicit per-client rate limiting in the kubelet for log query requests
Add metrics for tracking concurrent log query operations
Consider a maximum concurrent requests limit specifically for the /logs/ endpoint

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

The feature does not directly interact with the API server or etcd. However, the client (kubectl or other API client) needs to communicate with the API server to proxy the request to the kubelet. If the API server is unavailable, the client will not be able to reach the kubelet’s /logs endpoint through the proxy. Direct access to the kubelet (if configured) would still work.

What are other known failure modes?

[Log query timeout]
- Detection: Log queries that take longer than 30 seconds will timeout. The client will receive a timeout error. This can be detected through HTTP 504 Gateway Timeout responses or similar timeout errors in client logs.
- Mitigations: The 30-second timeout is built into the implementation to prevent resource exhaustion. Users should narrow their query scope (e.g., using time ranges or patterns) to retrieve logs within the timeout period.
- Diagnostics: Kubelet logs will show timeout errors. Error message: “context deadline exceeded” or similar timeout-related messages.
- Testing: Tested manually by attempting to query very large log files.
[Service/log file not found]
- Detection: If the requested service or log file does not exist, a 404 Not Found error will be returned. This can be observed in the API response.
- Mitigations: Users should verify the service name or log file path exists on the node. On Linux, use systemctl list-units to see available services. Check /var/log/ for available log files.
- Diagnostics: The error response will indicate “No such file or directory” or “Unit not found” depending on the underlying system.
- Testing: E2E tests include negative test cases for non-existent services.
[Permission denied accessing logs]
- Detection: If the kubelet process does not have permissions to read the requested logs, a permission error will be returned (typically HTTP 500 or 403).
- Mitigations: Ensure the kubelet runs with appropriate permissions to access system logs. On Linux systems with journald, the kubelet should be able to run journalctl. On Windows, the kubelet should have access to event logs.
- Diagnostics: Kubelet logs will show permission denied errors. Check kubelet process permissions and SELinux/AppArmor policies if applicable.
- Testing: Not explicitly tested in e2e tests, but covered through operational deployment testing.

What steps should be taken if SLOs are not being met to determine the problem?

If log queries are failing or timing out frequently:

Check kubelet health and resource usage (CPU, memory) on the affected nodes
Review kubelet logs for errors related to log query operations
Verify that the requested services/log files exist on the nodes
Check if log files are exceptionally large - consider using time filters or pattern matching to reduce query scope
Monitor the kubelet_http_requests_total metric filtered by the /logs endpoint to identify error rates
Verify that the NodeLogQuery feature gate and enableSystemLogQuery configuration option are properly enabled
Test with a simple query (e.g., querying kubelet logs from a recent time period) to isolate whether the issue is query-specific or systemic

Implementation History

Created on Jan 14, 2021
Updated on May 5th, 2021
Updated on Dec 13th, 2022
Updated on May 2nd, 2023
Updated on Feb 5th, 2024
Updated on Feb 2th, 2026: KEP updated to GA

Drawbacks

Alternatives

Alternatively we could use a client side reader on the nodes to redirect the logs. The Windows side would require privileged container support. However this would not help scenarios where containers are not launching successfully on the nodes.

Infrastructure Needed (Optional)

No additional infrastructure is needed for this enhancement beyond the existing Kubernetes testing infrastructure.