KEP-3638: Improve kubectl plugin resolution for non-shadowing subcommands

Release Signoff Checklist
Summary
Motivation
- Goals
- Non-Goals
Proposal
- Notes/Constraints/Caveats
- Risks and Mitigations
Design Details
Production Readiness Review Questionnaire
Implementation History
Drawbacks
Alternatives

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

(R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
(R) KEP approvers have approved the KEP status as implementable
(R) Design details are appropriately documented
(R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
- e2e Tests for all Beta API Operations (endpoints)
- (R) Ensure GA e2e tests meet requirements for Conformance Tests
- (R) Minimum Two Week Window for GA e2e tests to prove flake free
(R) Graduation criteria is in place
- (R) all GA Endpoints must be hit by Conformance Tests
(R) Production readiness review completed
(R) Production readiness review approved
“Implementation History” section is up-to-date for milestone
User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

This KEP allows external plugins to be used as subcommands of built-in commands when the subcommand does not exist.

Motivation

There is a high demand for enriching some built-in kubectl commands with new subcommands. More specifically take an instance of create command, even though it supports the creation of 18 resources(as of v1.25) with one line command execution(e.g. kubectl create job), new requests to support other resources as well are occasionally filed. These requests are understandable given the proliferation of CustomResourceDefinitions, but is not sustainable from the SIG CLI maintainers point of view. Because that might bring about not only increase the size of the binary but also create a maintenance burden.

This KEP proposes a viable alternative to users via plugins instead of creating static subcommands inside kubectl built-in commands.

Goals

Enable external plugin usage for kubectl built-in commands as subcommands, if subcommands does not exist.

Non-Goals

Implement a plugin

Proposal

When user executes a subcommand of a built-in command, instead returning an “unknown command” error, kubectl will search this subcommand in external plugins with the name structure similar to current plugin search mechanism(e.g. kubectl-create-foo).

$ kubectl create foo
(running kubectl-create-foo plugin)

If kubectl finds a match for the subcommand, it will execute this external plugin how it already does for external plugins currently.

Because of starting as alpha and this is an explicitly opted-in feature, shadowing functionality will be hidden behind the environment variable and will only be used after exporting;

export KUBECTL_ENABLE_CMD_SHADOW=true

Notes/Constraints/Caveats

This KEP in alpha and beta stage will only support shadowing for kubectl create to get feedback for other built-in commands as well. Currently, we know that there are requests for kubectl create command but we need to find out which other built-in commands will also be requested for this shadowing.

Risks and Mitigations

Security considerations are the same with executing other external plugins and there is no additional risk.

In terms of risk considerations, users may adopt external plugins that do not exist as subcommand(e.g. kubectl create foo) but afterwards built-in subcommand with the same name might be added. This brings about a risk about breaking users’ scripts which rely on external plugins. To overcome that problem, there should be a some form of forcing mechanism to select external plugin instead built-in subcommand. KUBECTL_ENABLE_CMD_SHADOW flag can also be used also for that purpose in later stages to provide a way to the users always using external plugin instead built-in subcommand.

Design Details

Test Plan

[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates

Unit tests

k8s.io/kubectl/pkg/cmd/cmd.go: 2023-01-16 - 63.5

Integration tests

Tests should include;

Create custom resource foo and foo2
Add external plugin only for kubectl-create-foo2 in test directory
Running kubectl create foo and expects error
Running kubectl create foo2 and expect it creates foo2 successfully

e2e tests

Test should include;

Add custom plugin kubectl-create-foo for custom resource foo
Apply custom resource foo into the test cluster
Execute kubeectl create foo command and expects resource is successfully created.

Graduation Criteria

Alpha

Feature implemented behind a KUBECTL_ENABLE_CMD_SHADOW=true environment variable
Initial unit tests

Beta

Gather feedback from developers and surveys
Add integration tests
set KUBECTL_ENABLE_CMD_SHADOW environment variable to true as default.

GA

Add e2e test
Remove KUBECTL_ENABLE_CMD_SHADOW environment variable
Enable shadowing for all non-existent subcommands

Upgrade / Downgrade Strategy

N/A

Version Skew Strategy

N/A

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?

Feature gate (also fill in values in kep.yaml)
- Feature gate name:
- Components depending on the feature gate:
Other
- Describe the mechanism: Exporting KUBECTL_ENABLE_CMD_SHADOW=true environment variable will enable the feature
- Will enabling / disabling the feature require downtime of the control plane? No
- Will enabling / disabling the feature require downtime or reprovisioning of a node? No

Does enabling the feature change any default behavior?

No, user will continue using their all commands without any change.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes, we can roll back to a previous release of kubectl or it can be simply done by user by unsetting environment variable.

What happens if we reenable the feature if it was previously rolled back?

Shadowing will continue being supported without any impact on default behavior.

Are there any tests for feature enablement/disablement?

No, users can only use this feature via enabling environment variable.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

What specific metrics should inform a rollback?

N/A

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

N/A

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

N/A

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?

N/A

How can someone using this feature know that it is working for their instance?

N/A

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

N/A

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

N/A

Are there any missing metrics that would be useful to have to improve observability of this feature?

N/A

Dependencies

Does this feature depend on any specific services running in the cluster?

Scalability

N/A

Will enabling / using this feature result in any new API calls?

Will enabling / using this feature result in introducing new API types?

Will enabling / using this feature result in any new calls to the cloud provider?

Will enabling / using this feature result in increasing size or count of the existing API objects?

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

Troubleshooting

Built-in commands and subcommands in kubectl always overrule external plugins so that it is not possible to shadow already existing built-ins. Thereby, same troubleshooting mechanism should be applied just troubleshooting the kubectl commands. If the user has a problem when using a plugin, there won’t be any change and user needs to contact with the plugin owner.

How does this feature react if the API server and/or etcd is unavailable?

What are other known failure modes?

What steps should be taken if SLOs are not being met to determine the problem?

Implementation History

The KEP was proposed on 2022-10-22 The KEP was promoted to beta on 2023-10-02

Drawbacks

Alternatives

Alternative of the proposed mechanism to overcome this request is to propose a new flag --use-plugin. User can pass this flag when they want to explicitly use the external plugin. Thanks to that, user also can shadow already existing subcommands. For example kubectl create job executes built-in job subcommand, on the other hand kubectl create job --use-plugin executes custom kubectl-create-job plugin.

However, downside of this alternative is that usability hurts without any gain;

Everytime user has to pass a new flag to execute external plugin.
This flag can not be passed to external plugins and this makes --use-plugin flag a special flag.
There is no clear use case allowing plugins for already existed subcommands which means that we want to disable this deliberately instead providing as a feature.