KEP-2299: Kustomize Plugin Composition API

Implementation History
ALPHA Implementable
Created 2021-01-20
Latest v1.24
Related Enhancements
Ownership
Owning SIG
SIG CLI
Participating SIGs
SIG CLI
Primary Authors
@knverey @campoy

KEP-2299: Kustomize Plugin Composition API

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

[kubernetes.io]: https://kubernetes.io/[kubernetes/enhancements] : https://git.k8s.io/enhancements [kubernetes/kubernetes]: https://git.k8s.io/kubernetes [kubernetes/website]: https://git.k8s.io/website[Configuration Functions Specification]: https://github.com/kubernetes-sigs/kustomize/blob/master/cmd/config/docs/api-conventions/functions-spec.md

Summary

Introduces a new Kustomize API (kind) that is oriented around Kustomize plugins, making them a first-class expression of a resource configuration bundle.

This new API will provide new sophisticated capabilities for using plugins as composable units. It will showcase plugins as a way to implement composable, declarative client-side abstractions.

Motivation

The Kustomization API was designed around built-in Kustomize transformer, generator and validator operations applied to Kustomize bases. While plugin transformers can be invoked via the appropriate field, Kustomization is suboptimal for workflows that are primarily composed of Kustomize plugins. Challenges with the current approach include:

  1. Sequencing of built-in and plugin transformers is static. The end user can choose to categorize a plugin as a generator, transformer or validator, but can only control relative execution order within those fields. Especially in the case of abstraction-based plugins, the user may not know which category they fall in, and indeed the plugin may change what it does based on input. This can be addressed by placing all plugins under transformers. However, doing so means they will all be executed after built-in transformers. The result is that an additional Kustomization layer is typically required to use built-ins fruitfully in abstraction-based workflows.

  2. Plugin config transformation must be isolated. Although it is possible to transform plugin configuration as resources before execution, it is not possible to have a sensible distribution of concerns in configuration sets composed this way. Notably, plugin configuration must be finalized (e.g. tailored to an environment) before it can be combined with built-in configuration at all, and built-in transformers cannot be specified in bases, or else they will not apply to the results of the plugins. example/kustomization demonstrates this difficulty.

The sequencing issue could be solved within Kustomization; it is the difficulty with plugin config transformation that motivates an entirely new Kind.

Goals

Develop an API (kind) for Kustomize that is focused on plugin-based workflows.

A successful implementation of this API should have the following characteristics:

  1. It must be easy to recursively compose and modify lists of plugins in a way that yields an overall configuration structure that reflects the user’s intent.

  2. Plugin orchestration must be central to the new API format. Built-in and plugin transformer config should be treated the same way in the new API.

  3. The new API must cleanly integrate with the existing Kustomize tool (i.e. kustomize build) as well as existing Kustomize kinds.

Non-Goals

  • Replace Kustomization for use cases that involve few or no plugins.
  • Expand the scope of the existing Kustomization API.
  • Introduce plugin distribution or discovery mechanisms.
  • Change the flags and parameters used for invoking plugins in any way.

Proposal

Introduce Composition as a new API kind recognized by kustomize build. A Composition must be defined in a composition.yaml file. Inclusion of both composition.yaml and kustomization.yaml in the target directory of a build will be considered an error.

The Composition API enables users to:

  • Define a list of transformer configuration that intermingles built-in and plugin transformers.
  • Import transformers from another Composition and add them to the list.
  • Override an imported transformer’s fields with new values.
  • Reorder the list of transformers prior to execution.

Once a user has written a Composition, they can run kustomize build to turn it into a list of Kubernetes resources that they can commit to git, the same way they would with Kustomization.

To provide this experience, Kustomize will need to do the following during builds:

  • Consolidate the Composition and its imports into a finalized list of transformers (which may be plugins).
  • Execute each transformer in sequence (no changes to plugin execution are proposed by this KEP).
  • Pass the output of each transformer as the input to the next.

User Stories (Optional)

Story 1

As a user, I can define my configuration as a sequence of declarative abstractions implemented by plugins.

# app/composition.yaml
kind: Composition

transformers:
# generate resources for a Java application
- apiVersion: example.com/v1
  kind: JavaApplication
  runtime: {container: {image: docker.example.co/kustomize_transformers/java:v1.0.0}}
  metadata:
    name: my-app
  spec:
    application: team/my-app
    version: v1.0
# transform resources to inject a logger
- apiVersion: example.com/v1
  kind: Logger
  runtime: {container: {image: docker.example.co/kustomize_transformers/logger:v1.0.3}}
  metadata:
    name: logger

Story 2

As a user, I can reuse and extend an existing Composition by importing its transformers and optionally overriding imported transformers’ fields.

# staging/composition.yaml
kind: Composition

transformersFrom:
# import the JavaApplication and Logger transformers
- path: ../app/composition.yaml
  importMode: prepend

transformerOverrides:
# override the JavaApplication version before the function is run
- apiVersion: example.com/v1
  kind: JavaApplication
  metadata:
    name: my-app
  spec:
    version: v1.1-beta

transformers:
# transform resources from imported transformers to inject metrics
- apiVersion: example.com/v1
  kind: Prometheus
  runtime: {container: {image: docker.example.co/kustomize_transformers/prometheus:v1.0.2}}
  metadata:
    name: metrics
  spec:
    prefix: my-app-

Notes/Constraints/Caveats (Optional)

Integration with Kustomization

By beta at the latest, Kustomizations should be able to reference Compositions in various fields as follows:

  • the resources and generators fields: the Composition will be executed with empty input to the first transformer in the consolidated Composition, and the results will be appended to the resource list.
  • the transformers field: the resource list will be provided as input to the first transformer in the consolidated Composition, and the output will replace the resource list.
  • the validators field: the resource list will be provided as input to the first transformer in the consolidated Composition, and the output (if any) will be discarded.

Integration with Kubectl Kustomize

Composition support will neither be required nor suppressed in kubectl kustomize during alpha. In other words, if a kubectl kustomize release happens during alpha, it can include the feature, but graduation to beta does not require this. Composition support MUST be included in at least two kubectl kustomize releases during beta.

Plugin execution flags

This KEP does not propose any changes to the flags used to gate plugins, or to the parameters available to configure container/exec plugin execution. All plugin execution from either Kind will remain gated behind --enable-alpha-plugins, as well as existing additional flags for particular plugin provider types. Any changes to plugin gating should apply identically to both Kinds. This does limit the usefulness of Composition within kubectl -k for the time being, since only a subset of plugins can be enabled in that context.

Risks and Mitigations

This proposal effectively takes a feature set that exists in Kustomize today and makes it much more usable in certain workflows. As such, it does not come with any novel technical risks.

Design Details

Key terminology

This terminology is not set in stone at this early stage, but should be used consistently within this proposal.

  • Composition: a new plugin-oriented API (kind) to be understood by kustomize build.
  • Transformer: a program that generates, manipulates and/or validates a stream of Kubernetes resources, along with the corresponding YAML config object used to configure that program.
    • Plugin transformer: A user-authored transformer.
    • Built-in transformer: A transformer whose configuration spec and implementation are part of Kustomize itself.
  • Plugin provider: The program that implements the plugin (e.g. container, script, Go program). Analogous to the controller for a custom resource.
  • Plugin/transformer config: The KRM-style YAML document that declares the desired state the transformer implements. In the case of a plugin, it includes the provider to execute and the specification to follow in doing so. Analogous to a custom resource object.

API schema

Example showing all fields:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Composition

# `transformersFrom` imports transformer lists from other Compositions.
# The target Compositions' own imports, overrides and reorderings are applied before it is imported.
transformersFrom:
- path: ../app/composition.yaml
  importMode: prepend # this is the default; append is also possible

# `transformerOverrides` allows fields of transformers imported via `transformersFrom` to be changed
# before execution.
# Entries are treated as strategic merge patches.
transformerOverrides:
- apiVersion: example.com/v1
  kind: JavaApplication
  metadata:
    name: my-app
  spec:
    application: team/my-app
    version: v3.1.3

# `transformers` adds new transformers to the list.
transformers:
- apiVersion: example.com/v1
  kind: Prometheus
  runtime:
    container:
      image: example/prometheus:v1.0.2
  metadata:
    name: metrics-injector

# `transformerOrder` allows advanced users to reorder the merged transformers list explicitly.
# transformers can be referred to by name alone as long as names are unique.
transformerOrder:
  - name: my-app
  - name: logger
  - name: metrics

Notes:

  • One notable difference from Kustomization is that in addition to being able to provide a path to a file containing the transformer config, you can specify the config itself (not strigified) inline.
  • The examples here use a new a top-level reserved runtime field instead of the JSON annotation currently used for this information. This field may be introduced as part of KEP 2953. It does not block this KEP, and usage within Kustomization and Composition will be identical regardless.
  • The metadata.name field is defaulted to Kind in kebab case. However, metadata.name MUST be specified when using multiple independent instances of the same GVK within a Composition tree.
Full schema
definitions:
  transformer:
    type: object
    additionalProperties: true
    required:
      - apiVersion
      - kind
      - metadata
    properties:
      kind:
        type: string
        minLength: 1
      apiVersion:
        type: string
        pattern: "^[a-z0-9][a-z0-9\\.]*\\/[a-z0-9]+$"
      metadata:
        type: object
        required:
          - name
        additionalProperties: false
        properties:
          name:
            type: string
            minLength: 1
            maxLength: 253
            pattern: "^[a-z0-9](https://raw.githubusercontent.com/kubernetes/enhancements/master/keps/sig-cli/2299-kustomize-plugin-composition/[-a-z0-9]*[a-z0-9])?(\\.[a-z0-9](https://raw.githubusercontent.com/kubernetes/enhancements/master/keps/sig-cli/2299-kustomize-plugin-composition/[-a-z0-9]*[a-z0-9])?)*$"
      runtime:
        type: object
        additionalProperties: false
        properties:
          container:
            type: object
            required:
            - image
            additionalProperties: false
            properties:
              image:
                type: string
                minLength: 1
          exec:
            type: object
            required:
            - path
            additionalProperties: false
            properties:
              path:
                type: string
                minLength: 1
type: object
required:
  - apiVersion
  - kind
additionalProperties: false
properties:
  apiVersion:
    type: string
    enum: ["kustomize.config.k8s.io/v1alpha1"]
  kind:
    type: string
    enum: ["Composition"]
  transformersFrom:
    type: array
    items:
      type: object
      required:
        - path
      additionalProperties: false
      properties:
        path:
          type: string
          minLength: 1
        importMode:
          type: string
          enum: ["append", "prepend"]
  transformers:
    type: array
    items:
      "$ref": "#/definitions/transformer"
  transformerOverrides:
    type: array
    items:
      "$ref": "#/definitions/transformer"
  transformerOrder:
    type: array
    items:
      type: object
      required:
      - name
      additionalProperties: false
      properties:
        kind:
          type: string
          minLength: 1
        apiVersion:
          type: string
          pattern: "^[a-z0-9][a-z0-9\\.]*\\/[a-z0-9]+$"
        name:
          type: string
          minLength: 1
          maxLength: 253
          pattern: "^[a-z0-9](https://raw.githubusercontent.com/kubernetes/enhancements/master/keps/sig-cli/2299-kustomize-plugin-composition/[-a-z0-9]*[a-z0-9])?(\\.[a-z0-9](https://raw.githubusercontent.com/kubernetes/enhancements/master/keps/sig-cli/2299-kustomize-plugin-composition/[-a-z0-9]*[a-z0-9])?)*$"

Built-in transformers

In Kustomize, everything is already a transformer . In a Composition, users can include any of the existing built-in transformers (including generators and validators) in their transformers list.

transformers:
- apiVersion: builtin
  kind: PrefixSuffixTransformer
  metadata:
    name: myFancyNamePrefixer
  prefix: bob-
  fieldSpecs:
  - path: metadata/name

One important exception to this is the resources: field, which is not already a generator. Work will need to be done to expose accumulateResources as one.

transformers:
- apiVersion: builtin
  kind: ResourceAccumulator
  paths:
  - path/to/file.yaml

In addition to that, kind: Kustomization itself may be inlined as a transformer. Because it is a transformer in this context, resources from previous transformers will be written to a temp file that will be automatically prepended to the Kustomization’s resources field.

transformers:
- apiVersion: kustomize.config.k8s.io/v1beta1
  kind: Kustomization
  commonLabels:
    foo: bar
  resources:
  - config.yaml

Composition evaluation

Composition topologically sorts imported transformers with its own transformers and then runs the combined list sequentially. This means that arbitrary depths of imports always result in a single consolidated Composition prior to execution. Once that consolidated Composition has been compiled, its transformers will be invoked the same way they are today from the transformers field in Kustomization.

For use cases that require resources to be evaluated eagerly before further processing, for example diamond-shaped configuration, the built-in ResourceAccumulator can be given a Composition to render into resources.

If it is deemed useful for debugging, a command for consolidating Composition without executing it may be introduced to the kustomize cfg command group (or equivalent, if cfg is replaced).

Test Plan

Testing can be done purely on the client-side without the need for a cluster.

Graduation Criteria

Alpha

  • Composition implemented with an alpha GV and supported by kustomize build. This includes implementing the transformer, transformersFrom, transformerOverrides and transformerOrder fields.
  • ResourceGenerator extracted as a transformer
  • Introduce a way to use default fieldSpecs when invoking builtin transformers from the transformers field (and generators from the generators field, etc.).
  • Built-in generators updated to work in transformers fields
  • Basic documentation added to the Kustomize website.

Beta

  • Composition beta GV supported by kustomize build
    • Kustomization transformer implemented
  • Composition supported by the resources, transformers, generators and validators fields in Kustomization and Component
  • Thorough documentation and examples published on the Kustomize website.

GA

  • At least two releases of kubectl kustomize have included Composition support, to allow for feedback from the wider community.
  • Inline transformer config support in the transformers, generators and validators fields of Kustomization
  • TBD

Deprecation

As with any Kustomize feature, deprecation post-alpha would need to take the kubectl kustomize release cycle into consideration as well as Kustomize’s own, to ensure all users are adequately warned before removal.

Upgrade / Downgrade Strategy

N/A – feature is client-side-only

Version Skew Strategy

N/A – feature is client-side-only

Production Readiness Review Questionnaire

Feature Enablement and Rollback

No special flags will be needed to use Composition, which will have a GV indicative of its maturity level. The same flags required to use plugins in existing contexts will be required to use them via Composition.

How can this feature be enabled / disabled in a live cluster?

N/A – feature is client-side-only

Does enabling the feature change any default behavior?

N/A – feature is client-side-only

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

N/A – feature is client-side-only

What happens if we reenable the feature if it was previously rolled back?

N/A – feature is client-side-only

Are there any tests for feature enablement/disablement?

N/A – no feature flag required

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

N/A – feature is client-side-only

What specific metrics should inform a rollback?

N/A – feature is client-side-only

Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

N/A – feature is client-side-only

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No

Monitoring Requirements

N/A – feature is client-side-only

How can an operator determine if the feature is in use by workloads?

N/A – feature is client-side-only

How can someone using this feature know that it is working for their instance?

N/A – feature is client-side-only

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

N/A – feature is client-side-only

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

N/A – feature is client-side-only

Are there any missing metrics that would be useful to have to improve observability of this feature?

N/A – feature is client-side-only

Dependencies

This feature does not require any new Golang dependencies.

Does this feature depend on any specific services running in the cluster?

No – feature is client-side-only

Scalability

This feature is not expected to have significantly different performance characteristics than the use of transformers in Kustomization. Performance in practice will depend largely on the type and implementation of plugins used.

Will enabling / using this feature result in any new API calls?

No

Will enabling / using this feature result in introducing new API types?

Yes, but only client-side within Kustomize

Will enabling / using this feature result in any new calls to the cloud provider?

No

Will enabling / using this feature result in increasing size or count of the existing API objects?

No

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

No

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

N/A – feature is client-side-only

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

N/A – feature is client-side-only

What are other known failure modes?

Kustomize plugins are immature (various alpha mechanisms) and when they fail to execute, it is not always graceful. Golang plugins in particular come with a long list of caveats. KRM Exec and Starlark plugins are not currently enabled in kubectl kustomize at all.

What steps should be taken if SLOs are not being met to determine the problem?

N/A – feature is client-side-only

Implementation History

  • April 27, 2021: Provisional KEP merged
  • August 2021: Proposal updated and marked implementable.
  • December 2021: Prototype PR opened at kustomize/#4323 and KEP updated to reflect additional work discovered.

Drawbacks

  • The new API provides another way to run plugins, and may be confusing to users trying to decide which to use. Creating separate APIs for Deployment, StatefulSet, DaemonSet have similar drawbacks.

  • Since this new API is oriented entirely around plugins, it will likely lead to workflows in which a considerable number of plugins must be invoked to render the resource list. Especially for container plugins, this may not be performant. Although the same issue would exist in today’s plugins via Kustomization, it may become more apparent if plugins get a substantial increase in ease of use.

Alternatives

  • Implement these capabilities in the Kustomization API. All of the fields proposed in the new API could instead be added as top-level fields, or grouped fields, in Kustomization. However, the result would not be very elegant. In particular, one of the primary proposed features–enabling transformer configuration to be overridden–requires a completely different evaluation model. Namely, a topological sort is applied to the entire tree of transformers prior to evaluation, as opposed to Kustomization’s current model of executing generators, transformers and plugins in each level in isolation. Combining these two mental models within a single API would be difficult and likely lead to considerable confusion among users.
  • Implement this as an independent tool. This is certainly possible; Composition itself is a transformer, and could be implemented as a plugin. However, this proposal solves a usability problem faced by existing Kustomize features. As such, it would be great if the Kustomize community could leverage it directly.