Kustomize Secret K:V Generator Plugins

Table of Contents

• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
• Risks and Mitigations
  • Several people want an exec-style plugin
    • mitigation
  • goplugin limitations
    • Not shareable as object code
      • mitigation
      • No current support for Windows
      • mitigation
    • General symbol execution from the plugin
      • mitigation
  • Two means to specify legacy KV generation
    • mitigation
• Graduation Criteria of plugin framework
  • Alpha status
  • Graduation to beta
• Implementation History

Summary

Kustomize users want to generate Kubernetes Secret objects from general key:value (KV) pairs where the value is supposed to be a secret. Users want to generate these pairs through integration with secret management tools (e.g. see comments on [692][execRemoval]).

Currently Kustomize only supports reading secret values from local files which raises security concerns about file lifetime and access. Reading secret values from the execution of arbitrary “commands” in a kustomization.yaml file introduces concerns in a world where kustomization files can be used directly from the internet when a user runs kustomize build. This proposal describes the syntax for a new key:value generator plugin framework supporting an arbitrary number of plugin types to generate key:value pairs.

Motivation

Not having a way to integrate SecretGenerator with secret management tools requires hacky, insecure workarounds.

Goals

• In the GeneratorArgs section of a kustomization file, a user may specify a plugin type, and a specific instance of that type, for generating key:value pairs.

• The specification will allow for any number of plugin types, and any number of instances of those types.

• The first type supported will be goplugins , to enable kustomize source code contributors to add custom KV generators without the need to maintain a kustomize source code fork.

  Kustomize maintainers expect developers who use a goplugin to understand that a kustomize binary and any goplugins expected to work with it must be compiled on the same machine with the same compiler against the same set of (transitive) libraries. It’s the developers responsibility to bundle the binary and shared libraries together in a container image for actual use.

• Other kinds of plugins, e.g. an execute this binary plugin, should be subsequently easy to add, and could be appropriate for end user use (but would require consideration in a KEP first).

Non-Goals

• Kustomize will not handle key:value generation plugin installation/management, or seek to build a plugin “ecosystem” of key:value generators.

Proposal

The current API of SecretGenerator looks like this:

secretGenerator:
- name: app-tls
  files:
  - secret/tls.cert
  - secret/tls.key
  type: "kubernetes.io/tls"

- name: env_file_secret
  env: env.txt
  type: Opaque

- name: myJavaServerEnvVars
  literals:
  - JAVA_HOME=/opt/java/jdk
  - JAVA_TOOL_OPTIONS=-agentlib:hprof

The proposed API would look like this:

secretGenerator:
- name: app-tls
  kvSources:
  - pluginType: builtin  // builtin is the default value of pluginType
    name: files
    args:
    - secret/tls.cert
    - secret/tls.key
- name: env_file_secret
  kvSources:
  - name: env  // this is a builtin
    args:
    - env.txt
- name: myJavaServerEnvVars
  kvSources:
  - name: literals    // this is a builtin
    args:
    - JAVA_HOME=/opt/java/jdk
    - JAVA_TOOL_OPTIONS=-agentlib:hprof
- name: secretFromPlugins
  kvSources:
  - pluginType: go      // described by this KEP
    name: myplugin
    args:
    - someArg
    - someOtherArg
  - pluginType: kubectl      // some future KEP can write this
    name: someKubectlPlugin
    args:
    - anotherArg
    - yetAnotherArg

The kvSources specified with pluginType: builtin are reformulations of existing key:value generators currently invoked by the existing dataSources specification.

A kvSource with pluginType: Go and name: myplugin would attempt to load the file

~/.config/kustomize/plugins/kvSources/myplugin.so

and access the loaded plugin via the interface

type KvSourcePlugin interface {
	Get() []kv.Pair
}

This clearly describes how the plugin must be formulated.

The loading implementation would look something like this:

func keyValuesFromPlugins(ldr ifc.Loader, plugins []types.Plugin) ([]kv.Pair, error) {
	var allKvs []kv.Pair

	for _, plugin := range plugins {
		secretFunc, err := findPlugin(plugin.Name)
		if err != nil {
			return nil, err
		}

		kvs, err := secretFunc(ldr.Root(), plugin.Args)
		if err != nil {
			return nil, err
		}

		allKvs = append(allKvs, kvs...)
	}

	return allKvs, nil
}

func findPlugin(name string) (func(string, []string) ([]kv.Pair, error), error) {
	allPlugins, err := filepath.Glob(os.ExpandEnv("$HOME/.config/kustomization_plugins/*.so"))
	if err != nil {
		return nil, fmt.Errorf("error loading plugins")
	}

	for _, filename := range allPlugins {
		p, err := plugin.Open(filename)
		if err != nil {
			return nil, err
		}

		symbol, err := p.Lookup(name)
		if err != nil {
			continue
		}

		if secretFunc, exists := symbol.(func(string, []string) ([]kv.Pair, error)); exists {
			return secretFunc, nil
		}
	}

	return nil, fmt.Errorf("plugin %s not found", name)
}

An example plugin would look like this:

package main

import (
	"os"

	"sigs.k8s.io/kustomize/pkg/kv"
)

func Env(root string, args []string) ([]kv.Pair, error) {
	var kvs []kv.Pair
	for _, arg := range args {
		kvs = append(kvs, kv.Pair{Key: arg, Value: os.Getenv(arg)})
	}
	return kvs, nil
}

A developer - not an end user - would compile the goplugin and main program like this:

go build -buildmode=plugin ~/.config/kustomize/plugins/kvSources/myplugin.so
go get sigs.k8s.io/kubernetes-sigs/kustomize

then bake the build artifacts into a container image for use by an end user or continuous delivery bot.

Risks and Mitigations

Several people want an exec-style plugin

exec-style means execute arbitrary code from some file, with key value pairs being captured from the stdout of a subprocess.

mitigation

Most the lines of code written to implement this KEP accommodate the notion of KV generation via a generic notion of a plugin, and a generic interface, in the kustomization file and associated structs and code.

This code recharactizes the three existing KV generation methods as pluginType: builtin (literals, env and files), introduces the new pluginType: Go, and leaves room for someone to easily add, say pluginType: kubectl to look for a kubectl plugin, and pluginType: whatever to handles some completely new style, e.g. look for the plugin name as an executable in some hardcoded location.

Actual implementation of these other kinds of plugins are out of scope for this KEP.

The ability to write a goplugin very specifically targets a kustomize contributor who understands their limitations. An exec-style plugin would enable a broader set of people, this time including end users. A KEP proposing such plugins would need to consider a different set of risks and maintenance requirements.

goplugin limitations

The first plugin mechanism will be goplugins, since by design they are trivial to make for a Go program (like kustomize). However, they have limitations, and are to some extent an experimental feature of Go tooling.

Not shareable as object code

The shared object (.so) files created via go build -buildmode=plugin cannot be reliably used by a loading program unless both the program and the plugin were compiled with the same version of Go, with the same transitive libs, on the same machine (see, e.g. this golang issue ).

Therefore, the Go developer who elects to write their key:value generator as a goplugin is obligated to compile a new kustomize binary as well as the plugin.

For this developer, this extra compilation step will be far easier than forking kustomize and modifying the code directly to introduce all the piping one would need to express a new key:value secret generation style in a kustomization file and integrate it with existing methods.

The risk to not compiling, and, say using an .so that was compiled long ago and far away is that the program will panic when the build command is executed.

mitigation

An attempt to kustomize build $target that includes a kustomization file specifying a pluginType: Go will fail with an error message explaining the panic risk and demanding the additional use of the command line flag:

kustomize build --alpha_enable_goplugin_and_accept_panic_risk $target

This flag signals that the feature 1) is alpha (it could be removed), and 2) the feature could panic if improperly used.

As noted above, developers who elect to use kustomize plus goplugins should bundle all compilation artifacts into a container image for end use. A bot using said image would always specify the flag.

No current support for Windows

This is mentioned in the plugin package overview section .

mitigation

The target use case for secret generation is gitops running in a container on any kubernetes cluster.

Developers on their workstations are not a target use case, as they are less likely to have the security requirements that necessitate using a plugin based generator.

If developers need this functionality, we need to better understand why. it is possible there is a better way of addressing their need. Of course, Windows users who want to support goplugins could contribute to the golang project.

General symbol execution from the plugin

mitigation

A Go plugin, when loaded, will be cast to a particular hardcoded interface, e.g.

type KvSourcePlugin interface {
	Get() []kv.Pair
}

and accessed exclusively through that interface.

Two means to specify legacy KV generation

The existing three mechanisms and syntax for generating KV pairs (literals, env and files) remain active, but additionally the work for this KEP will allow these mechanisms to be expressed as builtin plugins (see example above).

mitigation

If plugins - both goplugins and other styles - prove unpopular or problematic, we can remove them per API change policies.

If they do prove popular/useful, we can deprecate the legacy form for (literals, env and files), and help people convert to the new builtin form to access these KV sources.

Graduation Criteria of plugin framework

Alpha status

The kustomization fields that support a general plugin framework (which could support many kinds of plugins in addition to goplugins) will be implemented but documented as an alpha feature, which could be removed per API change policies. Loading of goplugins themselves, as noted above, will be protected by a flag denoting their alpha status. As these features target contributors, this will be mentioned in CONTRIBUTING.md.

Graduation to beta

• Exec-style plugins approaching GA.

  As goplugins are targeted to kustomize contributors, we’d like to see development of an exec-style plugin targeted to end users before deciding to graduate the framework to beta/GA.

  For goplugins themselves to reach beta/GA, we’d like exec-style based plugin implemented and still see some preference for the Go based approach.

• Testing and documentation.

  • High level feature test (like those in pkg/target )
  • Field documentarion in the canonical example file
  • Usage examples .

Implementation History

(TODO add PR’s here)