Kustomize Exec Secret Generator

Table of Contents

• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • New Flags
  • Example
    • Default Path
    • Flag Defined Absolute Path
    • Flag Defined Relative Path
    • Not Enabled
• Risks and Mitigations
  • Security
• Alternatives Considered
  • Git Style Plugins
    • Enable Plugins with Environment Variables
• Graduation Criteria
  • Graduation to GA
    • Audit Command
    • More OS Specific install locations
• Testing and documentation.
  • Testing
  • Documentation
• Implementation History

Summary

The ability to generate Secrets using exec was removed in kustomize v2 because of security concerns about users kustomizing malicious kustomization.yamls and thereby providing a path for kustomization.yaml’s publishers to execute arbitrary commands on the machines of any user who applies the kustomization.yaml.

Example goal to enable:

• Alice wants to develop an Application requiring a shared Secret, and to deploy it on Kubernetes using GitOps
• Alice wants her GitOps deployment mechanism to pull the Secrets that it deploys from an remote source without writing the Secrets as files to local disk.
• Alice’s organization configures the gitops deployment container to run Kustomize in the cluster and be capable of pulling Secrets from remote locations
• Alice writes her kustomization.yaml to use the generation options configured by her organization.

Example exploit to avoid:

• Alice wants to run a whitebox mysql instance on a test cluster
• Chuck publishes a whitebox mysql kustomization.yaml on GitHub, with a SecretGenerator that will read Alice’s ~/.kube/config and send it to Chuck’s server by executing sh will run a script to generate some Secret
• Alice runs kubectl apply -k https://github.com/chuck/mysql and has the credentials of all of her Kubernetes clusters sent to Chuck when the Secret is generated.

See kubernetes-sigs/kustomize#692 for more details.

Motivation

The ability to create Kubernetes Secrets generated by commands is commonly requested by users. This is useful for use cases where the user does not want the Secrets to be appear decrypted on disk before being applied to the cluster.

Examples:

• Secrets sources are encrypted and need to be decrypted before they are applied
• Secrets sources are not stored locally, and need to be fetched from some remote location
• Secrets are generated using some function (e.g. private-keys)

Note: For the target case, the command for generating the Secret already exists as an executable on the user’s machine. User’s are not expected to want a marketplace of solutions, rather instead they are expected to want to be able to invoke the tools they already use for addressing this task.

Goals

• Enable users to generate Secrets using the tools they already use to do so
• Secure by default - Alice must configure her environment in an insecure manner and run the command in an insecure way for it to be exploitable
• Support Linux / Mac / Windows OS’s

Non-Goals

• Support an ecosystem of users (i.e. not centralized within a single organization) authoring plugins and sharing them with one another.
• Eliminate all friction for publicly published whitebox kustomization.yamls to generate Secrets

Proposal

Re-introduce exec SecretGenerators, but with the following safeguards in place to address security concerns.

• Plugins must be explicitly enabled with a flag. If it is not enabled and is used, Kustomize will exit 1 without running exec.
  • This is to protect against the case where exploitable plugins are installed and used safely internally in an organization.
  • If exiting 1, Kustomize will first print the list of plugins exec commands that were not executed, and a message about the flag.
• Executables are restricted to flag defined paths - defaulted a location that is empty by default. Users must install / symlink executables to this location.

New Flags

Note: These flags were chosen so that they are clear when kustomize is embedded in other tools and so that it isn’t confused with plugins for those tools - e.g. kubect apply -k.

Example

Default Path

Note: my-exec-plugin exists at $XDG_CONFIG_HOME/kustomize/plugins/kvSources/my-exec-plugin

secretGenerator:
- name: fromPlugins
  kvSources:
  - pluginType: exec
    name: my-exec-plugin
    args:
    - anotherArg
    - yetAnotherArg

$ kustomize build ./ --enable-kust-plugins

Flag Defined Absolute Path

Note: my-exec-plugin exists at /foo/kvSources/my-exec-plugin

secretGenerator:
- name: fromPlugins
  kvSources:
  - pluginType: exec
    name: my-exec-plugin
    args:
    - anotherArg
    - yetAnotherArg

$ kustomize build ./ --enable-kust-plugins --kust-plugin-path=/foo/kvSources/

Flag Defined Relative Path

Note: my-exec-plugin exists at ./my-exec-plugin

secretGenerator:
- name: fromPlugins
  kvSources:
  - pluginType: exec
    name: my-exec-plugin
    args:
    - anotherArg
    - yetAnotherArg

$ kustomize build ./ --enable-kust-plugins --kust-plugin-path=./

Not Enabled

Note: my-exec-plugin exists at /usr/local/kustomize/plugins/kvSources/my-exec-plugin

secretGenerator:
- name: fromPlugins
  kvSources:
  - pluginType: exec
    name: my-exec-plugin
    args:
    - anotherArg
    - yetAnotherArg

$ kustomize build ./

Output:

secretGenerator plugins used, but plugins not enabled.  To enable plugins for trusted sources, specify --enable-kust-plugins.
secretGenerator plugins that will be run with --enable-kust-plugins:
$XDG_CONFIG_HOME/kustomize/plugins/builtin/my-exec-plugin anotherAg yetAnotherARg

Risks and Mitigations

Security

Risk: Chuck is able to exploit this feature to do something bad on Alice’s machine

Required steps to exploit:

• Alice executes (via -k or kustomize) a malicious kustomization.yaml
  • Alice does not run without enabling plugins with --enable-kust-plugins to see which plugins will be run and with which values.
• Alice opted-in to enable plugins by providing the flag --enable-kust-plugins
• Alice or her organization installed the targeted SecretGenerator to $XDG_CONFIG_HOME/kustomize/plugins/kvSources or another location she provided with --kust-plugin-path=<path-including-exploitable-binary>
• The specified SecretGenerator must be exploitable
  • Simple transformation functions would not be exploitable.
  • e.g. A command like cat would not be possible for Chuck to exploit in a meaningful way.

Example Exploit Commands:

In kustomize:

# User enables the plugin and allows it to exec sh
$ kustomize build https://github.com/chuck/maliciousapp --enable-kust-plugins --kust-plugin-path=/bin

In kubectl:

# User enables the plugin and allows it to exec sh
$ kubectl apply -k https://github.com/chuck/maliciousapp --enable-kust-plugins --kust-plugin-path=/bin

Analysis:

This is a low risk profile. Alice has to: kustomize a malicious kustomization file, explicitly enable plugins as command line arguments, and install an exploitable plugin to exploit the plugin system.

By default kustomize will not-execute the plugins, and print out which plugins were specified with their arguments.

Alternatives Considered

Git Style Plugins

Create a plugin mechanism similar to git or kubectl plugins which supports sub commands.

kubectl and git plugins are targeted at providing a clean interactive UX with extensible sub commands. Because the plugins will be invoked declaratively rather than imperatively support for this sort of UX is not necessary.

Enable Plugins with Environment Variables

Allow users to specify export KUSTOMIZE_ENABLE_PLUGINS=true instead of providing the flag.

Since this could be set in the environment, it doesn’t provide much additional security over the plugin directories.

Graduation Criteria

Target Launch Status: Beta

This is an integration of the previous execution SecretGenerator mechanism, and as such is relatively well understood.

Graduation to GA

Gather user feedback. Determine if there are addressable gaps.

Consider the following:

Audit Command

Add a new command in Kustomize: kustomize audit dir/

This will print out which commands (including args and flags) will be invoked when Kustomize is run, without actually invoking them. This will generally be helpful for users to view how secrets are generated.

More OS Specific install locations

Consider defaulting search path to include OS specific install locations such as /Library/ or ~/Library on OS X.

Testing and documentation.

Testing

TBD.

Documentation

TBD

Implementation History

(TODO add PR’s here)