KEP-692: Publish CRD OpenAPI

Publish CRD OpenAPI

Summary
Motivation
- Goals
- Non-Goals
Proposal
Graduation Criteria
Implementation History

Summary

In CustomResourceDefinition (CRD) we allow CRD author to define OpenAPI v3 schema, to enable server-side validation for CustomResources (CR) . The validation schema format is compatible for creating OpenAPI documentation for CRs, which can be used by clients like kubectl to perform client-side validation (e.g. kubectl create and kubectl apply), schema explanation (kubectl explain), and client generation. This KEP proposes using the OpenAPI v3 schema to create and publish OpenAPI documentation for CRs.

Motivation

Publishing CRD OpenAPI enables client-side validation, schema explanation and client generation for CRs. It covers the gap between CR and native Kubernetes APIs, which already support OpenAPI documentation.

Publishing CRD OpenAPI is also noted as potential followup in Validation for CustomResources Implementation Plan .

Goals

For every CRD that we serve, publish Paths (operations that we support on resource and subresources) and Definitions (for both CR object and CR list object) in OpenAPI documentation to fully demonstrate the existence of the API.
For CRDs with schema defined, the CR object Definition should include both CRD schema and native Kubernetes ObjectMeta and TypeMeta properties.
For CRDs without schema, the CR object definition will be as complete as possible while still maintaining compatibility with the openapi spec and with supported kubernetes components

Non-Goals

Finalize design on how to aggregate and publish CRD OpenAPI from kube-aggregator

Proposal

Ref: https://github.com/kubernetes/kubernetes/pull/71192

Build Schema/Definition

We drop the features that OpenAPI v2 don’t support from CRD OpenAPI v3 schema (listed in this spread-sheet by @nikhita and @sttts: oneOf, anyOf and not) (proposed here ).

We add documentation in CRD API pointing it out to CRD authors that if they want a complete and valid OpenAPI spec being generated for their Resource, they should not use v3 features like oneOf (proposed here ).

For CRD with schema, we append Kubernetes TypeMeta and ObjectMeta to the schema.

Build Spec

We template a WebService object for each CRD like what we do for built-in APIs , feed the WebService to kube-openapi builder to build OpenAPI spec.

Aggregate and Publish Spec from apiextensions-apiserver

We add a spec aggregator to apiextensions-apiserver like kube-aggregator does . The aggregator keeps a {CR GroupVersionKind (GVK) -> OpenAPI spec} mapping, aggregates specs, and serves the spec through “/openapi/v2” handler from apiextensions-apiserver.

All the code mentioned above (CRD OpenAPI builder, aggregator and controller) live within apiextensions-apiserver staging repo and get called within the same repo.

Aggregate and Publish Spec from kube-aggregator

We add apiextensions-apiserver into the resync queue in kube-aggregator openapi controller with a resync period of 1 second. Previously apiextensions-apiserver (delegation target) was treated as static and never get resynced.

Downloading spec and etag from apiextensions-apiserver is in-process function call. We skip aggregation when etag doesn’t change. So we are not adding much load.

Alternative solutions discussed in this doc . Finalizing the design is a non-goal for 1.14.

Risks and Mitigations

Performance: OpenAPI aggregation has performed poorly in the past. Introducing aggregation that occurs on a higher frequency could cause performance issues in the apiserver. To mitigate this, improvements in OpenAPI merging are being made (both in terms of memory and CPU cost), and benchmarking will be done prior to merge to ensure we do not regress API server performance.

Compatibility: some Kubernetes components (notably kubectl) perform client-side validation based on published openapi schemas. To ensure that publishing schemas for additional custom resources does not regress existing function, we will test creation of custom resources with and without validation with current and prior versions of kubectl.

We will add a feature gate CustomResourcePublishOpenAPI, which can be turned on to start CRD openapi creation & aggregation & publishing. The feature gate will be Alpha (defaulted to False) in 1.14, Beta (defaulted to True) in 1.15 and Stable in 1.16.

Graduation Criteria

In 1.14, publishing CRD OpenAPI should be e2e-tested to ensure:

for CRD with validation schema

client-side validation (kubectl create and kubectl apply) works to:
- allow requests with known and required properties
- reject requests with unknown properties when disallowed by the schema
- reject requests without required properties
kubectl explain works to:
- explain CR properties
- explain CR properties recursively
- return error when explain is called on property that doesn’t exist

for CRD without validation schema

client-side validation (kubectl create and kubectl apply) works to:
- allow requests with any unknown properties
kubectl explain works to:
- returns an understandable error or message when asked to explain a CR with no available schema

for any CRD with/without validation schema, we do not regress existing behavior for prior version of kubectl:

kubectl create client-side validation works to:
- allow requests with any unknown properties
kubectl explain works to:
- returns an understandable error or message when asked to explain a CR with no available schema

for multiple CRDs

CRs in different groups (two CRDs) show up in OpenAPI documentation
CRs in the same group but different version (one multiversion CRD or two CRDs) show up in OpenAPI documentation
CRs in the same group version but different kind (two CRDs) show up in OpenAPI documentation

for publishing latency and cpu load

does not consume significant delay to update OpenAPI Paths and Definitions after CRD gets created/updated/deleted
does not measurably increase server load at steady state

for kubectl apply

verify no regression using apply with CRs with and without schemas, on the current and previous version of kubectl