KEP-2523: consistent-resource-version-semantics

Implementation History
STABLE Implemented
Created 2020-03-09
Updated 2020-06-27
Latest 1.19
Ownership
Owning SIG
SIG API Machinery
Primary Authors
@jpbetz

Consistent Resource Version Semantics for List

Table of Contents

Summary

Make resource version semantics consistent for list requests regardless of pagination.

Motivation

Resource version semantics are inconsistent when using pagination. When a list request is made with a resourceVersion but no limit, “Not older than” semantics apply, but once a limit is set, “Exact” semantics apply. See API Concepts: Resource Version for details about the semantics.

The inconsistency is confusing and problematic. A client expecting “Not older than” semantics has no reason to expect the “410 Gone” HTTP responses from the server that can be sent when “Exact” semantics are applied and when the requested resource version is older than what is available in watch cache or the etcd compaction window. Even if the author of code to make a list request was aware of this inconsistency, the code to make a list request might be separated from the code setting a limit on the request. For example, ListerWatchers can be layered in to add pagination limits.

This was discussed in Feb 12th, 2020 SIG API machinery bi-weekly meeting .

Goals

  • Improve API to make resource version semantics consistent.
  • Backward compaibility. In particular, support the case of a client making a request using the updated API against an server serving the old API.

Proposal

Add a ResourceVersionMatch query parameter

Add an optional ResourceVersionMatch paramater to ListOptions with the enumeration values:

// ResourceVersionMatch specifies how the ResourceVersion parameter is applied. ResourceVersionMatch
// only applies to list calls and may only be set if ResourceVersion is also set.
type ResourceVersionMatch string

const (
	// ResourceVersionMatchNotOlderThan matches data at least as new as the provided
	// ResourceVersion. The newest available data is preferred, but any data not
	// older than this ResourceVersion may be served.
	// For list calls, this guarantees that ResourceVersion in the ListMeta is not older than the requested
	// ResourceVersion, but does not make any guarantee about the ResourceVersion in the ObjectMeta
	// of the list items since ObjectMeta.ResourceVersion tracks when an object was last updated,
	// not how up-to-date the object is when served.
	ResourceVersionMatchNotOlderThan ResourceVersionMatch = "NotOlderThan"
	// ResourceVersionMatchExact matches data at the exact ResourceVersion
	// provided. If the provided ResourceVersion is unavailable, the server responds with
	// HTTP 410 “Gone”.
	// For list calls, this guarantees that ResourceVersion in the ListMeta is the same as the requested
	// ResourceVersion, but does not make any guarantee about the ResourceVersion in the ObjectMeta
	// of the list items since ObjectMeta.ResourceVersion tracks when an object was last updated,
	// not how up-to-date the object is when served.
	ResourceVersionMatchExact ResourceVersionMatch = "Exact"
)

type ListOptions struct {
    ...
	// ResourceVersion sets a constraint on what resource versions a request may be served from.
	// If unset, the request is served from latest resource version to ensure strong consistency
	// (i.e. served from etcd via a quorum read). Setting a ResourceVersion is preferable in cases
	// where a ResourceVersion is known and can be used as a constraint since better performance
	// and scalability can be achieved for a cluster by avoiding quorum reads.
	// When ResourceVersion is set for list, it is highly recommended ResourceVersionMatch is also set.
	//
	// ResourceVersion for watch:
	// - if unset or 0, start a watch at any resource version, the most recent resource version available
	//   is preferred, but not required; any starting resource version is allowed. It is
	//   possible for the watch to start at a much older resource version that the client
	//   has previously observed, particularly in high availability configurations, due to
	//   partitions or stale caches. Clients that cannot tolerate this should not start a
	//   watch with this semantic. To establish initial state, the watch begins with synthetic
	//   “Added” events for all resources instances that exist at the starting resource version.
	//   All following watch events are for all changes that occurred after the resource version
	//   the watch started at;
	// - if non-zero, start a watch at an exact resource version. The watch events are for all
	//   changes after the provided resource version. Watch is not started with synthetic “Added”
	//   events for the provided resource version. The client is assumed to already have the
	//   initial state at the starting resource version since the client provided the resource version.
	//
	// ResourceVersion for list:
	// - if unset, then the result is returned at the most recent resource version. The returned
	//   data must be consistent (i.e. served from etcd via a quorum read);
	// - if set and ResourceVersionMatch is set, ResourceVersion is applied according to the ResourceVersionMatch rule;
	// - if set and ResourceVersionMatch is unset or the server ignores ResourceVersionMatch, the legacy behavior applies:
	//   - if 0, the result may be at any resource version. The newest available resource version
	//     is preferred, but strong consistency is not required; data at any resource version may
	//     be served. It is possible for the request to return data at a much older resource version
	//     that the client has previously observed, particularly in high availability configurations,
	//     due to partitions or stale caches. Clients that cannot tolerate this should not use this
	//     semantic;
	//   - if non-zero and Limit is unset, the ResourceVersionMatchNotOlderThan rule applies implicitly;
	//   - if non-zero and Limit is set, the ResourceVersionMatchExact rule applies implicitly.
	//
	// Defaults to unset
	// +optional
	ResourceVersion string `json:"resourceVersion,omitempty" protobuf:"bytes,4,opt,name=resourceVersion"`

	// ResourceVersionMatch determines how ResourceVersion is applied to list calls.
	// It is highly recommmend that ResourceVersionMatch be set for list calls where ResourceVersion is
	// set. If ResourceVersion is unset, ResourceVersionMatch is ignored.
	// For backward compatibility, clients must tolerate the server ignoring ResourceVersionMatch:
	// - When using ResourceVersionMatchNotOlderThan and Limit is set, clients must handle HTTP 410 “Gone” responses.
	//   For example, the client might retry with a newer ResourceVersion or fall back to ResourceVersion="".
	// - When using ResourceVersionMatchExact and Limit is unset, clients must verify that the ResourceVersion in the
	//   ListMeta of the response matches the requested ResourceVersion, and handle the case where it does not. For
	//   example, the client might fall back to the a request with Limit set.
	// Unless you have strong consistency requirements, using ResourceVersionMatchNotOlderThan and a known
	// ResourceVersion is preferable since it can achieve better performance and scalability of your cluster
	// than leaving ResourceVersion and ResourceVersionMatch unset, which requires quorum read to be served.
	// Defaults to unset
	// +optional
	ResourceVersionMatch ResourceVersionMatch `json:"resourceVersionMatch,omitempty" protobuf:"bytes,10,opt,name=resourceVersionMatch"`
    ...
}

Backward Compatibility

Versions of the kube-apiserver that pre-date the introduction of ResourceVersionMatch parameter will ignore it. Client will need to tolerate this as documented on ResourceVersionWatch:

  • When using ResourceVersionMatchNotOlderThan and Limit is set, clients must handle HTTP 410 “Gone” responses. For example, the client might retry with a newer ResourceVersion or fall back to a ResourceVersion="" request.
  • When using ResourceVersionMatchExact and Limit is unset, clients must verify that the ResourceVersion in the ListMeta of the response matches the requested ResourceVersion, and handle the case where it does not. For example, the client might fall back to the a request with Limit set.

When ‘ResourceVersionMatch’ is not provided, the behavior is the same as before it was introduced.

Impact on get calls

We will not add ResourceVersionMatch to get calls.

get provides consistent NotOlderThan semantics when ResourceVersion is set, which are easy to understand and doesn’t need to be changed.

get does not need the flexibility of Exact resource version lookup since it is already possible to use a list with fieldSelector=metadata.name=foo to get a single object with list.

Risks and Mitigations

The main risk to this proposal that it complicates the API surface area, resulting in an API that is more difficult to understand and use. But the existing behavior already complicates the API. With this proposal, clients at least have the ability to use a consistent set of semantics.

Another risk is that clients will either not realize, or not be sufficiently motivated, to update their code to move away from the legacy behavior. This can be mitigated a couple ways:

  • Update client libraries (client-go, …) to discourage using the legacy behavior, and eventually to disallow it.
  • At some point in the future, start providing warnings to the client that legacy behavior. E.g. via https://github.com/kubernetes/kubernetes/pull/73032 .

Alternatives Considered

Alternative: Introduce ExactResourceVersion and MinResourceVersion parameters

Deprecate ResourceVersion and introduce ExactResourceVersion and MinResourceVersion.

Backward Compatibility:

Versions of the kube-apiserver that pre-date the introduction of the ExactResourceVersion and MinResourceVersion parameters will ignore them, resulting in a quorum read. For clients to tolate responses from servers that ignore the new parameters in a backward compatible way:

  • When using MinResourceVersion, clients can either:
    • Also include ResourceVersion and get legacy semantics if MinResourceVersion is ignored. They will need to handle HTTP 410 “Gone” responses if Limit is set.
    • Tolerate a quorum read, which is guaranteed to provided the newest ResourceVersion but has scalability/performance implications.
  • When using ExactResourceVersion also set ResourceVersion and check the ResourceVersion in the ListMeta of the response. If it does not match, fall back to a request with a Limit set.

Advantages

  • Deprecating ResourceVersion highlights to API users that they need change calls that use ResourceVersion.

Disadvantages

  • Since the field will deprecated but never removed, in practice we have 3 options to understand instead of 2.
  • Clients have to include ResourceVersion even when using the new parameters for backward compatibility, resulting in ?ResourceVersion=3847&MinResourceVersion=3847 which is less readable than ?ResourceVersion=3847&ResourceVersionMatch=NotOlderThan.

Alternative: Use syntax in the query string

Introduce syntax (=N and >=N) instead of additional parameters.

Advantages

  • consise query parameters: resourceVersion=234, resourceVersion>=234.

Disadvantages

  • Still need to deal with backward compatibility, so we end up having to ask clients to do resourceVersion=234&resourceVersion>=234.
  • Not clear how we would support ‘Exact’ matches. == doesn’t work as well in query params so we’d need to select something else, and we already use = for the legacy case.

Implementation History