#### New Deprecations

* `metrics-server` is now the recommended way to provide the resource
  metrics API. Deploy `metrics-server` as an add-on in the same way that you deploy Heapster.

##### Cluster Autoscaler

* Cluster autoscaler is now GA
* Cluster support size is increased to 1000 nodes

- PodSecurityPolicy objects now support a `MayRunAs` rule for `fsGroup` and `supplementalGroups` options. This allows specifying ranges of allowed GIDs for pods/containers without forcing a default GID the way `MustRunAs` does. This means that a container to which such a policy applies to won't use any fsGroup/supplementalGroup GID if not explicitly specified, yet a specified GID must still fall in the GID range according to the policy. ([#65135](https:/...

This release also contains a number of bug fixes to the feature set. Aspiring plugin authors can look at [sample-exec-plugin](https://github.com/ankeesler/sample-exec-plugin) as a way to get started.

Related to this topic, the in-tree Azure and GCP authentication plugins have been [deprecated](https://github.com/kubernetes/kubernetes/pull/102181) in favor of out-of-tree implementations.

* nil admission options are now consistent with other nil options. ([#63993](https://github.com/kubernetes/kubernetes/pull/63993), [@tamalsaha](https://github.com/tamalsaha))
* Add a way to pass extra arguments to etcd. ([#63961](https://github.com/kubernetes/kubernetes/pull/63961), [@mborsz](https://github.com/mborsz))

## What’s New (Major Themes)

### Cloud Provider Labels reach General Availability

Added as a beta feature way back in v1.2, v1.17 sees the general availability of cloud provider labels.

### Volume Snapshot Moves to Beta

  * Rescheduler relies on taints, which were changed in a backward incompatible way. Rescheduler 0.3 shipped together with Kubernetes 1.6 understands the new taints and will clean up the old annotations, but Rescheduler 0.2 shipped together with Kubernetes 1.5 doesn't understand the new taints. In order to avoid eviction loop during...

## Changelog since v1.22.0

## What's New (Major Themes)

### Deprecation of FlexVolume

FlexVolume is deprecated. Out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes.
See [this doc](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors) for more information.

	<code>(c<sub>j</sub>, P<sub>k</sub>)</code> with that same type parameter
	were collected, determine the <a href="#Constant_expressions">constant kind</a>
	of the constants <code>c<sub>j</sub></code> in all those pairs the same way as for
	<a href="#Constant_expressions">constant expressions</a>.
	The type argument for <code>P<sub>k</sub></code> is the
	<a href="#Constants">default type</a> for the determined constant kind.

* Add a way to pass extra arguments to etcd. ([#63961](https://github.com/kubernetes/kubernetes/pull/63961), [@mborsz](https://github.com/mborsz))



# v1.9.8