* Make volume usage metrics available for Cinder ([#62668](https://github.com/kubernetes/kubernetes/pull/62668), [@zetaab](https://github.com/zetaab))

  * Dynamic Provisioning of PersistentVolumes: Kubernetes previously required all
volumes to be manually provisioned by a cluster administrator before use. With
this feature, volume plugins that support it (GCE PD, AWS EBS, and Cinder) can
automatically provision a PersistentVolume to bind to an unfulfilled
PersistentVolumeClaim.
  * Run multiple schedulers in parallel, e.g. one or more custom schedulers

             name: azure-cloud-provider
             namespace: kube-system
           - kind: ServiceAccount
             name: persistent-volume-binder
             namespace: kube-system
    ```

### Other notable changes

* Update metrics-server to v0.3.1 ([#68746](https://github.com/kubernetes/kubernetes/pull/68746), [@DirectXMan12](https://github.com/DirectXMan12))

				"application/ibe-pp-data",
				"application/iges",
				"application/illustrator",
				"application/im-iscomposing+xml",
				"application/index",
				"application/index.cmd",
				"application/index.obj",
				"application/index.response",
				"application/index.vnd",
				"application/iotp",
				"application/ipp",
				"application/isup",
				"application/java-archive",

  when the corresponding csi driver was installed:
  - `awsElasticBlockStore` for `ebs.csi.aws.com`
  - `azureDisk` for `disk.csi.azure.com`
  - `gcePersistentDisk` for `pd.csi.storage.googleapis.com`
  - `cinder` for `cinder.csi.openstack.org`
  - `csi`
  However it was still enforced using a limit in CSINode objects. ([#126924](https://github.com/kubernetes/kubernetes/pull/126924), [@carlory](https://github.com/carlory))

conjunction with SIG Cloud Provider, is focusing on moving internal APIs that the OpenStack (and other providers) depends upon to staging to guarantee API stability. This work also included abstracting the in-tree Cinder API and refactoring code to the external Cinder provider to remove additional Cinder volume provider code.

Additional work was also done to implement an OpenStack driver for the Cluster API effort lead by SIG Cluster Lifecycle. For the external Cloud-Provider-OpenStack code, the...

listen to her, still it was good practice to say it over) `--yes,
that's about the right distance--but then I wonder what Latitude
or Longitude I've got to?'  (Alice had no idea what Latitude was,
or Longitude either, but thought they were nice grand words to
say.)

  Presently she began again.  `I wonder if I shall fall right
THROUGH the earth!  How funny it'll seem to come out among the

KubeProxyConfiguration Before:

```
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
**featureGates: "SupportIPVSProxyMode=true"**
```

KubeProxyConfiguration After:

```
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
**featureGates:**
**  SupportIPVSProxyMode: true**
```

- OpenStack Cinder CSI migration is now GA and switched on by default, Cinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work (has been since v1.21). ([#107462](https://github.com/kubernetes/kubernetes/pull/107462), [@dims](https://github.com/dims))

* fix cinder volume dir umount issue [#24717](https://github.com/kubernetes/kubernetes/pull/24717) ([#24718](https://github.com/kubernetes/kubernetes/pull/24718), [@chengyli](https://github.com/chengyli))