- Added Custom Resource support to `kubectl autoscale` ([#72678](https://github.com/kubernetes/kubernetes/pull/72678), [@rmohr](https://github.com/rmohr))
- Cinder volume limit can now be configured from node too ([#74542](https://github.com/kubernetes/kubernetes/pull/74542), [@gnufied](https://github.com/gnufied))

### **OpenStack**

*   OpenStack Cinder support has been improved:
    *   Cinder version detection now works properly. ([#53115](https://github.com/kubernetes/kubernetes/pull/53115),[ @FengyunPan](https://github.com/FengyunPan))

 - OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. ([#98538](https://github.com/kubernetes/kubernetes/pull/98538), [@dims](https://github.com/dims)) [SIG Storage]

- Fix filtering of server status.
- Fix resizing PVC of Cinder volume.
- Disable load balancer configuration if it is not defined in cloud config.
- Add support for node shutdown taint.

The external provider includes all of the above with the additional fixes and features:
- Fix bug to prevent allocation of existing floating IP.
- Fix Cinder authentication bug when OS_DOMAIN_NAME not specified.

",groupCheckedEnd:" item(s)",badCreditCard:"The credit card number is not correct",badCVV:"The CVV number was not correct",wrongFileDim:"Incorrect image dimensions,",imageTooTall:"the image can not be taller than",imageTooWide:"the image can not be wider than",imageTooSmall:"the image was too small",min:"min",max:"max",imageRatioNotAccepted:"Image ratio is not be accepted",badBrazilTelephoneAnswer:"The phone number entered is invalid",badBrazilCEPAnswer:"The CEP entered is invalid",badBrazilCPFAnswer:"The...

  * Dynamic Provisioning of PersistentVolumes: Kubernetes previously required all
volumes to be manually provisioned by a cluster administrator before use. With
this feature, volume plugins that support it (GCE PD, AWS EBS, and Cinder) can
automatically provision a PersistentVolume to bind to an unfulfilled
PersistentVolumeClaim.
  * Run multiple schedulers in parallel, e.g. one or more custom schedulers

				"application/vnd.fujixerox.art4",
				"application/vnd.fujixerox.hbpl",
				"application/vnd.fujixerox.ddd",
				"application/vnd.fujixerox.docuworks",
				"application/vnd.fujixerox.docuworks.binder",
				"application/vnd.fut-misnet",
				"application/vnd.fuzzysheet",
				"application/vnd.genomatix.tuxedo",
				"application/vnd.geogebra.file",
				"application/vnd.geogebra.tool",

conjunction with SIG Cloud Provider, is focusing on moving internal APIs that the OpenStack (and other providers) depends upon to staging to guarantee API stability. This work also included abstracting the in-tree Cinder API and refactoring code to the external Cinder provider to remove additional Cinder volume provider code.

Additional work was also done to implement an OpenStack driver for the Cluster API effort lead by SIG Cluster Lifecycle. For the external Cloud-Provider-OpenStack code, the...

  when the corresponding csi driver was installed:
  - `awsElasticBlockStore` for `ebs.csi.aws.com`
  - `azureDisk` for `disk.csi.azure.com`
  - `gcePersistentDisk` for `pd.csi.storage.googleapis.com`
  - `cinder` for `cinder.csi.openstack.org`
  - `csi`
  However it was still enforced using a limit in CSINode objects. ([#126924](https://github.com/kubernetes/kubernetes/pull/126924), [@carlory](https://github.com/carlory))

* Scheduling conformance tests related to daemonsets should set the annotation that relaxes node selection restrictions, if any are set. This ensures conformance tests can run on a wider array of clusters. ([#68793](https://github.com/kubernetes/kubernetes/pull/68793), [@aveshagarwal](https://github.com/aveshagarwal))