- [CSI Health Monitoring](#csi-health-monitoring)
  - [Known Issues](#known-issues)
    - [<code>TopologyAwareHints</code> feature falls back to default behavior](#topologyawarehints-feature-falls-back-to-default-behavior)
  - [Urgent Upgrade Notes](#urgent-upgrade-notes)
    - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade)

- Prevent watch cache starvation by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior ([#123694](https://github.com/kubernetes/kubernetes/pull/123694), [@mengqiy](https://github.com/mengqiy)) [SIG API Machinery]

   - In version 1.8, enable this behavior by setting the
     `--horizontal-pod-autoscaler-use-rest-clients` flag to `true`.

   - In version 1.9, this behavior will be enabled by default, and must be explicitly
     disabled by setting the `--horizontal-pod-autoscaler-use-rest-clients` flag to `false`.

### Cluster Lifecycle

- The `auto-detect` behavior of the kubelet's `--cloud-provider` flag is deprecated.

- Fixed #118052: nodeAffinity on pods can change prior to scheduling gates being removed even when podSpec.affinity is nil in the initial spec, this matches the 1.28 behavior to allow consistent integrators to be written. ([#118199](https://github.com/kubernetes/kubernetes/pull/118199), [@aleskandro](https://github.com/aleskandro)) [SIG Apps, Scheduling and Testing]

- The current behavior for Services that `IPFamilyPolicy` set as `PreferDualstack`. The current behavior when the cluster is upgraded to dual-stack is:
  - Services that have been set to IPFamilyPolicy = PreferDualstack will be upgraded when the service object is updated. e.g., when a user change a label.
  
  This behavior will change to:

- Reverts graceful node shutdown to match 1.21 behavior of setting pods that have not yet successfully completed to "Failed" phase if the GracefulNodeShutdown feature is enabled in kubelet. The GracefulNodeShutdown feature is beta and must be explicitly configured via kubelet config to be enabled in 1.21+. This changes 1.22 and 1.23 behavior on node shutdown to match 1.21. If you do not want pods to be marked terminated on node shutdown in 1.22...

implementation is changed to not return an error when the specified target can be detected as not a mount point. On Linux, the behavior of detecting a mount point depends on `umount` command is validated when the mounter is created. Additionally, mount point checks will be skipped in CleanupMountPoint/CleanupMountWithForce if the mounter's Unmount having the changed behavior of not returning error when target is not a mount point. ([#109676](https://github.com/kubernetes/kubernetes/pull/109676), [...

implementation is changed to not return an error when the specified target can be detected as not a mount point. On Linux, the behavior of detecting a mount point depends on `umount` command is validated when the mounter is created. Additionally, mount point checks will be skipped in CleanupMountPoint/CleanupMountWithForce if the mounter's Unmount having the changed behavior of not returning error when target is not a mount point. ([#109676](https://github.com/kubernetes/kubernetes/pull/109676), [...

   ([#122276](https://github.com/kubernetes/kubernetes/pull/122276), [@pohly](https://github.com/pohly))

- Prevent watch cache starvation by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior ([#123693](https://github.com/kubernetes/kubernetes/pull/123693), [@mengqiy](https://github.com/mengqiy)) [SIG API Machinery]