// Optional: mode bits used to set permissions on this file, must be an octal value
  // between 0000 and 0777 or a decimal value between 0 and 511.
  // YAML accepts both octal and decimal values, JSON requires decimal values for mode bits.
  // If not specified, the volume defaultMode will be used.
  // This might be in conflict with other options that affect the file

- The userspace mode of kube-proxy no longer confusingly logs messages about deleting endpoints that it is actually adding. ([#83644](https://github.com/kubernetes/kubernetes/pull/83644), [@danwinship](https://github.com/danwinship))

volume availability, the API now enables volume resizing and snapshotting. These features are in alpha.
The SIG also focused on providing more control over storage: the ability to set requests and
limits on ephemeral storage, the ability to specify mount options, more metrics, and improvements to Flex driver deployments.

[SIG Storage]: https://github.com/kubernetes/community/tree/master/sig-storage

## Before Upgrading

*   deviceplugin has been extended to more gracefully handle the full device plugin lifecycle, including: ([#55088](https://github.com/kubernetes/kubernetes/pull/55088),[ @jiayingz](https://github.com/jiayingz))
    *   Kubelet now uses an explicit cm.GetDevicePluginResourceCapacity() function that makes it possible to more accurately determine what resources are inactive and return a more accurate view of available resources.

development of [ephemeral containers](https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/), it became more obvious how we can support developers with tooling built on top of `kubectl exec`. The addition of the `kubectl debug` [command](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/1441-kubectl-debug/README.md) (it is alpha but your feedback is more than welcome), allows developers to easily debug their Pods inside the cluster. We think this addition is invaluable....

  new callbacks for more granularity. Deprecated the `OnPodDeletedOrEvicted`
  node drain helper callback. ([#117502](https://github.com/kubernetes/kubernetes/pull/117502), [@adilGhaffarDev](https://github.com/adilGhaffarDev))

#### Extensibility

- The community has been asking for continuing support of extensibility, so this cycle features more work around CRDs and API Machinery. Most of the enhancements in this cycle were from SIG API Machinery and related areas.

### Extensibility around core Kubernetes APIs

where <code>P</code> is a type parameter and <code>C</code>
its corresponding constraint, the unification rules are bit
more complicated:
</p>

<ul>
<li>
	If <code>C</code> has a <a href="#Core_types">core type</a>
	<code>core(C)</code>
	and <code>P</code> has a known type argument <code>A</code>,
	<code>core(C)</code> and <code>A</code> must unify loosely.
	If <code>P</code> does not have a known type argument

* add goroutine to move unschedulable pods to activeq if they are not retried for more than 1 minute ([#72558](https://github.com/kubernetes/kubernetes/pull/72558), [@denkensk](https://github.com/denkensk))

* Collect metrics from Heapster in Stackdriver mode. ([#50517](https://github.com/kubernetes/kubernetes/pull/50517), [@piosz](https://github.com/piosz))
* fixes a bug around using the Global config ElbSecurityGroup where Kuberentes would modify the passed in Security Group. ([#49805](https://github.com/kubernetes/kubernetes/pull/49805), [@nbutton23](https://github.com/nbutton23))