In addition, support for NVMe disks has been added to Kubernetes, and a service of type LoadBalancer can now be backed with an NLB instead of an ELB (alpha).

### Azure

SIG Azure worked on improvements in the cloud provider, including significant work on the Azure Load Balancer implementation.

### Cluster Lifecycle

  
  ### Kubemark
  - Skip non-kubemark nodes while computing node information for node groups.
  
  ### Magnum
  - Add Magnum support in the Cluster Autoscaler helm chart
  
  ### Packet
  - Allow empty nodepools
  - Add support for multiple nodepools
  - Add pricing support
  
  ## Image

* APIserver backed by etcdv3 exports metric shows number of resources per kind. ([#59757](https://github.com/kubernetes/kubernetes/pull/59757), [@gmarek](https://github.com/gmarek))

  <mime-type type="application/vnd.tao.intent-module-archive">
    <glob pattern="*.tao"/>
  </mime-type>

  <mime-type type="application/vnd.tcpdump.pcap">
    <_comment>TCPDump pcap packet capture</_comment>
    <magic priority="60">
      <match value="0xa1b2c3d4" type="big32" offset="0" />
      <match value="0xd4c3b2a1" type="big32" offset="0" />
    </magic>
    <glob pattern="*.pcap"/>

- Kubelet config file will be backed up to `/etc/kubernetes/tmp/` folder with `kubeadm-kubelet-config` append with a random suffix as the filename ([#114695](https://github.com/kubernetes/kubernetes/pull/114695), [@chendave](https://github.com/chendave)) [SIG Cluster Lifecycle]

- The `EmptyDir` memory backed volumes are sized as the minimum of pod allocatable memory on a host and an optional explicit user provided value. ([#101048](https://github.com/kubernetes/kubernetes/pull/101048), [@dims](https://github.com/dims))

 - Fix: support removal of nodes backed by deleted non VMSS instances on Azure ([#91184](https://github.com/kubernetes/kubernetes/pull/91184), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider]

 - Fix: azure disk dangling attach issue which would cause API throttling ([#90749](https://github.com/kubernetes/kubernetes/pull/90749), [@andyzhangx](https://github.com/andyzhangx)) [SIG Cloud Provider]
 - Fix: support removal of nodes backed by deleted non VMSS instances on Azure ([#91184](https://github.com/kubernetes/kubernetes/pull/91184), [@bpineau](https://github.com/bpineau)) [SIG Cloud Provider]

* Registries backed by the generic Store's `Update` implementation support delete-on-update, which allows resources to be automatically deleted during an update provided: ([#48065](https://github.com/kubernetes/kubernetes/pull/48065), [@ironcladlou](https://github.com/ironcladlou))...

* Resolves an issue serving aggregated APIs backed by services that respond to requests to `/` with non-2xx HTTP responses ([#79895](https://github.com/kubernetes/kubernetes/pull/79895), [@deads2k](https://github.com/deads2k))