### Internal Storage Layer

    - [SIG Node](#sig-node)
    - [SIG Network](#sig-network)
    - [SIG Scalability](#sig-scalability)
    - [SIG Scheduling](#sig-scheduling)
    - [SIG Storage](#sig-storage)
  - [Before Upgrading](#before-upgrading)
  - [Known Issues](#known-issues)
  - [Deprecations](#deprecations)
    - [Apps](#apps)
    - [Auth](#auth)
    - [Autoscaling](#autoscaling)
    - [Cluster Lifecycle](#cluster-lifecycle)

    - [Windows](#windows)
    - [OpenStack](#openstack)
    - [API-machinery](#api-machinery)
    - [Auth](#auth)
    - [Azure](#azure)
    - [CLI](#cli)
    - [Network](#network)
  - [Before Upgrading](#before-upgrading)
  - [Known Issues](#known-issues)
  - [Deprecations](#deprecations)
  - [Other Notable Changes](#other-notable-changes-13)
    - [Apps](#apps)
    - [AWS](#aws)
    - [Auth](#auth-1)

    - [Client Binaries](#client-binaries-16)
    - [Server Binaries](#server-binaries-16)
    - [Node Binaries](#node-binaries-16)
  - [**Major Themes**](#major-themes)
  - [**Action Required Before Upgrading**](#action-required-before-upgrading)
    - [Network](#network)
    - [Storage](#storage)
    - [API Machinery](#api-machinery)
    - [Controller Manager](#controller-manager)
    - [kubectl (CLI)](#kubectl-cli)

    - [Instrumentation](#instrumentation)
    - [Network](#network)
    - [Node](#node)
    - [OpenStack](#openstack)
    - [Storage](#storage)
    - [Windows](#windows)
  - [Before Upgrading](#before-upgrading)
    - [**API Machinery**](#api-machinery-1)
    - [**Auth**](#auth-1)
    - [**CLI**](#cli)
    - [**Cluster Lifecycle**](#cluster-lifecycle-1)
    - [**Multicluster**](#multicluster)

    - [SIG OpenStack](#sig-openstack)
    - [SIG Scheduling](#sig-scheduling)
    - [SIG Storage](#sig-storage)
    - [SIG Windows](#sig-windows)
  - [Known Issues](#known-issues)
  - [Before Upgrading](#before-upgrading)
      - [New Deprecations](#new-deprecations)
      - [Removed Deprecations](#removed-deprecations)
      - [Graduated to Stable/GA](#graduated-to-stablega)
      - [Graduated to Beta](#graduated-to-beta)

## Urgent Upgrade Notes

### (No, really, you MUST do this before you upgrade)

Before upgrading to Kubernetes 1.13, you must keep the following in mind:

- kube-apiserver

  
  vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3.
  
  VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15.
  

- Multiple Kubernetes clusters can now share the same resource group
    - When upgrading from previous releases, issues will arise with public IPs if multiple clusters share the same resource group. To solve these problems, make the following changes to the cluster:

* First to FastAPI `0.68.2`, with no breaking changes, but upgrading all the sub-dependencies.
* Next to FastAPI `0.69.0`, which upgrades Starlette to `0.15.0`, with AnyIO support, and a higher chance of having breaking changes in your code.
* Finally to FastAPI `0.70.0`, just upgrading Starlette to the latest version `0.16.0` with additional bug fixes.