* Allow mounts to run in parallel for non-attachable volumes ([#28939](https://github.com/kubernetes/kubernetes/pull/28939), [@saad-ali](https://github.com/saad-ali))
* add enhanced volume and mount logging for block devices ([#24797](https://github.com/kubernetes/kubernetes/pull/24797), [@screeley44](https://github.com/screeley44))

### Non-recursive Volume Ownership (FSGroup) graduates to Beta

By default, the `fsgroup` setting, if specified, recursively updates permissions for every file in a volume on every mount. This can make mount, and pod startup, very slow if the volume has many files. 

    * - gcr.io/google-containers/kube-dnsmasq-amd64
* - Disable thin_ls due to excessive iops ([#43113](https://github.com/kubernetes/kubernetes/pull/43113), [@dashpole](https://github.com/dashpole))
    * - Ignore .mount cgroups, fixing dissappearing stats
    * - Fix wc goroutine leak
    * - Update aws-sdk-go dependency to 1.6.10

implementation is changed to not return an error when the specified target can be detected as not a mount point. On Linux, the behavior of detecting a mount point depends on `umount` command is validated when the mounter is created. Additionally, mount point checks will be skipped in CleanupMountPoint/CleanupMountWithForce if the mounter's Unmount having the changed behavior of not returning error when target is not a mount point. ([#109676](https://github.com/kubernetes/kubernetes/pull/109676), [@cartermckin...

		writeErrorResponse(ctx, w, errorCodes.ToAPIErr(s3Error), r.URL)
		return
	}

	// check if client is attempting to create more buckets, complain about it.
	if currBuckets := globalBucketMetadataSys.Count(); currBuckets+1 > maxBuckets {
		internalLogIf(ctx, fmt.Errorf("Please avoid creating more buckets %d beyond recommended %d", currBuckets+1, maxBuckets), logger.WarningKind)
	}

	opts := MakeBucketOptions{

	}
	versions, buf, err = msgp.ReadIntBytes(buf)
	if err != nil {
		return 0, 0, 0, buf, err
	}
	if versions < 0 {
		return 0, 0, 0, buf, fmt.Errorf("decodeXLHeaders: Negative version count %d", versions)
	}
	return versions, hdrVer, metaVer, buf, nil
}

// decodeVersions will decode a number of versions from a buffer
// and perform a callback for each version in order, newest first.

- Kubelet now hard rejects pods with AppArmor if the node does not have AppArmor. ([#125776](https://github.com/kubernetes/kubernetes/pull/125776), [@vinayakankugoyal](https://github.com/vinayakankugoyal))
- Mount-utils: treated syscall.ENODEV as corrupted mount. ([#126174](https://github.com/kubernetes/kubernetes/pull/126174), [@dobsonj](https://github.com/dobsonj))

#### Storage

* [stable] Mount options

  * The ability to specify mount options for volumes is moved from beta to stable.

  * A new `MountOptions` field in the `PersistentVolume` spec is available to specify mount options. This field replaces an annotation.

  * A new `MountOptions` field in the `StorageClass` spec allows configuration of mount options for dynamically provisioned volumes.

in containers are now `rslave` on Linux by default. To make this default work in all Linux environments the entire mount tree should be marked as shareable, e.g. via `mount --make-rshared /`. All Linux distributions that use systemd already have the root directory mounted as rshared and hence they need not do anything. In Linux environments without systemd we recommend running `mount --make-rshared /` during boot before docker is started, ([@jsafrane](https://github.com/jsafrane))

##...

     com.squareup.okio:okio:1.14.0
     ```

 *  Fix: Handle `HTTP/1.1 100 Continue` status lines, even on requests that did
    not send the `Expect: continue` request header.
 *  Fix: Do not count web sockets toward the dispatcher's per-host connection
    limit.
 *  Fix: Avoid using invalid HTTPS sessions. This prevents OkHttp from crashing
    with the error, `Unexpected TLS version: NONE`.