Kubernetes 1.20 introduces the new version of the CronJob controller, which uses the informer mechanism to replace polling, optimizing the performance. You can set
kube-controller -manager to enable the new version. The new version will be enabled by default on later Kubernetes versions.
Dockershim is being deprecated. Support for Docker is deprecated and will be removed from a future release. Docker-produced images will continue to work in your cluster with all CRI compliant runtimes as Docker images follow the Open Container Initiative (OCI) image specification.
For more information, see Don't Panic: Kubernetes and Docker and Dockershim Deprecation FAQ.
The log message and Kubernetes object reference structures are standardized to make log parsing, processing, storage, query, and analysis easier. Two methods are added to klog to support structured logs:
--logging-format parameter is added to all components, and its default value is
text in the previous format. You can set it to
json to support structured logs, and the following parameters will become invalid:
A longstanding bug regarding exec probe timeouts that may impact existing Pod definitions has been fixed. Prior to this fix, the
timeoutSeconds field was not respected for exec probes. Instead, probes would run indefinitely, even past their configured deadline, until a result was returned. With this change, the default value of
1 second will be applied if a value is not specified and existing Pod definitions may no longer be sufficient if a probe takes longer than one second. A feature gate, called
ExecProbeTimeout, has been added with this fix that enables you to revert to the previous behavior, but this will be locked and removed in subsequent releases. In order to revert to the previous behavior, you should set this feature gate to
For more information, see Configure Liveness, Readiness and Startup Probes.
This feature provides a standard way to trigger volume snapshot operations and allows you to incorporate snapshot operations in a portable manner on any Kubernetes environment and supported storage providers.
Additionally, these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced and enterprise-grade storage administration features for Kubernetes, including application or cluster level backup solutions.
Note that snapshot support requires Kubernetes distributors to bundle and deploy the snapshot controller, snapshot CRDs, and validation webhook. A CSI driver supporting the snapshot feature must also be deployed in the cluster.
kubectl alpha debug command graduates to beta, becoming
kubectl debug. It supports common debugging workflows directly from kubectl, for example:
Troubleshoot workloads that crash on startup by creating a copy of the Pod with a different container image or command.
Troubleshoot distroless containers by adding a new container with debugging tools, either in a new copy of the Pod or using an ephemeral container. (Ephemeral containers (
EphemeralContainers) are an alpha feature that are not enabled by default.)
Troubleshoot on a node by creating a container running in the host namespaces and with access to the host's file system.
Note that as a new built-in command,
kubectl debug takes priority over any kubectl plugin named
debug. You need to rename the affected plugins.
kubectl alpha debug is now deprecated and will be removed in a subsequent release. Update your scripts to use
kubectl debug. For more information, see Debug Running Pods.
Introduced on Kubernetes 1.18, the API priority and fairness feature is now enabled by default on Kubernetes 1.20. This allows
kube-apiserver to categorize incoming requests by priority.
SupportNodePidsLimit (node-to-Pod PID isolation) and
SupportPodPidsLimit (ability to limit PIDs per Pod) move to GA.
Users and cluster admins expect that Pods will adhere to expected Pod lifecycle, including Pod termination. Currently, when a node shuts down, Pods do not follow the expected Pod termination lifecycle and are not terminated gracefully, which may cause issues for some workloads. The
GracefulNodeShutdown feature is now in alpha on Kubernetes 1.20. It makes the kubelet aware of node system shutdowns, enabling graceful termination of Pods during a system shutdown.
CSIDrivers can use the
fsGroupPolicy field to control whether ownership and permissions (
None) can be modified during mounting.
The following can be set in a non-recursive manner: fsgroup -
The Cloud Controller Manager component is added.
Features graduating to GA:
node.k8s.io/v1beta1is deprecated and replaced with
networking.k8s.io/v1beta1is deprecated (it will be removed on Kubernetes 1.22) and replaced by
container.seccomp.security.alpha.kubernetes.io/...are deprecated (they will be removed on Kubernetes 1.22). You can directly specify the following fields for Pods and container specs:
securityContext: seccompProfile: type: RuntimeDefault|Localhost|Unconfined ## choose one of the three localhostProfile: my-profiles/profile-allow.json ## only necessary if type == Localhost
Kubernetes converts annotations and fields automatically, so no additional operations are required.
certificates.k8s.io/v1version is added to
CertificateSigningRequest. When using
spec.signerNameand stop using
spec.usages, which can contain only known and unique usages.
status.certificatemust be PEM encoded and can contain only the
Features graduating to beta:
The following features graduate to beta and are enabled by default:
|Deprecated Version||New Version|
componentstatusAPI is deprecated. This API provided status of etcd, kube-scheduler and kube-controller-manager components, but only worked when those components were local to apiserver, and when kube-scheduler and kube-controller-manager exposed unsecured health endpoints.
--insecure-bind-addressparameters can be set, but are invalid. The
--insecure-portparameters can be set to only
0. These parameters will be removed on Kubernetes 1.24.
TokenRequestProjectiongraduate to GA. You need to set the following parameters for kube-apiserver:
--service-account-issuer: Fixed URL of the cluster API server.
--service-account-key-file: One or multiple public keys for token verification.
--service-account-signing-key-file: Private key for service account issuing, which can use the same file as the
--cloud-config, which are replaced with
metrics/resource/v1alpha1endpoint is removed and replaced with
failure-domain.beta.kubernetes.io/regionlabels are deprecated and replaced with
topology.kubernetes.io/regionrespectively. All users of the
failure-domain.beta...labels should switch to the
basic authauthentication method is no longer supported.
When you upgrade from Kubernetes 1.18 to 1.20, successful mounting of CSI ephemeral (inline) volumes cannot be guaranteed. If your application uses a CSI ephemeral volume, we recommend you convert it to a persistent volume before upgrade.
Deprecated and new labels are as listed below:
|Deprecated Label||New Label|
VolumeSnapshotDataSource is enabled by default. For more information, see Kubernetes 1.17 Feature: Kubernetes Volume Snapshot Moves to Beta.
CSI Migration is enabled by default. For more information, see Kubernetes 1.17 Feature: Kubernetes In-Tree to CSI Volume Migration Moves to Beta.
The Topology Manager feature moves to beta on Kubernetes 1.18. This feature enables NUMA alignment of CPU and devices (such as SR-IOV VFs) that will allow your workload to run in an environment optimized for low latency.
Prior to the introduction of the Topology Manager, the CPU and Device Manager would make resource allocation decisions independent of each other. This could result in undesirable allocations on multi-socket CPU systems, causing degraded performance on latency critical applications.
Server-Side Apply was promoted to beta on Kubernetes 1.16, but is now introducing a second beta (Server-Side Apply) on Kubernetes 1.18. This new version will track and manage changes to fields of all new Kubernetes objects, allowing you to know what changed your resources and when.
IngressClass resource is used to describe a type of Ingress within a Kubernetes cluster.
Ingresses can specify the class they are associated with by using a new
ingressClassName field on Ingresses. This new resource and field replace the deprecated
kubectl debug: Alpha feature.
Windows CSI support: Alpha feature.
ImmutableEphemeralVolumes: Alpha feature (it supports immutable ConfigMaps and Secrets without refreshing the corresponding volumes).
EndpointSlices: Disabled by default
CSIMigrationAWS: Disabled by default
The following features, which are enabled by default and cannot be configured, are removed.
The following built-in cluster roles are removed:
10.0.0.0/24) is deprecated. It must be set through the
--service-cluster-ip-rangeparameter on kube-apiserver.
rbac.authorization.k8s.io/v1beta1API groups are deprecated and will be removed on Kubernetes 1.20. Therefore, migrate your resources to
CSINodeInfofeature gate is deprecated. This feature has graduated to GA and is enabled by default.
cacheSize: 0is specified in the configuration file, versions earlier than 1.18 are automatically configured to cache 1,000 keys, while version 1.18 will report a configuration verification error. You can disable the cache by setting
cacheSizeto a negative value.
--feature-gates: The following features are enabled by default and can no longer be configured through the command line.
apps/v1beta2, which are replaced with
replicasets, which are replaced with
networkpolicies, which is replaced with
podsecuritypolicies, which is replaced with
--enable-cadvisor-endpoints: This parameter is disabled by default. To access the
cAdvisor v1 JSONAPI, you must enable it.
--redirect-container-streamingparameter is deprecated and will be removed on later versions. Kubernetes 1.18 supports only the default behavior (kubelet proxy for streaming requests). If
--redirect-container-streaming=trueis set, it must be removed.
/metrics/resource/v1alpha1endpoint is deprecated and replaced with
--healthz-portis deprecated and replaced with
--metrics-portis deprecated and replaced with
EndpointSliceProxyingfeature gate (disabled by default) is added to control whether to enable EndpointSlices in kube-proxy. The
EndpointSlicefeature gate no longer affects the behaviors of kube-proxy.
scheduling_duration_secondsmetric is deprecated.
scheduling_algorithm_predicate_evaluation_secondsis deprecated and replaced with
scheduling_algorithm_priority_evaluation_secondsis deprecated and replaced with
AlwaysCheckAllPredicatesis deprecated in the scheduler policy API.
kube-scheduler, profiling is enabled by default. To disable profiling, specify the
--include-uninitializedparameter is removed.
k8s.io/client-gono longer use
http://localhost:8080as the default apiserver address.
kubectl runsupports Pod creation and no longer supports using the deprecated generator to create other types of resources.
kubectl rolling-updatecommand is removed and replaced with the
–dry-runsupports three parameter values:
–dry-run=serversupports the following commands:
rollout undo, and
kubectl alpha debugcommand is added, which can be used for debugging and troubleshooting on ephemeral containers in Pods (the
EphemeralContainersfeature introduced on version 1.16 needs to be enabled).
The implementation of hyperkube is changed from Go code to a bash script.
Production-ready features like bare metal cluster tool and high availability (HA) are improved and enhanced.
kubeadm support for HA capability moves to beta, allowing you to use the
kubeadm init and
kubeadm join commands to configure and deploy an HA control plane. Certificate management has become more robust, with kubeadm now seamlessly rotating all your certificates (on upgrades) before they expire. For more information, see Ability to create dynamic HA clusters with kubeadm and kubeadm: graduate the kubeadm configuration.
SIG Storage continues work to enable migration of in-tree volume plugins to Container Storage Interface (CSI). It works on bringing CSI to feature parity with in-tree functionality, including resizing and inline volumes. It introduces new alpha functionality in CSI that doesn't exist in the Kubernetes Storage subsystem yet, like volume cloning.
Volume cloning enables you to specify another PVC as a
DataSource when configuring a new volume. If the underlying storage system supports this functionality and implements the
CLONE_VOLUME capability in its CSI driver, then the new volume becomes a clone of the source volume. For more information, see In-tree storage plugin to CSI Driver Migration.
kubernetes/legacy-cloud-providersfor easier removal later and external usage.
apps/v1beta2APIs continue to be depreciated. These extensions will be retired on Kubernetes 1.16.
apps/v1beta2APIs are deprecated.
--log-file parameter is known to be problematic on Kubernetes 1.15. This presents as things being logged multiple times to the same file. For more information, see [Failing Test] timeouts in ci-kubernetes-e2e-gce-scale-performance.
node.kubernetes.io/masq-agent-ds-readyas the node selector and no longer uses
node.kubernetes.io/kube-proxy-ds-readyas the node selector and no longer uses
cloud.google.com/metadata-proxy-readyas the node selector and no longer uses
k8s.io/kubernetesand other published components (such as
k8s.io/api) now contain Go module files, including version information of the dependent library. For more information on consuming
k8s.io/client-gousing Go modules, see Installing client-go and add go module support, manage vendor directory using go mod vendor.
Hyperkube short aliases have been removed from source code, because hyperkube docker image currently creates these aliases. For more information, see fix Remove hyperkube short aliases.
v1alpha3configuration is totally removed.
kube-up.shno longer supports
Node.Status.Volumes.Attached.DevicePathfield is no longer set for CSI volumes. You must update any external controllers that depend on this field.
StorageObjectInUseProtectionadmission plugin is enabled by default. If you previously had not enabled it, your cluster behavior may change.
PodInfoOnMountis enabled for a CSI driver, the new
csi.storage.k8s.io/ephemeralparameter in the volume context allows a driver's
NodePublishVolumeimplementation to determine on a case-by-case basis whether the volume is ephemeral or a normal persistent volume. For more information, see persistent and ephemeral csi volumes.
VolumePVCDataSource(storage volume cloning feature) is promoted to beta. For more information, see Promote VolumePVCDataSource to beta for 1.16.
--enable-logs-handlerparameter is deprecated and will be removed on Kubernetes 1.19.
--basic-auth-fileflag and authentication mode are deprecated and will be removed from a future release.
10.0.0.0/24) is deprecated and will be removed in six months/two releases. The
--service-cluster-ip-rangeparameter is required to configure the service IP range.
v1beta1Event API is used. Any tool targeting scheduler events needs to use it.
--conntrack-maxparameter is removed and replaced with
--cleanup-iptablesparameter is removed.
--host-network-sourcesparameters are removed and replaced with the admission controller of
--node-labelsparameter can no longer be used to configure forbidden labels prefixed with
kubectl scale jobis removed.
--pod/-pparameter of the
kubectl execcommand is removed.
kubectl convertcommand is removed.
kubectl cpno longer supports copying symbolic links from containers. You can use the following commands instead:
local to pod:
tar cf - /tmp/foo | kubectl exec -i -n <some-namespace> <some-pod> -- tar xf - -C /tmp/bar
pod to local:
kubectl exec -n <some-namespace> <some-pod> -- tar cf - /tmp/foo | tar xf - -C /tmp/bar
kubeadm upgrade node configand
kubeadm upgrade node experimental-control-planecommands are deprecated and replaced with
kubeadm upgrade node.
--experimental-control-planeparameter is deprecated and replaced with
--experimental-upload-certsparameter is deprecated and replaced with
kubeadm config uploadcommand is deprecated and replaced with
kubeadm init phase upload-confi.
proxyplugin is deprecated and replaced with the
resyncperiodoption is removed from the
upstreamoption is deprecated. If it is specified, it will be ignored.
dry-rungraduates to beta (
dry-runenables you to simulate real API requests without actually changing the cluster status).
kubectl diffgraduates to beta.
CSIPersistentVolumegraduates to GA.
TaintBasedEvictiongraduates to beta.
RuntimeClassgraduates to beta.
runAsGroupgraduates to beta.
kubectl apply server-sidegraduates to alpha, allowing you to perform apply operations on the server side.
resolv.confcan be configured in Pods.
etcd2is no longer supported.
--storage-backend=etcd3is used by default.
--etcd-quorum-readparameter is deprecated.
--storage-versionsparameter is deprecated.
--repair-malformed-updatesparameter is deprecated.
--insecure-experimental-approve-all-kubelet-csrs-for-groupparameter is deprecated.
--google-json-keyparameter is deprecated.
--experimental-fail-swap-onparameter is deprecated.
componentconfig/v1alpha1is no longer supported.
run-containercommand is no longer supported.
node.alpha.kubernetes.io/unreachableare no longer supported and are replaced with
CustomResourcesare now beta and enabled by default. With this, updates to the
/statussubresource will disallow updates to all fields other than
.metadataas before). Also,
descriptioncan be used at the root of the CRD OpenAPI validation schema when the
/statussubresource is enabled. In addition, you can now create multiple versions of CustomResourceDefinitions, but without any kind of automatic conversion, and CustomResourceDefinitions now allow specification of additional columns for
kubectl getoutput via the
dry runfeature is supported. It allows you to view the execution results of some commands without having to submit relevant modifications.
client-go credentialsplugin graduates to beta, allowing you to get TLS authentication information from external plugins.
forbidauthorization decision) and
authorization.k8s.io/reason(the reason for this decision).
podsecuritypolicy.admission.k8s.io/validate-policyannotations containing the name of the policy that allows a Pod to be admitted. (
PodSecurityPolicyalso gains the ability to limit
hostPathvolume mounts to be read-only.)
CLI implements a new plugin mechanism, providing a library with common CLI tooling for plugin authors and further refactorings of the code.
DynamicKubeletConfiggraduates to beta.
cri-toolsgraduates to GA.
PodShareProcessNamespacegraduates to beta.
TaintNodeByConditiongraduates to beta.
--storage-versionparameter is removed and replaced with
--storage-versionsparameter is also deprecated.
--endpoint-reconciler-typeis changed to
--enable-admission-pluginsis used, it is contained by default. When the
--admission-controlparameter is used, it must be explicitly specified.
--rotate-certificatesparameter is deprecated and replaced with the
.RotateCertificatesfield in the configuration file.
kubectl rungenerators except
--interactiveparameter is removed from
--use-openapi-print-columnsis deprecated and replaced with