Release notes and version skew

• 1: v1.20 Release Notes
• 2: Kubernetes version and version skew support policy

1 - v1.20 Release Notes

v1.20.0

Documentation

Downloads for v1.20.0

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.19.0

What's New (Major Themes)

Dockershim deprecation

Docker as an underlying runtime is being deprecated. Docker-produced images will continue to work in your cluster with all runtimes, as they always have. The Kubernetes community has written a blog post about this in detail with a dedicated FAQ page for it.

External credential provider for client-go

The client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. Learn more about this on client-go credential plugins documentation.

CronJob controller v2 is available through feature gate

An alternative implementation of CronJob controller is now available as an alpha feature in this release, which has experimental performance improvement by using informers instead of polling. While this will be the default behavior in the future, you can try them in this release through a feature gate.

PID Limits graduates to General Availability

PID Limits features are now generally available on both SupportNodePidsLimit (node-to-pod PID isolation) and SupportPodPidsLimit (ability to limit PIDs per pod), after being enabled-by-default in beta stage for a year.

API Priority and Fairness graduates to Beta

Initially introduced in 1.18, Kubernetes 1.20 now enables API Priority and Fairness (APF) by default. This allows kube-apiserver to categorize incoming requests by priority levels.

IPv4/IPv6 run

IPv4/IPv6 dual-stack has been reimplemented for 1.20 to support dual-stack Services, based on user and community feedback. If your cluster has dual-stack enabled, you can create Services which can use IPv4, IPv6, or both, and you can change this setting for existing Services. Details are available in updated IPv4/IPv6 dual-stack docs, which cover the nuanced array of options.

We expect this implementation to progress from alpha to beta and GA in coming releases, so we’re eager to have you comment about your dual-stack experiences in #k8s-dual-stack or in enhancements #563.

go1.15.5

go1.15.5 has been integrated to Kubernetes project as of this release, including other infrastructure related updates on this effort.

CSI Volume Snapshot graduates to General Availability

CSI Volume Snapshot moves to GA in the 1.20 release. This feature provides a standard way to trigger volume snapshot operations in Kubernetes and allows Kubernetes users to incorporate snapshot operations in a portable manner on any Kubernetes environment regardless of supporting underlying storage providers. Additionally, these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced, enterprise grade, storage administration features for Kubernetes: including application or cluster level backup solutions. Note that snapshot support will require Kubernetes distributors to bundle the Snapshot controller, Snapshot CRDs, and validation webhook. In addition, a CSI driver supporting the snapshot functionality must also be deployed on the cluster.

Non-recursive Volume Ownership (FSGroup) graduates to Beta

By default, the fsgroup setting, if specified, recursively updates permissions for every file in a volume on every mount. This can make mount, and pod startup, very slow if the volume has many files. This setting enables a pod to specify a PodFSGroupChangePolicy that indicates that volume ownership and permissions will be changed only when permission and ownership of the root directory does not match with expected permissions on the volume.

CSIDriver policy for FSGroup graduates to Beta

The FSGroup's CSIDriver Policy is now beta in 1.20. This allows CSIDrivers to explicitly indicate if they want Kubernetes to manage permissions and ownership for their volumes via fsgroup.

Security Improvements for CSI Drivers (Alpha)

In 1.20, we introduce a new alpha feature CSIServiceAccountToken. This feature allows CSI drivers to impersonate the pods that they mount the volumes for. This improves the security posture in the mounting process where the volumes are ACL’ed on the pods’ service account without handing out unnecessary permissions to the CSI drivers’ service account. This feature is especially important for secret-handling CSI drivers, such as the secrets-store-csi-driver. Since these tokens can be rotated and short-lived, this feature also provides a knob for CSI drivers to receive NodePublishVolume RPC calls periodically with the new token. This knob is also useful when volumes are short-lived, e.g. certificates.

Introducing Graceful Node Shutdown (Alpha)

The GracefulNodeShutdown feature is now in Alpha. This allows kubelet to be aware of node system shutdowns, enabling graceful termination of pods during a system shutdown. This feature can be enabled through feature gate.

Runtime log sanitation

Logs can now be configured to use runtime protection from leaking sensitive data. Details for this experimental feature is available in documentation.

Pod resource metrics

On-demand metrics calculation is now available through /metrics/resources. When enabled, the endpoint will report the requested resources and the desired limits of all running pods.

Introducing RootCAConfigMap

RootCAConfigMap graduates to Beta, seperating from BoundServiceAccountTokenVolume. The kube-root-ca.crt ConfigMap is now available to every namespace, by default. It contains the Certificate Authority bundle for verify kube-apiserver connections.

kubectl debug graduates to Beta

kubectl alpha debug graduates from alpha to beta in 1.20, becoming kubectl debug. kubectl debug provides support for common debugging workflows directly from kubectl. Troubleshooting scenarios supported in this release of kubectl include: Troubleshoot workloads that crash on startup by creating a copy of the pod that uses a different container image or command. Troubleshoot distroless containers by adding a new container with debugging tools, either in a new copy of the pod or using an ephemeral container. (Ephemeral containers are an alpha feature that are not enabled by default.) Troubleshoot on a node by creating a container running in the host namespaces and with access to the host’s filesystem. Note that as a new builtin command, kubectl debug takes priority over any kubectl plugin named “debug”. You will need to rename the affected plugin. Invocations using kubectl alpha debug are now deprecated and will be removed in a subsequent release. Update your scripts to use kubectl debug instead of kubectl alpha debug! For more information about kubectl debug, see Debugging Running Pods on the Kubernetes website, kubectl help debug, or reach out to SIG CLI by visiting #sig-cli or commenting on enhancement #1441.

Removing deprecated flags in kubeadm

kubeadm applies a number of deprecations and removals of deprecated features in this release. More details are available in the Urgent Upgrade Notes and Kind / Deprecation sections.

Pod Hostname as FQDN graduates to Beta

Previously introduced in 1.19 behind a feature gate, SetHostnameAsFQDN is now enabled by default. More details on this behavior is available in documentation for DNS for Services and Pods

TokenRequest / TokenRequestProjection graduates to General Availability

Service account tokens bound to pod is now a stable feature. The feature gates will be removed in 1.21 release. For more information, refer to notes below on the changelogs.

RuntimeClass feature graduates to General Availability.

The node.k8s.io API groups are promoted from v1beta1 to v1. v1beta1 is now deprecated and will be removed in a future release, please start using v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]

Cloud Controller Manager now exclusively shipped by Cloud Provider

Kubernetes will no longer ship an instance of the Cloud Controller Manager binary. Each Cloud Provider is expected to ship their own instance of this binary. Details for a Cloud Provider to create an instance of such a binary can be found under here. Anyone with questions on building a Cloud Controller Manager should reach out to SIG Cloud Provider. Questions about the Cloud Controller Manager on a Managed Kubernetes solution should go to the relevant Cloud Provider. Questions about the Cloud Controller Manager on a non managed solution can be brought up with SIG Cloud Provider.

Known Issues

Summary API in kubelet doesn't have accelerator metrics

Currently, cadvisor_stats_provider provides AcceleratorStats but cri_stats_provider does not. As a result, when using cri_stats_provider, kubelet's Summary API does not have accelerator metrics. There is an open work in progress to fix this.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• A bug was fixed in kubelet where exec probe timeouts were not respected. This may result in unexpected behavior since the default timeout (if not specified) is 1s which may be too small for some exec probes. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts. See configure probe section of the documentation for more details.

  • This change in behavior may be unexpected for some clusters and can be disabled by turning off the ExecProbeTimeout feature gate. This gate will be locked and removed in future releases so that exec probe timeouts are always respected. (#94115, @andrewsykim) [SIG Node and Testing]

• RuntimeClass feature graduates to General Availability. Promote node.k8s.io API groups from v1beta1 to v1. v1beta1 is now deprecated and will be removed in a future release, please start using v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]

• API priority and fairness graduated to beta. 1.19 servers with APF turned on should not be run in a multi-server cluster with 1.20+ servers. (#96527, @adtac) [SIG API Machinery and Testing]

• For CSI drivers, kubelet no longer creates the target_path for NodePublishVolume in accordance with the CSI spec. Kubelet also no longer checks if staging and target paths are mounts or corrupted. CSI drivers need to be idempotent and do any necessary mount verification. (#88759, @andyzhangx) [SIG Storage]

• Kubeadm: http://git.k8s.io/enhancements/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md (#95382, @neolit123) [SIG Cluster Lifecycle]

  • The label applied to control-plane nodes "node-role.kubernetes.io/master" is now deprecated and will be removed in a future release after a GA deprecation period.
  • Introduce a new label "node-role.kubernetes.io/control-plane" that will be applied in parallel to "node-role.kubernetes.io/master" until the removal of the "node-role.kubernetes.io/master" label.
  • Make "kubeadm upgrade apply" add the "node-role.kubernetes.io/control-plane" label on existing nodes that only have the "node-role.kubernetes.io/master" label during upgrade.
  • Please adapt your tooling built on top of kubeadm to use the "node-role.kubernetes.io/control-plane" label.
  • The taint applied to control-plane nodes "node-role.kubernetes.io/master:NoSchedule" is now deprecated and will be removed in a future release after a GA deprecation period.
  • Apply toleration for a new, future taint "node-role.kubernetes.io/control-plane:NoSchedule" to the kubeadm CoreDNS / kube-dns managed manifests. Note that this taint is not yet applied to kubeadm control-plane nodes.
  • Please adapt your workloads to tolerate the same future taint preemptively.

• Kubeadm: improve the validation of serviceSubnet and podSubnet. ServiceSubnet has to be limited in size, due to implementation details, and the mask can not allocate more than 20 bits. PodSubnet validates against the corresponding cluster "--node-cidr-mask-size" of the kube-controller-manager, it fail if the values are not compatible. kubeadm no longer sets the node-mask automatically on IPv6 deployments, you must check that your IPv6 service subnet mask is compatible with the default node mask /64 or set it accordenly. Previously, for IPv6, if the podSubnet had a mask lower than /112, kubeadm calculated a node-mask to be multiple of eight and splitting the available bits to maximise the number used for nodes. (#95723, @aojea) [SIG Cluster Lifecycle]

• The deprecated flag --experimental-kustomize is now removed from kubeadm commands. Use --experimental-patches instead, which was introduced in 1.19. Migration information available in --help description for --experimental-patches. (#94871, @neolit123)

• Windows hyper-v container featuregate is deprecated in 1.20 and will be removed in 1.21 (#95505, @wawa0210) [SIG Node and Windows]

• The kube-apiserver ability to serve on an insecure port, deprecated since v1.10, has been removed. The insecure address flags --address and --insecure-bind-address have no effect in kube-apiserver and will be removed in v1.24. The insecure port flags --port and --insecure-port may only be set to 0 and will be removed in v1.24. (#95856, @knight42, [SIG API Machinery, Node, Testing])

• Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. It changes the dual-stack API wrt Service from a single ipFamily field to 3 fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), ipFamilies (a list of families assigned), and clusterIPs (inclusive of clusterIP). Most users do not need to set anything at all, defaulting will handle it for them. Services are single-stack unless the user asks for dual-stack. This is all gated by the "IPv6DualStack" feature gate. (#91824, @khenidak) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]

• TokenRequest and TokenRequestProjection are now GA features. The following flags are required by the API server:

  • --service-account-issuer, should be set to a URL identifying the API server that will be stable over the cluster lifetime.
  • --service-account-key-file, set to one or more files containing one or more public keys used to verify tokens.
  • --service-account-signing-key-file, set to a file containing a private key to use to sign service account tokens. Can be the same file given to kube-controller-manager with --service-account-private-key-file. (#95896, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle]

• kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags:

  • apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration.
  • cluster-name: use clusterName from ClusterConfiguration
  • cert-dir: use certificatesDir from ClusterConfiguration (#94879, @knight42) [SIG Cluster Lifecycle]

• Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected. The kubectl-check-ownerreferences tool can be run prior to upgrading to locate existing objects with invalid ownerReferences.

  • A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted.
  • A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. (#92743, @liggitt) [SIG API Machinery, Apps and Testing]

Changes by Kind

Deprecation

• Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. (#94624, @dims) [SIG Node]
• Kubeadm: deprecate self-hosting support. The experimental command "kubeadm alpha self-hosting" is now deprecated and will be removed in a future release. (#95125, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: graduate the "kubeadm alpha certs" command to a parent command "kubeadm certs". The command "kubeadm alpha certs" is deprecated and will be removed in a future release. Please migrate. (#94938, @yagonobre) [SIG Cluster Lifecycle]
• Kubeadm: remove the deprecated "kubeadm alpha kubelet config enable-dynamic" command. To continue using the feature please defer to the guide for "Dynamic Kubelet Configuration" at k8s.io. This change also removes the parent command "kubeadm alpha kubelet" as there are no more sub-commands under it for the time being. (#94668, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: remove the deprecated --kubelet-config flag for the command "kubeadm upgrade node" (#94869, @neolit123) [SIG Cluster Lifecycle]
• Kubectl: deprecate --delete-local-data (#95076, @dougsland) [SIG CLI, Cloud Provider and Scalability]
• Kubelet's deprecated endpoint metrics/resource/v1alpha1 has been removed, please adopt metrics/resource. (#94272, @RainbowMango) [SIG Instrumentation and Node]
• Removes deprecated scheduler metrics DeprecatedSchedulingDuration, DeprecatedSchedulingAlgorithmPredicateEvaluationSecondsDuration, DeprecatedSchedulingAlgorithmPriorityEvaluationSecondsDuration (#94884, @arghya88) [SIG Instrumentation and Scheduling]
• Scheduler alpha metrics binding_duration_seconds and scheduling_algorithm_preemption_evaluation_seconds are deprecated, Both of those metrics are now covered as part of framework_extension_point_duration_seconds, the former as a PostFilter the latter and a Bind plugin. The plan is to remove both in 1.21 (#95001, @arghya88) [SIG Instrumentation and Scheduling]
• Support 'controlplane' as a valid EgressSelection type in the EgressSelectorConfiguration API. 'Master' is deprecated and will be removed in v1.22. (#95235, @andrewsykim) [SIG API Machinery]
• The v1alpha1 PodPreset API and admission plugin has been removed with no built-in replacement. Admission webhooks can be used to modify pods on creation. (#94090, @deads2k) [SIG API Machinery, Apps, CLI, Cloud Provider, Scalability and Testing]

API Change

• TokenRequest and TokenRequestProjection features have been promoted to GA. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection for details on configuring and using this feature. The TokenRequest and TokenRequestProjection feature gates will be removed in v1.21.
  • kubeadm's kube-apiserver Pod manifest now includes the following flags by default "--service-account-key-file", "--service-account-signing-key-file", "--service-account-issuer". (#93258, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle, Storage and Testing]
• A new nofuzz go build tag now disables gofuzz support. Release binaries enable this. (#92491, @BenTheElder) [SIG API Machinery]
• Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest (#95741, @katiewasnothere) [SIG Node]
• Add a serving and terminating condition to the EndpointSlice API. serving tracks the readiness of endpoints regardless of their terminating state. This is distinct from ready since ready is only true when pods are not terminating. terminating is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. (#92968, @andrewsykim) [SIG Apps and Network]
• Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. It changes the dual-stack API wrt Service from a single ipFamily field to 3 fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), ipFamilies (a list of families assigned), and clusterIPs (inclusive of clusterIP). Most users do not need to set anything at all, defaulting will handle it for them. Services are single-stack unless the user asks for dual-stack. This is all gated by the "IPv6DualStack" feature gate. (#91824, @khenidak) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
• Add support for hugepages to downward API (#86102, @derekwaynecarr) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
• Adds kubelet alpha feature, GracefulNodeShutdown which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. (#96129, @bobbypage) [SIG Node]
• AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. (#96327, @robscott) [SIG Apps and Network]
• Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. (#92744, @uablrek) [SIG Apps and Network]
• Certain fields on Service objects will be automatically cleared when changing the service's type to a mode that does not need those fields. For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. (#95196, @thockin) [SIG API Machinery, Apps, Network and Testing]
• Document that ServiceTopology feature is required to use service.spec.topologyKeys. (#96528, @andrewsykim) [SIG Apps]
• EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate.
  • EndpointSlice topology field will be deprecated in an upcoming release.
  • EndpointSlice "IP" address type is formally removed after being deprecated in Kubernetes 1.17.
  • The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. (#96440, @robscott) [SIG API Machinery, Apps and Network]
• External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ (#92632, @RenaudWasTaken) [SIG Node and Testing]
• Fewer candidates are enumerated for preemption to improve performance in large clusters. (#94814, @adtac)
• Fix conversions for custom metrics. (#94481, @wojtek-t) [SIG API Machinery and Instrumentation]
• GPU metrics provided by kubelet are now disabled by default. (#95184, @RenaudWasTaken)
• If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric serviceaccount_stale_tokens_total to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting kube-apiserver with flag --service-account-extend-token-expiration=false (#96273, @zshihang) [SIG API Machinery and Auth]
• Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. (#94196, @andrewsykim) [SIG Node and Release]
• Introduces a metric source for HPAs which allows scaling based on container resource usage. (#90691, @arjunrn) [SIG API Machinery, Apps, Autoscaling and CLI]
• Kube-apiserver now deletes expired kube-apiserver Lease objects:
  • The feature is under feature gate APIServerIdentity.
  • A flag is added to kube-apiserver: identity-lease-garbage-collection-check-period-seconds (#95895, @roycaihw) [SIG API Machinery, Apps, Auth and Testing]
• Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting --volume-host-allow-local-loopback=false, or from contacting specific CIDR ranges by setting --volume-host-cidr-denylist (for example, --volume-host-cidr-denylist=127.0.0.1/28,feed::/16) (#91785, @mattcary) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
• Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock (#94603, @wojtek-t) [SIG API Machinery, Apps, Cloud Provider and Scheduling]
• Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly (#94182, @mattfenwick) [SIG API Machinery, Apps, Auth, Network and Node]
• Move configurable fsgroup change policy for pods to beta (#96376, @gnufied) [SIG Apps and Storage]
• New flag is introduced, i.e. --topology-manager-scope=container|pod. The default value is the "container" scope. (#92967, @cezaryzukowski) [SIG Instrumentation, Node and Testing]
• New parameter defaultingType for PodTopologySpread plugin allows to use k8s defined or user provided default constraints (#95048, @alculquicondor) [SIG Scheduling]
• NodeAffinity plugin can be configured with AddedAffinity. (#96202, @alculquicondor) [SIG Node, Scheduling and Testing]
• Promote RuntimeClass feature to GA. Promote node.k8s.io API groups from v1beta1 to v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]
• Reminder: The labels "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" are deprecated in favor of "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" respectively. All users of the "failure-domain.beta..." labels should switch to the "topology..." equivalents. (#96033, @thockin) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing]
• Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. (#93901, @jpbetz) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
• Services will now have a clusterIPs field to go with clusterIP. clusterIPs[0] is a synonym for clusterIP and will be syncronized on create and update operations. (#95894, @thockin) [SIG Network]
• The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. (#91921, @mtaufen) [SIG Auth]
• The status of v1beta1 CRDs without "preserveUnknownFields:false" now shows a violation, "spec.preserveUnknownFields: Invalid value: true: must be false". (#93078, @vareti)
• The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBService is enabled. The feature gate is disabled by default. The user has to enable it for the API Server. (#94028, @janosi) [SIG API Machinery and Apps]
• This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in CSIDriverSpec. (#93130, @zshihang) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
• Users can try the cronjob controller v2 using the feature gate. This will be the default controller in future releases. (#93370, @alaypatel07) [SIG API Machinery, Apps, Auth and Testing]
• VolumeSnapshotDataSource moves to GA in 1.20 release (#95282, @xing-yang) [SIG Apps]
• WinOverlay feature graduated to beta (#94807, @ksubrmnn) [SIG Windows]

Feature

• Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

• A new metric apiserver_request_filter_duration_seconds has been introduced that measures request filter latency in seconds. (#95207, @tkashem) [SIG API Machinery and Instrumentation]

• A new set of alpha metrics are reported by the Kubernetes scheduler under the /metrics/resources endpoint that allow administrators to easily see the resource consumption (requests and limits for all resources on the pods) and compare it to actual pod usage or node capacity. (#94866, @smarterclayton) [SIG API Machinery, Instrumentation, Node and Scheduling]

• Add --experimental-logging-sanitization flag enabling runtime protection from leaking sensitive data in logs (#96370, @serathius) [SIG API Machinery, Cluster Lifecycle and Instrumentation]

• Add a StorageVersionAPI feature gate that makes API server update storageversions before serving certain write requests. This feature allows the storage migrator to manage storage migration for built-in resources. Enabling internal.apiserver.k8s.io/v1alpha1 API and APIServerIdentity feature gate are required to use this feature. (#93873, @roycaihw) [SIG API Machinery, Auth and Testing]

• Add a metric for time taken to perform recursive permission change (#95866, @JornShen) [SIG Instrumentation and Storage]

• Add a new vSphere metric: cloudprovider_vsphere_vcenter_versions. It's content show vCenter hostnames with the associated server version. (#94526, @Danil-Grigorev) [SIG Cloud Provider and Instrumentation]

• Add a new flag to set priority for the kubelet on Windows nodes so that workloads cannot overwhelm the node there by disrupting kubelet process. (#96051, @ravisantoshgudimetla) [SIG Node and Windows]

• Add feature to size memory backed volumes (#94444, @derekwaynecarr) [SIG Storage and Testing]

• Add foreground cascading deletion to kubectl with the new kubectl delete foreground|background|orphan option. (#93384, @zhouya0)

• Add metrics for azure service operations (route and loadbalancer). (#94124, @nilo19) [SIG Cloud Provider and Instrumentation]

• Add network rule support in Azure account creation. (#94239, @andyzhangx)

• Add node_authorizer_actions_duration_seconds metric that can be used to estimate load to node authorizer. (#92466, @mborsz) [SIG API Machinery, Auth and Instrumentation]

• Add pod_ based CPU and memory metrics to Kubelet's /metrics/resource endpoint (#95839, @egernst) [SIG Instrumentation, Node and Testing]

• Added get-users and delete-user to the kubectl config subcommand (#89840, @eddiezane) [SIG CLI]

• Added counter metric "apiserver_request_self" to count API server self-requests with labels for verb, resource, and subresource. (#94288, @LogicalShark) [SIG API Machinery, Auth, Instrumentation and Scheduling]

• Added new k8s.io/component-helpers repository providing shared helper code for (core) components. (#92507, @ingvagabund) [SIG Apps, Node, Release and Scheduling]

• Adds create ingress command to kubectl (#78153, @amimof) [SIG CLI and Network]

• Adds a headless service on node-local-cache addon. (#88412, @stafot) [SIG Cloud Provider and Network]

• Allow cross compilation of kubernetes on different platforms. (#94403, @bnrjee) [SIG Release]

• Azure: Support multiple services sharing one IP address (#94991, @nilo19) [SIG Cloud Provider]

• CRDs: For structural schemas, non-nullable null map fields will now be dropped and defaulted if a default is available. null items in list will continue being preserved, and fail validation if not nullable. (#95423, @apelisse) [SIG API Machinery]

• Changed: default "Accept: /" header added to HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (https://github.com/kubernetes/website/pull/24756) (#95641, @fonsecas72) [SIG Network and Node]

• Client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. (#95489, @ankeesler) [SIG API Machinery and Auth]

• Command to start network proxy changes from 'KUBE_ENABLE_EGRESS_VIA_KONNECTIVITY_SERVICE ./cluster/kube-up.sh' to 'KUBE_ENABLE_KONNECTIVITY_SERVICE=true ./hack/kube-up.sh' (#92669, @Jefftree) [SIG Cloud Provider]

• Configure AWS LoadBalancer health check protocol via service annotations. (#94546, @kishorj)

• DefaultPodTopologySpread graduated to Beta. The feature gate is enabled by default. (#95631, @alculquicondor) [SIG Scheduling and Testing]

• E2e test for PodFsGroupChangePolicy (#96247, @saikat-royc) [SIG Storage and Testing]

• Ephemeral containers now apply the same API defaults as initContainers and containers (#94896, @wawa0210) [SIG Apps and CLI]

• Gradudate the Pod Resources API to G.A Introduces the pod_resources_endpoint_requests_total metric which tracks the total number of requests to the pod resources API (#92165, @RenaudWasTaken) [SIG Instrumentation, Node and Testing]

• In dual-stack bare-metal clusters, you can now pass dual-stack IPs to kubelet --node-ip. eg: kubelet --node-ip 10.1.0.5,fd01::0005. This is not yet supported for non-bare-metal clusters.

  In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods will now get dual-stack PodIPs. (#95239, @danwinship) [SIG Network and Node]

• Introduce api-extensions category which will return: mutating admission configs, validating admission configs, CRDs and APIServices when used in kubectl get, for example. (#95603, @soltysh) [SIG API Machinery]

• Introduces a new GCE specific cluster creation variable KUBE_PROXY_DISABLE. When set to true, this will skip over the creation of kube-proxy (whether the daemonset or static pod). This can be used to control the lifecycle of kube-proxy separately from the lifecycle of the nodes. (#91977, @varunmar) [SIG Cloud Provider]

• Kube-apiserver now maintains a Lease object to identify itself:

  • The feature is under feature gate APIServerIdentity.
  • Two flags are added to kube-apiserver: identity-lease-duration-seconds, identity-lease-renew-interval-seconds (#95533, @roycaihw) [SIG API Machinery]

• Kube-apiserver: The timeout used when making health check calls to etcd can now be configured with --etcd-healthcheck-timeout. The default timeout is 2 seconds, matching the previous behavior. (#93244, @Sh4d1) [SIG API Machinery]

• Kube-apiserver: added support for compressing rotated audit log files with --audit-log-compress (#94066, @lojies) [SIG API Machinery and Auth]

• Kubeadm now prints warnings instead of throwing errors if the current system time is outside of the NotBefore and NotAfter bounds of a loaded certificate. (#94504, @neolit123)

• Kubeadm: Add a preflight check that the control-plane node has at least 1700MB of RAM (#93275, @xlgao-zju) [SIG Cluster Lifecycle]

• Kubeadm: add the "--cluster-name" flag to the "kubeadm alpha kubeconfig user" to allow configuring the cluster name in the generated kubeconfig file (#93992, @prabhu43) [SIG Cluster Lifecycle]

• Kubeadm: add the "--kubeconfig" flag to the "kubeadm init phase upload-certs" command to allow users to pass a custom location for a kubeconfig file. (#94765, @zhanw15) [SIG Cluster Lifecycle]

• Kubeadm: make etcd pod request 100m CPU, 100Mi memory and 100Mi ephemeral_storage by default (#94479, @knight42) [SIG Cluster Lifecycle]

• Kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags:

  • apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration.
  • cluster-name: use clusterName from ClusterConfiguration
  • cert-dir: use certificatesDir from ClusterConfiguration (#94879, @knight42) [SIG Cluster Lifecycle]

• Kubectl create now supports creating ingress objects. (#94327, @rikatz) [SIG CLI and Network]

• Kubectl rollout history sts/sts-name --revision=some-revision will start showing the detailed view of the sts on that specified revision (#86506, @dineshba) [SIG CLI]

• Kubectl: Previously users cannot provide arguments to a external diff tool via KUBECTL_EXTERNAL_DIFF env. This release now allow users to specify args to KUBECTL_EXTERNAL_DIFF env. (#95292, @dougsland) [SIG CLI]

• Kubemark now supports both real and hollow nodes in a single cluster. (#93201, @ellistarn) [SIG Scalability]

• Kubernetes E2E test image manifest lists now contain Windows images. (#77398, @claudiubelu) [SIG Testing and Windows]

• Kubernetes is now built using go1.15.2

  • build: Update to k/repo-infra@v0.1.1 (supports go1.15.2)

  • build: Use go-runner:buster-v2.0.1 (built using go1.15.1)

  • bazel: Replace --features with Starlark build settings flag

  • hack/lib/util.sh: some bash cleanups

    • switched one spot to use kube::logging
    • make kube::util::find-binary return an error when it doesn't find anything so that hack scripts fail fast instead of with '' binary not found errors.
    • this required deleting some genfeddoc stuff. the binary no longer exists in k/k repo since we removed federation/, and I don't see it in https://github.com/kubernetes-sigs/kubefed/ either. I'm assuming that it's gone for good now.

  • bazel: output go_binary rule directly from go_binary_conditional_pure

    From: @mikedanese: Instead of aliasing. Aliases are annoying in a number of ways. This is specifically bugging me now because they make the action graph harder to analyze programmatically. By using aliases here, we would need to handle potentially aliased go_binary targets and dereference to the effective target.

    The comment references an issue with pure = select(...) which appears to be resolved considering this now builds.

  • make kube::util::find-binary not dependent on bazel-out/ structure

    Implement an aspect that outputs go_build_mode metadata for go binaries, and use that during binary selection. (#94449, @justaugustus) [SIG Architecture, CLI, Cluster Lifecycle, Node, Release and Testing]

• Kubernetes is now built using go1.15.5

  • build: Update to k/repo-infra@v0.1.2 (supports go1.15.5) (#95776, @justaugustus) [SIG Cloud Provider, Instrumentation, Release and Testing]

• New default scheduling plugins order reduces scheduling and preemption latency when taints and node affinity are used (#95539, @soulxu) [SIG Scheduling]

• Only update Azure data disks when attach/detach (#94265, @andyzhangx) [SIG Cloud Provider]

• Promote SupportNodePidsLimit to GA to provide node-to-pod PID isolation. Promote SupportPodPidsLimit to GA to provide ability to limit PIDs per pod. (#94140, @derekwaynecarr)

• SCTP support in API objects (Pod, Service, NetworkPolicy) is now GA. Note that this has no effect on whether SCTP is enabled on nodes at the kernel level, and note that some cloud platforms and network plugins do not support SCTP traffic. (#95566, @danwinship) [SIG Apps and Network]

• Scheduler now ignores Pod update events if the resourceVersion of old and new Pods are identical. (#96071, @Huang-Wei) [SIG Scheduling]

• Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extention point. (#93534, @everpeace) [SIG Scheduling and Testing]

• SelectorSpreadPriority maps to PodTopologySpread plugin when DefaultPodTopologySpread feature is enabled (#95448, @alculquicondor) [SIG Scheduling]

• Send GCE node startup scripts logs to console and journal. (#95311, @karan)

• SetHostnameAsFQDN has been graduated to Beta and therefore it is enabled by default. (#95267, @javidiaz) [SIG Node]

• Support [service.beta.kubernetes.io/azure-pip-ip-tags] annotations to allow customers to specify ip-tags to influence public-ip creation in Azure [Tag1=Value1, Tag2=Value2, etc.] (#94114, @MarcPow) [SIG Cloud Provider]

• Support custom tags for cloud provider managed resources (#96450, @nilo19) [SIG Cloud Provider]

• Support customize load balancer health probe protocol and request path (#96338, @nilo19) [SIG Cloud Provider]

• Support for Windows container images (OS Versions: 1809, 1903, 1909, 2004) was added the pause:3.4 image. (#91452, @claudiubelu) [SIG Node, Release and Windows]

• Support multiple standard load balancers in one cluster (#96111, @nilo19) [SIG Cloud Provider]

• The beta RootCAConfigMap feature gate is enabled by default and causes kube-controller-manager to publish a "kube-root-ca.crt" ConfigMap to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. (#96197, @zshihang) [SIG API Machinery, Apps, Auth and Testing]

• The kubelet_runtime_operations_duration_seconds metric buckets were set to 0.005 0.0125 0.03125 0.078125 0.1953125 0.48828125 1.220703125 3.0517578125 7.62939453125 19.073486328125 47.6837158203125 119.20928955078125 298.0232238769531 and 745.0580596923828 seconds (#96054, @alvaroaleman) [SIG Instrumentation and Node]

• There is a new pv_collector_total_pv_count metric that counts persistent volumes by the volume plugin name and volume mode. (#95719, @tsmetana) [SIG Apps, Instrumentation, Storage and Testing]

• Volume snapshot e2e test to validate PVC and VolumeSnapshotContent finalizer (#95863, @RaunakShah) [SIG Cloud Provider, Storage and Testing]

• Warns user when executing kubectl apply/diff to resource currently being deleted. (#95544, @SaiHarshaK) [SIG CLI]

• kubectl alpha debug has graduated to beta and is now kubectl debug. (#96138, @verb) [SIG CLI and Testing]

• kubectl debug gains support for changing container images when copying a pod for debugging, similar to how kubectl set image works. See kubectl help debug for more information. (#96058, @verb) [SIG CLI]

Documentation

• Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList (#95117, @andrewsykim) [SIG API Machinery]
• Kubelet: remove alpha warnings for CNI flags. (#94508, @andrewsykim) [SIG Network and Node]
• Updates docs and guidance on cloud provider InstancesV2 and Zones interface for external cloud providers:
  • removes experimental warning for InstancesV2
  • document that implementation of InstancesV2 will disable calls to Zones
  • deprecate Zones in favor of InstancesV2 (#96397, @andrewsykim) [SIG Cloud Provider]

Failing Test

• Resolves an issue running Ingress conformance tests on clusters which use finalizers on Ingress objects to manage releasing load balancer resources (#96742, @spencerhance) [SIG Network and Testing]
• The Conformance test "validates that there is no conflict between pods with same hostPort but different hostIP and protocol" now validates the connectivity to each hostPort, in addition to the functionality. (#96627, @aojea) [SIG Scheduling and Testing]

Bug or Regression

• Add kubectl wait --ignore-not-found flag (#90969, @zhouya0) [SIG CLI]

• Added support to kube-proxy for externalTrafficPolicy=Local setting via Direct Server Return (DSR) load balancers on Windows. (#93166, @elweb9858) [SIG Network]

• Alter wording to describe pods using a pvc (#95635, @RaunakShah) [SIG CLI]

• An issues preventing volume expand controller to annotate the PVC with volume.kubernetes.io/storage-resizer when the PVC StorageClass is already updated to the out-of-tree provisioner is now fixed. (#94489, @ialidzhikov) [SIG API Machinery, Apps and Storage]

• Azure ARM client: don't segfault on empty response and http error (#94078, @bpineau) [SIG Cloud Provider]

• Azure armclient backoff step defaults to 1 (no retry). (#94180, @feiskyer)

• Azure: fix a bug that kube-controller-manager would panic if wrong Azure VMSS name is configured (#94306, @knight42) [SIG Cloud Provider]

• Both apiserver_request_duration_seconds metrics and RequestReceivedTimestamp fields of an audit event now take into account the time a request spends in the apiserver request filters. (#94903, @tkashem)

• Build/lib/release: Explicitly use '--platform' in building server images

  When we switched to go-runner for building the apiserver, controller-manager, and scheduler server components, we no longer reference the individual architectures in the image names, specifically in the 'FROM' directive of the server image Dockerfiles.

  As a result, server images for non-amd64 images copy in the go-runner amd64 binary instead of the go-runner that matches that architecture.

  This commit explicitly sets the '--platform=linux/${arch}' to ensure we're pulling the correct go-runner arch from the manifest list.

  Before: FROM ${base_image}

  After: FROM --platform=linux/${arch} ${base_image} (#94552, @justaugustus) [SIG Release]

• Bump node-problem-detector version to v0.8.5 to fix OOM detection in with Linux kernels 5.1+ (#96716, @tosi3k) [SIG Cloud Provider, Scalability and Testing]

• CSIDriver object can be deployed during volume attachment. (#93710, @Jiawei0227) [SIG Apps, Node, Storage and Testing]

• Ceph RBD volume expansion now works even when ceph.conf was not provided. (#92027, @juliantaylor)

• Change plugin name in fsgroupapplymetrics of csi and flexvolume to distinguish different driver (#95892, @JornShen) [SIG Instrumentation, Storage and Testing]

• Change the calculation of pod UIDs so that static pods get a unique value - will cause all containers to be killed and recreated after in-place upgrade. (#87461, @bboreham) [SIG Node]

• Change the mount way from systemd to normal mount except ceph and glusterfs intree-volume. (#94916, @smileusd) [SIG Apps, Cloud Provider, Network, Node, Storage and Testing]

• Changes to timeout parameter handling in 1.20.0-beta.2 have been reverted to avoid breaking backwards compatibility with existing clients. (#96727, @liggitt) [SIG API Machinery and Testing]

• Clear UDP conntrack entry on endpoint changes when using nodeport (#71573, @JacobTanenbaum) [SIG Network]

• Cloud node controller: handle empty providerID from getProviderID (#95342, @nicolehanjing) [SIG Cloud Provider]

• Disable watchcache for events (#96052, @wojtek-t) [SIG API Machinery]

• Disabled LocalStorageCapacityIsolation feature gate is honored during scheduling. (#96092, @Huang-Wei) [SIG Scheduling]

• Do not fail sorting empty elements. (#94666, @soltysh) [SIG CLI]

• Dual-stack: make nodeipam compatible with existing single-stack clusters when dual-stack feature gate become enabled by default (#90439, @SataQiu) [SIG API Machinery]

• Duplicate owner reference entries in create/update/patch requests now get deduplicated by the API server. The client sending the request now receives a warning header in the API response. Clients should stop sending requests with duplicate owner references. The API server may reject such requests as early as 1.24. (#96185, @roycaihw) [SIG API Machinery and Testing]

• Endpoint slice controller now mirrors parent's service label to its corresponding endpoint slices. (#94443, @aojea)

• Ensure getPrimaryInterfaceID not panic when network interfaces for Azure VMSS are null (#94355, @feiskyer) [SIG Cloud Provider]

• Exposes and sets a default timeout for the SubjectAccessReview client for DelegatingAuthorizationOptions (#95725, @p0lyn0mial) [SIG API Machinery and Cloud Provider]

• Exposes and sets a default timeout for the TokenReview client for DelegatingAuthenticationOptions (#96217, @p0lyn0mial) [SIG API Machinery and Cloud Provider]

• Fix CVE-2020-8555 for Quobyte client connections. (#95206, @misterikkit) [SIG Storage]

• Fix IP fragmentation of UDP and TCP packets not supported issues on LoadBalancer rules (#96464, @nilo19) [SIG Cloud Provider]

• Fix a bug that DefaultPreemption plugin is disabled when using (legacy) scheduler policy. (#96439, @Huang-Wei) [SIG Scheduling and Testing]

• Fix a bug where loadbalancer deletion gets stuck because of missing resource group. (#93962, @phiphi282)

• Fix a concurrent map writes error in kubelet (#93773, @knight42) [SIG Node]

• Fix a panic in kubectl debug when a pod has multiple init or ephemeral containers. (#94580, @kiyoshim55)

• Fix a regression where kubeadm bails out with a fatal error when an optional version command line argument is supplied to the "kubeadm upgrade plan" command (#94421, @rosti) [SIG Cluster Lifecycle]

• Fix azure disk attach failure for disk size bigger than 4TB (#95463, @andyzhangx) [SIG Cloud Provider]

• Fix azure disk data loss issue on Windows when unmount disk (#95456, @andyzhangx) [SIG Cloud Provider and Storage]

• Fix azure file migration panic (#94853, @andyzhangx) [SIG Cloud Provider]

• Fix bug in JSON path parser where an error occurs when a range is empty (#95933, @brianpursley) [SIG API Machinery]

• Fix client-go prometheus metrics to correctly present the API path accessed in some environments. (#74363, @aanm) [SIG API Machinery]

• Fix detach azure disk issue when vm not exist (#95177, @andyzhangx) [SIG Cloud Provider]

• Fix etcd_object_counts metric reported by kube-apiserver (#94773, @tkashem) [SIG API Machinery]

• Fix incorrectly reported verbs for kube-apiserver metrics for CRD objects (#93523, @wojtek-t) [SIG API Machinery and Instrumentation]

• Fix k8s.io/apimachinery/pkg/api/meta.SetStatusCondition to update ObservedGeneration (#95961, @KnicKnic) [SIG API Machinery]

• Fix kubectl SchemaError on CRDs with schema using x-kubernetes-preserve-unknown-fields on array types. (#94888, @sttts) [SIG API Machinery]

• Fix memory leak in kube-apiserver when underlying time goes forth and back. (#96266, @chenyw1990) [SIG API Machinery]

• Fix missing csi annotations on node during parallel csinode update. (#94389, @pacoxu) [SIG Storage]

• Fix network_programming_latency metric reporting for Endpoints/EndpointSlice deletions, where we don't have correct timestamp (#95363, @wojtek-t) [SIG Network and Scalability]

• Fix paging issues when Azure API returns empty values with non-empty nextLink (#96211, @feiskyer) [SIG Cloud Provider]

• Fix pull image error from multiple ACRs using azure managed identity (#96355, @andyzhangx) [SIG Cloud Provider]

• Fix race condition on timeCache locks. (#94751, @auxten)

• Fix regression on kubectl portforward when TCP and UCP services were configured on the same port. (#94728, @amorenoz)

• Fix scheduler cache snapshot when a Node is deleted before its Pods (#95130, @alculquicondor) [SIG Scheduling]

• Fix the cloudprovider_azure_api_request_duration_seconds metric buckets to correctly capture the latency metrics. Previously, the majority of the calls would fall in the "+Inf" bucket. (#94873, @marwanad) [SIG Cloud Provider and Instrumentation]

• Fix vSphere volumes that could be erroneously attached to wrong node (#96224, @gnufied) [SIG Cloud Provider and Storage]

• Fix verb & scope reporting for kube-apiserver metrics (LIST reported instead of GET) (#95562, @wojtek-t) [SIG API Machinery and Testing]

• Fix vsphere detach failure for static PVs (#95447, @gnufied) [SIG Cloud Provider and Storage]

• Fix: azure disk resize error if source does not exist (#93011, @andyzhangx) [SIG Cloud Provider]

• Fix: detach azure disk broken on Azure Stack (#94885, @andyzhangx) [SIG Cloud Provider]

• Fix: resize Azure disk issue when it's in attached state (#96705, @andyzhangx) [SIG Cloud Provider]

• Fix: smb valid path error (#95583, @andyzhangx) [SIG Storage]

• Fix: use sensitiveOptions on Windows mount (#94126, @andyzhangx) [SIG Cloud Provider and Storage]

• Fixed a bug causing incorrect formatting of kubectl describe ingress. (#94985, @howardjohn) [SIG CLI and Network]

• Fixed a bug in client-go where new clients with customized Dial, Proxy, GetCert config may get stale HTTP transports. (#95427, @roycaihw) [SIG API Machinery]

• Fixed a bug that prevents kubectl to validate CRDs with schema using x-kubernetes-preserve-unknown-fields on object fields. (#96369, @gautierdelorme) [SIG API Machinery and Testing]

• Fixed a bug that prevents the use of ephemeral containers in the presence of a validating admission webhook. (#94685, @verb) [SIG Node and Testing]

• Fixed a bug where aggregator_unavailable_apiservice metrics were reported for deleted apiservices. (#96421, @dgrisonnet) [SIG API Machinery and Instrumentation]

• Fixed a bug where improper storage and comparison of endpoints led to excessive API traffic from the endpoints controller (#94112, @damemi) [SIG Apps, Network and Testing]

• Fixed a regression which prevented pods with docker/default seccomp annotations from being created in 1.19 if a PodSecurityPolicy was in place which did not allow runtime/default seccomp profiles. (#95985, @saschagrunert) [SIG Auth]

• Fixed bug in reflector that couldn't recover from "Too large resource version" errors with API servers 1.17.0-1.18.5 (#94316, @janeczku) [SIG API Machinery]

• Fixed bug where kubectl top pod output is not sorted when --sort-by and --containers flags are used together (#93692, @brianpursley) [SIG CLI]

• Fixed kubelet creating extra sandbox for pods with RestartPolicyOnFailure after all containers succeeded (#92614, @tnqn) [SIG Node and Testing]

• Fixes an issue proxying to ipv6 pods without specifying a port (#94834, @liggitt) [SIG API Machinery and Network]

• Fixes code generation for non-namespaced create subresources fake client test. (#96586, @Doude) [SIG API Machinery]

• Fixes high CPU usage in kubectl drain (#95260, @amandahla) [SIG CLI]

• For vSphere Cloud Provider, If VM of worker node is deleted, the node will also be deleted by node controller (#92608, @lubronzhan) [SIG Cloud Provider]

• Gracefully delete nodes when their parent scale set went missing (#95289, @bpineau) [SIG Cloud Provider]

• HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. (#95981, @caesarxuchao) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node]

• If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400.

  • If the user specifies a timeout in the request URL that exceeds the maximum request deadline allowed by the apiserver, the request will be aborted with an HTTP 400. (#96061, @tkashem) [SIG API Machinery, Network and Testing]

• If we set SelectPolicy MinPolicySelect on scaleUp behavior or scaleDown behavior,Horizontal Pod Autoscaler doesn`t automatically scale the number of pods correctly (#95647, @JoshuaAndrew) [SIG Apps and Autoscaling]

• Ignore apparmor for non-linux operating systems (#93220, @wawa0210) [SIG Node and Windows]

• Ignore root user check when windows pod starts (#92355, @wawa0210) [SIG Node and Windows]

• Improve error messages related to nodePort endpoint changes conntrack entries cleanup. (#96251, @ravens) [SIG Network]

• In dual-stack clusters, kubelet will now set up both IPv4 and IPv6 iptables rules, which may fix some problems, eg with HostPorts. (#94474, @danwinship) [SIG Network and Node]

• Increase maximum IOPS of AWS EBS io1 volume to current maximum (64,000). (#90014, @jacobmarble)

• Ipvs: ensure selected scheduler kernel modules are loaded (#93040, @cmluciano) [SIG Network]

• K8s.io/apimachinery: runtime.DefaultUnstructuredConverter.FromUnstructured now handles converting integer fields to typed float values (#93250, @liggitt) [SIG API Machinery]

• Kube-proxy now trims extra spaces found in loadBalancerSourceRanges to match Service validation. (#94107, @robscott) [SIG Network]

• Kubeadm ensures "kubeadm reset" does not unmount the root "/var/lib/kubelet" directory if it is mounted by the user. (#93702, @thtanaka)

• Kubeadm now makes sure the etcd manifest is regenerated upon upgrade even when no etcd version change takes place (#94395, @rosti) [SIG Cluster Lifecycle]

• Kubeadm now warns (instead of error out) on missing "ca.key" files for root CA, front-proxy CA and etcd CA, during "kubeadm join --control-plane" if the user has provided all certificates, keys and kubeconfig files which require signing with the given CA keys. (#94988, @neolit123)

• Kubeadm: add missing "--experimental-patches" flag to "kubeadm init phase control-plane" (#95786, @Sh4d1) [SIG Cluster Lifecycle]

• Kubeadm: avoid a panic when determining if the running version of CoreDNS is supported during upgrades (#94299, @zouyee) [SIG Cluster Lifecycle]

• Kubeadm: ensure the etcd data directory is created with 0700 permissions during control-plane init and join (#94102, @neolit123) [SIG Cluster Lifecycle]

• Kubeadm: fix coredns migration should be triggered when there are newdefault configs during kubeadm upgrade (#96907, @pacoxu) [SIG Cluster Lifecycle]

• Kubeadm: fix the bug that kubeadm tries to call 'docker info' even if the CRI socket was for another CR (#94555, @SataQiu) [SIG Cluster Lifecycle]

• Kubeadm: for Docker as the container runtime, make the "kubeadm reset" command stop containers before removing them (#94586, @BedivereZero) [SIG Cluster Lifecycle]

• Kubeadm: make the kubeconfig files for the kube-controller-manager and kube-scheduler use the LocalAPIEndpoint instead of the ControlPlaneEndpoint. This makes kubeadm clusters more reseliant to version skew problems during immutable upgrades: https://kubernetes.io/docs/setup/release/version-skew-policy/#kube-controller-manager-kube-scheduler-and-cloud-controller-manager (#94398, @neolit123) [SIG Cluster Lifecycle]

• Kubeadm: relax the validation of kubeconfig server URLs. Allow the user to define custom kubeconfig server URLs without erroring out during validation of existing kubeconfig files (e.g. when using external CA mode). (#94816, @neolit123) [SIG Cluster Lifecycle]

• Kubectl: print error if users place flags before plugin name (#92343, @knight42) [SIG CLI]

• Kubelet: assume that swap is disabled when /proc/swaps does not exist (#93931, @SataQiu) [SIG Node]

• New Azure instance types do now have correct max data disk count information. (#94340, @ialidzhikov) [SIG Cloud Provider and Storage]

• Port mapping now allows the same containerPort of different containers to different hostPort without naming the mapping explicitly. (#94494, @SergeyKanzhelev)

• Print go stack traces at -v=4 and not -v=2 (#94663, @soltysh) [SIG CLI]

• Recreate EndpointSlices on rapid Service creation. (#94730, @robscott)

• Reduce volume name length for vsphere volumes (#96533, @gnufied) [SIG Storage]

• Remove ready file and its directory (which is created during volume SetUp) during emptyDir volume TearDown. (#95770, @jingxu97) [SIG Storage]

• Reorganized iptables rules to fix a performance issue (#95252, @tssurya) [SIG Network]

• Require feature flag CustomCPUCFSQuotaPeriod if setting a non-default cpuCFSQuotaPeriod in kubelet config. (#94687, @karan) [SIG Node]

• Resolves a regression in 1.19+ with workloads targeting deprecated beta os/arch labels getting stuck in NodeAffinity status on node startup. (#96810, @liggitt) [SIG Node]

• Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected. The kubectl-check-ownerreferences tool can be run prior to upgrading to locate existing objects with invalid ownerReferences.

  • A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted.
  • A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. (#92743, @liggitt) [SIG API Machinery, Apps and Testing]

• Skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:162]: Driver azure-disk doesn't support snapshot type DynamicSnapshot -- skipping skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:185]: Driver azure-disk doesn't support ntfs -- skipping (#96144, @qinpingli) [SIG Storage and Testing]

• StatefulSet Controller now waits for PersistentVolumeClaim deletion before creating pods. (#93457, @ymmt2005)

• StreamWatcher now calls HandleCrash at appropriate sequence. (#93108, @lixiaobing1)

• Support the node label node.kubernetes.io/exclude-from-external-load-balancers (#95542, @nilo19) [SIG Cloud Provider]

• The AWS network load balancer attributes can now be specified during service creation (#95247, @kishorj) [SIG Cloud Provider]

• The /debug/api_priority_and_fairness/dump_requests path at an apiserver will no longer return a phantom line for each exempt priority level. (#93406, @MikeSpreitzer) [SIG API Machinery]

• The kube-apiserver will no longer serve APIs that should have been deleted in GA non-alpha levels. Alpha levels will continue to serve the removed APIs so that CI doesn't immediately break. (#96525, @deads2k) [SIG API Machinery]

• The kubelet recognizes the --containerd-namespace flag to configure the namespace used by cadvisor. (#87054, @changyaowei) [SIG Node]

• Unhealthy pods covered by PDBs can be successfully evicted if enough healthy pods are available. (#94381, @michaelgugino) [SIG Apps]

• Update Calico to v3.15.2 (#94241, @lmm) [SIG Cloud Provider]

• Update default etcd server version to 3.4.13 (#94287, @jingyih) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing]

• Update max azure data disk count map (#96308, @andyzhangx) [SIG Cloud Provider and Storage]

• Update the PIP when it is not in the Succeeded provisioning state during the LB update. (#95748, @nilo19) [SIG Cloud Provider]

• Update the frontend IP config when the service's pipName annotation is changed (#95813, @nilo19) [SIG Cloud Provider]

• Update the route table tag in the route reconcile loop (#96545, @nilo19) [SIG Cloud Provider]

• Use NLB Subnet CIDRs instead of VPC CIDRs in Health Check SG Rules (#93515, @t0rr3sp3dr0) [SIG Cloud Provider]

• Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. (#92817, @kmala) [SIG Node]

• Very large patches may now be specified to kubectl patch with the --patch-file flag instead of including them directly on the command line. The --patch and --patch-file flags are mutually exclusive. (#93548, @smarterclayton) [SIG CLI]

• Volume binding: report UnschedulableAndUnresolvable status instead of an error when bound PVs not found (#95541, @cofyc) [SIG Apps, Scheduling and Storage]

• Warn instead of fail when creating Roles and ClusterRoles with custom verbs via kubectl (#92492, @eddiezane) [SIG CLI]

• When creating a PVC with the volume.beta.kubernetes.io/storage-provisioner annotation already set, the PV controller might have incorrectly deleted the newly provisioned PV instead of binding it to the PVC, depending on timing and system load. (#95909, @pohly) [SIG Apps and Storage]

• [kubectl] Fail when local source file doesn't exist (#90333, @bamarni) [SIG CLI]

Other (Cleanup or Flake)

• Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

  ([#96443](https://github.com/kubernetes/kubernetes/pull/96443), [@alaypatel07](https://github.com/alaypatel07)) [SIG Apps]

• --redirect-container-streaming is no longer functional. The flag will be removed in v1.22 (#95935, @tallclair) [SIG Node]

• A new metric requestAbortsTotal has been introduced that counts aborted requests for each group, version, verb, resource, subresource and scope. (#95002, @p0lyn0mial) [SIG API Machinery, Cloud Provider, Instrumentation and Scheduling]

• API priority and fairness metrics use snake_case in label names (#96236, @adtac) [SIG API Machinery, Cluster Lifecycle, Instrumentation and Testing]

• Add fine grained debugging to intra-pod conformance test to troubleshoot networking issues for potentially unhealthy nodes when running conformance or sonobuoy tests. (#93837, @jayunit100)

• Add the following metrics:

  • network_plugin_operations_total
  • network_plugin_operations_errors_total (#93066, @AnishShah)

• Adds a bootstrapping ClusterRole, ClusterRoleBinding and group for /metrics, /livez/, /readyz/, & /healthz/- endpoints. (#93311, @logicalhan) [SIG API Machinery, Auth, Cloud Provider and Instrumentation]

• AdmissionReview objects sent for the creation of Namespace API objects now populate the namespace attribute consistently (previously the namespace attribute was empty for Namespace creation via POST requests, and populated for Namespace creation via server-side-apply PATCH requests) (#95012, @nodo) [SIG API Machinery and Testing]

• Applies translations on all command descriptions (#95439, @HerrNaN) [SIG CLI]

• Base-images: Update to debian-iptables:buster-v1.3.0

  • Uses iptables 1.8.5
  • base-images: Update to debian-base:buster-v1.2.0
  • cluster/images/etcd: Build etcd:3.4.13-1 image
    • Uses debian-base:buster-v1.2.0 (#94733, @justaugustus) [SIG API Machinery, Release and Testing]

• Changed: default "Accept-Encoding" header removed from HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (#96127, @fonsecas72) [SIG Network and Node]

• Client-go header logging (at verbosity levels >= 9) now masks Authorization header contents (#95316, @sfowl) [SIG API Machinery]

• Decrease warning message frequency on setting volume ownership for configmap/secret. (#92878, @jvanz)

• Enhance log information of verifyRunAsNonRoot, add pod, container information (#94911, @wawa0210) [SIG Node]

• Fix func name NewCreateCreateDeploymentOptions (#91931, @lixiaobing1) [SIG CLI]

• Fix kubelet to properly log when a container is started. Previously, kubelet may log that container is dead and was restarted when it was actually started for the first time. This behavior only happened on pods with initContainers and regular containers. (#91469, @rata)

• Fixes the message about no auth for metrics in scheduler. (#94035, @zhouya0) [SIG Scheduling]

• Generators for services are removed from kubectl (#95256, @Git-Jiro) [SIG CLI]

• Introduce kubectl-convert plugin. (#96190, @soltysh) [SIG CLI and Testing]

• Kube-scheduler now logs processed component config at startup (#96426, @damemi) [SIG Scheduling]

• Kubeadm: Separate argument key/value in log msg (#94016, @mrueg) [SIG Cluster Lifecycle]

• Kubeadm: remove the CoreDNS check for known image digests when applying the addon (#94506, @neolit123) [SIG Cluster Lifecycle]

• Kubeadm: update the default pause image version to 1.4.0 on Windows. With this update the image supports Windows versions 1809 (2019LTS), 1903, 1909, 2004 (#95419, @jsturtevant) [SIG Cluster Lifecycle and Windows]

• Kubectl: the generator flag of kubectl autoscale has been deprecated and has no effect, it will be removed in a feature release (#92998, @SataQiu) [SIG CLI]

• Lock ExternalPolicyForExternalIP to default, this feature gate will be removed in 1.22. (#94581, @knabben) [SIG Network]

• Mask ceph RBD adminSecrets in logs when logLevel >= 4. (#95245, @sfowl)

• Remove offensive words from kubectl cluster-info command. (#95202, @rikatz)

• Remove support for "ci/k8s-master" version label in kubeadm, use "ci/latest" instead. See kubernetes/test-infra#18517. (#93626, @vikkyomkar)

• Remove the dependency of csi-translation-lib module on apiserver/cloud-provider/controller-manager (#95543, @wawa0210) [SIG Release]

• Scheduler framework interface moved from pkg/scheduler/framework/v1alpha to pkg/scheduler/framework (#95069, @farah) [SIG Scheduling, Storage and Testing]

• Service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset is removed. All Standard load balancers will always enable tcp resets. (#94297, @MarcPow) [SIG Cloud Provider]

• Stop propagating SelfLink (deprecated in 1.16) in kube-apiserver (#94397, @wojtek-t) [SIG API Machinery and Testing]

• Strip unnecessary security contexts on Windows (#93475, @ravisantoshgudimetla) [SIG Node, Testing and Windows]

• To ensure the code be strong, add unit test for GetAddressAndDialer (#93180, @FreeZhang61) [SIG Node]

• UDP and SCTP protocols can left stale connections that need to be cleared to avoid services disruption, but they can cause problems that are hard to debug. Kubernetes components using a loglevel greater or equal than 4 will log the conntrack operations and its output, to show the entries that were deleted. (#95694, @aojea) [SIG Network]

• Update CNI plugins to v0.8.7 (#94367, @justaugustus) [SIG Cloud Provider, Network, Node, Release and Testing]

• Update cri-tools to v1.19.0 (#94307, @xmudrii) [SIG Cloud Provider]

• Update etcd client side to v3.4.13 (#94259, @jingyih) [SIG API Machinery and Cloud Provider]

• Users will now be able to configure all supported values for AWS NLB health check interval and thresholds for new resources. (#96312, @kishorj) [SIG Cloud Provider]

• V1helpers.MatchNodeSelectorTerms now accepts just a Node and a list of Terms (#95871, @damemi) [SIG Apps, Scheduling and Storage]

• Vsphere: improve logging message on node cache refresh event (#95236, @andrewsykim) [SIG Cloud Provider]

• MatchNodeSelectorTerms function moved to k8s.io/component-helpers (#95531, @damemi) [SIG Apps, Scheduling and Storage]

• kubectl api-resources now prints the API version (as 'API group/version', same as output of kubectl api-versions). The column APIGROUP is now APIVERSION (#95253, @sallyom) [SIG CLI]

• kubectl get ingress now prefers the networking.k8s.io/v1 over extensions/v1beta1 (deprecated since v1.14). To explicitly request the deprecated version, use kubectl get ingress.v1beta1.extensions. (#94309, @liggitt) [SIG API Machinery and CLI]

Dependencies

Added

• cloud.google.com/go/firestore: v1.1.0
• github.com/Azure/go-autorest: v14.2.0+incompatible
• github.com/armon/go-metrics: f0300d1
• github.com/armon/go-radix: 7fddfc3
• github.com/bketelsen/crypt: 5cbc8cc
• github.com/form3tech-oss/jwt-go: v3.2.2+incompatible
• github.com/fvbommel/sortorder: v1.0.1
• github.com/hashicorp/consul/api: v1.1.0
• github.com/hashicorp/consul/sdk: v0.1.1
• github.com/hashicorp/errwrap: v1.0.0
• github.com/hashicorp/go-cleanhttp: v0.5.1
• github.com/hashicorp/go-immutable-radix: v1.0.0
• github.com/hashicorp/go-msgpack: v0.5.3
• github.com/hashicorp/go-multierror: v1.0.0
• github.com/hashicorp/go-rootcerts: v1.0.0
• github.com/hashicorp/go-sockaddr: v1.0.0
• github.com/hashicorp/go-uuid: v1.0.1
• github.com/hashicorp/go.net: v0.0.1
• github.com/hashicorp/logutils: v1.0.0
• github.com/hashicorp/mdns: v1.0.0
• github.com/hashicorp/memberlist: v0.1.3
• github.com/hashicorp/serf: v0.8.2
• github.com/jmespath/go-jmespath/internal/testify: v1.5.1
• github.com/mitchellh/cli: v1.0.0
• github.com/mitchellh/go-testing-interface: v1.0.0
• github.com/mitchellh/gox: v0.4.0
• github.com/mitchellh/iochan: v1.0.0
• github.com/pascaldekloe/goe: 57f6aae
• github.com/posener/complete: v1.1.1
• github.com/ryanuber/columnize: 9b3edd6
• github.com/sean-/seed: e2103e2
• github.com/subosito/gotenv: v1.2.0
• github.com/willf/bitset: d5bec33
• gopkg.in/ini.v1: v1.51.0
• gopkg.in/yaml.v3: 9f266ea
• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0

Changed

• cloud.google.com/go/bigquery: v1.0.1 → v1.4.0
• cloud.google.com/go/datastore: v1.0.0 → v1.1.0
• cloud.google.com/go/pubsub: v1.0.1 → v1.2.0
• cloud.google.com/go/storage: v1.0.0 → v1.6.0
• cloud.google.com/go: v0.51.0 → v0.54.0
• github.com/Azure/go-autorest/autorest/adal: v0.8.2 → v0.9.5
• github.com/Azure/go-autorest/autorest/date: v0.2.0 → v0.3.0
• github.com/Azure/go-autorest/autorest/mocks: v0.3.0 → v0.4.1
• github.com/Azure/go-autorest/autorest: v0.9.6 → v0.11.1
• github.com/Azure/go-autorest/logger: v0.1.0 → v0.2.0
• github.com/Azure/go-autorest/tracing: v0.5.0 → v0.6.0
• github.com/Microsoft/go-winio: fc70bd9 → v0.4.15
• github.com/aws/aws-sdk-go: v1.28.2 → v1.35.24
• github.com/blang/semver: v3.5.0+incompatible → v3.5.1+incompatible
• github.com/checkpoint-restore/go-criu/v4: v4.0.2 → v4.1.0
• github.com/containerd/containerd: v1.3.3 → v1.4.1
• github.com/containerd/ttrpc: v1.0.0 → v1.0.2
• github.com/containerd/typeurl: v1.0.0 → v1.0.1
• github.com/coreos/etcd: v3.3.10+incompatible → v3.3.13+incompatible
• github.com/docker/docker: aa6a989 → bd33bbf
• github.com/go-gl/glfw/v3.3/glfw: 12ad95a → 6f7a984
• github.com/golang/groupcache: 215e871 → 8c9f03a
• github.com/golang/mock: v1.3.1 → v1.4.1
• github.com/golang/protobuf: v1.4.2 → v1.4.3
• github.com/google/cadvisor: v0.37.0 → v0.38.5
• github.com/google/go-cmp: v0.4.0 → v0.5.2
• github.com/google/pprof: d4f498a → 1ebb73c
• github.com/google/uuid: v1.1.1 → v1.1.2
• github.com/gorilla/mux: v1.7.3 → v1.8.0
• github.com/gorilla/websocket: v1.4.0 → v1.4.2
• github.com/jmespath/go-jmespath: c2b33e8 → v0.4.0
• github.com/karrick/godirwalk: v1.7.5 → v1.16.1
• github.com/opencontainers/go-digest: v1.0.0-rc1 → v1.0.0
• github.com/opencontainers/runc: 819fcc6 → v1.0.0-rc92
• github.com/opencontainers/runtime-spec: 237cc4f → 4d89ac9
• github.com/opencontainers/selinux: v1.5.2 → v1.6.0
• github.com/prometheus/procfs: v0.1.3 → v0.2.0
• github.com/quobyte/api: v0.1.2 → v0.1.8
• github.com/spf13/cobra: v1.0.0 → v1.1.1
• github.com/spf13/viper: v1.4.0 → v1.7.0
• github.com/storageos/go-api: 343b3ef → v2.2.0+incompatible
• github.com/stretchr/testify: v1.4.0 → v1.6.1
• github.com/vishvananda/netns: 52d707b → db3c7e5
• go.etcd.io/etcd: 17cef6e → dd1b699
• go.opencensus.io: v0.22.2 → v0.22.3
• golang.org/x/crypto: 75b2880 → 7f63de1
• golang.org/x/exp: da58074 → 6cc2880
• golang.org/x/lint: fdd1cda → 738671d
• golang.org/x/net: ab34263 → 69a7880
• golang.org/x/oauth2: 858c2ad → bf48bf1
• golang.org/x/sys: ed371f2 → 5cba982
• golang.org/x/text: v0.3.3 → v0.3.4
• golang.org/x/time: 555d28b → 3af7569
• golang.org/x/xerrors: 9bdfabe → 5ec99f8
• google.golang.org/api: v0.15.1 → v0.20.0
• google.golang.org/genproto: cb27e3a → 8816d57
• google.golang.org/grpc: v1.27.0 → v1.27.1
• google.golang.org/protobuf: v1.24.0 → v1.25.0
• honnef.co/go/tools: v0.0.1-2019.2.3 → v0.0.1-2020.1.3
• k8s.io/gengo: 8167cfd → 83324d8
• k8s.io/klog/v2: v2.2.0 → v2.4.0
• k8s.io/kube-openapi: 6aeccd4 → d219536
• k8s.io/system-validators: v1.1.2 → v1.2.0
• k8s.io/utils: d5654de → 67b214c
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.9 → v0.0.14
• sigs.k8s.io/structured-merge-diff/v4: v4.0.1 → v4.0.2

Removed

• github.com/armon/consul-api: eb2c6b5
• github.com/go-ini/ini: v1.9.0
• github.com/ugorji/go: v1.1.4
• github.com/xlab/handysort: fb3537e
• github.com/xordataexchange/crypt: b2862e3
• vbom.ml/util: db5cfe1

Dependencies

Added

• cloud.google.com/go/firestore: v1.1.0
• github.com/Azure/go-autorest: v14.2.0+incompatible
• github.com/armon/go-metrics: f0300d1
• github.com/armon/go-radix: 7fddfc3
• github.com/bketelsen/crypt: 5cbc8cc
• github.com/form3tech-oss/jwt-go: v3.2.2+incompatible
• github.com/fvbommel/sortorder: v1.0.1
• github.com/hashicorp/consul/api: v1.1.0
• github.com/hashicorp/consul/sdk: v0.1.1
• github.com/hashicorp/errwrap: v1.0.0
• github.com/hashicorp/go-cleanhttp: v0.5.1
• github.com/hashicorp/go-immutable-radix: v1.0.0
• github.com/hashicorp/go-msgpack: v0.5.3
• github.com/hashicorp/go-multierror: v1.0.0
• github.com/hashicorp/go-rootcerts: v1.0.0
• github.com/hashicorp/go-sockaddr: v1.0.0
• github.com/hashicorp/go-uuid: v1.0.1
• github.com/hashicorp/go.net: v0.0.1
• github.com/hashicorp/logutils: v1.0.0
• github.com/hashicorp/mdns: v1.0.0
• github.com/hashicorp/memberlist: v0.1.3
• github.com/hashicorp/serf: v0.8.2
• github.com/jmespath/go-jmespath/internal/testify: v1.5.1
• github.com/mitchellh/cli: v1.0.0
• github.com/mitchellh/go-testing-interface: v1.0.0
• github.com/mitchellh/gox: v0.4.0
• github.com/mitchellh/iochan: v1.0.0
• github.com/pascaldekloe/goe: 57f6aae
• github.com/posener/complete: v1.1.1
• github.com/ryanuber/columnize: 9b3edd6
• github.com/sean-/seed: e2103e2
• github.com/subosito/gotenv: v1.2.0
• github.com/willf/bitset: d5bec33
• gopkg.in/ini.v1: v1.51.0
• gopkg.in/yaml.v3: 9f266ea
• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0

Changed

• cloud.google.com/go/bigquery: v1.0.1 → v1.4.0
• cloud.google.com/go/datastore: v1.0.0 → v1.1.0
• cloud.google.com/go/pubsub: v1.0.1 → v1.2.0
• cloud.google.com/go/storage: v1.0.0 → v1.6.0
• cloud.google.com/go: v0.51.0 → v0.54.0
• github.com/Azure/go-autorest/autorest/adal: v0.8.2 → v0.9.5
• github.com/Azure/go-autorest/autorest/date: v0.2.0 → v0.3.0
• github.com/Azure/go-autorest/autorest/mocks: v0.3.0 → v0.4.1
• github.com/Azure/go-autorest/autorest: v0.9.6 → v0.11.1
• github.com/Azure/go-autorest/logger: v0.1.0 → v0.2.0
• github.com/Azure/go-autorest/tracing: v0.5.0 → v0.6.0
• github.com/Microsoft/go-winio: fc70bd9 → v0.4.15
• github.com/aws/aws-sdk-go: v1.28.2 → v1.35.24
• github.com/blang/semver: v3.5.0+incompatible → v3.5.1+incompatible
• github.com/checkpoint-restore/go-criu/v4: v4.0.2 → v4.1.0
• github.com/containerd/containerd: v1.3.3 → v1.4.1
• github.com/containerd/ttrpc: v1.0.0 → v1.0.2
• github.com/containerd/typeurl: v1.0.0 → v1.0.1
• github.com/coreos/etcd: v3.3.10+incompatible → v3.3.13+incompatible
• github.com/docker/docker: aa6a989 → bd33bbf
• github.com/go-gl/glfw/v3.3/glfw: 12ad95a → 6f7a984
• github.com/golang/groupcache: 215e871 → 8c9f03a
• github.com/golang/mock: v1.3.1 → v1.4.1
• github.com/golang/protobuf: v1.4.2 → v1.4.3
• github.com/google/cadvisor: v0.37.0 → v0.38.5
• github.com/google/go-cmp: v0.4.0 → v0.5.2
• github.com/google/pprof: d4f498a → 1ebb73c
• github.com/google/uuid: v1.1.1 → v1.1.2
• github.com/gorilla/mux: v1.7.3 → v1.8.0
• github.com/gorilla/websocket: v1.4.0 → v1.4.2
• github.com/jmespath/go-jmespath: c2b33e8 → v0.4.0
• github.com/karrick/godirwalk: v1.7.5 → v1.16.1
• github.com/opencontainers/go-digest: v1.0.0-rc1 → v1.0.0
• github.com/opencontainers/runc: 819fcc6 → v1.0.0-rc92
• github.com/opencontainers/runtime-spec: 237cc4f → 4d89ac9
• github.com/opencontainers/selinux: v1.5.2 → v1.6.0
• github.com/prometheus/procfs: v0.1.3 → v0.2.0
• github.com/quobyte/api: v0.1.2 → v0.1.8
• github.com/spf13/cobra: v1.0.0 → v1.1.1
• github.com/spf13/viper: v1.4.0 → v1.7.0
• github.com/storageos/go-api: 343b3ef → v2.2.0+incompatible
• github.com/stretchr/testify: v1.4.0 → v1.6.1
• github.com/vishvananda/netns: 52d707b → db3c7e5
• go.etcd.io/etcd: 17cef6e → dd1b699
• go.opencensus.io: v0.22.2 → v0.22.3
• golang.org/x/crypto: 75b2880 → 7f63de1
• golang.org/x/exp: da58074 → 6cc2880
• golang.org/x/lint: fdd1cda → 738671d
• golang.org/x/net: ab34263 → 69a7880
• golang.org/x/oauth2: 858c2ad → bf48bf1
• golang.org/x/sys: ed371f2 → 5cba982
• golang.org/x/text: v0.3.3 → v0.3.4
• golang.org/x/time: 555d28b → 3af7569
• golang.org/x/xerrors: 9bdfabe → 5ec99f8
• google.golang.org/api: v0.15.1 → v0.20.0
• google.golang.org/genproto: cb27e3a → 8816d57
• google.golang.org/grpc: v1.27.0 → v1.27.1
• google.golang.org/protobuf: v1.24.0 → v1.25.0
• honnef.co/go/tools: v0.0.1-2019.2.3 → v0.0.1-2020.1.3
• k8s.io/gengo: 8167cfd → 83324d8
• k8s.io/klog/v2: v2.2.0 → v2.4.0
• k8s.io/kube-openapi: 6aeccd4 → d219536
• k8s.io/system-validators: v1.1.2 → v1.2.0
• k8s.io/utils: d5654de → 67b214c
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.9 → v0.0.14
• sigs.k8s.io/structured-merge-diff/v4: v4.0.1 → v4.0.2

Removed

• github.com/armon/consul-api: eb2c6b5
• github.com/go-ini/ini: v1.9.0
• github.com/ugorji/go: v1.1.4
• github.com/xlab/handysort: fb3537e
• github.com/xordataexchange/crypt: b2862e3
• vbom.ml/util: db5cfe1

v1.20.0-rc.0

Downloads for v1.20.0-rc.0

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0-beta.2

Changes by Kind

Feature

• Kubernetes is now built using go1.15.5
  • build: Update to k/repo-infra@v0.1.2 (supports go1.15.5) (#95776, @justaugustus) [SIG Cloud Provider, Instrumentation, Release and Testing]

Failing Test

• Resolves an issue running Ingress conformance tests on clusters which use finalizers on Ingress objects to manage releasing load balancer resources (#96742, @spencerhance) [SIG Network and Testing]
• The Conformance test "validates that there is no conflict between pods with same hostPort but different hostIP and protocol" now validates the connectivity to each hostPort, in addition to the functionality. (#96627, @aojea) [SIG Scheduling and Testing]

Bug or Regression

• Bump node-problem-detector version to v0.8.5 to fix OOM detection in with Linux kernels 5.1+ (#96716, @tosi3k) [SIG Cloud Provider, Scalability and Testing]
• Changes to timeout parameter handling in 1.20.0-beta.2 have been reverted to avoid breaking backwards compatibility with existing clients. (#96727, @liggitt) [SIG API Machinery and Testing]
• Duplicate owner reference entries in create/update/patch requests now get deduplicated by the API server. The client sending the request now receives a warning header in the API response. Clients should stop sending requests with duplicate owner references. The API server may reject such requests as early as 1.24. (#96185, @roycaihw) [SIG API Machinery and Testing]
• Fix: resize Azure disk issue when it's in attached state (#96705, @andyzhangx) [SIG Cloud Provider]
• Fixed a bug where aggregator_unavailable_apiservice metrics were reported for deleted apiservices. (#96421, @dgrisonnet) [SIG API Machinery and Instrumentation]
• Fixes code generation for non-namespaced create subresources fake client test. (#96586, @Doude) [SIG API Machinery]
• HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. (#95981, @caesarxuchao) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node]
• Kubeadm: fix coredns migration should be triggered when there are newdefault configs during kubeadm upgrade (#96907, @pacoxu) [SIG Cluster Lifecycle]
• Reduce volume name length for vsphere volumes (#96533, @gnufied) [SIG Storage]
• Resolves a regression in 1.19+ with workloads targeting deprecated beta os/arch labels getting stuck in NodeAffinity status on node startup. (#96810, @liggitt) [SIG Node]

Dependencies

Added

Nothing has changed.

Changed

• github.com/google/cadvisor: v0.38.4 → v0.38.5

Removed

Nothing has changed.

v1.20.0-beta.2

Downloads for v1.20.0-beta.2

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0-beta.1

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• A bug was fixed in kubelet where exec probe timeouts were not respected. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts.

This change in behavior may be unexpected for some clusters and can be disabled by turning off the ExecProbeTimeout feature gate. This gate will be locked and removed in future releases so that exec probe timeouts are always respected. (#94115, @andrewsykim) [SIG Node and Testing]

• For CSI drivers, kubelet no longer creates the target_path for NodePublishVolume in accordance with the CSI spec. Kubelet also no longer checks if staging and target paths are mounts or corrupted. CSI drivers need to be idempotent and do any necessary mount verification. (#88759, @andyzhangx) [SIG Storage]

• Kubeadm:

• The label applied to control-plane nodes "node-role.kubernetes.io/master" is now deprecated and will be removed in a future release after a GA deprecation period.

• Introduce a new label "node-role.kubernetes.io/control-plane" that will be applied in parallel to "node-role.kubernetes.io/master" until the removal of the "node-role.kubernetes.io/master" label.

• Make "kubeadm upgrade apply" add the "node-role.kubernetes.io/control-plane" label on existing nodes that only have the "node-role.kubernetes.io/master" label during upgrade.

• Please adapt your tooling built on top of kubeadm to use the "node-role.kubernetes.io/control-plane" label.

• The taint applied to control-plane nodes "node-role.kubernetes.io/master:NoSchedule" is now deprecated and will be removed in a future release after a GA deprecation period.

• Apply toleration for a new, future taint "node-role.kubernetes.io/control-plane:NoSchedule" to the kubeadm CoreDNS / kube-dns managed manifests. Note that this taint is not yet applied to kubeadm control-plane nodes.

• Please adapt your workloads to tolerate the same future taint preemptively.

For more details see: http://git.k8s.io/enhancements/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md (#95382, @neolit123) [SIG Cluster Lifecycle]

Changes by Kind

Deprecation

• Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance issues in the Kubernetes community. We encourage you to evaluate moving to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available. (#94624, @dims) [SIG Node]
• Kubectl: deprecate --delete-local-data (#95076, @dougsland) [SIG CLI, Cloud Provider and Scalability]

API Change

• API priority and fairness graduated to beta 1.19 servers with APF turned on should not be run in a multi-server cluster with 1.20+ servers. (#96527, @adtac) [SIG API Machinery and Testing]

• Add LoadBalancerIPMode feature gate (#92312, @Sh4d1) [SIG Apps, CLI, Cloud Provider and Network]

• Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest (#95741, @katiewasnothere) [SIG Node]

• Add a 'serving' and terminating condition to the EndpointSlice API.

  serving tracks the readiness of endpoints regardless of their terminating state. This is distinct from ready since ready is only true when pods are not terminating. terminating is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. (#92968, @andrewsykim) [SIG Apps and Network]

• Add support for hugepages to downward API (#86102, @derekwaynecarr) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]

• Adds kubelet alpha feature, GracefulNodeShutdown which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. (#96129, @bobbypage) [SIG Node]

• AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. (#96327, @robscott) [SIG Apps and Network]

• Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. (#92744, @uablrek) [SIG Apps and Network]

• Document that ServiceTopology feature is required to use service.spec.topologyKeys. (#96528, @andrewsykim) [SIG Apps]

• EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate.

  • EndpointSlice topology field will be deprecated in an upcoming release.
  • EndpointSlice "IP" address type is formally removed after being deprecated in Kubernetes 1.17.
  • The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. (#96440, @robscott) [SIG API Machinery, Apps and Network]

• Fewer candidates are enumerated for preemption to improve performance in large clusters (#94814, @adtac) [SIG Scheduling]

• If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric serviceaccount_stale_tokens_total to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting kube-apiserver with flag --service-account-extend-token-expiration=false (#96273, @zshihang) [SIG API Machinery and Auth]

• Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. (#94196, @andrewsykim) [SIG Node and Release]

• Kube-apiserver now deletes expired kube-apiserver Lease objects:

  • The feature is under feature gate APIServerIdentity.
  • A flag is added to kube-apiserver: identity-lease-garbage-collection-check-period-seconds (#95895, @roycaihw) [SIG API Machinery, Apps, Auth and Testing]

• Move configurable fsgroup change policy for pods to beta (#96376, @gnufied) [SIG Apps and Storage]

• New flag is introduced, i.e. --topology-manager-scope=container|pod. The default value is the "container" scope. (#92967, @cezaryzukowski) [SIG Instrumentation, Node and Testing]

• NodeAffinity plugin can be configured with AddedAffinity. (#96202, @alculquicondor) [SIG Node, Scheduling and Testing]

• Promote RuntimeClass feature to GA. Promote node.k8s.io API groups from v1beta1 to v1. (#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]

• Reminder: The labels "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" are deprecated in favor of "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" respectively. All users of the "failure-domain.beta..." labels should switch to the "topology..." equivalents. (#96033, @thockin) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing]

• The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBSVC is enabled. "action required" The feature gate is disabled by default. The user has to enable it for the API Server. (#94028, @janosi) [SIG API Machinery and Apps]

• This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in CSIDriverSpec. (#93130, @zshihang) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]

• Users can try the cronjob controller v2 using the feature gate. This will be the default controller in future releases. (#93370, @alaypatel07) [SIG API Machinery, Apps, Auth and Testing]

• VolumeSnapshotDataSource moves to GA in 1.20 release (#95282, @xing-yang) [SIG Apps]

Feature

• Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: (#95896, @zshihang) [SIG API Machinery and Cluster Lifecycle]
• A new set of alpha metrics are reported by the Kubernetes scheduler under the /metrics/resources endpoint that allow administrators to easily see the resource consumption (requests and limits for all resources on the pods) and compare it to actual pod usage or node capacity. (#94866, @smarterclayton) [SIG API Machinery, Instrumentation, Node and Scheduling]
• Add --experimental-logging-sanitization flag enabling runtime protection from leaking sensitive data in logs (#96370, @serathius) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• Add a StorageVersionAPI feature gate that makes API server update storageversions before serving certain write requests. This feature allows the storage migrator to manage storage migration for built-in resources. Enabling internal.apiserver.k8s.io/v1alpha1 API and APIServerIdentity feature gate are required to use this feature. (#93873, @roycaihw) [SIG API Machinery, Auth and Testing]
• Add a new vSphere metric: cloudprovider_vsphere_vcenter_versions. It's content show vCenter hostnames with the associated server version. (#94526, @Danil-Grigorev) [SIG Cloud Provider and Instrumentation]
• Add feature to size memory backed volumes (#94444, @derekwaynecarr) [SIG Storage and Testing]
• Add node_authorizer_actions_duration_seconds metric that can be used to estimate load to node authorizer. (#92466, @mborsz) [SIG API Machinery, Auth and Instrumentation]
• Add pod_ based CPU and memory metrics to Kubelet's /metrics/resource endpoint (#95839, @egernst) [SIG Instrumentation, Node and Testing]
• Adds a headless service on node-local-cache addon. (#88412, @stafot) [SIG Cloud Provider and Network]
• CRDs: For structural schemas, non-nullable null map fields will now be dropped and defaulted if a default is available. null items in list will continue being preserved, and fail validation if not nullable. (#95423, @apelisse) [SIG API Machinery]
• E2e test for PodFsGroupChangePolicy (#96247, @saikat-royc) [SIG Storage and Testing]
• Gradudate the Pod Resources API to G.A Introduces the pod_resources_endpoint_requests_total metric which tracks the total number of requests to the pod resources API (#92165, @RenaudWasTaken) [SIG Instrumentation, Node and Testing]
• Introduce api-extensions category which will return: mutating admission configs, validating admission configs, CRDs and APIServices when used in kubectl get, for example. (#95603, @soltysh) [SIG API Machinery]
• Kube-apiserver now maintains a Lease object to identify itself:
  • The feature is under feature gate APIServerIdentity.
  • Two flags are added to kube-apiserver: identity-lease-duration-seconds, identity-lease-renew-interval-seconds (#95533, @roycaihw) [SIG API Machinery]
• Kube-apiserver: The timeout used when making health check calls to etcd can now be configured with --etcd-healthcheck-timeout. The default timeout is 2 seconds, matching the previous behavior. (#93244, @Sh4d1) [SIG API Machinery]
• Kubectl: Previously users cannot provide arguments to a external diff tool via KUBECTL_EXTERNAL_DIFF env. This release now allow users to specify args to KUBECTL_EXTERNAL_DIFF env. (#95292, @dougsland) [SIG CLI]
• Scheduler now ignores Pod update events if the resourceVersion of old and new Pods are identical. (#96071, @Huang-Wei) [SIG Scheduling]
• Support custom tags for cloud provider managed resources (#96450, @nilo19) [SIG Cloud Provider]
• Support customize load balancer health probe protocol and request path (#96338, @nilo19) [SIG Cloud Provider]
• Support multiple standard load balancers in one cluster (#96111, @nilo19) [SIG Cloud Provider]
• The beta RootCAConfigMap feature gate is enabled by default and causes kube-controller-manager to publish a "kube-root-ca.crt" ConfigMap to every namespace. This ConfigMap contains a CA bundle used for verifying connections to the kube-apiserver. (#96197, @zshihang) [SIG API Machinery, Apps, Auth and Testing]
• The kubelet_runtime_operations_duration_seconds metric got additional buckets of 60, 300, 600, 900 and 1200 seconds (#96054, @alvaroaleman) [SIG Instrumentation and Node]
• There is a new pv_collector_total_pv_count metric that counts persistent volumes by the volume plugin name and volume mode. (#95719, @tsmetana) [SIG Apps, Instrumentation, Storage and Testing]
• Volume snapshot e2e test to validate PVC and VolumeSnapshotContent finalizer (#95863, @RaunakShah) [SIG Cloud Provider, Storage and Testing]
• Warns user when executing kubectl apply/diff to resource currently being deleted. (#95544, @SaiHarshaK) [SIG CLI]
• kubectl alpha debug has graduated to beta and is now kubectl debug. (#96138, @verb) [SIG CLI and Testing]
• kubectl debug gains support for changing container images when copying a pod for debugging, similar to how kubectl set image works. See kubectl help debug for more information. (#96058, @verb) [SIG CLI]

Documentation

• Updates docs and guidance on cloud provider InstancesV2 and Zones interface for external cloud providers:
  • removes experimental warning for InstancesV2
  • document that implementation of InstancesV2 will disable calls to Zones
  • deprecate Zones in favor of InstancesV2 (#96397, @andrewsykim) [SIG Cloud Provider]

Bug or Regression

• Change plugin name in fsgroupapplymetrics of csi and flexvolume to distinguish different driver (#95892, @JornShen) [SIG Instrumentation, Storage and Testing]
• Clear UDP conntrack entry on endpoint changes when using nodeport (#71573, @JacobTanenbaum) [SIG Network]
• Exposes and sets a default timeout for the TokenReview client for DelegatingAuthenticationOptions (#96217, @p0lyn0mial) [SIG API Machinery and Cloud Provider]
• Fix CVE-2020-8555 for Quobyte client connections. (#95206, @misterikkit) [SIG Storage]
• Fix IP fragmentation of UDP and TCP packets not supported issues on LoadBalancer rules (#96464, @nilo19) [SIG Cloud Provider]
• Fix a bug that DefaultPreemption plugin is disabled when using (legacy) scheduler policy. (#96439, @Huang-Wei) [SIG Scheduling and Testing]
• Fix bug in JSON path parser where an error occurs when a range is empty (#95933, @brianpursley) [SIG API Machinery]
• Fix client-go prometheus metrics to correctly present the API path accessed in some environments. (#74363, @aanm) [SIG API Machinery]
• Fix memory leak in kube-apiserver when underlying time goes forth and back. (#96266, @chenyw1990) [SIG API Machinery]
• Fix paging issues when Azure API returns empty values with non-empty nextLink (#96211, @feiskyer) [SIG Cloud Provider]
• Fix pull image error from multiple ACRs using azure managed identity (#96355, @andyzhangx) [SIG Cloud Provider]
• Fix vSphere volumes that could be erroneously attached to wrong node (#96224, @gnufied) [SIG Cloud Provider and Storage]
• Fixed a bug that prevents kubectl to validate CRDs with schema using x-kubernetes-preserve-unknown-fields on object fields. (#96369, @gautierdelorme) [SIG API Machinery and Testing]
• For vSphere Cloud Provider, If VM of worker node is deleted, the node will also be deleted by node controller (#92608, @lubronzhan) [SIG Cloud Provider]
• HTTP/2 connection health check is enabled by default in all Kubernetes clients. The feature should work out-of-the-box. If needed, users can tune the feature via the HTTP2_READ_IDLE_TIMEOUT_SECONDS and HTTP2_PING_TIMEOUT_SECONDS environment variables. The feature is disabled if HTTP2_READ_IDLE_TIMEOUT_SECONDS is set to 0. (#95981, @caesarxuchao) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node]
• If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400.
  • If the user specifies a timeout in the request URL that exceeds the maximum request deadline allowed by the apiserver, the request will be aborted with an HTTP 400. (#96061, @tkashem) [SIG API Machinery, Network and Testing]
• Improve error messages related to nodePort endpoint changes conntrack entries cleanup. (#96251, @ravens) [SIG Network]
• Print go stack traces at -v=4 and not -v=2 (#94663, @soltysh) [SIG CLI]
• Remove ready file and its directory (which is created during volume SetUp) during emptyDir volume TearDown. (#95770, @jingxu97) [SIG Storage]
• Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of OwnerRefInvalidNamespace are recorded when namespace mismatches between child and owner objects are detected.
  • A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted.
  • A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector. (#92743, @liggitt) [SIG API Machinery, Apps and Testing]
• Skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:162]: Driver azure-disk doesn't support snapshot type DynamicSnapshot -- skipping skip [k8s.io/kubernetes@v1.19.0/test/e2e/storage/testsuites/base.go:185]: Driver azure-disk doesn't support ntfs -- skipping (#96144, @qinpingli) [SIG Storage and Testing]
• The AWS network load balancer attributes can now be specified during service creation (#95247, @kishorj) [SIG Cloud Provider]
• The kube-apiserver will no longer serve APIs that should have been deleted in GA non-alpha levels. Alpha levels will continue to serve the removed APIs so that CI doesn't immediately break. (#96525, @deads2k) [SIG API Machinery]
• Update max azure data disk count map (#96308, @andyzhangx) [SIG Cloud Provider and Storage]
• Update the route table tag in the route reconcile loop (#96545, @nilo19) [SIG Cloud Provider]
• Volume binding: report UnschedulableAndUnresolvable status instead of an error when bound PVs not found (#95541, @cofyc) [SIG Apps, Scheduling and Storage]
• [kubectl] Fail when local source file doesn't exist (#90333, @bamarni) [SIG CLI]

Other (Cleanup or Flake)

• Handle slow cronjob lister in cronjob controller v2 and improve memory footprint. (#96443, @alaypatel07) [SIG Apps]
• --redirect-container-streaming is no longer functional. The flag will be removed in v1.22 (#95935, @tallclair) [SIG Node]
• A new metric requestAbortsTotal has been introduced that counts aborted requests for each group, version, verb, resource, subresource and scope. (#95002, @p0lyn0mial) [SIG API Machinery, Cloud Provider, Instrumentation and Scheduling]
• API priority and fairness metrics use snake_case in label names (#96236, @adtac) [SIG API Machinery, Cluster Lifecycle, Instrumentation and Testing]
• Applies translations on all command descriptions (#95439, @HerrNaN) [SIG CLI]
• Changed: default "Accept-Encoding" header removed from HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (#96127, @fonsecas72) [SIG Network and Node]
• Generators for services are removed from kubectl (#95256, @Git-Jiro) [SIG CLI]
• Introduce kubectl-convert plugin. (#96190, @soltysh) [SIG CLI and Testing]
• Kube-scheduler now logs processed component config at startup (#96426, @damemi) [SIG Scheduling]
• NONE (#96179, @bbyrne5) [SIG Network]
• Users will now be able to configure all supported values for AWS NLB health check interval and thresholds for new resources. (#96312, @kishorj) [SIG Cloud Provider]

Dependencies

Added

• cloud.google.com/go/firestore: v1.1.0
• github.com/armon/go-metrics: f0300d1
• github.com/armon/go-radix: 7fddfc3
• github.com/bketelsen/crypt: 5cbc8cc
• github.com/hashicorp/consul/api: v1.1.0
• github.com/hashicorp/consul/sdk: v0.1.1
• github.com/hashicorp/errwrap: v1.0.0
• github.com/hashicorp/go-cleanhttp: v0.5.1
• github.com/hashicorp/go-immutable-radix: v1.0.0
• github.com/hashicorp/go-msgpack: v0.5.3
• github.com/hashicorp/go-multierror: v1.0.0
• github.com/hashicorp/go-rootcerts: v1.0.0
• github.com/hashicorp/go-sockaddr: v1.0.0
• github.com/hashicorp/go-uuid: v1.0.1
• github.com/hashicorp/go.net: v0.0.1
• github.com/hashicorp/logutils: v1.0.0
• github.com/hashicorp/mdns: v1.0.0
• github.com/hashicorp/memberlist: v0.1.3
• github.com/hashicorp/serf: v0.8.2
• github.com/mitchellh/cli: v1.0.0
• github.com/mitchellh/go-testing-interface: v1.0.0
• github.com/mitchellh/gox: v0.4.0
• github.com/mitchellh/iochan: v1.0.0
• github.com/pascaldekloe/goe: 57f6aae
• github.com/posener/complete: v1.1.1
• github.com/ryanuber/columnize: 9b3edd6
• github.com/sean-/seed: e2103e2
• github.com/subosito/gotenv: v1.2.0
• github.com/willf/bitset: d5bec33
• gopkg.in/ini.v1: v1.51.0
• gopkg.in/yaml.v3: 9f266ea
• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0

Changed

• cloud.google.com/go/bigquery: v1.0.1 → v1.4.0
• cloud.google.com/go/datastore: v1.0.0 → v1.1.0
• cloud.google.com/go/pubsub: v1.0.1 → v1.2.0
• cloud.google.com/go/storage: v1.0.0 → v1.6.0
• cloud.google.com/go: v0.51.0 → v0.54.0
• github.com/Microsoft/go-winio: fc70bd9 → v0.4.15
• github.com/aws/aws-sdk-go: v1.35.5 → v1.35.24
• github.com/blang/semver: v3.5.0+incompatible → v3.5.1+incompatible
• github.com/checkpoint-restore/go-criu/v4: v4.0.2 → v4.1.0
• github.com/containerd/containerd: v1.3.3 → v1.4.1
• github.com/containerd/ttrpc: v1.0.0 → v1.0.2
• github.com/containerd/typeurl: v1.0.0 → v1.0.1
• github.com/coreos/etcd: v3.3.10+incompatible → v3.3.13+incompatible
• github.com/docker/docker: aa6a989 → bd33bbf
• github.com/go-gl/glfw/v3.3/glfw: 12ad95a → 6f7a984
• github.com/golang/groupcache: 215e871 → 8c9f03a
• github.com/golang/mock: v1.3.1 → v1.4.1
• github.com/golang/protobuf: v1.4.2 → v1.4.3
• github.com/google/cadvisor: v0.37.0 → v0.38.4
• github.com/google/go-cmp: v0.4.0 → v0.5.2
• github.com/google/pprof: d4f498a → 1ebb73c
• github.com/google/uuid: v1.1.1 → v1.1.2
• github.com/gorilla/mux: v1.7.3 → v1.8.0
• github.com/gorilla/websocket: v1.4.0 → v1.4.2
• github.com/karrick/godirwalk: v1.7.5 → v1.16.1
• github.com/opencontainers/runc: 819fcc6 → v1.0.0-rc92
• github.com/opencontainers/runtime-spec: 237cc4f → 4d89ac9
• github.com/opencontainers/selinux: v1.5.2 → v1.6.0
• github.com/prometheus/procfs: v0.1.3 → v0.2.0
• github.com/quobyte/api: v0.1.2 → v0.1.8
• github.com/spf13/cobra: v1.0.0 → v1.1.1
• github.com/spf13/viper: v1.4.0 → v1.7.0
• github.com/stretchr/testify: v1.4.0 → v1.6.1
• github.com/vishvananda/netns: 52d707b → db3c7e5
• go.opencensus.io: v0.22.2 → v0.22.3
• golang.org/x/exp: da58074 → 6cc2880
• golang.org/x/lint: fdd1cda → 738671d
• golang.org/x/net: ab34263 → 69a7880
• golang.org/x/oauth2: 858c2ad → bf48bf1
• golang.org/x/sys: ed371f2 → 5cba982
• golang.org/x/text: v0.3.3 → v0.3.4
• golang.org/x/time: 555d28b → 3af7569
• golang.org/x/xerrors: 9bdfabe → 5ec99f8
• google.golang.org/api: v0.15.1 → v0.20.0
• google.golang.org/genproto: cb27e3a → 8816d57
• google.golang.org/grpc: v1.27.0 → v1.27.1
• google.golang.org/protobuf: v1.24.0 → v1.25.0
• honnef.co/go/tools: v0.0.1-2019.2.3 → v0.0.1-2020.1.3
• k8s.io/gengo: 8167cfd → 83324d8
• k8s.io/klog/v2: v2.2.0 → v2.4.0
• k8s.io/kube-openapi: 8b50664 → d219536
• k8s.io/utils: d5654de → 67b214c
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.12 → v0.0.14
• sigs.k8s.io/structured-merge-diff/v4: b3cf1e8 → v4.0.2

Removed

• github.com/armon/consul-api: eb2c6b5
• github.com/go-ini/ini: v1.9.0
• github.com/ugorji/go: v1.1.4
• github.com/xordataexchange/crypt: b2862e3

v1.20.0-beta.1

Downloads for v1.20.0-beta.1

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0-beta.0

Changes by Kind

Deprecation

• ACTION REQUIRED: The kube-apiserver ability to serve on an insecure port, deprecated since v1.10, has been removed. The insecure address flags --address and --insecure-bind-address have no effect in kube-apiserver and will be removed in v1.24. The insecure port flags --port and --insecure-port may only be set to 0 and will be removed in v1.24. (#95856, @knight42) [SIG API Machinery, Node and Testing]

API Change

• • TokenRequest and TokenRequestProjection features have been promoted to GA. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection for details on configuring and using this feature. The TokenRequest and TokenRequestProjection feature gates will be removed in v1.21.
  • kubeadm's kube-apiserver Pod manifest now includes the following flags by default "--service-account-key-file", "--service-account-signing-key-file", "--service-account-issuer". (#93258, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle, Storage and Testing]
• Certain fields on Service objects will be automatically cleared when changing the service's type to a mode that does not need those fields. For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. (#95196, @thockin) [SIG API Machinery, Apps, Network and Testing]
• Services will now have a clusterIPs field to go with clusterIP. clusterIPs[0] is a synonym for clusterIP and will be syncronized on create and update operations. (#95894, @thockin) [SIG Network]

Feature

• A new metric apiserver_request_filter_duration_seconds has been introduced that measures request filter latency in seconds. (#95207, @tkashem) [SIG API Machinery and Instrumentation]
• Add a new flag to set priority for the kubelet on Windows nodes so that workloads cannot overwhelm the node there by disrupting kubelet process. (#96051, @ravisantoshgudimetla) [SIG Node and Windows]
• Changed: default "Accept: /" header added to HTTP probes. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes (https://github.com/kubernetes/website/pull/24756) (#95641, @fonsecas72) [SIG Network and Node]
• Client-go credential plugins can now be passed in the current cluster information via the KUBERNETES_EXEC_INFO environment variable. (#95489, @ankeesler) [SIG API Machinery and Auth]
• Kube-apiserver: added support for compressing rotated audit log files with --audit-log-compress (#94066, @lojies) [SIG API Machinery and Auth]

Documentation

• Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList (#95117, @andrewsykim) [SIG API Machinery]

Bug or Regression

• Added support to kube-proxy for externalTrafficPolicy=Local setting via Direct Server Return (DSR) load balancers on Windows. (#93166, @elweb9858) [SIG Network]
• Disable watchcache for events (#96052, @wojtek-t) [SIG API Machinery]
• Disabled LocalStorageCapacityIsolation feature gate is honored during scheduling. (#96092, @Huang-Wei) [SIG Scheduling]
• Fix bug in JSON path parser where an error occurs when a range is empty (#95933, @brianpursley) [SIG API Machinery]
• Fix k8s.io/apimachinery/pkg/api/meta.SetStatusCondition to update ObservedGeneration (#95961, @KnicKnic) [SIG API Machinery]
• Fixed a regression which prevented pods with docker/default seccomp annotations from being created in 1.19 if a PodSecurityPolicy was in place which did not allow runtime/default seccomp profiles. (#95985, @saschagrunert) [SIG Auth]
• Kubectl: print error if users place flags before plugin name (#92343, @knight42) [SIG CLI]
• When creating a PVC with the volume.beta.kubernetes.io/storage-provisioner annotation already set, the PV controller might have incorrectly deleted the newly provisioned PV instead of binding it to the PVC, depending on timing and system load. (#95909, @pohly) [SIG Apps and Storage]

Other (Cleanup or Flake)

• Kubectl: the generator flag of kubectl autoscale has been deprecated and has no effect, it will be removed in a feature release (#92998, @SataQiu) [SIG CLI]
• V1helpers.MatchNodeSelectorTerms now accepts just a Node and a list of Terms (#95871, @damemi) [SIG Apps, Scheduling and Storage]
• MatchNodeSelectorTerms function moved to k8s.io/component-helpers (#95531, @damemi) [SIG Apps, Scheduling and Storage]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.20.0-beta.0

Downloads for v1.20.0-beta.0

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0-alpha.3

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Kubeadm: improve the validation of serviceSubnet and podSubnet. ServiceSubnet has to be limited in size, due to implementation details, and the mask can not allocate more than 20 bits. PodSubnet validates against the corresponding cluster "--node-cidr-mask-size" of the kube-controller-manager, it fail if the values are not compatible. kubeadm no longer sets the node-mask automatically on IPv6 deployments, you must check that your IPv6 service subnet mask is compatible with the default node mask /64 or set it accordenly. Previously, for IPv6, if the podSubnet had a mask lower than /112, kubeadm calculated a node-mask to be multiple of eight and splitting the available bits to maximise the number used for nodes. (#95723, @aojea) [SIG Cluster Lifecycle]
• Windows hyper-v container featuregate is deprecated in 1.20 and will be removed in 1.21 (#95505, @wawa0210) [SIG Node and Windows]

Changes by Kind

Deprecation

• Support 'controlplane' as a valid EgressSelection type in the EgressSelectorConfiguration API. 'Master' is deprecated and will be removed in v1.22. (#95235, @andrewsykim) [SIG API Machinery]

API Change

• Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API. It changes the dual-stack API wrt Service from a single ipFamily field to 3 fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack), ipFamilies (a list of families assigned), and clusterIPs (inclusive of clusterIP). Most users do not need to set anything at all, defaulting will handle it for them. Services are single-stack unless the user asks for dual-stack. This is all gated by the "IPv6DualStack" feature gate. (#91824, @khenidak) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
• Introduces a metric source for HPAs which allows scaling based on container resource usage. (#90691, @arjunrn) [SIG API Machinery, Apps, Autoscaling and CLI]

Feature

• Add a metric for time taken to perform recursive permission change (#95866, @JornShen) [SIG Instrumentation and Storage]
• Allow cross compilation of kubernetes on different platforms. (#94403, @bnrjee) [SIG Release]
• Command to start network proxy changes from 'KUBE_ENABLE_EGRESS_VIA_KONNECTIVITY_SERVICE ./cluster/kube-up.sh' to 'KUBE_ENABLE_KONNECTIVITY_SERVICE=true ./hack/kube-up.sh' (#92669, @Jefftree) [SIG Cloud Provider]
• DefaultPodTopologySpread graduated to Beta. The feature gate is enabled by default. (#95631, @alculquicondor) [SIG Scheduling and Testing]
• Kubernetes E2E test image manifest lists now contain Windows images. (#77398, @claudiubelu) [SIG Testing and Windows]
• Support for Windows container images (OS Versions: 1809, 1903, 1909, 2004) was added the pause:3.4 image. (#91452, @claudiubelu) [SIG Node, Release and Windows]

Documentation

• Fake dynamic client: document that List does not preserve TypeMeta in UnstructuredList (#95117, @andrewsykim) [SIG API Machinery]

Bug or Regression

• Exposes and sets a default timeout for the SubjectAccessReview client for DelegatingAuthorizationOptions. (#95725, @p0lyn0mial) [SIG API Machinery and Cloud Provider]
• Alter wording to describe pods using a pvc (#95635, @RaunakShah) [SIG CLI]
• If we set SelectPolicy MinPolicySelect on scaleUp behavior or scaleDown behavior,Horizontal Pod Autoscaler doesn`t automatically scale the number of pods correctly (#95647, @JoshuaAndrew) [SIG Apps and Autoscaling]
• Ignore apparmor for non-linux operating systems (#93220, @wawa0210) [SIG Node and Windows]
• Ipvs: ensure selected scheduler kernel modules are loaded (#93040, @cmluciano) [SIG Network]
• Kubeadm: add missing "--experimental-patches" flag to "kubeadm init phase control-plane" (#95786, @Sh4d1) [SIG Cluster Lifecycle]
• Reorganized iptables rules to fix a performance issue (#95252, @tssurya) [SIG Network]
• Unhealthy pods covered by PDBs can be successfully evicted if enough healthy pods are available. (#94381, @michaelgugino) [SIG Apps]
• Update the PIP when it is not in the Succeeded provisioning state during the LB update. (#95748, @nilo19) [SIG Cloud Provider]
• Update the frontend IP config when the service's pipName annotation is changed (#95813, @nilo19) [SIG Cloud Provider]

Other (Cleanup or Flake)

• NO (#95690, @nikhita) [SIG Release]

Dependencies

Added

• github.com/form3tech-oss/jwt-go: v3.2.2+incompatible

Changed

• github.com/Azure/go-autorest/autorest/adal: v0.9.0 → v0.9.5
• github.com/Azure/go-autorest/autorest/mocks: v0.4.0 → v0.4.1
• golang.org/x/crypto: 75b2880 → 7f63de1

Removed

Nothing has changed.

v1.20.0-alpha.3

Downloads for v1.20.0-alpha.3

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0-alpha.2

Changes by Kind

API Change

• New parameter defaultingType for PodTopologySpread plugin allows to use k8s defined or user provided default constraints (#95048, @alculquicondor) [SIG Scheduling]

Feature

• Added new k8s.io/component-helpers repository providing shared helper code for (core) components. (#92507, @ingvagabund) [SIG Apps, Node, Release and Scheduling]
• Adds create ingress command to kubectl (#78153, @amimof) [SIG CLI and Network]
• Kubectl create now supports creating ingress objects. (#94327, @rikatz) [SIG CLI and Network]
• New default scheduling plugins order reduces scheduling and preemption latency when taints and node affinity are used (#95539, @soulxu) [SIG Scheduling]
• SCTP support in API objects (Pod, Service, NetworkPolicy) is now GA. Note that this has no effect on whether SCTP is enabled on nodes at the kernel level, and note that some cloud platforms and network plugins do not support SCTP traffic. (#95566, @danwinship) [SIG Apps and Network]
• Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extention point. (#93534, @everpeace) [SIG Scheduling and Testing]
• SelectorSpreadPriority maps to PodTopologySpread plugin when DefaultPodTopologySpread feature is enabled (#95448, @alculquicondor) [SIG Scheduling]
• SetHostnameAsFQDN has been graduated to Beta and therefore it is enabled by default. (#95267, @javidiaz) [SIG Node]

Bug or Regression

• An issues preventing volume expand controller to annotate the PVC with volume.kubernetes.io/storage-resizer when the PVC StorageClass is already updated to the out-of-tree provisioner is now fixed. (#94489, @ialidzhikov) [SIG API Machinery, Apps and Storage]
• Change the mount way from systemd to normal mount except ceph and glusterfs intree-volume. (#94916, @smileusd) [SIG Apps, Cloud Provider, Network, Node, Storage and Testing]
• Fix azure disk attach failure for disk size bigger than 4TB (#95463, @andyzhangx) [SIG Cloud Provider]
• Fix azure disk data loss issue on Windows when unmount disk (#95456, @andyzhangx) [SIG Cloud Provider and Storage]
• Fix verb & scope reporting for kube-apiserver metrics (LIST reported instead of GET) (#95562, @wojtek-t) [SIG API Machinery and Testing]
• Fix vsphere detach failure for static PVs (#95447, @gnufied) [SIG Cloud Provider and Storage]
• Fix: smb valid path error (#95583, @andyzhangx) [SIG Storage]
• Fixed a bug causing incorrect formatting of kubectl describe ingress. (#94985, @howardjohn) [SIG CLI and Network]
• Fixed a bug in client-go where new clients with customized Dial, Proxy, GetCert config may get stale HTTP transports. (#95427, @roycaihw) [SIG API Machinery]
• Fixes high CPU usage in kubectl drain (#95260, @amandahla) [SIG CLI]
• Support the node label node.kubernetes.io/exclude-from-external-load-balancers (#95542, @nilo19) [SIG Cloud Provider]

Other (Cleanup or Flake)

• Fix func name NewCreateCreateDeploymentOptions (#91931, @lixiaobing1) [SIG CLI]
• Kubeadm: update the default pause image version to 1.4.0 on Windows. With this update the image supports Windows versions 1809 (2019LTS), 1903, 1909, 2004 (#95419, @jsturtevant) [SIG Cluster Lifecycle and Windows]
• Upgrade snapshot controller to 3.0.0 (#95412, @saikat-royc) [SIG Cloud Provider]
• Remove the dependency of csi-translation-lib module on apiserver/cloud-provider/controller-manager (#95543, @wawa0210) [SIG Release]
• Scheduler framework interface moved from pkg/scheduler/framework/v1alpha to pkg/scheduler/framework (#95069, @farah) [SIG Scheduling, Storage and Testing]
• UDP and SCTP protocols can left stale connections that need to be cleared to avoid services disruption, but they can cause problems that are hard to debug. Kubernetes components using a loglevel greater or equal than 4 will log the conntrack operations and its output, to show the entries that were deleted. (#95694, @aojea) [SIG Network]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.20.0-alpha.2

Downloads for v1.20.0-alpha.2

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0-alpha.1

Changes by Kind

Deprecation

• Action-required: kubeadm: graduate the "kubeadm alpha certs" command to a parent command "kubeadm certs". The command "kubeadm alpha certs" is deprecated and will be removed in a future release. Please migrate. (#94938, @yagonobre) [SIG Cluster Lifecycle]
• Action-required: kubeadm: remove the deprecated feature --experimental-kustomize from kubeadm commands. The feature was replaced with --experimental-patches in 1.19. To migrate see the --help description for the --experimental-patches flag. (#94871, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: deprecate self-hosting support. The experimental command "kubeadm alpha self-hosting" is now deprecated and will be removed in a future release. (#95125, @neolit123) [SIG Cluster Lifecycle]
• Removes deprecated scheduler metrics DeprecatedSchedulingDuration, DeprecatedSchedulingAlgorithmPredicateEvaluationSecondsDuration, DeprecatedSchedulingAlgorithmPriorityEvaluationSecondsDuration (#94884, @arghya88) [SIG Instrumentation and Scheduling]
• Scheduler alpha metrics binding_duration_seconds and scheduling_algorithm_preemption_evaluation_seconds are deprecated, Both of those metrics are now covered as part of framework_extension_point_duration_seconds, the former as a PostFilter the latter and a Bind plugin. The plan is to remove both in 1.21 (#95001, @arghya88) [SIG Instrumentation and Scheduling]

API Change

• GPU metrics provided by kubelet are now disabled by default (#95184, @RenaudWasTaken) [SIG Node]
• New parameter defaultingType for PodTopologySpread plugin allows to use k8s defined or user provided default constraints (#95048, @alculquicondor) [SIG Scheduling]
• Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. (#93901, @jpbetz) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
• Status of v1beta1 CRDs without "preserveUnknownFields:false" will show violation "spec.preserveUnknownFields: Invalid value: true: must be false" (#93078, @vareti) [SIG API Machinery]

Feature

• Added get-users and delete-user to the kubectl config subcommand (#89840, @eddiezane) [SIG CLI]

• Added counter metric "apiserver_request_self" to count API server self-requests with labels for verb, resource, and subresource. (#94288, @LogicalShark) [SIG API Machinery, Auth, Instrumentation and Scheduling]

• Added new k8s.io/component-helpers repository providing shared helper code for (core) components. (#92507, @ingvagabund) [SIG Apps, Node, Release and Scheduling]

• Adds create ingress command to kubectl (#78153, @amimof) [SIG CLI and Network]

• Allow configuring AWS LoadBalancer health check protocol via service annotations (#94546, @kishorj) [SIG Cloud Provider]

• Azure: Support multiple services sharing one IP address (#94991, @nilo19) [SIG Cloud Provider]

• Ephemeral containers now apply the same API defaults as initContainers and containers (#94896, @wawa0210) [SIG Apps and CLI]

• In dual-stack bare-metal clusters, you can now pass dual-stack IPs to kubelet --node-ip. eg: kubelet --node-ip 10.1.0.5,fd01::0005. This is not yet supported for non-bare-metal clusters.

  In dual-stack clusters where nodes have dual-stack addresses, hostNetwork pods will now get dual-stack PodIPs. (#95239, @danwinship) [SIG Network and Node]

• Introduces a new GCE specific cluster creation variable KUBE_PROXY_DISABLE. When set to true, this will skip over the creation of kube-proxy (whether the daemonset or static pod). This can be used to control the lifecycle of kube-proxy separately from the lifecycle of the nodes. (#91977, @varunmar) [SIG Cloud Provider]

• Kubeadm: do not throw errors if the current system time is outside of the NotBefore and NotAfter bounds of a loaded certificate. Print warnings instead. (#94504, @neolit123) [SIG Cluster Lifecycle]

• Kubeadm: make the command "kubeadm alpha kubeconfig user" accept a "--config" flag and remove the following flags:

  • apiserver-advertise-address / apiserver-bind-port: use either localAPIEndpoint from InitConfiguration or controlPlaneEndpoint from ClusterConfiguration.
  • cluster-name: use clusterName from ClusterConfiguration
  • cert-dir: use certificatesDir from ClusterConfiguration (#94879, @knight42) [SIG Cluster Lifecycle]

• Kubectl rollout history sts/sts-name --revision=some-revision will start showing the detailed view of the sts on that specified revision (#86506, @dineshba) [SIG CLI]

• Scheduling Framework: expose Run[Pre]ScorePlugins functions to PreemptionHandle which can be used in PostFilter extention point. (#93534, @everpeace) [SIG Scheduling and Testing]

• Send gce node startup scripts logs to console and journal (#95311, @karan) [SIG Cloud Provider and Node]

• Support kubectl delete orphan/foreground/background options (#93384, @zhouya0) [SIG CLI and Testing]

Bug or Regression

• Change the mount way from systemd to normal mount except ceph and glusterfs intree-volume. (#94916, @smileusd) [SIG Apps, Cloud Provider, Network, Node, Storage and Testing]
• Cloud node controller: handle empty providerID from getProviderID (#95342, @nicolehanjing) [SIG Cloud Provider]
• Fix a bug where the endpoint slice controller was not mirroring the parent service labels to its corresponding endpoint slices (#94443, @aojea) [SIG Apps and Network]
• Fix azure disk attach failure for disk size bigger than 4TB (#95463, @andyzhangx) [SIG Cloud Provider]
• Fix azure disk data loss issue on Windows when unmount disk (#95456, @andyzhangx) [SIG Cloud Provider and Storage]
• Fix detach azure disk issue when vm not exist (#95177, @andyzhangx) [SIG Cloud Provider]
• Fix network_programming_latency metric reporting for Endpoints/EndpointSlice deletions, where we don't have correct timestamp (#95363, @wojtek-t) [SIG Network and Scalability]
• Fix scheduler cache snapshot when a Node is deleted before its Pods (#95130, @alculquicondor) [SIG Scheduling]
• Fix vsphere detach failure for static PVs (#95447, @gnufied) [SIG Cloud Provider and Storage]
• Fixed a bug that prevents the use of ephemeral containers in the presence of a validating admission webhook. (#94685, @verb) [SIG Node and Testing]
• Gracefully delete nodes when their parent scale set went missing (#95289, @bpineau) [SIG Cloud Provider]
• In dual-stack clusters, kubelet will now set up both IPv4 and IPv6 iptables rules, which may fix some problems, eg with HostPorts. (#94474, @danwinship) [SIG Network and Node]
• Kubeadm: for Docker as the container runtime, make the "kubeadm reset" command stop containers before removing them (#94586, @BedivereZero) [SIG Cluster Lifecycle]
• Kubeadm: warn but do not error out on missing "ca.key" files for root CA, front-proxy CA and etcd CA, during "kubeadm join --control-plane" if the user has provided all certificates, keys and kubeconfig files which require signing with the given CA keys. (#94988, @neolit123) [SIG Cluster Lifecycle]
• Port mapping allows to map the same containerPort to multiple hostPort without naming the mapping explicitly. (#94494, @SergeyKanzhelev) [SIG Network and Node]
• Warn instead of fail when creating Roles and ClusterRoles with custom verbs via kubectl (#92492, @eddiezane) [SIG CLI]

Other (Cleanup or Flake)

• Added fine grained debugging to the intra-pod conformance test for helping easily resolve networking issues for nodes that might be unhealthy when running conformance or sonobuoy tests. (#93837, @jayunit100) [SIG Network and Testing]
• AdmissionReview objects sent for the creation of Namespace API objects now populate the namespace attribute consistently (previously the namespace attribute was empty for Namespace creation via POST requests, and populated for Namespace creation via server-side-apply PATCH requests) (#95012, @nodo) [SIG API Machinery and Testing]
• Client-go header logging (at verbosity levels >= 9) now masks Authorization header contents (#95316, @sfowl) [SIG API Machinery]
• Enhance log information of verifyRunAsNonRoot, add pod, container information (#94911, @wawa0210) [SIG Node]
• Errors from staticcheck:
  vendor/k8s.io/client-go/discovery/cached/memory/memcache_test.go:94:2: this value of g is never used (SA4006) (#95098, @phunziker) [SIG API Machinery]
• Kubeadm: update the default pause image version to 1.4.0 on Windows. With this update the image supports Windows versions 1809 (2019LTS), 1903, 1909, 2004 (#95419, @jsturtevant) [SIG Cluster Lifecycle and Windows]
• Masks ceph RBD adminSecrets in logs when logLevel >= 4 (#95245, @sfowl) [SIG Storage]
• Upgrade snapshot controller to 3.0.0 (#95412, @saikat-royc) [SIG Cloud Provider]
• Remove offensive words from kubectl cluster-info command (#95202, @rikatz) [SIG Architecture, CLI and Testing]
• The following new metrics are available.
  • network_plugin_operations_total
  • network_plugin_operations_errors_total (#93066, @AnishShah) [SIG Instrumentation, Network and Node]
• Vsphere: improve logging message on node cache refresh event (#95236, @andrewsykim) [SIG Cloud Provider]
• kubectl api-resources now prints the API version (as 'API group/version', same as output of kubectl api-versions). The column APIGROUP is now APIVERSION (#95253, @sallyom) [SIG CLI]

Dependencies

Added

• github.com/jmespath/go-jmespath/internal/testify: v1.5.1

Changed

• github.com/aws/aws-sdk-go: v1.28.2 → v1.35.5
• github.com/jmespath/go-jmespath: c2b33e8 → v0.4.0
• k8s.io/kube-openapi: 6aeccd4 → 8b50664
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.9 → v0.0.12
• sigs.k8s.io/structured-merge-diff/v4: v4.0.1 → b3cf1e8

Removed

Nothing has changed.

v1.20.0-alpha.1

Downloads for v1.20.0-alpha.1

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0-alpha.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Azure blob disk feature(kind: Shared, Dedicated) has been deprecated, you should use kind: Managed in kubernetes.io/azure-disk storage class. (#92905, @andyzhangx) [SIG Cloud Provider and Storage]
• CVE-2020-8559 (Medium): Privilege escalation from compromised node to cluster. See https://github.com/kubernetes/kubernetes/issues/92914 for more details. The API Server will no longer proxy non-101 responses for upgrade requests. This could break proxied backends (such as an extension API server) that respond to upgrade requests with a non-101 response code. (#92941, @tallclair) [SIG API Machinery]

Changes by Kind

Deprecation

• Kube-apiserver: the componentstatus API is deprecated. This API provided status of etcd, kube-scheduler, and kube-controller-manager components, but only worked when those components were local to the API server, and when kube-scheduler and kube-controller-manager exposed unsecured health endpoints. Instead of this API, etcd health is included in the kube-apiserver health check and kube-scheduler/kube-controller-manager health checks can be made directly against those components' health endpoints. (#93570, @liggitt) [SIG API Machinery, Apps and Cluster Lifecycle]
• Kubeadm: deprecate the "kubeadm alpha kubelet config enable-dynamic" command. To continue using the feature please defer to the guide for "Dynamic Kubelet Configuration" at k8s.io. (#92881, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: remove the deprecated "kubeadm alpha kubelet config enable-dynamic" command. To continue using the feature please defer to the guide for "Dynamic Kubelet Configuration" at k8s.io. This change also removes the parent command "kubeadm alpha kubelet" as there are no more sub-commands under it for the time being. (#94668, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: remove the deprecated --kubelet-config flag for the command "kubeadm upgrade node" (#94869, @neolit123) [SIG Cluster Lifecycle]
• Kubelet's deprecated endpoint metrics/resource/v1alpha1 has been removed, please adopt to metrics/resource. (#94272, @RainbowMango) [SIG Instrumentation and Node]
• The v1alpha1 PodPreset API and admission plugin has been removed with no built-in replacement. Admission webhooks can be used to modify pods on creation. (#94090, @deads2k) [SIG API Machinery, Apps, CLI, Cloud Provider, Scalability and Testing]

API Change

• A new nofuzz go build tag now disables gofuzz support. Release binaries enable this. (#92491, @BenTheElder) [SIG API Machinery]
• A new alpha-level field, SupportsFsGroup, has been introduced for CSIDrivers to allow them to specify whether they support volume ownership and permission modifications. The CSIVolumeSupportFSGroup feature gate must be enabled to allow this field to be used. (#92001, @huffmanca) [SIG API Machinery, CLI and Storage]
• Added pod version skew strategy for seccomp profile to synchronize the deprecated annotations with the new API Server fields. Please see the corresponding section in the KEP for more detailed explanations. (#91408, @saschagrunert) [SIG Apps, Auth, CLI and Node]
• Adds the ability to disable Accelerator/GPU metrics collected by Kubelet (#91930, @RenaudWasTaken) [SIG Node]
• Custom Endpoints are now mirrored to EndpointSlices by a new EndpointSliceMirroring controller. (#91637, @robscott) [SIG API Machinery, Apps, Auth, Cloud Provider, Instrumentation, Network and Testing]
• External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ (#92632, @RenaudWasTaken) [SIG Node and Testing]
• Fix conversions for custom metrics. (#94481, @wojtek-t) [SIG API Machinery and Instrumentation]
• Generic ephemeral volumes, a new alpha feature under the GenericEphemeralVolume feature gate, provide a more flexible alternative to EmptyDir volumes: as with EmptyDir, volumes are created and deleted for each pod automatically by Kubernetes. But because the normal provisioning process is used (PersistentVolumeClaim), storage can be provided by third-party storage vendors and all of the usual volume features work. Volumes don't need to be empt; for example, restoring from snapshot is supported. (#92784, @pohly) [SIG API Machinery, Apps, Auth, CLI, Instrumentation, Node, Scheduling, Storage and Testing]
• Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting --volume-host-allow-local-loopback=false, or from contacting specific CIDR ranges by setting --volume-host-cidr-denylist (for example, --volume-host-cidr-denylist=127.0.0.1/28,feed::/16) (#91785, @mattcary) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
• Kubernetes is now built with golang 1.15.0-rc.1.
  • The deprecated, legacy behavior of treating the CommonName field on X.509 serving certificates as a host name when no Subject Alternative Names are present is now disabled by default. It can be temporarily re-enabled by adding the value x509ignoreCN=0 to the GODEBUG environment variable. (#93264, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Storage and Testing]
• Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock (#94603, @wojtek-t) [SIG API Machinery, Apps, Cloud Provider and Scheduling]
• Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly (#94182, @mattfenwick) [SIG API Machinery, Apps, Auth, Network and Node]
• The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. (#91921, @mtaufen) [SIG Auth]
• The kube-controller-manager managed signers can now have distinct signing certificates and keys. See the help about --cluster-signing-[signer-name]-{cert,key}-file. --cluster-signing-{cert,key}-file is still the default. (#90822, @deads2k) [SIG API Machinery, Apps and Auth]
• When creating a networking.k8s.io/v1 Ingress API object, spec.tls[*].secretName values are required to pass validation rules for Secret API object names. (#93929, @liggitt) [SIG Network]
• WinOverlay feature graduated to beta (#94807, @ksubrmnn) [SIG Windows]

Feature

• ACTION REQUIRED : In CoreDNS v1.7.0, metrics names have been changed which will be backward incompatible with existing reporting formulas that use the old metrics' names. Adjust your formulas to the new names before upgrading.

  Kubeadm now includes CoreDNS version v1.7.0. Some of the major changes include:

  • Fixed a bug that could cause CoreDNS to stop updating service records.
  • Fixed a bug in the forward plugin where only the first upstream server is always selected no matter which policy is set.
  • Remove already deprecated options resyncperiod and upstream in the Kubernetes plugin.
  • Includes Prometheus metrics name changes (to bring them in line with standard Prometheus metrics naming convention). They will be backward incompatible with existing reporting formulas that use the old metrics' names.
  • The federation plugin (allows for v1 Kubernetes federation) has been removed. More details are available in https://coredns.io/2020/06/15/coredns-1.7.0-release/ (#92651, @rajansandeep) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation]

• Add metrics for azure service operations (route and loadbalancer). (#94124, @nilo19) [SIG Cloud Provider and Instrumentation]

• Add network rule support in Azure account creation (#94239, @andyzhangx) [SIG Cloud Provider]

• Add tags support for Azure File Driver (#92825, @ZeroMagic) [SIG Cloud Provider and Storage]

• Added kube-apiserver metrics: apiserver_current_inflight_request_measures and, when API Priority and Fairness is enable, windowed_request_stats. (#91177, @MikeSpreitzer) [SIG API Machinery, Instrumentation and Testing]

• Audit events for API requests to deprecated API versions now include a "k8s.io/deprecated": "true" audit annotation. If a target removal release is identified, the audit event includes a "k8s.io/removal-release": "<majorVersion>.<minorVersion>" audit annotation as well. (#92842, @liggitt) [SIG API Machinery and Instrumentation]

• Cloud node-controller use InstancesV2 (#91319, @gongguan) [SIG Apps, Cloud Provider, Scalability and Storage]

• Kubeadm: Add a preflight check that the control-plane node has at least 1700MB of RAM (#93275, @xlgao-zju) [SIG Cluster Lifecycle]

• Kubeadm: add the "--cluster-name" flag to the "kubeadm alpha kubeconfig user" to allow configuring the cluster name in the generated kubeconfig file (#93992, @prabhu43) [SIG Cluster Lifecycle]

• Kubeadm: add the "--kubeconfig" flag to the "kubeadm init phase upload-certs" command to allow users to pass a custom location for a kubeconfig file. (#94765, @zhanw15) [SIG Cluster Lifecycle]

• Kubeadm: deprecate the "--csr-only" and "--csr-dir" flags of the "kubeadm init phase certs" subcommands. Please use "kubeadm alpha certs generate-csr" instead. This new command allows you to generate new private keys and certificate signing requests for all the control-plane components, so that the certificates can be signed by an external CA. (#92183, @wallrj) [SIG Cluster Lifecycle]

• Kubeadm: make etcd pod request 100m CPU, 100Mi memory and 100Mi ephemeral_storage by default (#94479, @knight42) [SIG Cluster Lifecycle]

• Kubemark now supports both real and hollow nodes in a single cluster. (#93201, @ellistarn) [SIG Scalability]

• Kubernetes is now built using go1.15.2

  • build: Update to k/repo-infra@v0.1.1 (supports go1.15.2)

  • build: Use go-runner:buster-v2.0.1 (built using go1.15.1)

  • bazel: Replace --features with Starlark build settings flag

  • hack/lib/util.sh: some bash cleanups

    • switched one spot to use kube::logging
    • make kube::util::find-binary return an error when it doesn't find anything so that hack scripts fail fast instead of with '' binary not found errors.
    • this required deleting some genfeddoc stuff. the binary no longer exists in k/k repo since we removed federation/, and I don't see it in https://github.com/kubernetes-sigs/kubefed/ either. I'm assuming that it's gone for good now.

  • bazel: output go_binary rule directly from go_binary_conditional_pure

    From: @mikedanese: Instead of aliasing. Aliases are annoying in a number of ways. This is specifically bugging me now because they make the action graph harder to analyze programmatically. By using aliases here, we would need to handle potentially aliased go_binary targets and dereference to the effective target.

    The comment references an issue with pure = select(...) which appears to be resolved considering this now builds.

  • make kube::util::find-binary not dependent on bazel-out/ structure

    Implement an aspect that outputs go_build_mode metadata for go binaries, and use that during binary selection. (#94449, @justaugustus) [SIG Architecture, CLI, Cluster Lifecycle, Node, Release and Testing]

• Only update Azure data disks when attach/detach (#94265, @andyzhangx) [SIG Cloud Provider]

• Promote SupportNodePidsLimit to GA to provide node to pod pid isolation Promote SupportPodPidsLimit to GA to provide ability to limit pids per pod (#94140, @derekwaynecarr) [SIG Node and Testing]

• Rename pod_preemption_metrics to preemption_metrics. (#93256, @ahg-g) [SIG Instrumentation and Scheduling]

• Server-side apply behavior has been regularized in the case where a field is removed from the applied configuration. Removed fields which have no other owners are deleted from the live object, or reset to their default value if they have one. Safe ownership transfers, such as the transfer of a replicas field from a user to an HPA without resetting to the default value are documented in Transferring Ownership (#92661, @jpbetz) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Testing]

• Set CSIMigrationvSphere feature gates to beta. Users should enable CSIMigration + CSIMigrationvSphere features and install the vSphere CSI Driver (https://github.com/kubernetes-sigs/vsphere-csi-driver) to move workload from the in-tree vSphere plugin "kubernetes.io/vsphere-volume" to vSphere CSI Driver.

  Requires: vSphere vCenter/ESXi Version: 7.0u1, HW Version: VM version 15 (#92816, @divyenpatel) [SIG Cloud Provider and Storage]

• Support [service.beta.kubernetes.io/azure-pip-ip-tags] annotations to allow customers to specify ip-tags to influence public-ip creation in Azure [Tag1=Value1, Tag2=Value2, etc.] (#94114, @MarcPow) [SIG Cloud Provider]

• Support a smooth upgrade from client-side apply to server-side apply without conflicts, as well as support the corresponding downgrade. (#90187, @julianvmodesto) [SIG API Machinery and Testing]

• Trace output in apiserver logs is more organized and comprehensive. Traces are nested, and for all non-long running request endpoints, the entire filter chain is instrumented (e.g. authentication check is included). (#88936, @jpbetz) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Scheduling]

• kubectl alpha debug now supports debugging nodes by creating a debugging container running in the node's host namespaces. (#92310, @verb) [SIG CLI]

Documentation

• Kubelet: remove alpha warnings for CNI flags. (#94508, @andrewsykim) [SIG Network and Node]

Failing Test

• Kube-proxy iptables min-sync-period defaults to 1 sec. Previously, it was 0. (#92836, @aojea) [SIG Network]

Bug or Regression

• A panic in the apiserver caused by the informer-sync health checker is now fixed. (#93600, @ialidzhikov) [SIG API Machinery]

• Add kubectl wait --ignore-not-found flag (#90969, @zhouya0) [SIG CLI]

• Adding fix to the statefulset controller to wait for pvc deletion before creating pods. (#93457, @ymmt2005) [SIG Apps]

• Azure ARM client: don't segfault on empty response and http error (#94078, @bpineau) [SIG Cloud Provider]

• Azure: fix a bug that kube-controller-manager would panic if wrong Azure VMSS name is configured (#94306, @knight42) [SIG Cloud Provider]

• Azure: per VMSS VMSS VMs cache to prevent throttling on clusters having many attached VMSS (#93107, @bpineau) [SIG Cloud Provider]

• Both apiserver_request_duration_seconds metrics and RequestReceivedTimestamp field of an audit event take into account the time a request spends in the apiserver request filters. (#94903, @tkashem) [SIG API Machinery, Auth and Instrumentation]

• Build/lib/release: Explicitly use '--platform' in building server images

  When we switched to go-runner for building the apiserver, controller-manager, and scheduler server components, we no longer reference the individual architectures in the image names, specifically in the 'FROM' directive of the server image Dockerfiles.

  As a result, server images for non-amd64 images copy in the go-runner amd64 binary instead of the go-runner that matches that architecture.

  This commit explicitly sets the '--platform=linux/${arch}' to ensure we're pulling the correct go-runner arch from the manifest list.

  Before: FROM ${base_image}

  After: FROM --platform=linux/${arch} ${base_image} (#94552, @justaugustus) [SIG Release]

• CSIDriver object can be deployed during volume attachment. (#93710, @Jiawei0227) [SIG Apps, Node, Storage and Testing]

• CVE-2020-8557 (Medium): Node-local denial of service via container /etc/hosts file. See https://github.com/kubernetes/kubernetes/issues/93032 for more details. (#92916, @joelsmith) [SIG Node]

• Do not add nodes labeled with kubernetes.azure.com/managed=false to backend pool of load balancer. (#93034, @matthias50) [SIG Cloud Provider]

• Do not fail sorting empty elements. (#94666, @soltysh) [SIG CLI]

• Do not retry volume expansion if CSI driver returns FailedPrecondition error (#92986, @gnufied) [SIG Node and Storage]

• Dockershim security: pod sandbox now always run with no-new-privileges and runtime/default seccomp profile dockershim seccomp: custom profiles can now have smaller seccomp profiles when set at pod level (#90948, @pjbgf) [SIG Node]

• Dual-stack: make nodeipam compatible with existing single-stack clusters when dual-stack feature gate become enabled by default (#90439, @SataQiu) [SIG API Machinery]

• Endpoint controller requeues service after an endpoint deletion event occurs to confirm that deleted endpoints are undesired to mitigate the effects of an out of sync endpoint cache. (#93030, @swetharepakula) [SIG Apps and Network]

• EndpointSlice controllers now return immediately if they encounter an error creating, updating, or deleting resources. (#93908, @robscott) [SIG Apps and Network]

• EndpointSliceMirroring controller now copies labels from Endpoints to EndpointSlices. (#93442, @robscott) [SIG Apps and Network]

• EndpointSliceMirroring controller now mirrors Endpoints that do not have a Service associated with them. (#94171, @robscott) [SIG Apps, Network and Testing]

• Ensure backoff step is set to 1 for Azure armclient. (#94180, @feiskyer) [SIG Cloud Provider]

• Ensure getPrimaryInterfaceID not panic when network interfaces for Azure VMSS are null (#94355, @feiskyer) [SIG Cloud Provider]

• Eviction requests for pods that have a non-zero DeletionTimestamp will always succeed (#91342, @michaelgugino) [SIG Apps]

• Extended DSR loadbalancer feature in winkernel kube-proxy to HNS versions 9.3-9.max, 10.2+ (#93080, @elweb9858) [SIG Network]

• Fix HandleCrash order (#93108, @lixiaobing1) [SIG API Machinery]

• Fix a concurrent map writes error in kubelet (#93773, @knight42) [SIG Node]

• Fix a regression where kubeadm bails out with a fatal error when an optional version command line argument is supplied to the "kubeadm upgrade plan" command (#94421, @rosti) [SIG Cluster Lifecycle]

• Fix azure file migration panic (#94853, @andyzhangx) [SIG Cloud Provider]

• Fix bug where loadbalancer deletion gets stuck because of missing resource group #75198 (#93962, @phiphi282) [SIG Cloud Provider]

• Fix calling AttachDisk on a previously attached EBS volume (#93567, @gnufied) [SIG Cloud Provider, Storage and Testing]

• Fix detection of image filesystem, disk metrics for devicemapper, detection of OOM Kills on 5.0+ linux kernels. (#92919, @dashpole) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Node]

• Fix etcd_object_counts metric reported by kube-apiserver (#94773, @tkashem) [SIG API Machinery]

• Fix incorrectly reported verbs for kube-apiserver metrics for CRD objects (#93523, @wojtek-t) [SIG API Machinery and Instrumentation]

• Fix instance not found issues when an Azure Node is recreated in a short time (#93316, @feiskyer) [SIG Cloud Provider]

• Fix kube-apiserver /readyz to contain "informer-sync" check ensuring that internal informers are synced. (#93670, @wojtek-t) [SIG API Machinery and Testing]

• Fix kubectl SchemaError on CRDs with schema using x-kubernetes-preserve-unknown-fields on array types. (#94888, @sttts) [SIG API Machinery]

• Fix memory leak in EndpointSliceTracker for EndpointSliceMirroring controller. (#93441, @robscott) [SIG Apps and Network]

• Fix missing csi annotations on node during parallel csinode update. (#94389, @pacoxu) [SIG Storage]

• Fix the cloudprovider_azure_api_request_duration_seconds metric buckets to correctly capture the latency metrics. Previously, the majority of the calls would fall in the "+Inf" bucket. (#94873, @marwanad) [SIG Cloud Provider and Instrumentation]

• Fix: azure disk resize error if source does not exist (#93011, @andyzhangx) [SIG Cloud Provider]

• Fix: detach azure disk broken on Azure Stack (#94885, @andyzhangx) [SIG Cloud Provider]

• Fix: determine the correct ip config based on ip family (#93043, @aramase) [SIG Cloud Provider]

• Fix: initial delay in mounting azure disk & file (#93052, @andyzhangx) [SIG Cloud Provider and Storage]

• Fix: use sensitiveOptions on Windows mount (#94126, @andyzhangx) [SIG Cloud Provider and Storage]

• Fixed Ceph RBD volume expansion when no ceph.conf exists (#92027, @juliantaylor) [SIG Storage]

• Fixed a bug where improper storage and comparison of endpoints led to excessive API traffic from the endpoints controller (#94112, @damemi) [SIG Apps, Network and Testing]

• Fixed a bug whereby the allocation of reusable CPUs and devices was not being honored when the TopologyManager was enabled (#93189, @klueska) [SIG Node]

• Fixed a panic in kubectl debug when pod has multiple init containers or ephemeral containers (#94580, @kiyoshim55) [SIG CLI]

• Fixed a regression that sometimes prevented kubectl portforward to work when TCP and UDP services were configured on the same port (#94728, @amorenoz) [SIG CLI]

• Fixed bug in reflector that couldn't recover from "Too large resource version" errors with API servers 1.17.0-1.18.5 (#94316, @janeczku) [SIG API Machinery]

• Fixed bug where kubectl top pod output is not sorted when --sort-by and --containers flags are used together (#93692, @brianpursley) [SIG CLI]

• Fixed kubelet creating extra sandbox for pods with RestartPolicyOnFailure after all containers succeeded (#92614, @tnqn) [SIG Node and Testing]

• Fixed memory leak in endpointSliceTracker (#92838, @tnqn) [SIG Apps and Network]

• Fixed node data lost in kube-scheduler for clusters with imbalance on number of nodes across zones (#93355, @maelk) [SIG Scheduling]

• Fixed the EndpointSliceController to correctly create endpoints for IPv6-only pods.

  Fixed the EndpointController to allow IPv6 headless services, if the IPv6DualStack feature gate is enabled, by specifying ipFamily: IPv6 on the service. (This already worked with the EndpointSliceController.) (#91399, @danwinship) [SIG Apps and Network]

• Fixes a bug evicting pods after a taint with a limited tolerationSeconds toleration is removed from a node (#93722, @liggitt) [SIG Apps and Node]

• Fixes a bug where EndpointSlices would not be recreated after rapid Service recreation. (#94730, @robscott) [SIG Apps, Network and Testing]

• Fixes a race condition in kubelet pod handling (#94751, @auxten) [SIG Node]

• Fixes an issue proxying to ipv6 pods without specifying a port (#94834, @liggitt) [SIG API Machinery and Network]

• Fixes an issue that can result in namespaced custom resources being orphaned when their namespace is deleted, if the CRD defining the custom resource is removed concurrently with namespaces being deleted, then recreated. (#93790, @liggitt) [SIG API Machinery and Apps]

• Ignore root user check when windows pod starts (#92355, @wawa0210) [SIG Node and Windows]

• Increased maximum IOPS of AWS EBS io1 volumes to 64,000 (current AWS maximum). (#90014, @jacobmarble) [SIG Cloud Provider and Storage]

• K8s.io/apimachinery: runtime.DefaultUnstructuredConverter.FromUnstructured now handles converting integer fields to typed float values (#93250, @liggitt) [SIG API Machinery]

• Kube-aggregator certificates are dynamically loaded on change from disk (#92791, @p0lyn0mial) [SIG API Machinery]

• Kube-apiserver: fixed a bug returning inconsistent results from list requests which set a field or label selector and set a paging limit (#94002, @wojtek-t) [SIG API Machinery]

• Kube-apiserver: jsonpath expressions with consecutive recursive descent operators are no longer evaluated for custom resource printer columns (#93408, @joelsmith) [SIG API Machinery]

• Kube-proxy now trims extra spaces found in loadBalancerSourceRanges to match Service validation. (#94107, @robscott) [SIG Network]

• Kube-up now includes CoreDNS version v1.7.0. Some of the major changes include:

  • Fixed a bug that could cause CoreDNS to stop updating service records.
  • Fixed a bug in the forward plugin where only the first upstream server is always selected no matter which policy is set.
  • Remove already deprecated options resyncperiod and upstream in the Kubernetes plugin.
  • Includes Prometheus metrics name changes (to bring them in line with standard Prometheus metrics naming convention). They will be backward incompatible with existing reporting formulas that use the old metrics' names.
  • The federation plugin (allows for v1 Kubernetes federation) has been removed. More details are available in https://coredns.io/2020/06/15/coredns-1.7.0-release/ (#92718, @rajansandeep) [SIG Cloud Provider]

• Kubeadm now makes sure the etcd manifest is regenerated upon upgrade even when no etcd version change takes place (#94395, @rosti) [SIG Cluster Lifecycle]

• Kubeadm: avoid a panic when determining if the running version of CoreDNS is supported during upgrades (#94299, @zouyee) [SIG Cluster Lifecycle]

• Kubeadm: ensure "kubeadm reset" does not unmount the root "/var/lib/kubelet" directory if it is mounted by the user (#93702, @thtanaka) [SIG Cluster Lifecycle]

• Kubeadm: ensure the etcd data directory is created with 0700 permissions during control-plane init and join (#94102, @neolit123) [SIG Cluster Lifecycle]

• Kubeadm: fix the bug that kubeadm tries to call 'docker info' even if the CRI socket was for another CR (#94555, @SataQiu) [SIG Cluster Lifecycle]

• Kubeadm: make the kubeconfig files for the kube-controller-manager and kube-scheduler use the LocalAPIEndpoint instead of the ControlPlaneEndpoint. This makes kubeadm clusters more reseliant to version skew problems during immutable upgrades: https://kubernetes.io/docs/setup/release/version-skew-policy/#kube-controller-manager-kube-scheduler-and-cloud-controller-manager (#94398, @neolit123) [SIG Cluster Lifecycle]

• Kubeadm: relax the validation of kubeconfig server URLs. Allow the user to define custom kubeconfig server URLs without erroring out during validation of existing kubeconfig files (e.g. when using external CA mode). (#94816, @neolit123) [SIG Cluster Lifecycle]

• Kubeadm: remove duplicate DNS names and IP addresses from generated certificates (#92753, @QianChenglong) [SIG Cluster Lifecycle]

• Kubelet: assume that swap is disabled when /proc/swaps does not exist (#93931, @SataQiu) [SIG Node]

• Kubelet: fix race condition in pluginWatcher (#93622, @knight42) [SIG Node]

• Kuberuntime security: pod sandbox now always runs with runtime/default seccomp profile kuberuntime seccomp: custom profiles can now have smaller seccomp profiles when set at pod level (#90949, @pjbgf) [SIG Node]

• NONE (#71269, @DeliangFan) [SIG Node]

• New Azure instance types do now have correct max data disk count information. (#94340, @ialidzhikov) [SIG Cloud Provider and Storage]

• Pods with invalid Affinity/AntiAffinity LabelSelectors will now fail scheduling when these plugins are enabled (#93660, @damemi) [SIG Scheduling]

• Require feature flag CustomCPUCFSQuotaPeriod if setting a non-default cpuCFSQuotaPeriod in kubelet config. (#94687, @karan) [SIG Node]

• Reverted devicemanager for Windows node added in 1.19rc1. (#93263, @liggitt) [SIG Node and Windows]

• Scheduler bugfix: Scheduler doesn't lose pod information when nodes are quickly recreated. This could happen when nodes are restarted or quickly recreated reusing a nodename. (#93938, @alculquicondor) [SIG Scalability, Scheduling and Testing]

• The EndpointSlice controller now waits for EndpointSlice and Node caches to be synced before starting. (#94086, @robscott) [SIG Apps and Network]

• The /debug/api_priority_and_fairness/dump_requests path at an apiserver will no longer return a phantom line for each exempt priority level. (#93406, @MikeSpreitzer) [SIG API Machinery]

• The kubelet recognizes the --containerd-namespace flag to configure the namespace used by cadvisor. (#87054, @changyaowei) [SIG Node]

• The terminationGracePeriodSeconds from pod spec is respected for the mirror pod. (#92442, @tedyu) [SIG Node and Testing]

• Update Calico to v3.15.2 (#94241, @lmm) [SIG Cloud Provider]

• Update default etcd server version to 3.4.13 (#94287, @jingyih) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing]

• Updated Cluster Autoscaler to 1.19.0; (#93577, @vivekbagade) [SIG Autoscaling and Cloud Provider]

• Use NLB Subnet CIDRs instead of VPC CIDRs in Health Check SG Rules (#93515, @t0rr3sp3dr0) [SIG Cloud Provider]

• Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. (#92817, @kmala) [SIG Node]

• Very large patches may now be specified to kubectl patch with the --patch-file flag instead of including them directly on the command line. The --patch and --patch-file flags are mutually exclusive. (#93548, @smarterclayton) [SIG CLI]

• When creating a networking.k8s.io/v1 Ingress API object, spec.rules[*].http values are now validated consistently when the host field contains a wildcard. (#93954, @Miciah) [SIG CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing]

Other (Cleanup or Flake)

• --cache-dir sets cache directory for both http and discovery, defaults to $HOME/.kube/cache (#92910, @soltysh) [SIG API Machinery and CLI]
• Adds a bootstrapping ClusterRole, ClusterRoleBinding and group for /metrics, /livez/, /readyz/, & /healthz/- endpoints. (#93311, @logicalhan) [SIG API Machinery, Auth, Cloud Provider and Instrumentation]
• Base-images: Update to debian-iptables:buster-v1.3.0
  • Uses iptables 1.8.5
  • base-images: Update to debian-base:buster-v1.2.0
  • cluster/images/etcd: Build etcd:3.4.13-1 image
    • Uses debian-base:buster-v1.2.0 (#94733, @justaugustus) [SIG API Machinery, Release and Testing]
• Build: Update to debian-base@v2.1.2 and debian-iptables@v12.1.1 (#93667, @justaugustus) [SIG API Machinery, Release and Testing]
• Build: Update to debian-base@v2.1.3 and debian-iptables@v12.1.2 (#93916, @justaugustus) [SIG API Machinery, Release and Testing]
• Build: Update to go-runner:buster-v2.0.0 (#94167, @justaugustus) [SIG Release]
• Fix kubelet to properly log when a container is started. Before, sometimes the log said that a container is dead and was restarted when it was started for the first time. This only happened when using pods with initContainers and regular containers. (#91469, @rata) [SIG Node]
• Fix: license issue in blob disk feature (#92824, @andyzhangx) [SIG Cloud Provider]
• Fixes the flooding warning messages about setting volume ownership for configmap/secret volumes (#92878, @jvanz) [SIG Instrumentation, Node and Storage]
• Fixes the message about no auth for metrics in scheduler. (#94035, @zhouya0) [SIG Scheduling]
• Kube-up: defaults to limiting critical pods to the kube-system namespace to match behavior prior to 1.17 (#93121, @liggitt) [SIG Cloud Provider and Scheduling]
• Kubeadm: Separate argument key/value in log msg (#94016, @mrueg) [SIG Cluster Lifecycle]
• Kubeadm: remove support for the "ci/k8s-master" version label. This label has been removed in the Kubernetes CI release process and would no longer work in kubeadm. You can use the "ci/latest" version label instead. See kubernetes/test-infra#18517 (#93626, @vikkyomkar) [SIG Cluster Lifecycle]
• Kubeadm: remove the CoreDNS check for known image digests when applying the addon (#94506, @neolit123) [SIG Cluster Lifecycle]
• Kubernetes is now built with go1.15.0 (#93939, @justaugustus) [SIG Release and Testing]
• Kubernetes is now built with go1.15.0-rc.2 (#93827, @justaugustus) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing]
• Lock ExternalPolicyForExternalIP to default, this feature gate will be removed in 1.22. (#94581, @knabben) [SIG Network]
• Service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset is removed. All Standard load balancers will always enable tcp resets. (#94297, @MarcPow) [SIG Cloud Provider]
• Stop propagating SelfLink (deprecated in 1.16) in kube-apiserver (#94397, @wojtek-t) [SIG API Machinery and Testing]
• Strip unnecessary security contexts on Windows (#93475, @ravisantoshgudimetla) [SIG Node, Testing and Windows]
• To ensure the code be strong, add unit test for GetAddressAndDialer (#93180, @FreeZhang61) [SIG Node]
• Update CNI plugins to v0.8.7 (#94367, @justaugustus) [SIG Cloud Provider, Network, Node, Release and Testing]
• Update Golang to v1.14.5
  • Update repo-infra to 0.0.7 (to support go1.14.5 and go1.13.13)
    • Includes:
      • bazelbuild/bazel-toolchains@3.3.2
      • bazelbuild/rules_go@v0.22.7 (#93088, @justaugustus) [SIG Release and Testing]
• Update Golang to v1.14.6
  • Update repo-infra to 0.0.8 (to support go1.14.6 and go1.13.14)
    • Includes:
      • bazelbuild/bazel-toolchains@3.4.0
      • bazelbuild/rules_go@v0.22.8 (#93198, @justaugustus) [SIG Release and Testing]
• Update cri-tools to v1.19.0 (#94307, @xmudrii) [SIG Cloud Provider]
• Update default etcd server version to 3.4.9 (#92349, @jingyih) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing]
• Update etcd client side to v3.4.13 (#94259, @jingyih) [SIG API Machinery and Cloud Provider]
• kubectl get ingress now prefers the networking.k8s.io/v1 over extensions/v1beta1 (deprecated since v1.14). To explicitly request the deprecated version, use kubectl get ingress.v1beta1.extensions. (#94309, @liggitt) [SIG API Machinery and CLI]

Dependencies

Added

• github.com/Azure/go-autorest: v14.2.0+incompatible
• github.com/fvbommel/sortorder: v1.0.1
• github.com/yuin/goldmark: v1.1.27
• sigs.k8s.io/structured-merge-diff/v4: v4.0.1

Changed

• github.com/Azure/go-autorest/autorest/adal: v0.8.2 → v0.9.0
• github.com/Azure/go-autorest/autorest/date: v0.2.0 → v0.3.0
• github.com/Azure/go-autorest/autorest/mocks: v0.3.0 → v0.4.0
• github.com/Azure/go-autorest/autorest: v0.9.6 → v0.11.1
• github.com/Azure/go-autorest/logger: v0.1.0 → v0.2.0
• github.com/Azure/go-autorest/tracing: v0.5.0 → v0.6.0
• github.com/Microsoft/hcsshim: v0.8.9 → 5eafd15
• github.com/cilium/ebpf: 9f1617e → 1c8d4c9
• github.com/containerd/cgroups: bf292b2 → 0dbf7f0
• github.com/coredns/corefile-migration: v1.0.8 → v1.0.10
• github.com/evanphx/json-patch: e83c0a1 → v4.9.0+incompatible
• github.com/google/cadvisor: 8450c56 → v0.37.0
• github.com/json-iterator/go: v1.1.9 → v1.1.10
• github.com/opencontainers/go-digest: v1.0.0-rc1 → v1.0.0
• github.com/opencontainers/runc: 1b94395 → 819fcc6
• github.com/prometheus/client_golang: v1.6.0 → v1.7.1
• github.com/prometheus/common: v0.9.1 → v0.10.0
• github.com/prometheus/procfs: v0.0.11 → v0.1.3
• github.com/rubiojr/go-vhd: 0bfd3b3 → 02e2102
• github.com/storageos/go-api: 343b3ef → v2.2.0+incompatible
• github.com/urfave/cli: v1.22.1 → v1.22.2
• go.etcd.io/etcd: 54ba958 → dd1b699
• golang.org/x/crypto: bac4c82 → 75b2880
• golang.org/x/mod: v0.1.0 → v0.3.0
• golang.org/x/net: d3edc99 → ab34263
• golang.org/x/tools: c00d67e → c1934b7
• k8s.io/kube-openapi: 656914f → 6aeccd4
• k8s.io/system-validators: v1.1.2 → v1.2.0
• k8s.io/utils: 6e3d28b → d5654de

Removed

• github.com/godbus/dbus: ade71ed
• github.com/xlab/handysort: fb3537e
• sigs.k8s.io/structured-merge-diff/v3: v3.0.0
• vbom.ml/util: db5cfe1

2 - Kubernetes version and version skew support policy

This document describes the maximum version skew supported between various Kubernetes components. Specific cluster deployment tools may place additional restrictions on version skew.

Supported versions

Kubernetes versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology. For more information, see Kubernetes Release Versioning.

The Kubernetes project maintains release branches for the most recent three minor releases (1.24, 1.23, 1.22). Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support.

Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases, when required.

The Release Managers group owns this decision.

For more information, see the Kubernetes patch releases page.

Supported version skew

kube-apiserver

In highly-available (HA) clusters, the newest and oldest kube-apiserver instances must be within one minor version.

Example:

• newest kube-apiserver is at 1.20
• other kube-apiserver instances are supported at 1.20 and 1.19

kubelet

kubelet must not be newer than kube-apiserver, and may be up to two minor versions older.

Example:

• kube-apiserver is at 1.20
• kubelet is supported at 1.20, 1.19, and 1.18

Note: If version skew exists between kube-apiserver instances in an HA cluster, this narrows the allowed kubelet versions.

Example:

• kube-apiserver instances are at 1.20 and 1.19
• kubelet is supported at 1.19, and 1.18 (1.20 is not supported because that would be newer than the kube-apiserver instance at version 1.19)

kube-controller-manager, kube-scheduler, and cloud-controller-manager

kube-controller-manager, kube-scheduler, and cloud-controller-manager must not be newer than the kube-apiserver instances they communicate with. They are expected to match the kube-apiserver minor version, but may be up to one minor version older (to allow live upgrades).

Example:

• kube-apiserver is at 1.20
• kube-controller-manager, kube-scheduler, and cloud-controller-manager are supported at 1.20 and 1.19

Note: If version skew exists between kube-apiserver instances in an HA cluster, and these components can communicate with any kube-apiserver instance in the cluster (for example, via a load balancer), this narrows the allowed versions of these components.

Example:

• kube-apiserver instances are at 1.20 and 1.19
• kube-controller-manager, kube-scheduler, and cloud-controller-manager communicate with a load balancer that can route to any kube-apiserver instance
• kube-controller-manager, kube-scheduler, and cloud-controller-manager are supported at 1.19 (1.20 is not supported because that would be newer than the kube-apiserver instance at version 1.19)

kubectl

kubectl is supported within one minor version (older or newer) of kube-apiserver.

Example:

• kube-apiserver is at 1.20
• kubectl is supported at 1.21, 1.20, and 1.19

Note: If version skew exists between kube-apiserver instances in an HA cluster, this narrows the supported kubectl versions.

Example:

• kube-apiserver instances are at 1.20 and 1.19
• kubectl is supported at 1.20 and 1.19 (other versions would be more than one minor version skewed from one of the kube-apiserver components)

Supported component upgrade order

The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing cluster from version 1.19 to version 1.20.

kube-apiserver

Pre-requisites:

• In a single-instance cluster, the existing kube-apiserver instance is 1.19
• In an HA cluster, all kube-apiserver instances are at 1.19 or 1.20 (this ensures maximum skew of 1 minor version between the oldest and newest kube-apiserver instance)
• The kube-controller-manager, kube-scheduler, and cloud-controller-manager instances that communicate with this server are at version 1.19 (this ensures they are not newer than the existing API server version, and are within 1 minor version of the new API server version)
• kubelet instances on all nodes are at version 1.19 or 1.18 (this ensures they are not newer than the existing API server version, and are within 2 minor versions of the new API server version)
• Registered admission webhooks are able to handle the data the new kube-apiserver instance will send them:
  • ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects are updated to include any new versions of REST resources added in 1.20 (or use the matchPolicy: Equivalent option available in v1.15+)
  • The webhooks are able to handle any new versions of REST resources that will be sent to them, and any new fields added to existing versions in 1.20

Upgrade kube-apiserver to 1.20

Note: Project policies for API deprecation and API change guidelines require kube-apiserver to not skip minor versions when upgrading, even in single-instance clusters.

kube-controller-manager, kube-scheduler, and cloud-controller-manager

Pre-requisites:

• The kube-apiserver instances these components communicate with are at 1.20 (in HA clusters in which these control plane components can communicate with any kube-apiserver instance in the cluster, all kube-apiserver instances must be upgraded before upgrading these components)

Upgrade kube-controller-manager, kube-scheduler, and cloud-controller-manager to 1.20

kubelet

Pre-requisites:

• The kube-apiserver instances the kubelet communicates with are at 1.20

Optionally upgrade kubelet instances to 1.20 (or they can be left at 1.19 or 1.18)

Note: Before performing a minor version kubelet upgrade, drain pods from that node. In-place minor version kubelet upgrades are not supported.

Warning:

Running a cluster with kubelet instances that are persistently two minor versions behind kube-apiserver is not recommended:

• they must be upgraded within one minor version of kube-apiserver before the control plane can be upgraded
• it increases the likelihood of running kubelet versions older than the three maintained minor releases

kube-proxy

• kube-proxy must be the same minor version as kubelet on the node.
• kube-proxy must not be newer than kube-apiserver.
• kube-proxy must be at most two minor versions older than kube-apiserver.

Example:

If kube-proxy version is 1.18:

• kubelet version must be at the same minor version as 1.18.
• kube-apiserver version must be between 1.18 and 1.20, inclusive.