You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
IPVS: Disable graceful termination for UDP traffic to solve issues with high number of UDP connections (DNS / syslog in particular) (#77802, @lbernail)
Fix broken detection of non-root image user ID (#78261, @tallclair)
Active watches of custom resources now terminate properly if the CRD is modified. (#78029, @liggitt)
fix azure retry issue when return 2XX with error (#78298, @andyzhangx)
Fixes a bug where dry-run is not honored for pod/eviction sub-resource. (#76969, @apelisse)
Fixes bug in DaemonSetController causing it to stop processing some DaemonSets for 5 minutes after node removal. (#76060, @krzysztof-jastrzebski)
Check if container memory stats are available before accessing it (#77656, @yastij)
client-go and kubectl no longer write cached discovery files with world-accessible file permissions (#77874, @yuchengwu)
Fixed a bug in the apiserver storage that could cause just-added finalizers to be ignored on an immediately following delete request, leading to premature deletion. (#77619, @caesarxuchao)
If a pod has a running instance, the stats of its previously terminated instances will not show up in the kubelet summary stats any more for CRI runtimes like containerd and cri-o. (#77426, @Random-Liu)
This keeps the behavior consistent with Docker integration, and fixes an issue that some container Prometheus metrics don't work when there are summary stats for multiple instances of the same pod.
Kubelet: add usageNanoCores from CRI stats provider (#73659, @feiskyer)
fix detach azure disk back off issue which has too big lock in failure retry condition (#76573, @andyzhangx)
Increase Azure default maximumLoadBalancerRuleCount to 250. (#72621, @feiskyer)
fix race condition issue for smb mount on windows (#75371, @andyzhangx)
Ensure the backend pools are set correctly for Azure SLB with multiple backend pools (e.g. outbound rules) (#76691, @feiskyer)
[metrics-server addon] Restore connecting to nodes via IP addresses (#76819, @serathius)
Fixes a NPD bug on GCI, so that it disables glog writing to files for log-counter (#76211, @wangzhen127)
Fixed parsing of fsType in AWS StorageClass parameters (#75943, @jsafrane)
[stackdriver addon] Bump prometheus-to-sd to v0.5.0 to pick up security fixes. (#75362, @serathius)
[fluentd-gcp addon] Bump fluentd-gcp-scaler to v0.5.1 to pick up security fixes.
[fluentd-gcp addon] Bump event-exporter to v0.2.4 to pick up security fixes.
[fluentd-gcp addon] Bump prometheus-to-sd to v0.5.0 to pick up security fixes.
[metatada-proxy addon] Bump prometheus-to-sd v0.5.0 to pick up security fixes.
Node-Problem-Detector configuration is now decoupled from the Kubernetes release on GKE/GCE. (#73288, @wangzhen127)
[IPVS] Allow for transparent kube-proxy restarts (#75283, @lbernail)
[IPVS] Introduces flag ipvs-strict-arp to configure stricter ARP sysctls, defaulting to false to preserve existing behaviors. This was enabled by default in 1.13.0, which impacted a few CNI plugins. (#75295, @lbernail)
Fix AAD support for Azure sovereign cloud in kubectl (#72143, @karataliu)
Delay CSI client initialization to make reconstruction of CSI volume possible because clients may not be available on kubelet restart. (#74652, @cofyc)
Fixes an issue with missing apiVersion/kind in object data sent to admission webhooks (#74448, @liggitt)
Because some plugins mount volume on sub-directory of volume path, we need to distinguish between volume path and mount path to fix issue in reconstruction. (#74653, @cofyc)
Allow disable outbound SNAT when Azure standard load balancer is used together with outbound rules. (#75282, @feiskyer)
Ensure Azure load balancer cleaned up on 404 or 403 when deleting LoadBalancer services. (#75256, @feiskyer)
Fix kubelet start failure issue on Azure Stack due to InstanceMetadata setting (#74936, @rjaini)
While this is a backwards-incompatible change, it would have been impossible to setup reliable monitoring around these metrics since the labels were not stable.
scheduler: use incremental scheduling cycle in PriorityQueue to put all in-flight unschedulable pods back to active queue if we received move request (#73309, @cofyc)
Add metrics-port to kube-proxy cmd flags. (#72682, @whypro)
kube-apiserver: a request body of a CREATE/UPDATE/PATCH/DELETE resource operation larger than 100 MB will return a 413 "request entity too large" error. (#73805, @caesarxuchao)
Custom apiservers built with the latest apiserver library will have the 100MB limit on the body of resource requests as well. The limit can be altered via ServerRunOptions.MaxRequestBodyBytes.
The body size limit does not apply to subresources like pods/proxy that proxy request content to another server.
The apiserver, including both the kube-apiserver and apiservers built with the generic apiserver library, will now return 413 RequestEntityTooLarge error if a json patch contains more than 10,000 operations. (#74000, @caesarxuchao)
Fix watch to not send the same set of events multiple times causing watcher to go back in time (#73845, @wojtek-t)
fixes an error processing watch events when running skewed apiservers (#73482, @liggitt)
MAC Address filter has been fixed in vSphere Cloud Provider, it no longer ignores 00:1c:14 and 00:05:69 prefixes (#73721, @frapposelli)
add goroutine to move unschedulable pods to activeq if they are not retried for more than 1 minute (#72558, @denkensk)
A new TaintNodesByCondition admission plugin taints newly created Node objects as "not ready", to fix a race condition that could cause pods to be scheduled on new nodes before their taints were updated to accurately reflect their reported conditions. This admission plugin is enabled by default if the TaintNodesByCondition feature is enabled. (#73097, @bsalamat)
kubeadm: add back --cert-dir option for kubeadm init phase certs sa (#73239, @mattkelly)
Scale max-inflight limits together with master VM sizes. (#73268, @wojtek-t)
kubeadm: explicitly wait for etcd to have grown when joining a new control plane (#72984, @ereslibre)
Improve efficiency of preemption logic in clusters with many pending pods. (#72895, @bsalamat)
Fix AWS NLB security group updates where valid security group ports were incorrectly removed (#68422, @kellycampbell)
when updating a service or when node changes occur.
Allow for watching objects larger than 1MB given etcd accepts objects of size up to 1.5MB (#72053, @wojtek-t)
kubectl: fixed an issue with "too old resource version" errors continuously appearing when calling kubectl delete (#72825, @liggitt)
Fix scheduling starvation of pods in cluster with large number of unschedulable pods. (#72619, @everpeace)
Fixes spurious 0-length API responses. (#72856, @liggitt)
client-go: shortens refresh period for token files to 1 minute to ensure auto-rotated projected service account tokens are read frequently enough. (#72437, @liggitt)
Updates the kubernetes dashboard add-on to v1.10.1. Skipping dashboard login is no longer enabled by default. (#72495, @liggitt)
Fixes a bug in HPA controller so HPAs are always updated every resyncPeriod (15 seconds). (#72373, @krzysztof-jastrzebski)
Fix device mountable volume names in DSW to prevent races in device mountable plugin, e.g. local. (#71509, @cofyc)
change azure disk host cache to ReadOnly by default (#72229, @andyzhangx)
Fixes issue with cleaning up stale NFS subpath mounts (#71804, @msau42)
Fix a race condition in the scheduler preemption logic that could cause nominatedNodeName of a pod not to be considered in one or more scheduling cycles. (#72259, @bsalamat)
Fix race condition introduced by graceful termination which can lead to a deadlock in kube-proxy (#72361, @lbernail)
Support graceful termination with IPVS when deleting a service (#71895, @lbernail)
Fixes issue where subpath volume content was deleted during orphaned pod cleanup for Local volumes that are directories (and not mount points) on the root filesystem. (#72291, @msau42)
kube-proxy in IPVS mode will stop initiating connections to terminating pods for services with sessionAffinity set. (#71834, @lbernail)
fix race condition when attach azure disk in vmss (#71992, @andyzhangx)
Update to use go1.11.3 with fix for CVE-2018-16875 (#72035, @seemethere)
Fix a race condition in which kubeadm only waits for the kubelets kubeconfig file when it has performed the TLS bootstrap, but wasn't waiting for certificates to be present in the filesystem (#72030, @ereslibre)
kubeadm: fix a possible panic when joining a new control plane node in HA scenarios (#72123, @anitgandhi)
kubeadm: fix a bug when syncing etcd endpoints (#71945, @pytimer)
kube-scheduler: restores ability to run without authentication configuration lookup permissions (#71755, @liggitt)
client-go: restores behavior of populating the BearerToken field in rest.Config objects constructed from kubeconfig files containing tokenFile config, or from in-cluster configuration. An additional BearerTokenFile field is now populated to enable constructed clients to periodically refresh tokens. (#71713, @liggitt)
apply: fix detection of non-dry-run enabled servers (#71854, @apelisse)
Scheduler only activates unschedulable pods if node's scheduling related properties change. (#71551, @mlmhl)
Fixes pod deletion when cleaning old cronjobs (#71802, @soltysh)
fix issue: vm sku restriction policy does not work in azure disk attach/detach (#71941, @andyzhangx)
Include CRD for BGPConfigurations, needed for calico 2.x to 3.x upgrade. (#71868, @satyasm)
UDP connections now support graceful termination in IPVS mode (#71515, @lbernail)
kubeadm: use kubeconfig flag instead of kubeconfig-dir on init phase bootstrap-token (#71803, @yagonobre)
On GCI, NPD starts to monitor kubelet, docker, containerd crashlooping, read-only filesystem and corrupt docker overlay2 issues. (#71522, @wangzhen127)
Fixes an issue where Portworx volumes cannot be mounted if 9001 port is already in use on the host and users remap 9001 to another port. (#70392, @harsh-px)
Only use the first IP address got from instance metadata. This is because Azure CNI would set up a list of IP addresses in instance metadata, while only the first one is the Node's IP. (#71736, @feiskyer)
kube-controller-manager: fixed issue display help for the deprecated insecure --port flag (#71601, @liggitt)
CVE-2018-1002105, a critical security issue in the Kubernetes API Server, is resolved in v1.13.0 (and in v1.10.11, v1.11.5, and v1.12.3). We recommend all clusters running previous versions update to one of these releases immediately. See issue #71411 for details.
Urgent Upgrade Notes
(No, really, you MUST do this before you upgrade)
Before upgrading to Kubernetes 1.13, you must keep the following in mind:
kube-apiserver
The deprecated etcd2 storage backend has been removed. Before upgrading a kube-apiserver using --storage-backend=etcd2, etcd v2 data must be migrated to the v3 storage backend, and kube-apiserver invocations changed to use --storage-backend=etcd3. Please consult the installation procedure used to set up etcd for specific migration instructions. Backups prior to upgrade are always a good practice, but since the etcd2 to etcd3 migration is not reversible, an etcd backup prior to migration is essential.
The deprecated --etcd-quorum-read flag has been removed. Quorum reads are now always enabled when fetching data from etcd. Remove the --etcd-quorum-read flag from kube-apiserver invocations before upgrading.
kube-controller-manager
The deprecated --insecure-experimental-approve-all-kubelet-csrs-for-group flag has been removed.
kubelet
The deprecated --google-json-key flag has been removed. Remove the --google-json-key flag from kubelet invocations before upgrading. (#69354, @yujuhong)
DaemonSet pods now make use of scheduling features that require kubelets to be at 1.11 or above. Ensure all kubelets in the cluster are at 1.11 or above before upgrading kube-controller-manager to 1.13.
The schema for the alpha CSINodeInfo CRD has been split into spec and status fields, and new fields status.available and status.volumePluginMechanism added. Clusters using the previous alpha schema must delete and recreate the CRD using the new schema. (#70515, @davidz627)
kube-scheduler dropped support for configuration files with apiVersion componentconfig/v1alpha1. Ensure kube-scheduler is configured using command-line flags or a configuration file with apiVersion kubescheduler.config.k8s.io/v1alpha1 before upgrading to 1.13.
kubectl
The deprecated command run-container has been removed. Invocations should use kubectl run instead (#70728, @Pingan2017)
client-go releases will no longer have bootstrap (k8s.io/client-go/tools/bootstrap) related code. Any reference to it will break. Please redirect all references to k8s.io/bootstrap instead. (#67356, @yliaog)
Kubernetes cannot distinguish between GCE Zonal PDs and Regional PDs with the same name. To workaround this issue, precreate PDs with unique names. PDs that are dynamically provisioned do not encounter this issue. (#70716, @msau42)
Known Issues
If kubelet plugin registration for a driver fails, kubelet will not retry. The driver must delete and recreate the driver registration socket in order to force kubelet to attempt registration again. Restarting only the driver container may not be sufficient to trigger recreation of the socket, instead a pod restart may be required. (#71487)
In some cases, a Flex volume resize may leave a PVC with erroneous Resizing condition even after volume has been successfully expanded. Users may choose to delete the condition, but it is not required. (#71470)
The CSI driver-registrar external sidecar container v1.0.0-rc2 is known to take up to 1 minute to start in some cases. We expect this issue to be resolved in a future release of the sidecar container. For verification, please see the release notes of future releases of the external sidecar container. (#76)
When using IPV6-only, be sure to use proxy-mode=iptables as proxy-mode=ipvs is known to not work. (#68437)
Deprecations
kube-apiserver
The --service-account-api-audiences flag is deprecated in favor of --api-audiences. The old flag is accepted with a warning but will be removed in a future release. (#70105, @mikedanese)
The --experimental-encryption-provider-config flag is deprecated in favor of --encryption-provider-config. The old flag is accepted with a warning but will be removed in 1.14. (#71206, @stlaz)
As part of graduating the etcd encryption feature to beta, the configuration file referenced by --encryption-provider-config now uses kind: EncryptionConfiguration and apiVersion: apiserver.config.k8s.io/v1. Support for kind: EncryptionConfig and apiVersion: v1 is deprecated and will be removed in a future release. (#67383, @stlaz)
The --deserialization-cache-size flag is deprecated, and will be removed in a future release. The flag is inactive since the etcd2 storage backend was removed. (#69842, @liggitt)
The Node authorization mode no longer allows kubelets to delete their Node API objects (prior to 1.11, in rare circumstances related to cloudprovider node ID changes, kubelets would attempt to delete/recreate their Node object at startup) (#71021, @liggitt)
The built-in system:csi-external-provisioner and system:csi-external-attacher cluster roles are deprecated and will not be auto-created in a future release. CSI deployments should provide their own RBAC role definitions with required permissions. (#69868, @pohly)
The built-in system:aws-cloud-provider cluster role is deprecated and will not be auto-created in a future release. Deployments using the AWS cloud provider should grant required permissions to the aws-cloud-provider service account in the kube-system namespace as part of deployment. (#66635, @wgliang)
kubelet
Use of the beta plugin registration directory {kubelet_root_dir}/plugins/ for registration of external drivers via the kubelet plugin registration protocol is deprecated in favor of {kubelet_root_dir}/plugins_registry/. Support for the old directory is planned to be removed in v1.15. Device plugin and CSI storage drivers should switch to the new directory prior to v1.15. Only CSI storage drivers that support 0.x versions of the CSI API are allowed in the old directory. (#70494 by @RenaudWasTaken and #71314 by @saad-ali)
With the release of the CSI 1.0 API, support for CSI drivers using 0.3 and older releases of the CSI API is deprecated, and is planned to be removed in Kubernetes v1.15. CSI drivers should be updated to support the CSI 1.0 API, and deployed in the new kubelet plugin registration directory ({kubelet_root_dir}/plugins_registry/) once all nodes in the cluster are at 1.13 or higher (#71020 and #71314, both by @saad-ali)
Use of the --node-labels flag to set labels under the kubernetes.io/ and k8s.io/ prefix will be subject to restriction by the NodeRestriction admission plugin in future releases. See admission plugin documentation for allowed labels. (#68267, @liggitt)
kube-scheduler
The alpha critical pod annotation (scheduler.alpha.kubernetes.io/critical-pod) is deprecated. Pod priority should be used instead to mark pods as critical. (#70298, @bsalamat)
The following features are now GA, and the associated feature gates are deprecated and will be removed in a future release:
CSIPersistentVolume
GCERegionalPersistentDisk
KubeletPluginsWatcher
VolumeScheduling
kubeadm
The DynamicKubeletConfig feature gate is deprecated. The functionality is still accessible by using the kubeadm alpha kubelet enable-dynamic command.
The command kubeadm config print-defaults is deprecated in favor of kubeadm config print init-defaults and kubeadm config print join-defaults (#69617, @rosti)
support for the v1alpha3 configuration file format is deprecated and will be removed in 1.14. Use kubeadm config migrate to migrate v1alpha3 configuration files to v1beta1, which provides improvements in image repository management, addons configuration, and other areas. The documentation for v1beta1 can be found here: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1
The node.status.volumes.attached.devicePath field is deprecated for CSI volumes and will not be set in future releases (#71095, @msau42)
kubectl
The kubectl convert command is deprecated and will be removed in a future release (#70820, @seans3)
Support for passing unknown provider names to the E2E test binaries is deprecated and will be removed in a future release. Use --provider=skeleton (no ssh access) or --provider=local (local cluster with ssh) instead. (#70141, @pohly)
Major Themes
SIG API Machinery
For the 1.13 release, SIG API Machinery is happy to announce that the dry-run functionality is now beta.
In v1.13 we worked on tighter integrations of Kubernetes API objects with AWS services. These include three out-of-tree alpha feature releases:
Alpha for AWS ALB (Application Load Balancer) integration to Kubernetes Ingress resources.
Alpha for CSI specification 0.3 integration to AWS EBS (Elastic Block Store)
Alpha for the cloudprovider-aws cloud controller manager binary. Additionally we added aws-k8s-tester, deployer interface for kubetest, to the test-infra repository. This plugin allowed us to integrate Prow to the 3 subprojects defined above in order to provide CI signal for all 3 features. The CI signal is visible here under SIG-AWS.
For detailed release notes on the three alpha features from SIG AWS, please refer to the following Changelogs:
For 1.13 SIG Azure was focused on adding additional Azure Disk support for Ultra SSD, Standard SSD, and Premium Azure Files. Azure Availability Zones and cross resource group nodes were also moved from Alpha to Beta in 1.13.
SIG Big Data
During the 1.13 release cycle, SIG Big Data has been focused on community engagements relating to 3rd-party project integrations with Kubernetes. There have been no impacts on the 1.13 release.
SIG CLI
Over the course of 1.13 release SIG CLI mostly focused on stabilizing the items we’ve been working on over the past releases such as server-side printing and its support in kubectl, as well as finishing kubectl diff which is based on server-side dry-run feature. We’ve continued separating kubectl code to prepare for extraction out of main repository. Finally, thanks to the awesome support and feedback from community we’ve managed to promote the new plugin mechanism to Beta.
SIG Cloud Provider
For v1.13, SIG Cloud Provider has been focused on stabilizing the common APIs and interfaces consumed by cloud providers today. This involved auditing the cloud provider APIs for anything that should be deprecated as well as adding changes where necessary. In addition, SIG Cloud Provider has begun exploratory work around having a “cloud provider” e2e test suite which can be used to test common cloud provider functionalities with resources such as nodes and load balancers.
We are also continuing our long running effort to extract all the existing cloud providers that live in k8s.io/kubernetes into their own respective repos. Along with this migration, we are slowly transitioning users to use the cloud-controller-manager for any cloud provider features instead of the kube-controller-manager.
SIG Cluster Lifecycle
For 1.13 SIG Cluster Lifecycle is pleased to announce the long awaited promotion of kubeadm to stable GA, and the promotion of kubeadm’s configuration API to v1beta1.
In this release the SIG again focused on further improving the user experience on cluster creation and also fixing a number of bugs and other assorted improvements.
Some notable changes in kubeadm since Kubernetes 1.12:
kubeadm’s configuration API is now v1beta1. The new configuration format provides improvements in - image repository management, addons configuration, and other areas. We encourage v1alpha3 users to migrate to this configuration API using kubeadm config migrate, as v1alpha3 will be removed in 1.14. The documentation for v1beta1 can be found here: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1
kubeadm has graduated kubeadm alpha phase commands to kubeadm init phase. This means that the phases of creating a control-plane node are now tightly integrated as part of the init command. Alpha features, not yet ready for GA are still kept under kubeadm alpha and we appreciate feedback on them.
kubeadm init and kubeadm init phase now have a --image-repository flag, improving support for environments with limited access to official kubernetes repository.
The DynamicKubeletConfig and SelfHosting functionality was moved outside of kubeadm init and feature gates and is now exposed under kubeadm alpha.
Kubeadm init phase certs now support the --csr-only option, simplifying custom CA creation.
kubeadm join --experimental-control-plane now automatically adds a new etcd member for local etcd mode, further simplifying required tasks for HA clusters setup.
Improvements were made to kubeadm reset related to cleaning etcd and notifying the user about the state of iptables.
kubeadm commands now print warnings if input YAML documents contain unknown or duplicate fields.
kubeadm now properly recognizes Docker 18.09.0 and newer, but still treats 18.06 as the default supported version.
kubeadm now automatically sets the --pod-infra-container-image flag when starting the kubelet.
SIG IBM Cloud
The IBM Cloud SIG was focused on defining its charter and working towards moving its cloud provider code to an external repository with a goal to have this work done by the end of Kubernetes 1.14 release cycle. In the SIG meetings, we also made sure to share updates on the latest Kubernetes developments in the IBM Cloud like the availability of Kubernetes v1.12.2 in the IBM Cloud Kubernetes Service (IKS). The SIG updates were provided in the Kubernetes community weekly call and at the KubeCon China 2018.
SIG Multicluster
Moving Federation v2 from Alpha towards Beta has been the focus of our effort over the past quarter. To this end we engaged with end users, and successfully enlisted additional contributors from companies including IBM, Amadeus, Cisco and others. Federation v2 provides a suite of decoupled API’s and re-usable components for building multi-cluster control planes. We plan to start releasing Beta components in late 2018. In addition, more minor updates were made to our cluster-registry and multi-cluster ingress sub-projects.
SIG Network
For 1.13, the areas of focus were in IPv6, DNS improvements and some smaller items:
CoreDNS is now the default cluster DNS passing all of the scale/resource usage tests
Node-local DNS cache feature is available in Alpha. This feature deploys a lightweight DNS caching Daemonset that avoids the conntrack and converts queries from UDP to more reliable TCP.
PodReady++ feature now has kubectl CLI support.
Progress was made towards finalizing the IPv6 dual stack support KEP and support for topological routing of services.
SIG Node
SIG Node focused on stability and performance improvements in the 1.13 release. A new alpha feature is introduced to improve the mechanism that nodes heartbeat back to the control plane. The NodeLease feature results in the node using a Lease resource in the kube-node-lease namespace that is renewed periodically. The NodeStatus that was used previously to heartbeat back to the control plane is only updated when it changes. This reduces load on the control plane for large clusters. The Kubelet plugin registration mechanism, which enables automatic discovery of external plugins (including CSI and device plugins) has been promoted to stable in this release (introduced as alpha in 1.11 and promoted to beta in 1.12).
SIG Openstack
The major theme for the SIG OpenStack release is the work-in-progress for removing the in-tree provider. This work, being done in conjunction with SIG Cloud Provider, is focusing on moving internal APIs that the OpenStack (and other providers) depends upon to staging to guarantee API stability. This work also included abstracting the in-tree Cinder API and refactoring code to the external Cinder provider to remove additional Cinder volume provider code.
Additional work was also done to implement an OpenStack driver for the Cluster API effort lead by SIG Cluster Lifecycle. For the external Cloud-Provider-OpenStack code, the SIG largely focused on bug fixes and updates to match K8s 1.13 development.
SIG Scalability
SIG Scalability has mostly focused on stability and deflaking our tests, investing into framework for writing scalability tests (ClusterLoader v2) with a goal to migrate all tests to it by the end of 2018 and on the work towards extending definition of Kubernetes scalability by providing more/better user-friendly SLIs/SLOs.
SIG Scheduling
SIG Scheduling has mostly focused on stability in 1.13 and has postponed some of the major features to the next versions. There are still two notable changes: 1. TaintBasedEviction is moved to Beta and will be enabled by default. With this feature enabled, condition taints are automatically added to the nodes and pods can add tolerations for them if needed. 2. Pod critical annotation is deprecated. Pods should use pod priority instead of the annotation.
It is worth noting again that kube-scheduler will use apiVersion kubescheduler.config.k8s.io/v1alpha1 instead of componentconfig/v1alpha1 in its configuration files in 1.13.
SIG Service Catalog
The Service Plan Defaults feature is still under active development.
We continue to improve the UX for the svcat CLI, specifically filling in gaps for the new Namespaced Service Broker feature.
SIG Storage
Over the last year, SIG Storage has been focused on adding support for the Container Storage Interface (CSI) to Kubernetes. The specification recently moved to 1.0, and on the heels of this achievement, Kubernetes v1.13 moves CSI support for PersistentVolumes to GA.
With CSI the Kubernetes volume layer becomes truly extensible, allowing third party storage developers to write drivers making their storage systems available in Kubernetes without having to touch the core code.
CSI was first introduction as alpha in Kubernetes v1.9 and moved to beta in Kubernetes v1.10.
You can find a list of sample and production drivers in the CSI Documentation.
SIG Storage also moves support for Block Volumes to beta (introduced as alpha in v1.9) and support for Topology Aware Volume Scheduling to stable (introduced as alpha in v1.9 and promoted to beta in 1.10).
SIG UI
The migration to the newest version of Angular is still under active development as it is most important thing on the roadmap at the moment. We are getting closer to to the new release. We continue fixing bugs and adding other improvements.
SIG VMWare
Major focus for SIG VMware for this release is the work on moving internal APIs that the vSphere provider depends upon to staging to guarantee API stability. This work is being done in conjunction with SIG Cloud Provider and includes the creation of a brand new vsphere-csi plugin to replace the current volume functionalities in-tree.
Additional work was also done to implement a vSphere provider for the Cluster API effort lead by SIG Cluster Lifecycle. For the out-of-tree vSphere cloud provider, the SIG largely focused on bug fixes and updates to match K8s 1.13 development.
SIG Windows
SIG Windows focused on improving reliability for Windows and Kubernetes support
New Features
kubelet: When node lease feature is enabled, kubelet reports node status to api server only if there is some change or it didn't report over last report interval. (#69753, @wangzhen127)
vSphereVolume implements Raw Block Volume Support (#68761, @fanzhangio)
CRD supports multi-version Schema, Subresources and AdditionalPrintColumns (NOTE that CRDs created prior to 1.13 populated the top-level additionalPrinterColumns field by default. To apply an updated that changes to per-version additionalPrinterColumns, the top-level additionalPrinterColumns field must be explicitly set to null). (#70211, @roycaihw)
New addon in addon manager that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true. (#70193, @saad-ali)
Delegated authorization can now allow unrestricted access for system:masters like the main kube-apiserver (#70671, @deads2k)
Added dns capabilities for Windows CNI plugins: (#67435, @feiskyer)
kube-apiserver: --audit-webhook-version and --audit-log-version now default to audit.k8s.io/v1 if unspecified (#70476, @charrywanganthony)
kubeadm: timeoutForControlPlane is introduced as part of the API Server config, that controls the timeout for the wait for control plane to be up. Default value is 4 minutes. (#70480, @rosti)
--api-audiences now defaults to the --service-account-issuer if the issuer is provided but the API audience is not. (#70308, @mikedanese)
Added support for projected volume in describe function (#70158, @WanLinghao)
kubeadm now automatically creates a new stacked etcd member when joining a new control plane node (does not applies to external etcd) (#69486, @fabriziopandini)
Display the usage of ephemeral-storage when using kubectl describe node (#70268, @Pingan2017)
Added functionality to enable br_netfilter and ip_forward for debian packages to improve kubeadm support for CRI runtime besides Docker. (#70152, @ashwanikhemani)
Added regions ap-northeast-3 and eu-west-3 to the list of well known AWS regions. (#70252, @nckturner)
kubeadm: Implemented preflight check to ensure that number of CPUs (#70048, @bart0sh)
CoreDNS is now the default DNS server in kube-up deployments. (#69883, @chrisohaver)
Opt out of chowning and chmoding from kubectl cp. (#69573, @bjhaid)
Failed to provision volume with StorageClass "azurefile-premium": failed to create share andy-mg1121-dynamic-pvc-1a7b2813-d1b7-11e8-9e96-000d3a03e16b in account f7228f99bcde411e8ba4900: failed to create file share, err: storage: service returned error: StatusCode=400, ErrorCode=InvalidHeaderValue, ErrorMessage=The value for one of the HTTP headers is not in the correct format. (#69718, @andyzhangx)
TaintBasedEvictions feature is promoted to beta. (#69824, @Huang-Wei)
Dry-run is promoted to Beta and will be enabled by default. (#69644, @apelisse)
kubectl get priorityclass now prints value column by default. (#69431, @Huang-Wei)
Added a new container based image for running e2e tests (#69368, @dims)
The LC_ALL and LC_MESSAGES env vars can now be used to set desired locale for kubectl while keeping LANG unchanged. (#69500, @m1kola)
NodeLifecycleController: Now node lease renewal is treated as the heartbeat signal from the node, in addition to NodeStatus Update. (#69241, @wangzhen127)
It is now possible to use named ports in the kubectl port-forward command (#69477, @m1kola)
kubectl wait now supports condition value checks other than true using --for condition=available=false (#69295, @deads2k)
Updated defaultbackend image to 1.5. Users should concentrate on updating scripts to the new version. (#69120, @aledbf)
Bumped Dashboard version to v1.10.0 (#68450, @jeefy)
Added env variables to control CPU requests of kube-controller-manager and kube-scheduler. (#68823, @loburm)
PodSecurityPolicy objects now support a MayRunAs rule for fsGroup and supplementalGroups options. This allows specifying ranges of allowed GIDs for pods/containers without forcing a default GID the way MustRunAs does. This means that a container to which such a policy applies to won't use any fsGroup/supplementalGroup GID if not explicitly specified, yet a specified GID must still fall in the GID range according to the policy. (#65135, @stlaz)
Upgrade Stackdriver Logging Agent addon image to 0.6-1.6.0-1 to use Fluentd v1.2. This provides nanoseconds timestamp granularity for logs. (#70954, @qingling128)
When the BoundServiceAccountTokenVolumes Alpha feature is enabled, ServiceAccount volumes now use a projected volume source and their names have the prefix "kube-api-access". (#69848, @mikedanese)
Raw block volume support is promoted to beta, and enabled by default. This is accessible via the volumeDevices container field in pod specs, and the volumeMode field in persistent volume and persistent volume claims definitions. (#71167, @msau42)
TokenReview now supports audience validation of tokens with audiences other than the kube-apiserver. (#62692, @mikedanese)
StatefulSet is supported in kubectl autoscale command (#71103, @Pingan2017)
Kubernetes v1.13 moves support for Container Storage Interface to GA. As part of this move Kubernetes now supports CSI v1.0.0 and deprecates support for CSI 0.3 and older releases. Older CSI drivers must be updated to CSI 1.0 and moved to the new kubelet plugin registration directory in order to work with Kubernetes 1.15+. (#71020, @saad-ali)
Added option to create CSRs instead of certificates for kubeadm init phase certs and kubeadm alpha certs renew (#70809, @liztio)
Added a kubelet socket which serves an grpc service containing the devices used by containers on the node. (#70508, @dashpole)
Added DynamicAuditing feature which allows for the configuration of audit webhooks through the use of an AuditSink API object. (#67257, @pbarker)
The kube-apiserver's healthz now takes in an optional query parameter which allows you to disable health checks from causing healthz failures. (#70676, @logicalhan)
Introduced support for running a nodelocal dns cache. It is disabled by default, can be enabled by setting KUBE_ENABLE_NODELOCAL_DNS=true (#70555, @prameshj)
Added readiness gates in extended output for pods (#70775, @freehan)
Added Ready column and improve human-readable output of Deployments and StatefulSets (#70466, @Pingan2017)
Added kubelet_container_log_size_bytes metric representing the log file size of a container. (#70749, @brancz)
NodeLifecycleController: When node lease feature is enabled, node lease will be deleted when the corresponding node is deleted. (#70034, @wangzhen127)
GCERegionalPersistentDisk feature is GA now! (#70716, @jingxu97)
Added secure port 10259 to the kube-scheduler (enabled by default) and deprecate old insecure port 10251. Without further flags self-signed certs are created on startup in memory. (#69663, @sttts)
Release Notes From SIGs
SIG API Machinery
The OwnerReferencesPermissionEnforcement admission plugin now checks authorization for the correct scope (namespaced or cluster-scoped) of the owner resource type. Previously, it always checked permissions at the same scope as the child resource. (#70389, @caesarxuchao)
OpenAPI spec now correctly marks delete request's body parameter as optional (#70032, @iamneha)
The rules for incrementing metadata.generation of custom resources changed: (#69059, @caesarxuchao)
If the custom resource participates the spec/status convention, the metadata.generation of the CR increments when there is any change, except for the changes to the metadata or the changes to the status.
If the custom resource does not participate the spec/status convention, the metadata.generation of the CR increments when there is any change to the CR, except for changes to the metadata.
A custom resource is considered to participate the spec/status convention if and only if the "CustomResourceSubresources" feature gate is turned on and the CRD has .spec.subresources.status={}.
Fixed patch/update operations on multi-version custom resources (#70087, @liggitt)
Reduced memory utilization of admission webhook metrics by removing resource related labels. (#69895, @jpbetz)
Kubelet can now parse PEM file containing both TLS certificate and key in arbitrary order. Previously key was always required to be first. (#69536, @awly)
Code-gen: Removed lowercasing for project imports (#68484, @jsturtevant)
Fixed client cert setup in delegating authentication logic (#69430, @DirectXMan12)
OpenAPI spec and API reference now reflect dryRun query parameter for POST/PUT/PATCH operations (#69359, @roycaihw)
Fixed the sample-apiserver so that its BanFlunder admission plugin can be used. (#68417, @MikeSpreitzer)
APIService availability related to networking glitches are corrected faster (#68678, @deads2k)
Fixed an issue with stuck connections handling error responses (#71412, @liggitt)
apiserver: fixed handling and logging of panics in REST handlers (#71076, @liggitt)
kube-controller-manager no longer removes ownerReferences from ResourceQuota objects (#70035, @liggitt)
"unfinished_work_microseconds" is added to the workqueue metrics; it can be used to detect stuck worker threads. (kube-controller-manager runs many workqueues.) (#70884, @lavalamp)
Timeouts set in ListOptions for clients are also be respected locally (#70998, @deads2k)
Added support for CRD conversion webhook (#67006, @mbohlool)
client-go: fixed sending oversized data frames to spdystreams in remotecommand.NewSPDYExecutor (#70999, @liggitt)
Fixed missing flags in -controller-manager --help. (#71298, @stewart-yu)
Fixed missing flags in kube-apiserver --help. (#70204, @imjching)
The caBundle and service fields in admission webhook API objects now correctly indicate they are optional (#70138, @liggitt)
Fixed an issue with stuck connections handling error responses (#71419, @liggitt)
kube-controller-manager and cloud-controller-manager now hold generated serving certificates in-memory unless a writeable location is specified with --cert-dir (#69884, @liggitt)
CCM server will not listen insecurely if secure port is specified (#68982, @aruneli)
List operations against the API now return internal server errors instead of partially complete lists when a value cannot be transformed from storage. The updated behavior is consistent with all other operations that require transforming data from storage such as watch and get. (#69399, @mikedanese)
SIG Auth
API Server can be configured to reject requests that cannot be audit-logged. (#65763, @x13n)
Go clients created from a kubeconfig that specifies a TokenFile now periodically reload the token from the specified file. (#70606, @mikedanese)
When --rotate-server-certificates is enabled, kubelet will no longer request a new certificate on startup if the current certificate on disk is satisfactory. (#69991, @agunnerson-ibm)
Added dynamic audit configuration api (#67547, @pbarker)
Added ability to control primary GID of containers through Pod Spec and PodSecurityPolicy (#67802, @krmayankk)
kube-apiserver: the NodeRestriction admission plugin now prevents kubelets from modifying Node labels prefixed with node-restriction.kubernetes.io/. The node-restriction.kubernetes.io/ label prefix is reserved for cluster administrators to use for labeling Node objects to target workloads to nodes in a way that kubelets cannot modify or spoof. (#68267, @liggitt)
service.beta.kubernetes.io/aws-load-balancer-internal now supports true and false values, previously it only supported non-empty strings (#69436, @mcrute)
Added service.beta.kubernetes.io/aws-load-balancer-security-groups annotation to set the security groups to the AWS ELB to be the only ones specified in the annotation in case this is present (does not add 0.0.0.0/0). (#62774, @Raffo)
SIG Azure
Ensured orphan public IPs on Azure deleted when service recreated with the same name. (#70463, @feiskyer)
Improved Azure instance metadata handling by adding caches. (#70353, @feiskyer)
Corrected check for non-Azure managed nodes with the Azure cloud provider (#70135, @marc-sensenich)
Fixed azure disk attach/detach failed forever issue (#71377, @andyzhangx)
kubectl apply can now change a deployment strategy from rollout to recreate without explicitly clearing the rollout-related fields (#70436, @liggitt)
The kubectl plugin list command now displays discovered plugin paths in the same order as they are found in a user's PATH variable. (#70443, @juanvallejo)
kubectl get no longer exits before printing all of its results if an error is found (#70311, @juanvallejo)
Fixed a runtime error occuring when sorting the output of kubectl get with empty results (#70740, @mfpierre)
kubectl: support multiple arguments for cordon/uncordon and drain (#68655, @goodluckbot)
Fixed ability for admin/edit/view users to see controller revisions, needed for kubectl rollout commands (#70699, @liggitt)
kubectl rollout undo now returns errors when attempting to rollback a deployment to a non-existent revision (#70039, @liggitt)
kubectl run now generates apps/v1 deployments by default (#71006, @liggitt)
The "kubectl cp" command now supports path shortcuts (../) in remote paths. (#65189, @juanvallejo)
Fixed dry-run output in kubectl apply --prune (#69344, @zegl)
The kubectl wait command must handle when a watch returns an error vs closing by printing out the error and retrying the watch. (#69389, @smarterclayton)
kubectl: support multiple arguments for cordon/uncordon and drain (#68655, @goodluckbot)
SIG Cloud Provider
Added deprecation warning for all cloud providers (#69171, @andrewsykim)
SIG Cluster Lifecycle
kubeadm: Updates version of CoreDNS to 1.2.6 (#70796, @detiber)
kubeadm: Validate kubeconfig files in case of external CA mode. (#70537, @yagonobre)
kubeadm: The writable config file option for extra volumes is renamed to readOnly with a reversed meaning. With readOnly defaulted to false (as in pod specs). (#70495, @rosti)
kubeadm: Multiple API server endpoints support upon join is removed as it is now redundant. (#69812, @rosti)
kubeadm reset now cleans up custom etcd data path (#70003, @yagonobre)
kubeadm: Fixed unnecessary upgrades caused by undefined order of Volumes and VolumeMounts in manifests (#70027, @bart0sh)
Fixed cluster autoscaler addon permissions so it can access batch/job. (#69858, @losipiuk)
kubeadm: JoinConfiguration now houses the discovery options in a nested Discovery structure, which in turn has a couple of other nested structures to house more specific options (BootstrapTokenDiscovery and FileDiscovery) (#67763, @rosti)
kubeadm: Fixed a possible scenario where kubeadm can pull much newer control-plane images (#69301, @neolit123)
kubeadm now allows mixing of init/cluster and join configuration in a single YAML file (although a warning gets printed in this case). (#69426, @rosti)
kubeadm init correctly uses --node-name and --cri-socket when --config option is also used (#71323, @bart0sh)
kubeadm: Always pass spec.nodeName as --hostname-override for kube-proxy (#71283, @Klaven)
kubeadm join correctly uses --node-name and --cri-socket when --config option is also used (#71270, @bart0sh)
kubeadm now supports the --image-repository flag for customizing what registry to pull images from (#71135, @luxas)
kubeadm: The writable config file option for extra volumes is renamed to readOnly with a reversed meaning. With readOnly defaulted to false (as in pod specs). (#70495, @rosti)
kubeadm: Multiple API server endpoints support upon join is removed as it is now redundant. (#69812, @rosti)
kubeadm: JoinConfiguration now houses the discovery options in a nested Discovery structure, which in turn has a couple of other nested structures to house more specific options (BootstrapTokenDiscovery and FileDiscovery) (#67763, @rosti)
kubeadm: Use advertise-client-urls instead of listen-client-urls as and etcd-servers options for apiserver. (#69827, @tomkukral)
Kubeadm now respects the custom image registry configuration across joins and upgrades. Kubeadm passes the custom registry to the kubelet for a custom pause container. (#70603, @chuckha)
kubeadm reset now outputs instructions about manual iptables rules cleanup. (#70874, @rdodev)
kubeadm: remove the AuditPolicyConfiguration feature gate (#70807, @Klaven)
kubeadm pre-pulls Etcd image only if external Etcd is not used and --etcd-upgrade=false is not specified (#70743, @bart0sh)
kubeadm: UnifiedControlPlaneImage is replaced by UseHyperKubeImage boolean value. (#70793, @rosti)
For kube-up and derived configurations, CoreDNS will honor master taints, for consistency with kube-dns behavior. (#70868, @justinsb)
Recognize newer docker versions without -ce/-ee suffix: 18.09.0 (#71001, @thomas-riccardi)
Any external provider should be aware the cloud-provider interface should be imported from :- cloudprovider "k8s.io/cloud-provider" (#68310, @cheftako)
Fixed 'kubeadm upgrade' infinite loop waiting for pod restart (#69886, @bart0sh)
Added tolerations for Stackdriver Logging and Metadata Agents. (#69737, @qingling128)
Enabled insertId generation, and updated Stackdriver Logging Agent image to 0.5-1.5.36-1-k8s. This help reduce log duplication and guarantee log order. (#68920, @qingling128)
Corrected family type (inet6) for ipsets in ipv6-only clusters (#68436, @uablrek)
kube-proxy argument hostname-override can be used to override hostname defined in the configuration file (#69340, @stevesloka)
CoreDNS correctly implements DNS spec for Services with externalNames that look like IP addresses. Kube-dns does not follow the spec for the same case, resulting in a behavior change when moving from Kube-dns to CoreDNS. See: coredns/coredns#2324
IPVS proxier now set net/ipv4/vs/conn_reuse_mode to 0 by default, which will highly improve IPVS proxier performance. (#71114, @Lion-Wei)
Addon configuration is introduced in the kubeadm config API, while feature flag CoreDNS is now deprecated. (#70024, @fabriziopandini)
SIG Node
Fixed a bug in previous releases where a pod could be placed inside another pod's cgroup when specifying --cgroup-root (#70678, @dashpole)
Optimized calculating stats when only CPU and Memory stats are returned from Kubelet stats/summary http endpoint. (#68841, @krzysztof-jastrzebski)
kubelet now supports log-file option to write logs directly to a specific file (#70917, @dims)
Do not detach volume if mount in progress (#71145, @gnufied)
The runtimeHandler field on the RuntimeClass resource now accepts the empty string. (#69550, @tallclair)
kube-apiserver: fixes procMount field incorrectly being marked as required in openapi schema (#69694, @jessfraz)
SIG OpenStack
Fixed cloud-controller-manager crash when using OpenStack provider and PersistentVolume initializing controller (#70459, @mvladev)
SIG Release
Use debian-base instead of busybox as base image for server images (#70245, @ixdy)
Images for cloud-controller-manager, kube-apiserver, kube-controller-manager, and kube-scheduler now contain a minimal /etc/nsswitch.conf and should respect /etc/hosts for lookups (#69238, @BenTheElder)
SIG Scheduling
Added metrics for volume scheduling operations (#59529, @wackxu)
Improved memory use and performance when processing large numbers of pods containing tolerations (#65350, @liggitt)
Fixed a bug in the scheduler that could cause the scheduler to go to an infinite loop when all nodes in a zone are removed. (#69758, @bsalamat)
Clear pod binding cache on bind error to make sure stale pod binding cache will not be used. (#71212, @cofyc)
Fixed a scheduler panic due to internal cache inconsistency (#71063, @Huang-Wei)
Report kube-scheduler unhealthy if leader election is deadlocked. (#71085, @bsalamat)
Fixed a potential bug that scheduler preempts unnecessary pods. (#70898, @Huang-Wei)
SIG Storage
Fixed CSI volume limits not showing up in node's capacity and allocatable (#70540, @gnufied)
CSI drivers now have access to mountOptions defined on the storage class when attaching volumes. (#67898, @bswartz)
change default azure file mount permission to 0777 (#69854, @andyzhangx)
CSIPersistentVolume feature, i.e. PersistentVolumes with CSIPersistentVolumeSource, is GA. (#69929, @jsafrane)
Fixed CSIDriver API object to allow missing fields. (#69331, @jsafrane)
Flex volume plugins now support expandvolume (to increase underlying volume capacity) and expanfs (resize filesystem) commands that Flex plugin authors can implement to support expanding in use Flex PersistentVolumes (#67851, @aniket-s-kulkarni)
The default storage class annotation for the storage addons has been changed to use the GA variant (#68345, @smelchior)
GlusterFS PersistentVolumes sources can now reference endpoints in any namespace using the spec.glusterfs.endpointsNamespace field. Ensure all kubelets are upgraded to 1.13+ before using this capability. (#60195, @humblec)
Upgrade Stackdriver Logging Agent addon image to 0.6-1.6.0-1 to use Fluentd v1.2. This provides nanoseconds timestamp granularity for logs. (#70954, @qingling128)
fixes a runtime error occuring when sorting the output of kubectl get with empty results (#70740, @mfpierre)
fix azure disk attach/detach failed forever issue (#71377, @andyzhangx)
Do not detach volume if mount in progress (#71145, @gnufied)
Fix missing flags in kube-apiserver --help. (#70204, @imjching)
kubeadm init correctly uses --node-name and --cri-socket when --config option is also used (#71323, @bart0sh)
API server flag --experimental-encryption-provider-config was renamed to --encryption-provider-config. The old flag is accepted with a warning but will be removed in 1.14. (#71206, @stlaz)
Fix missing flags in *-controller-manager --help. (#71298, @stewart-yu)
Clear pod binding cache on bind error to make sure stale pod binding cache will not be used. (#71212, @cofyc)
kubeadm: always pass spec.nodeName as --hostname-override for kube-proxy (#71283, @Klaven)
kubeadm join correctly uses --node-name and --cri-socket when --config option is also used (#71270, @bart0sh)
apiserver can be configured to reject requests that cannot be audit-logged. (#65763, @x13n)
Kubelet Device Plugin Registration directory changed from from {kubelet_root_dir}/plugins/ to {kubelet_root_dir}/plugins_registry/. Any drivers (CSI or device plugin) that were using the old path must be updated to work with this version. (#70494, @RenaudWasTaken)
When the BoundServiceAccountTokenVolumes Alpha feature is enabled, ServiceAccount volumes now use a projected volume source and their names have the prefix "kube-api-access". (#69848, @mikedanese)
ACTION REQUIRED: The Node.Status.Volumes.Attached.DevicePath fields is deprecated for CSI volumes and will be unset in a future release (#71095, @msau42)
Other notable changes
Raw block volume support is promoted to beta, and enabled by default. This is accessible via the volumeDevices container field in pod specs, and the volumeMode field in persistent volume and persistent volume claims definitions. (#71167, @msau42)
Fix a scheduler panic due to internal cache inconsistency (#71063, @Huang-Wei)
Fix a potential bug that scheduler preempts unnecessary pods. (#70898, @Huang-Wei)
The API server encryption configuration file format has graduated to stable and moved to apiVersion: apiserver.config.k8s.io/v1 and kind: EncryptionConfiguration. (#67383, @stlaz)
kubelet now supports log-file option to write logs directly to a specific file (#70917, @dims)
kubeadm now supports the --image-repository flag for customizing what registry to pull images from (#71135, @luxas)
timeouts set in ListOptions for clients will also be respected locally (#70998, @deads2k)
IPVS proxier now set net/ipv4/vs/conn_reuse_mode to 0 by default, which will highly improve IPVS proxier performance. (#71114, @Lion-Wei)
StatefulSet is supported in kubectl autoscale command (#71103, @Pingan2017)
Report kube-scheduler unhealthy if leader election is deadlocked. (#71085, @bsalamat)
apiserver: fixes handling and logging of panics in REST handlers (#71076, @liggitt)
kubelets are no longer allowed to delete their own Node API object. Prior to 1.11, in rare circumstances related to cloudprovider node ID changes, kubelets would attempt to delete/recreate their Node object at startup. Kubelets older than 1.11 are not supported running against a v1.13+ API server. If an unsupported legacy kubelet encounters this situation, a cluster admin can remove the Node object: (#71021, @liggitt)
Kubernetes v1.13 moves support for Container Storage Interface to GA. As part of this move Kubernetes now supports CSI v1.0.0 and drops support for CSI 0.3 and older releases. Older CSI drivers must be updated to CSI 1.0 in order to work with Kubernetes 1.13+. (#71020, @saad-ali)
kubeadm: Use advertise-client-urls instead of listen-client-urls as and etcd-servers options for apiserver. (#69827, @tomkukral)
Add option to create CSRs instead of certificates for kubeadm init phase certs and kubeadm alpha certs renew (#70809, @liztio)
Add a kubelet socket which serves an grpc service containing the devices used by containers on the node. (#70508, @dashpole)
kube-apiserver: the NodeRestriction admission plugin now prevents kubelets from modifying Node labels prefixed with node-restriction.kubernetes.io/. The node-restriction.kubernetes.io/ label prefix is reserved for cluster administrators to use for labeling Node objects to target workloads to nodes in a way that kubelets cannot modify or spoof. (#68267, @liggitt)
kubelet: it is now deprecated to use the --node-labels flag to set kubernetes.io/ and k8s.io/-prefixed labels other than the following labels:
kubernetes.io/hostname
kubernetes.io/instance-type
kubernetes.io/os
kubernetes.io/arch
beta.kubernetes.io/instance-type
beta.kubernetes.io/os
beta.kubernetes.io/arch
failure-domain.kubernetes.io/zone
failure-domain.kubernetes.io/region
failure-domain.beta.kubernetes.io/zone
failure-domain.beta.kubernetes.io/region
[*.]kubelet.kubernetes.io/*
[*.]node.kubernetes.io/*
Setting other kubernetes.io/- and k8s.io/-prefixed labels using the --node-labels flag will produce a warning in v1.13, and be disallowed in v1.15. Setting labels that are not prefixed with kubernetes.io/ or k8s.io/ is still permitted.
Adds DynamicAuditing feature which allows for the configuration of audit webhooks through the use of an AuditSink API object. (#67257, @pbarker)
The Kubelet plugin registration mechanism used by device plugins and CSI plugins is now GA (#70559, @vladimirvivien)
CSIPersistentVolume feature, i.e. PersistentVolumes with CSIPersistentVolumeSource, is GA. (#69929, @jsafrane)
CSIPersistentVolume feature gate is now deprecated and will be removed according to deprecation policy.
kubectl: support multiple arguments for cordon/uncordon and drain (#68655, @goodluckbot)
The kube-apiserver's healthz now takes in an optional query parameter which allows you to disable health checks from causing healthz failures. (#70676, @logicalhan)
client-go: fixes sending oversized data frames to spdystreams in remotecommand.NewSPDYExecutor (#70999, @liggitt)
kube-controller-manager no longer removes ownerReferences from ResourceQuota objects (#70035, @liggitt)
Introduces support for running a nodelocal dns cache. It is disabled by default, can be enabled by setting KUBE_ENABLE_NODELOCAL_DNS=true (#70555, @prameshj)
An ip address is required for the cache instance to listen for requests on, default is a link local ip address of value 169.254.20.10
Fix dry-run output in kubectl apply --prune (#69344, @zegl)
kubectl run now generates apps/v1 deployments by default (#71006, @liggitt)
kubeadm reset now outputs instructions about manual iptables rules cleanup. (#70874, @rdodev)
Recognize newer docker versions without -ce/-ee suffix: 18.09.0 (#71001, @thomas-riccardi)
"unfinished_work_microseconds" is added to the workqueue metrics; it can be used to detect stuck worker threads. (kube-controller-manager runs many workqueues.) (#70884, @lavalamp)
add readiness gates in extended output for pods (#70775, @freehan)
add Ready column and improve human-readable output of Deployments and StatefulSets (#70466, @Pingan2017)
Kubeadm now respects the custom image registry configuration across joins and upgrades. Kubeadm passes the custom registry to the kubelet for a custom pause container. (#70603, @chuckha)
kubeadm: deprecate the DynamicKubeletConfig feature gate. The functionality is still accessible by using the kubeadm alpha kubelet enable-dynamic command. (#70849, @yagonobre)
Add kubelet_container_log_size_bytes metric representing the log file size of a container. (#70749, @brancz)
kubeadm: remove the AuditPolicyConfiguration feature gate (#70807, @Klaven)
Kubeadm: attributes for join --control-plane workflow are now grouped into a dedicated JoinControlPlane struct (#70870, @fabriziopandini)
Addon configuration is introduced in the kubeadm config API, while feature flag CoreDNS is now deprecated. (#70024, @fabriziopandini)
Fixes ability for admin/edit/view users to see controller revisions, needed for kubectl rollout commands (#70699, @liggitt)
kubeadm pre-pulls Etcd image only if external Etcd is not used and --etcd-upgrade=false is not specified (#70743, @bart0sh)
Delete node lease if the corresponding node is deleted (#70034, @wangzhen127)
In a future release the kubectl convert command will be deprecated. (#70820, @seans3)
kubeadm: UnifiedControlPlaneImage is replaced by UseHyperKubeImage boolean value. (#70793, @rosti)
kubeadm v1beta1 API: InitConfiguration.APIEndpoint has been renamed to .LocalAPIEndpoint (#70761, @luxas)
Breaking change: CSINodeInfo split into Spec and Status. New fields Available and VolumePluginMechanism added to CSINodeInfo csi-api object. CSIDriverInfo no longer deleted on Driver uninstallation, instead Available flag is set to false. (#70515, @davidz627)
GCERegionalPersistentDisk feature is GA now! (#70716, @jingxu97)
Add secure port 10259 to the kube-scheduler (enabled by default) and deprecate old insecure port 10251. Without further flags self-signed certs are created on startup in memory. (#69663, @sttts)
--feature-gates argument has been removed from the kubeadm join command. Feature gates will be retrieved from the cluster configuration during the join process. (#70755, @ereslibre)
[kubeadm] Updates version of CoreDNS to 1.2.6 (#70796, @detiber)
kubelet: When node lease feature is enabled, kubelet reports node status to api server only if there is some change or it didn't report over last report interval. (#69753, @wangzhen127)
Self hosted is no longer supported in the standard workflow. The feature flags have been removed and your self hosted cluster is no longer able to upgrade via kubeadm. (#69878, @Klaven)
vSphereVolume implements Raw Block Volume Support (#68761, @fanzhangio)
[GCE] Filter out spammy audit logs from cluster autoscaler. (#70696, @loburm)
CRD supports multi-version Schema, Subresources and AdditionalPrintColumns (NOTE that CRDs created prior to 1.13 populated the top-level additionalPrinterColumns field by default. To apply an update that changes to per-version additionalPrinterColumns, the top-level additionalPrinterColumns field must be explicitly set to null). (#70211, @roycaihw)
Fixes a bug in previous releases where a pod could be placed inside another pod's cgroup when specifying --cgroup-root (#70678, @dashpole)
Upgrade golang.org/x/net image to release-branch.go1.10 (#70663, @wenjiaswe)
New addon in addon manager that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true. (#70193, @saad-ali)
delegated authorization can now allow unrestricted access for system:masters like the main kube-apiserver (#70671, @deads2k)
The VolumeScheduling feature is GA. The VolumeScheduling feature gate is deprecated and will be removed in a future release. (#70673, @msau42)
Go clients created from a kubeconfig that specifies a TokenFile now periodically reload the token from the specified file. (#70606, @mikedanese)
kubeadm: validate kubeconfig files in case of external CA mode. (#70537, @yagonobre)
kube-apiserver: --audit-webhook-version and --audit-log-version now default to audit.k8s.io/v1 if unspecified (#70476, @charrywanganthony)
kubeadm: timeoutForControlPlane is introduced as part of the API Server config, that controls the timeout for the wait for control plane to be up. Default value is 4 minutes. (#70480, @rosti)
kubeadm: The writable config file option for extra volumes is renamed to readOnly with a reversed meaning. With readOnly defaulted to false (as in pod specs). (#70495, @rosti)
remove retry operation on attach/detach azure disk (#70568, @andyzhangx)
Fix CSI volume limits not showing up in node's capacity and allocatable (#70540, @gnufied)
Flex volume plugins now support expandvolume (to increase underlying volume capacity) and expanfs (resize filesystem) commands that Flex plugin authors can implement to support expanding in use Flex PersistentVolumes (#67851, @aniket-s-kulkarni)
kubeadm: Control plane component configs are separated into ClusterConfiguration sub-structs. (#70371, @rosti)
The MountPropagation feature is unconditionally enabled in v1.13, and can no longer be disabled. (#68230, @bertinatto)
add azure UltraSSD, StandardSSD disk type support (#70477, @andyzhangx)
The OwnerReferencesPermissionEnforcement admission plugin now checks authorization for the correct scope (namespaced or cluster-scoped) of the owner resource type. Previously, it always checked permissions at the same scope as the child resource. (#70389, @caesarxuchao)
Ensure orphan public IPs on Azure deleted when service recreated with the same name. (#70463, @feiskyer)
kubectl apply can now change a deployment strategy from rollout to recreate without explicitly clearing the rollout-related fields (#70436, @liggitt)
Fix cloud-controller-manager crash when using OpenStack provider and PersistentVolume initializing controller (#70459, @mvladev)
kubelet --system-reserved and --kube-reserved are supported now on Windows nodes (#69960, @feiskyer)
CSI drivers now have access to mountOptions defined on the storage class when attaching volumes. (#67898, @bswartz)
The kubectl plugin list command will now display discovered plugin paths in the same order as they are found in a user's PATH variable. (#70443, @juanvallejo)
Handle Windows named pipes in host mounts. (#69484, @ddebroy)
kubeadm: Multiple API server endpoints support upon join is removed as it is now redundant. (#69812, @rosti)
OpenAPI spec marks delete request's body parameter as optional (#70032, @iamneha)
kube-controller-manager and cloud-controller-manager now hold generated serving certificates in-memory unless a writeable location is specified with --cert-dir (#69884, @liggitt)
Scheduler only activates unschedulable pods if node's scheduling related properties change. (#70366, @mlmhl)
--api-audiences now defaults to the --service-account-issuer if the issuer is provided but the API audience is not. (#70308, @mikedanese)
kubectl rollout undo now returns errors when attempting to rollback a deployment to a non-existent revision (#70039, @liggitt)
kubectl rollout undo no longer uses the deprecated extensions/v1beta1 rollback API, which means that Events are no longer emitted when rolling back a deployment
The builtin system:csi-external-provisioner and system:csi-external-attacher cluster roles are deprecated and will not be updated for deployments of CSI sidecar container versions >= 0.4. Deployments with the current CSI sidecar containers have to provide their own RBAC definitions. The reason is that the rules depend on how the sidecar containers are used, which is defined by the deployment. (#69868, @pohly)
Use debian-base instead of busybox as base image for server images (#70245, @ixdy)
add support for projected volume in describe function (#70158, @WanLinghao)
Kubeadm reset now clean up custom etcd data path (#70003, @yagonobre)
We changed when the metadata.generation of a custom resource (CR) increments. (#69059, @caesarxuchao)
If the CR participates the spec/status convention, the metadata.generation of the CR increments when there is any change, except for the changes to the metadata or the changes to the status.
If the CR does not participate the spec/status convention, the metadata.generation of the CR increments when there is any change to the CR, except for changes to the metadata.
A CR is considered to participate the spec/status convention if and only if the "CustomResourceSubresources" feature gate is turned on and the CRD has .spec.subresources.status={}.
Improve Azure instance metadata handling by adding caches. (#70353, @feiskyer)
adding cn-northwest-1 for AWS China Ningxia region (#70155, @pahud)
"kubectl get" no longer exits before printing all of its results if an error is found (#70311, @juanvallejo)
kubeadm now automatically creates a new stacked etcd member when joining a new control plane node (does not applies to external etcd) (#69486, @fabriziopandini)
Critical pod annotation is deprecated. Pod priority should be used instead to mark pods as critical. (#70298, @bsalamat)
Display the usage of ephemeral-storage when using kubectl describe node (#70268, @Pingan2017)
Added functionality to enable br_netfilter and ip_forward for debian packages to improve kubeadm support for CRI runtime besides Docker. (#70152, @ashwanikhemani)
Add regions ap-northeast-3 and eu-west-3 to the list of well known AWS regions. (#70252, @nckturner)
Remove kube-controller-manager flag '--insecure-experimental-approve-all-kubelet-csrs-for-group'(deprecated in v1.7) (#69209, @Pingan2017)
GCE/GKE load balancer health check default interval changes from 2 seconds to 8 seconds, unhealthyThreshold to 3. (#70099, @grayluck)
Health check parameters are configurable to be bigger than default values.
The kubectl wait command must handle when a watch returns an error vs closing by printing out the error and retrying the watch. (#69389, @smarterclayton)
Updates to use debian-iptables v11.0, debian-hyperkube-base 0.12.0, and kube-addon-manager:v8.9. (#70209, @ixdy)
Fixed patch/update operations on multi-version custom resources (#70087, @liggitt)
When --rotate-server-certificates is enabled, kubelet will no longer request a new certificate on startup if the current certificate on disk is satisfactory. (#69991, @agunnerson-ibm)
Support for passing unknown provider names to the E2E test binaries is going to be deprecated. Use --provider=skeleton (no ssh access) or --provider=local (local cluster with ssh) instead. (#70141, @pohly)
Add scheduler benchmark tests for PodAffinity and NodeAffinity. (#69898, @Huang-Wei)
Fix cluster autoscaler addon permissions so it can access batch/job. (#69858, @losipiuk)
change default azure file mount permission to 0777 (#69854, @andyzhangx)
kubeadm: JoinConfiguration now houses the discovery options in a nested Discovery structure, which in turn has a couple of other nested structures to house more specific options (BootstrapTokenDiscovery and FileDiscovery) (#67763, @rosti)
Fix tests to use fsync instead of sync (#69755, @mrunalp)
kube-proxy argument hostname-override can be used to override hostname defined in the configuration file (#69340, @stevesloka)
kube-apiserver: the --deserialization-cache-size flag is no longer used, is deprecated, and will be removed in a future release (#69842, @liggitt)
kube-apiserver: the deprecated --etcd-quorum-read flag has been removed, and quorum reads are always enabled when fetching data from etcd. (#69527, @liggitt)
Moved staging/src/k8s.io/client-go/tools/bootstrap to staging/src/k8s… (#67356, @yliaog)
[action required] kubeadm: The v1alpha2 config API has been removed. (#69055, @fabriziopandini)
Please convert your v1alpha2 configuration files to v1alpha3 using the
kubeadm: fix a case where fetching a kubernetesVersion from the internet still happened even if some commands don't need it. (#69645, @neolit123)
Add tolerations for Stackdriver Logging and Metadata Agents. (#69737, @qingling128)
Fix a bug in the scheduler that could cause the scheduler to go to an infinite loop when all nodes in a zone are removed. (#69758, @bsalamat)
Dry-run is promoted to Beta and will be enabled by default. (#69644, @apelisse)
kubectl get priorityclass now prints value column by default. (#69431, @Huang-Wei)
Added a new container based image for running e2e tests (#69368, @dims)
Remove the deprecated --google-json-key flag from kubelet. (#69354, @yujuhong)
kube-apiserver: fixes procMount field incorrectly being marked as required in openapi schema (#69694, @jessfraz)
The LC_ALL and LC_MESSAGES env vars can now be used to set desired locale for kubectl while keeping LANG unchanged. (#69500, @m1kola)
Add ability to control primary GID of containers through Pod Spec and PodSecurityPolicy (#67802, @krmayankk)
NodeLifecycleController: Now node lease renewal is treated as the heartbeat signal from the node, in addition to NodeStatus Update. (#69241, @wangzhen127)
[GCE] Enable by default audit logging truncating backend. (#68288, @loburm)
Enable insertId generation, and update Stackdriver Logging Agent image to 0.5-1.5.36-1-k8s. This help reduce log duplication and guarantee log order. (#68920, @qingling128)
Move NodeInfo utils into pkg/scheduler/cache. (#69495, @wgliang)
-viper-config can be used to set also the options defined by command line flags
the default config file is "e2e.yaml/toml/json/..." and the test starts when no such config is found (as before) but if -viper-config is used, the config file must exist
-viper-config can be used to select a file with full path, with or without file suffix
the csiImageVersion/Registry flags were renamed to storage.csi.imageVersion/Registry
Move FakeCache to pkg/scheduler/internal/cache/fake. (#69318, @wgliang)
The "kubectl cp" command now supports path shortcuts (../) in remote paths. (#65189, @juanvallejo)
The runtimeHandler field on the RuntimeClass resource now accepts the empty string. (#69550, @tallclair)
Kubelet can now parse PEM file containing both TLS certificate and key in arbitrary order. Previously key was always required to be first. (#69536, @awly)
Scheduling conformance tests related to daemonsets should set the annotation that relaxes node selection restrictions, if any are set. This ensures conformance tests can run on a wider array of clusters. (#68793, @aveshagarwal)
Replace Parallelize with function ParallelizeUntil and formally deprecate the Parallelize. (#68403, @wgliang)
Move scheduler cache interface and implementation to pkg/scheduler/internal/cache. (#68968, @wgliang)
Any external provider should be aware the cloud-provider interface should be imported from :- cloudprovider "k8s.io/cloud-provider" (#68310, @cheftako)
kubeadm: Fix a crash if the etcd local alpha phase is called when the configuration contains an external etcd cluster (#69420, @ereslibre)
kubeadm now allows mixing of init/cluster and join configuration in a single YAML file (although a warning gets printed in this case). (#69426, @rosti)
Fix client cert setup in delegating authentication logic (#69430, @DirectXMan12)
service.beta.kubernetes.io/aws-load-balancer-internal now supports true and false values, previously it only supported non-empty strings (#69436, @mcrute)
OpenAPI spec and API reference now reflect dryRun query parameter for POST/PUT/PATCH operations (#69359, @roycaihw)
kube-apiserver has removed support for the etcd2 storage backend (deprecated since v1.9). Existing clusters must migrate etcd v2 data to etcd v3 storage before upgrading to v1.13. (#69310, @liggitt)
List operations against the API now return internal server errors instead of partially complete lists when a value cannot be transformed from storage. The updated behavior is consistent with all other operations that require transforming data from storage such as watch and get. (#69399, @mikedanese)
kubectl wait now supports condition value checks other than true using --for condition=available=false (#69295, @deads2k)
CCM server will not listen insecurely if secure port is specified (#68982, @aruneli)
Bump cluster-proportional-autoscaler to 1.3.0 (#69338, @MrHohn)
Rebase docker image on scratch.
fix inconsistency in windows kernel proxy when updating HNS policy. (#68923, @delulu)
Fixes the sample-apiserver so that its BanFlunder admission plugin can be used. (#68417, @MikeSpreitzer)
Fixed CSIDriver API object to allow missing fields. (#69331, @jsafrane)
Wait for pod failed event in subpath test. (#69300, @mrunalp)
[GCP] Added env variables to control CPU requests of kube-controller-manager and kube-scheduler. (#68823, @loburm)
Bump up pod short start timeout to 2 minutes. (#69291, @mrunalp)
Use the mounted "/var/run/secrets/kubernetes.io/serviceaccount/token" as the token file for running in-cluster based e2e testing. (#69273, @dims)
apiservice availability related to networking glitches are corrected faster (#68678, @deads2k)
extract volume attachment status checking operation as a common function when attaching a CSI volume (#68931, @mlmhl)
PodSecurityPolicy objects now support a MayRunAs rule for fsGroup and supplementalGroups options. This allows specifying ranges of allowed GIDs for pods/containers without forcing a default GID the way MustRunAs does. This means that a container to which such a policy applies to won't use any fsGroup/supplementalGroup GID if not explicitly specified, yet a specified GID must still fall in the GID range according to the policy. (#65135, @stlaz)
Images for cloud-controller-manager, kube-apiserver, kube-controller-manager, and kube-scheduler now contain a minimal /etc/nsswitch.conf and should respect /etc/hosts for lookups (#69238, @BenTheElder)
kubectl: add the --no-headers flag to kubectl top ... (#67890, @WanLinghao)
Restrict redirect following from the apiserver to same-host redirects, and ignore redirects in some cases. (#66516, @tallclair)
Fixed pod cleanup when /var/lib/kubelet is a symlink. (#68741, @jsafrane)
Add "only_cpu_and_memory" GET parameter to /stats/summary http handler in kubelet. If parameter is true then only cpu and memory will be present in response. (#67829, @krzysztof-jastrzebski)
Flex drivers by default do not produce metrics. Flex plugins can enable metrics collection by setting the capability 'supportsMetrics' to true. Make sure the file system can support fs stat to produce metrics in this case. (#67508, @brahmaroutu)
Use monotonically increasing generation to prevent scheduler equivalence cache race. (#67308, @cofyc)
Fix kubelet service file permission warning (#66669, @daixiang0)