Catalog / Kubernetes

1.35.0

Kubernetes v1.35.0

10 months agobreakingaddedfixedOriginal notes

Changelog since v1.34.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • ACTION REQUIRED:

    Removed the --pod-infra-container-image flag from kubelet command line. For non-kubeadm clusters, users must manually remove this flag from their kubelet configuration to prevent startup failures before upgrading kubelet. For kubeadm clusters, if users pass extra arguments to the kubelet like --pod-infra-container-image, it will be written to the kubelet env file during the init phase. kubeadm does not remove it during the init or join phase, so users must manually remove it from extraArgs in the kubelet configuration file. (#133779, @carlory)

  • ACTION REQUIRED:

vendor: Updated k8s.io/system-validators to v1.12.1. The cgroups validator now throws an error instead of a warning if cgroups v1 is detected on the host and the provided KubeletVersion is v1.35 or newer.

kubeadm: Started using k8s.io/system-validators v1.12.1 in kubeadm v1.35. During kubeadm init, kubeadm join, and kubeadm upgrade, the SystemVerification preflight check throws an error if cgroups v1 is detected and the detected kubelet version is v1.35 or newer. For older versions of kubelet, a preflight warning is displayed.

To allow cgroups v1 with kubeadm and kubelet version v1.35 or newer, you must:

  • Ignore the error from the SystemVerification preflight check by kubeadm.
  • Edit the kube-system/kubelet-config ConfigMap and add the failCgroupV1: false field before upgrading. (#134744, @neolit123) [SIG Cluster Lifecycle and Node]

Changes by Kind

Deprecation

  • ACTION REQUIRED: failCgroupV1 will be set to true from 1.35. This means that nodes will not start on a cgroup v1 by default. This puts cgroup v1 into a deprecated state. (#134298, @kannon92)
  • Marked ipvs mode in kube-proxy as deprecated, which will be removed in a future version of Kubernetes. Users are encouraged to migrate to nftables. (#134539, @adrianmoisey)

API Change

  • Added ObservedGeneration to CustomResourceDefinition conditions. (#134984, @michaelasp)

  • Added WithOrigin within apis/core/validation with adjusted tests. (#132825, @PatrickLaabs)

  • Added scoring for the prioritized list feature so nodes that best satisfy the highest-ranked subrequests were chosen. (#134711, @mortent) [SIG Node, Scheduling and Testing]

  • Added the --min-compatibility-version flag to kube-apiserver, kube-controller-manager, and kube-scheduler. (#133980, @siyuanfoundation) [SIG API Machinery, Architecture, Cluster Lifecycle, Etcd, Scheduling and Testing]

  • Added the StorageVersionMigration v1beta1 API and removed the v1alpha1 API.

    ACTION REQUIRED: The v1alpha1 API is no longer supported. Users must remove any v1alpha1 resources before upgrading. (#134784, @michaelasp) [SIG API Machinery, Apps, Auth, Etcd and Testing]

  • Added validation to ensure log-flush-frequency is a positive value, returning an error instead of causing a panic. (#133540, @BenTheElder) [SIG Architecture, Instrumentation, Network and Node]

  • All containers are restarted when a source container in a restart policy rule exits. This alpha feature is gated behind RestartAllContainersOnContainerExit. (#134345, @yuanwang04) [SIG Apps, Node and Testing]

  • CSI drivers can now opt in to receive service account tokens via the secrets field instead of volume context by setting spec.serviceAccountTokenInSecrets: true in the CSIDriver object. This prevents tokens from being exposed in logs and other outputs. The feature is gated by the CSIServiceAccountTokenSecrets feature gate (beta in v1.35). (#134826, @aramase) [SIG API Machinery, Auth, Storage and Testing]

  • Changed kuberc configuration schema. Two new optional fields added to kuberc configuration, credPluginPolicy and credPluginAllowlist. This is documented in KEP-3104 and documentation is added to the website by kubernetes/website#52877 (#134870, @pmengelbert) [SIG API Machinery, Architecture, Auth, CLI, Instrumentation and Testing]

  • DRA device taints: DeviceTaintRule status provides information about the rule, including whether Pods still need to be evicted (EvictionInProgress condition). The newly added None effect can be used to preview what a DeviceTaintRule would do if it used the NoExecute effect and to taint devices (device health) without immediately affecting scheduling or running Pods. (#134152, @pohly) [SIG API Machinery, Apps, Auth, Node, Release, Scheduling and Testing]

  • DRA: The DynamicResourceAllocation feature gate for the core functionality (GA in v1.34) has now been locked to enabled-by-default and cannot be disabled anymore. (#134452, @pohly) [SIG Auth, Node, Scheduling and Testing]

  • Enabled kubectl get -o kyaml by default. To disable it, set KUBECTL_KYAML=false. (#133327, @thockin)

  • Enabled in-place resizing of pod-level resources.

    • Added Resources in PodStatus to capture resources set in the pod-level cgroup.
    • Added AllocatedResources in PodStatus to capture resources requested in the PodSpec. (#132919, @ndixita) [SIG API Machinery, Apps, Architecture, Auth, CLI, Instrumentation, Node, Scheduling and Testing]

  • Enabled the NominatedNodeNameForExpectation feature in kube-scheduler by default.

    • Enabled the ClearingNominatedNodeNameAfterBinding feature in kube-apiserver by default. (#135103, @ania-borowiec) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scheduling, Storage and Testing]

  • Enhanced discovery responses to merge API groups and resources from all peer apiservers when the UnknownVersionInteroperabilityProxy feature is enabled. (#133648, @richabanker) [SIG API Machinery, Auth, Cloud Provider, Node, Scheduling and Testing]

  • Extended core/v1 Toleration to support numeric comparison operators (Gt,Lt). (#134665, @helayoty) [SIG API Machinery, Apps, Node, Scheduling, Testing and Windows]

  • Feature gate dependencies are now explicit, and validated at startup. A feature can no longer be enabled if it depends on a disabled feature. In particular, this means that AllAlpha=true will no longer work without enabling disabled-by-default beta features that are depended on (either with AllBeta=true or explicitly enumerating the disabled dependencies). (#133697, @tallclair) [SIG API Machinery, Architecture, Cluster Lifecycle and Node]

  • Generated OpenAPI model packages for API types into zz_generated.model_name.go files, accessible via the OpenAPIModelName() function. This allows API authors to declare desired OpenAPI model packages instead of relying on the Go package path of API types. (#131755, @jpbetz) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling, Storage and Testing]

  • Implemented constrained impersonation as described in KEP-5284. (#134803, @enj) [SIG API Machinery, Auth and Testing]

  • Introduced a new declarative validation tag +k8s:customUnique to control listmap uniqueness. (#134279, @yongruilin) [SIG API Machinery and Auth]

  • Introduced a structured and versioned v1alpha1 response for the statusz endpoint. (#134313, @richabanker) [SIG API Machinery, Architecture, Instrumentation, Network, Node, Scheduling and Testing]

  • Introduced a structured and versioned v1alpha1 response format for the flagz endpoint. (#134995, @yongruilin) [SIG API Machinery, Architecture, Instrumentation, Network, Node, Scheduling and Testing]

  • Introduced the GangScheduling kube-scheduler plugin to support "all-or-nothing" scheduling using the scheduling.k8s.io/v1alpha1 Workload API. (#134722, @macsko) [SIG API Machinery, Apps, Auth, CLI, Etcd, Scheduling and Testing]

  • Introduced the Node Declared Features capability (alpha), which includes:

    • A new Node.Status.DeclaredFeatures field for publishing node-specific features.
    • A component-helpers library for feature registration and inference.
    • A NodeDeclaredFeatures scheduler plugin to match pods with nodes that provide required features.
    • A NodeDeclaredFeatureValidator admission plugin to validate pod updates against a node's declared features. (#133389, @pravk03) [SIG API Machinery, Apps, Node, Release, Scheduling and Testing]

  • Introduced the scheduling.k8s.io/v1alpha1 Workload API to express workload-level scheduling requirements and allow the kube-scheduler to act on them. (#134564, @macsko) [SIG API Machinery, Apps, CLI, Etcd, Scheduling and Testing]

  • Introduced the alpha MutableSchedulingDirectivesForSuspendedJobs feature gate (disabled by default), which allows mutating a Job's scheduling directives while the Job is suspended. It also updates the Job controller to clears the status.startTime field for suspended Jobs. (#135104, @mimowo) [SIG Apps and Testing]

  • Kube-apiserver: Fixed a v1.34 regression in CustomResourceDefinition handling that incorrectly warned about unrecognized formats on number and integer properties. (#133896, @yongruilin) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Contributor Experience, Network, Node and Scheduling]

  • Kube-apiserver: Fixed a possible panic validating a custom resource whose CustomResourceDefinition indicates a status subresource exists, but which does not define a status property in the openAPIV3Schema. (#133721, @fusida) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]

  • Kubernetes API Go types removed runtime use of the github.com/gogo/protobuf library, and are no longer registered into the global gogo type registry. Kubernetes API Go types were not suitable for use with the google.golang.org/protobuf library, and no longer implement ProtoMessage() by default to avoid accidental incompatible use. If removal of these marker methods impacts your use, it can be re-enabled for one more release with a kubernetes_protomessage_one_more_release build tag, but will be removed in v1.36. (#134256, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling and Storage]

  • Made node affinity in Persistent Volume mutable. (#134339, @huww98) [SIG API Machinery, Apps and Node]

  • Moved the ImagePullIntent and ImagePulledRecord objects used by the kubelet to track image pulls to the v1beta1 API version. (#132579, @stlaz) [SIG Auth and Node]

  • Pod resize now only allows CPU and memory resources; other resource types are forbidden. (#135084, @tallclair) [SIG Apps, Node and Testing]

  • Prevented Pods from being scheduled onto nodes that lack the required CSI driver. (#135012, @gnufied) [SIG API Machinery, Scheduling, Storage and Testing]

  • Promoted HPA configurable tolerance to beta. The HPAConfigurableTolerance feature gate has now been enabled by default. (#133128, @jm-franc) [SIG API Machinery and Autoscaling]

  • Promoted ReplicaSet and Deployment .status.terminatingReplicas tracking to beta. The DeploymentReplicaSetTerminatingReplicas feature gate is now enabled by default. (#133087, @atiratree) [SIG API Machinery, Apps and Testing]

  • Promoted PodObservedGenerationTracking to GA. (#134948, @natasha41575) [SIG API Machinery, Apps, Node, Scheduling and Testing]

  • Promoted the JobManagedBy feature to general availability. The JobManagedBy feature gate was locked to true and will be removed in a future Kubernetes release. (#135080, @dejanzele) [SIG API Machinery, Apps and Testing]

  • Promoted the MaxUnavailableStatefulSet feature to beta and enabling it by default. (#133153, @helayoty) [SIG API Machinery and Apps]

  • Removed the StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks feature gates, which were locked since v1.32. (#134994, @liggitt) [SIG API Machinery, Auth, Node and Testing]

  • Scheduler: Added the bindingTimeout argument to the DynamicResources plugin configuration, allowing customization of the wait duration in PreBind for device binding conditions. Defaults to 10 minutes when DRADeviceBindingConditions and DRAResourceClaimDeviceStatus are both enabled. (#134905, @fj-naji) [SIG Node and Scheduling]

  • The DRA device taints and toleration feature received a separate feature gate, DRADeviceTaintRules, which controlled support for DeviceTaintRules. This allowed disabling it while keeping DRADeviceTaints enabled so that tainting via ResourceSlices continued to work. (#135068, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

  • The Pod Certificates feature moved to beta. The PodCertificateRequest feature gate is set disabled by default. To use the feature, users must enable the certificates API groups in v1beta1 and enable the PodCertificateRequest feature gate. The UserAnnotations field was added to the PodCertificateProjection API and the corresponding UnverifiedUserAnnotations field was added to the PodCertificateRequest API. (#134624, @yt2985) [SIG API Machinery, Apps, Auth, Etcd, Instrumentation, Node and Testing]

  • The KubeletEnsureSecretPulledImages feature was promoted to Beta and enabled by default. (#135228, @aramase) [SIG Auth, Node and Testing]

  • The PreferSameZone and PreferSameNode values for the Service trafficDistribution field graduated to general availability. The PreferClose value is now deprecated in favor of the more explicit PreferSameZone. (#134457, @danwinship) [SIG API Machinery, Apps, Network and Testing]

  • Updated ResourceQuota to count device class requests within a ResourceClaim as two additional quotas when the DRAExtendedResource feature is enabled:

    • requests.deviceclass.resource.k8s.io/<deviceclass> is charged based on the worst-case number of devices requested.
    • Device classes mapping to an extended resource now consume requests.<extended resource name>. (#134210, @yliaog) [SIG API Machinery, Apps, Node, Scheduling and Testing]

  • Updated storage version for MutatingAdmissionPolicy to v1beta1. (#133715, @cici37) [SIG API Machinery, Etcd and Testing]

  • Updated the Partitionable Devices feature to support referencing counter sets across ResourceSlices within the same resource pool. Devices from incomplete pools were no longer considered for allocation. This change introduced backwards-incompatible updates to the alpha feature, requiring any ResourceSlices using it to be removed before upgrading or downgrading between v1.34 and v1.35. (#134189, @mortent) [SIG API Machinery, Node, Scheduling and Testing]

  • Upgraded the PodObservedGenerationTracking feature to beta in v1.34 and removed the alpha version description from the OpenAPI specification. (#133883, @yangjunmyfm192085)

Feature

  • Added k8s-short-name and k8s-long-name format validation tags to enforce DNS label and DNS subdomain compliance. (#133894, @lalitc375)

  • Added kubectl kuberc view and kubectl kuberc set commands to perform operations against the kuberc file. (#135003, @ardaguclu) [SIG CLI and Testing]

  • Added kubelet stress test for pod cleanup when rejection due to VolumeAttachmentLimitExceeded. (#133357, @torredil) [SIG Node and Storage]

  • Added paths section to kubelet statusz endpoint. (#133239, @Peac36)

  • Added a source label to the resourceclaim_controller_resource_claims metric. Added the scheduler_resourceclaim_creates_total metric for DRAExtendedResource. (#134523, @bitoku) [SIG Apps, Instrumentation, Node and Scheduling]

  • Added a counter metric kubelet_image_manager_ensure_image_requests_total{present_locally, pull_policy, pull_required} that exposes details about kubelet ensuring an image exists on the node. (#132644, @stlaz) [SIG Auth and Node]

  • Added additional event emissions during Pod resizing to provide clearer visibility when a Pod’s resize status changes. (#134825, @natasha41575)

  • Added configurable per-device health check timeouts to the DRA health monitoring API. (#135147, @harche) [SIG Node]

  • Added metrics for the MaxUnavailable feature in StatefulSet. (#130951, @Edwinhr716) [SIG Apps and Instrumentation]

  • Added paths section to scheduler statusz endpoint. (#132606, @Peac36) [SIG API Machinery, Architecture, Instrumentation, Network, Node, Scheduling and Testing]

  • Added remote runtime and image Close() method to be able to close the connection. (#133211, @saschagrunert) [SIG Node]

  • Added support for tracing in kubectl with the --profile=trace flag. (#134709, @tchap)

  • Added support for validating UUID format. (#133948, @lalitc375)

  • Added the -n flag as a shorthand for --namespace in the kubectl config set-context command. (#134384, @tchap) [SIG CLI and Testing]

  • Added the ChangeContainerStatusOnKubeletRestart feature gate, which defaults to disabled. When the feature gate is disabled, kubelet does not change the Pod status upon restart, and Pods do not re-run startup probes after the kubelet restarts. (#134746, @HirazawaUi) [SIG Node and Testing]

  • Added the CloudControllerManagerWatchBasedRoutesReconciliation feature gate. (#131220, @lukasmetzner) [SIG API Machinery and Cloud Provider]

  • Added the UserNamespacesHostNetworkSupport feature gate. This gate is disabled by default, and when enabled, allowed hostNetwork pods to use user namespaces. (#134893, @HirazawaUi) [SIG Apps, Node and Testing]

  • After fixing regressions detected in v1.34, the SchedulerAsyncAPICalls feature gate was re-enabled by default. (#135059, @macsko)

  • Changed WaitForNamedCacheSync to WaitForNamedCacheSyncWithContext. (#133904, @aditigupta96) [SIG API Machinery, Apps, Auth and Network]

  • DRA: the resource.k8s.io API now uses the v1 API version (introduced in 1.34) as default storage version. Downgrading to 1.33 is not supported. (#133876, @kei01234kei) [SIG API Machinery, Etcd and Testing]

  • Enabled the MutableCSINodeAllocatableCount feature gate by default in beta. (#134647, @torredil)

  • Enabled the WatchListClient feature gate. (#134180, @p0lyn0mial) [SIG API Machinery, Apps, Auth, CLI, Instrumentation, Node and Testing]

  • Enabled the feature gate ContainerRestartRules by default. The ContainerRestartRules feature has been promoted to beta. Fixed a bug in this feature that caused probes to continue to run even if the container has terminated and is not restartable. (#134631, @yuanwang04)

  • Graduated the PodTopologyLabelsAdmission feature gate to Beta and enabled it by default. Pods now receive topology.kubernetes.io/zone and topology.kubernetes.io/region labels automatically when their assigned Node has these labels. (#135158, @andrewsykim)

  • Graduated the fine-grained supplemental groups policy (KEP-3619) to GA. (#135088, @everpeace) [SIG Node and Testing]

  • Graduated the image volume source feature to Beta and enabled it by default. (#135195, @haircommander) [SIG Apps, Instrumentation, Node and Testing]

  • Implemented opportunistic batching (KEP-5598) to optimize scheduling for pods with identical scheduling requirements. (#135231, @bwsalmon) [SIG Node, Scheduling, Storage and Testing]

  • Implemented scoring for DRA-backed extended resources. (#134058, @bart0sh) [SIG Node, Scheduling and Testing]

  • Improved throughput in the real-FIFO queue used by informers and controllers by adding batch handling for processing watch events. (#132240, @yue9944882) [SIG API Machinery, Scheduling and Storage]

  • Introduced end-to-end tests to verify component invariant metrics across the entire test suite. (#133394, @BenTheElder)

  • Introduced new kubelet metrics for the Ensure Secret Pulled Images KEP, including: - kubelet_imagemanager_ondisk_pullintents for tracking pull intent records on disk - kubelet_imagemanager_ondisk_pulledrecords for tracking pulled image records on disk - kubelet_imagemanager_image_mustpull_checks_total{result} for counting image must-pull verification checks. (#132812, @stlaz) [SIG Auth and Node]

  • Introduced the --as-user-extra persistent flag in kubectl, which allows passing extra arguments during impersonation. (#134378, @ardaguclu) [SIG CLI and Testing]

  • K8s.io/apimachinery: Introduced a helper function to compare resourceVersion strings between two objects of the same resource. (#134330, @michaelasp) [SIG API Machinery, Apps, Auth, Instrumentation, Network, Node, Scheduling, Storage and Testing]

  • KEP-5440: Enabled support for resizing resources while a Job is suspended. This feature is alpha. (#132441, @kannon92) [SIG Apps and Testing]

  • Kube-apiserver: Made the subresources pods/exec, pods/attach, and pods/portforward require create permission for both SPDY and Websocket API requests. Previously, SPDY requests required create permission, but Websocket requests only required get permission. This change is gated by the AuthorizePodWebsocketUpgradeCreatePermission feature-gate, which is enabled by default.

    Before upgrading to 1.35, ensure any custom ClusterRoles and Roles intended to grant pods/exec, pods/attach, or pods/portforward permission include the create verb. (#134577, @seans3) [SIG API Machinery, Auth, Node and Testing]

  • Kubeadm: Added error printing during retries related to the WaitForAllControlPlaneComponents functionality at verbosity level 5. (#134433, @neolit123)

  • Kubeadm: Added the HTTPEndpoints field to ClusterConfiguration.Etcd.ExternalEtcd to configure HTTP endpoints for etcd communication in v1beta4. This separates HTTP traffic (e.g., /metrics, /health) from gRPC traffic, improving access control. Mirrors etcd’s --listen-client-http-urls behavior; if not set, the Endpoints field handles both traffic types. (#134890, @SataQiu)

  • Kubeadm: Graduated the kubeadm-specific feature gate ControlPlaneKubeletLocalMode to GA and locked it to enabled by default. To opt out, patch the server field in /etc/kubernetes/kubelet.conf. Deprecated the subphase of kubeadm join phase control-plane-join called etcd, which is now hidden and replaced by subphase with identical functionality etcd-join. The etcd subphase will be removed in a future release. The subphase kubelet-wait-bootstrap of kubeadm join is no longer experimental and will now always run. (#134106, @neolit123)

  • Kubernetes is now built using Go 1.25.1 (#134095, @dims) [SIG Release and Testing]

  • Kubernetes is now built using Go 1.25.4 (#135492, @cpanato) [SIG Release and Testing]

  • Kubernetes now uses Go Language Version 1.25, including https://go.dev/blog/container-aware-gomaxprocs (#134120, @BenTheElder) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling and Storage]

  • Locked down the AllowOverwriteTerminationGracePeriodSeconds feature gate. (#133792, @HirazawaUi)

  • Locked the (generally available) feature gate ExecProbeTimeout to true. (#134635, @vivzbansal) [SIG Node and Testing]

  • Metrics: Excluded dryRun requests from apiserver_request_sli_duration_seconds. (#131092, @aldudko) [SIG API Machinery and Instrumentation]

  • Migrated validation in resource.k8s.io to declarative validation. When the DeclarativeValidation feature gate is enabled, mismatches with existing validation are reported via metrics. when DeclarativeValidationTakeover feature gate is enabled, declarative validation becomes the primary source of errors for migrated fields. (#134072, @yongruilin) [SIG API Machinery, Apps and Auth]

  • Moved the Pod Certificates feature to beta. Added UserAnnotations to the PodCertificateProjection API and UnverifiedUserAnnotations to the PodCertificateRequest API. The PodCertificateRequest feature gate remains disabled by default and requires enabling the v1beta1 certificates API groups. (#134790, @yt2985) [SIG Auth, Instrumentation and Testing]

  • Promoted ImageGCMaximumAge to stable. (#134736, @haircommander) [SIG Node and Testing]

  • Promoted InPlacePodVerticalScaling to GA. (#134949, @natasha41575) [SIG API Machinery, Node and Scheduling]

  • Promoted kubectl command headers to stable. (#134777, @soltysh) [SIG CLI and Testing]

  • Promoted the EnvF

These notes run past the length kept in the archive. The rest is on the publisher’s page.