Catalog / Kubernetes

1.34.0

Kubernetes v1.34.0

1.1 years agosecurityaddedfixedOriginal notes

Changelog since v1.33.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • For metrics apiserver_cache_list_fetched_objects_total, apiserver_cache_list_returned_objects_total, apiserver_cache_list_total replace resource_prefix label with API group and resource labels. For metrics etcd_request_duration_seconds, etcd_requests_total and etcd_request_errors_total replace type label with API resource and group label. For metric apiserver_selfrequest_total add a API group label. For metrics apiserver_watch_events_sizes and apiserver_watch_events_total replace API kind label with resource label. For metrics apiserver_request_body_size_bytes, apiserver_storage_events_received_total, apiserver_storage_list_evaluated_objects_total, apiserver_storage_list_fetched_objects_total, apiserver_storage_list_returned_objects_total, apiserver_storage_list_total, apiserver_watch_cache_events_dispatched_total, apiserver_watch_cache_events_received_total, apiserver_watch_cache_initializations_total, apiserver_watch_cache_resource_version, watch_cache_capacity, apiserver_init_events_total, apiserver_terminated_watchers_total, watch_cache_capacity_increase_total, watch_cache_capacity_decrease_total, apiserver_watch_cache_read_wait_seconds, apiserver_watch_cache_consistent_read_total, apiserver_storage_consistency_checks_total, etcd_bookmark_counts, storage_decode_errors_total extract the API group from resource label and put it in new group label. (#131845, @serathius) [SIG API Machinery, Etcd, Instrumentation and Testing]
  • Kubelet: removed the deprecated flag --cloud-config from the command line. (#130161, @carlory) [SIG Cloud Provider, Node and Scalability]
  • Static pods that reference API objects were denied admission by the kubelet so that static pods would not be silently running even after the mirror pod creation failed. (#131837, @sreeram-venkitesh) [SIG Auth, Node and Testing]
  • The Scheduling Framework exposed NodeInfos to the PreFilter plugins. The PreFilter plugins now accepted the NodeInfo list from the arguments. (#130720, @saintube) [SIG Node, Scheduling, Storage and Testing]

Changes by Kind

Deprecation

  • Apimachinery: Deprecated MessageCountMap and CreateAggregateFromMessageCountMap. (#132376, @tico88612)
  • DRA kubelet: gRPC API graduated to v1, v1beta1 was deprecated starting in 1.34. Updating DRA drivers to the k8s.io/dynamic-resource-allocation/kubeletplugin helper from 1.34 added support for both API versions. (#132700, @pohly) [SIG Node and Testing]
  • Deprecated the preferences field in kubeconfig in favor of kuberc. (#131741, @soltysh) [SIG API Machinery, CLI, Cluster Lifecycle and Testing]
  • Kubeadm: Consistently prefixed errors with error: when printing them. (#132080, @neolit123)
  • Kubeadm: Exposed only the non-deprecated klog flags (-v and -vmodule), in line with KEP https://features.k8s.io/2845. (#131647, @carsontham)
  • [cloud-provider] Respected the exclude-from-external-load-balancers=false label. (#131085, @kayrus) [SIG Cloud Provider and Network]

API Change

  • Added omitempty and opt tag to the API v1beta2 AdminAccess type in the DeviceRequestAllocationResult struct. (#132338, @PatrickLaabs)

  • Added a runtime.ApplyConfiguration interface implemented by all generated apply configuration types. (#132194, @alvaroaleman) [SIG API Machinery and Instrumentation]

  • Added a detailed event for in-place pod vertical scaling completed, improving cluster management and debugging. (#130387, @shiya0705) [SIG API Machinery, Apps, Autoscaling, Node, Scheduling and Testing]

  • Added a mechanism for configurable container restarts: container-level restart rules. This was an alpha feature behind the ContainerRestartRules feature gate. (#132642, @yuanwang04) [SIG API Machinery, Apps, Node and Testing]

  • Added a new FileKeyRef field to containers, allowing them to load variables from files by setting this field.

    Introduced the EnvFiles feature gate to govern activation of this functionality. (#132626, @HirazawaUi) [SIG API Machinery, Apps, Node and Testing]

  • Added driver-owned fields in ResourceSlice to mark whether the device was shareable among multiple resource claims (or requests) and to specify how each capacity could be shared between different requests.

    • Added user-owned fields in ResourceClaim to specify resource requirements against each device capacity.
    • Added scheduler-owned field in ResourceClaim.Status to specify how much device capacity is reserved for a specific request.
    • Added an additional identifier to ResourceClaim.Status for the device supports multiple allocations.
    • Added a new constraint type to enforce uniqueness of specified attributes across all allocated devices. (#132522, @sunya-ch) [SIG API Machinery, Apps, Architecture, CLI, Cluster Lifecycle, Network, Node, Release, Scheduling and Testing]

  • Added new optional APIs in ResouceSlice.Basic and ResourceClaim.Status.AllocatedDeviceStatus. (#130160, @KobayashiD27) [SIG API Machinery, Apps, Architecture, Node, Release, Scheduling and Testing]

  • Added support for specifying controlplane or cluster egress selectors in JWT authenticators via the issuer.egressSelectorType field in the AuthenticationConfiguration.jwt array. If unset, the previous behavior of using no egress selector is preserved. This functionality requires the StructuredAuthenticationConfigurationEgressSelector beta feature gate (enabled by default). (#132768, @enj) [SIG API Machinery, Auth and Testing]

  • Added support in the Kubelet for monitoring the health of devices allocated via Dynamic Resource Allocation (DRA) and report it in the pod.status.containerStatuses.allocatedResourcesStatus field. This required the DRA plugin to implement the new v1alpha1 NodeHealth gRPC service. This feature was controlled by the ResourceHealthStatus feature gate. (#130606, @Jpsassine) [SIG Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Network, Node, Release, Scheduling, Storage and Testing]

  • Added support in the kubelet's image pull credential tracking for service account-based verification. When an image was pulled using service account credentials via external credential providers, subsequent Pods using the same service account (UID, name, and namespace) could access the cached image without re-authentication for the lifetime of that service account. (#132771, @aramase) [SIG Auth, Node and Testing]

  • Added validation to reject Pods using the PodLevelResources feature on Windows OS due to lack of support. The API server rejected Pods with pod-level resources and a Pod.spec.os.name targeting Windows. Kubelet on nodes running Windows also rejected Pods with pod-level resources at the admission phase. (#133046, @toVersus) [SIG Apps and Node]

  • Added warnings when creating headless service with set loadBalancerIP,externalIPs and/or SessionAffinity. (#132214, @Peac36)

  • Allowed pvc.spec.VolumeAttributesClassName to change from non-nil to nil. (#132106, @AndrewSirenko)

  • Allowed setting the hostnameOverride field in PodSpec to specify any RFC 1123 DNS subdomain as the pod's hostname. The HostnameOverride feature gate was introduced to control enablement of this functionality. (#132558, @HirazawaUi) [SIG API Machinery, Apps, Network, Node and Testing]

  • Changed underlying logic for Eviction Manager helper functions. (#132277, @KevinTMtz) [SIG Node, Scheduling and Testing]

  • Changed underlying logic to propagate pod-level hugepage cgroup to containers when they did not specify hugepage resources.

    • Added validation to enforce the hugepage aggregated container limits to be smaller than or equal to pod-level limits. This was already enforced with the defaulted requests from the specified limits, however it did not make it clear about both hugepage requests and limits. (#131089, @KevinTMtz) [SIG Apps, Node and Testing]

  • Corrected the documentation to clarify that podSelector is optional and described its default behavior. (#131354, @tomoish)

  • DRA API: resource.k8s.io/v1alpha3 now only contains DeviceTaintRule. All other types got removed because they became obsolete when introducing the v1beta1 API in 1.32. before updating a cluster where resourceclaims, resourceclaimtemplates, deviceclasses, or resourceslices might have been stored using Kubernetes < 1.32, delete all of those resources before updating and recreate them as needed while running Kubernetes >= 1.32. (#132000, @pohly) [SIG Etcd, Node, Scheduling and Testing]

  • DRA: Starting with Kubernetes 1.34, the alpha-level resource.k8s.io/admin-access label has been updated to resource.kubernetes.io/admin-access. Admins using the alpha feature and updating from 1.33 can set both labels, upgrade, then remove resource.k8s.io/admin-access when no downgrade is going to happen anymore. (#131996, @ritazh) [SIG Node and Testing]

  • DRA: The scheduler plugin prevented abnormal filter runtimes by timing out after 10 seconds. This was configurable via the plugin configuration's FilterTimeout. Setting it to zero disabled the timeout and restored the behavior of Kubernetes <= 1.33. (#132033, @pohly) [SIG Node, Scheduling and Testing]

  • DRA: When the prioritized list feature was used in a request and the resulting number of allocated devices exceeded the number of allowed devices per claim, the scheduler aborted the attempt to allocate devices early. Previously, it tried to many different combinations, which could take a long time. (#130593, @mortent) [SIG Apps, Node, Scheduling and Testing]

  • DRA: removed support for the v1alpha4 kubelet gRPC API (added in 1.31, superseded in 1.32). DRA drivers using the helper package from Kubernetes >= 1.32 use the v1beta1 API and continue to be supported. (#132574, @pohly)

  • Deprecated StreamingConnectionIdleTimeout field of the kubelet config. (#131992, @lalitc375)

  • Dynamic Resource Allocation: Graduated core functionality to general availability (GA). This newly stable feature uses the structured parameters flavor of DRA. (#132706, @pohly) [SIG API Machinery, Apps, Auth, Autoscaling, Etcd, Node, Scheduling and Testing]

  • Enabled kube-apiserver support for PodCertificateRequest and PodCertificate projected volumes (behind the PodCertificateRequest feature gate). (#128010, @ahmedtd) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Storage and Testing]

  • Extended resources backed by DRA feature allowed cluster operator to specify extendedResourceName in DeviceClass, and application operator to continue using extended resources in pod's requests to request for DRA devices matching the DeviceClass.

    NodeResourcesFit plugin scoring didn't work for extended resources backed by DRA. (#130653, @yliaog) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]

  • Extended the NodePorts scheduling plugin to consider hostPorts used by restartable init containers. (#132040, @avrittrohwer) [SIG Scheduling and Testing]

  • Fixed a 1.33 regression that causes a nil panic in kube-scheduler when aggregating resource requested across container's spec and status. (#132895, @yue9944882) [SIG Node and Scheduling]

  • Fixed prerelease lifecycle for PodCertificateRequest. (#133350, @carlory)

  • Introduced OpenAPI format support for k8s-short-name and k8s-long-name in CustomResourceDefinition schemas. (#132504, @jpbetz) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling and Storage]

  • Introduced the admissionregistration.k8s.io/v1beta1/MutatingAdmissionPolicy API type. To enable, enable the MutatingAdmissionPolicy feature gate (which was off by default) and set --runtime-config=admissionregistration.k8s.io/v1beta1=true on the kube-apiserver. Note that the default stored version remained alpha in 1.34, and whoever enabled beta during 1.34 needed to run a storage migration yourself to ensure you don't depend on alpha data in etcd. (#132821, @cici37) [SIG API Machinery, Etcd and Testing]

  • Kube-apiserver: Added support for disabling caching of authorization webhook decisions in the --authorization-config file. The new fields cacheAuthorizedRequests and cacheUnauthorizedRequests could be set to false to prevent caching for authorized or unauthorized requests. See the https://kubernetes.io/docs/reference/access-authn-authz/authorization/#using-configuration-file-for-authorization for more details. (#129237, @rfranzke) [SIG API Machinery and Auth]

  • Kube-apiserver: Promoted the StructuredAuthenticationConfiguration feature gate to GA. (#131916, @aramase) [SIG API Machinery, Auth and Testing]

  • Kube-apiserver: the AuthenticationConfiguration type accepted in --authentication-config files has been promoted to apiserver.config.k8s.io/v1. (#131752, @aramase) [SIG API Machinery, Auth and Testing]

  • Kube-log-runner: Added the -log-file-size parameter to rotate log output into a new file once it reached a certain size. Introduced -log-file-age to enable automatic removal of old output files, and -flush-interval to support periodic flushing. (#127667, @zylxjtu) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Release, Scheduling, Storage, Testing and Windows]

  • Kubectl: Graduated kuberc support to beta. A kuberc configuration file provided a mechanism for customizing kubectl behavior (distinct from kubeconfig, which configures cluster access across different clients). (#131818, @soltysh) [SIG CLI and Testing]

  • Promoted Job Pod Replacement Policy to general availability. The JobPodReplacementPolicy feature gate was locked to true and will be removed in a future Kubernetes release. (#132173, @dejanzele) [SIG Apps and Testing]

  • Promoted MutableCSINodeAllocatableCount to beta. (#132429, @torredil)

  • Promoted feature-gate VolumeAttributesClass to GA

    • Promoted API VolumeAttributesClass and VolumeAttributesClassList to storage.k8s.io/v1. (#131549, @carlory) [SIG API Machinery, Apps, Auth, CLI, Etcd, Storage and Testing]

  • Promoted the APIServerTracing feature gate to GA. The --tracing-config-file flag accepted TracingConfiguration in version apiserver.config.k8s.io/v1 (with no changes from apiserver.config.k8s.io/v1beta1). (#132340, @dashpole) [SIG API Machinery and Testing]

  • Promoted the AuthorizeWithSelectors and AuthorizeNodeWithSelectors feature gates to stable and locked on. (#132656, @liggitt) [SIG API Machinery, Auth and Testing]

  • Promoted the KubeletTracing feature gate to GA. (#132341, @dashpole) [SIG Instrumentation and Node]

  • Promoted the RelaxedEnvironmentVariableValidation feature gate to GA and locked it in the enabled state by default. (#132054, @HirazawaUi) [SIG Apps, Architecture, Node and Testing]

  • Removed an inaccurate statement about requiring ports when the Pod spec hostNetwork field was set. (#130994, @BenTheElder) [SIG Network and Node]

  • Removed deprecated gogo protocol definitions from k8s.io/kubelet/pkg/apis/pluginregistration in favor of google.golang.org/protobuf. (#132773, @saschagrunert)

  • Removed deprecated gogo protocol definitions from k8s.io/cri-api in favor of google.golang.org/protobuf. (#128653, @saschagrunert) [SIG API Machinery, Auth, Instrumentation, Node and Testing]

  • Replaced Boolean-pointer-helper functions with the k8s.io/utils/ptr implementations. (#132794, @PatrickLaabs) [SIG API Machinery, Auth, CLI, Node and Testing]

  • Replaced boolPtrFn helper functions with the "k8s.io/utils/ptr" implementation. (#132907, @PatrickLaabs)

  • Replaced deprecated package k8s.io/utils/pointer with k8s.io/utils/ptr for the apiextensions-apiserver apiextensions. (#132723, @PatrickLaabs)

  • Replaced deprecated package k8s.io/utils/pointer with k8s.io/utils/ptr for the apiserver (1/2). (#132751, @PatrickLaabs) [SIG API Machinery and Auth]

  • Replaced deprecated package k8s.io/utils/pointer with k8s.io/utils/ptr for the component-base. (#132754, @PatrickLaabs) [SIG API Machinery, Architecture, Instrumentation and Scheduling]

  • Replaced deprecated package k8s.io/utils/pointer with k8s.io/utils/ptr for the kube-aggregator apiregistration. (#132701, @PatrickLaabs)

  • Simplied validation error message for invalid fields by removing redundant field name. (#132513, @xiaoweim) [SIG API Machinery, Apps, Auth, Node and Scheduling]

  • Simplied validation error message for required fields by removing redundant messages. (#132472, @xiaoweim) [SIG API Machinery, Apps, Architecture, Auth, Cloud Provider, Network, Node and Storage]

  • The KubeletServiceAccountTokenForCredentialProviders feature was beta and enabled by default. (#133017, @aramase) [SIG Auth and Node]

  • The conditionType is "oneof" approved/denied check of CertificateSigningRequest's .status.conditions field was migrated to declarative validation. If the DeclarativeValidation feature gate was enabled, mismatches with existing validation are reported via metrics. If the DeclarativeValidationTakeover feature gate was enabled, declarative validation was the primary source of errors for migrated fields. (#133013, @aaron-prindle) [SIG API Machinery and Auth]

  • The fallback behavior of the Downward API's resourceFieldRef field was updated to account for pod-level resources: if container-level limits were not set, pod-level limits were now used before falling back to node allocatable resources. (#132605, @toVersus) [SIG Node, Scheduling and Testing]

  • The validation of replicas field in the ReplicationController /scale subresource has been migrated to declarative validation. If the DeclarativeValidation feature gate is enabled, mismatches with existing validation are reported via metrics. If the DeclarativeValidationTakeover feature gate is enabled, declarative validation is the primary source of errors for migrated fields. (#131664, @jpbetz) [SIG API Machinery and Apps]

  • The validation-gen code generator generated validation code that supported validation ratcheting. (#132236, @yongruilin) [SIG API Machinery, Apps, Auth and Node]

  • Updated IsDNS1123SubdomainWithUnderscore so that, when it returned an error, it also returned the correct regex information (dns1123SubdomainFmtWithUnderscore). (#132034, @ChosenFoam)

  • Updated etcd version to v3.6.0. (#131501, @joshjms) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing]

  • Updated the v1 credential provider configuration to include the tokenAttributes.cacheType field. This field is required and must be set to either ServiceAccount or Token when configuring a provider that uses a service account to fetch registry credentials. (#132617, @aramase) [SIG Auth, Node and Testing]

  • Zero-value metadata.creationTimestamp values are now omitted and no longer serialize an explicit null in JSON, YAML, and CBOR output (#130989, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Etcd, Instrumentation, Network, Node, Scheduling, Storage and Testing]

  • AppArmor profiles specified in the Pod or container SecurityContext were no longer copied to deprecated AppArmor annotations (prefix container.apparmor.security.beta.kubernetes.io/). Anything that inspected the deprecated annotations must be migrated to use the SecurityContext fields instead. (#131989, @tallclair)

  • MultiCIDRServiceAllocator was locked and enabled by default, DisableAllocatorDualWrite was enabled by default. (#131318, @aojea) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Network and Testing]

Feature

  • Added 3 new metrics for monitoring async API calls in the scheduler when the SchedulerAsyncAPICalls feature gate was enabled:

    • scheduler_async_api_call_execution_total: tracks executed API calls by call type and result (success/error)
    • scheduler_async_api_call_duration_seconds: histogram of API call execution duration by call type and result
    • scheduler_pending_async_api_calls: gauge showing current number of pending API calls in the queue. (#133120, @utam0k) [SIG Release and Scheduling]

  • Added HPA support to pod-level resource specifications. When the pod-level resource feature was enabled, HPAs configured with Resource type metrics calculated the pod resources from pod.Spec.Resources field, if specified. (#132430, @laoj2) [SIG Apps, Autoscaling and Testing]

  • Added Traffic Distribution field to kubectl describe service output (#131491, @tchap) [SIG CLI]

  • Added SizeBasedListCostEstimate feature gate that allowed apiserver to estimate sizes of objects to calculate cost of LIST requests. (#132355, @serathius) [SIG API Machinery and Etcd]

  • Added apiserver_resource_size_estimate_bytes metric to API server. (#132893, @serathius) [SIG API Machinery, Etcd and Instrumentation]

  • Added started_user_namespaced_pods_total and started_user_namespaced_pods_errors_total for tracking the successes and failures in creating pods if a user namespace was requested. (#132902, @haircommander) [SIG Node and Testing]

  • Added a --show-swap option to kubectl top subcommands (#129458, @iholder101) [SIG CLI]

  • Added a container_swap_limit_bytes metric to expose the swap limit assigned to containers under the LimitedSwap swap behavior. (#132348, @iholder101) [SIG Node and Testing]

  • Added a delay to node updates after kubelet startup. A random offset, based on the configured nodeStatusReportFrequency, helped distribute traffic and load from node status updates more evenly over time. The initial status update could occur up to 50% earlier or later than the regular schedule. (#130919, @mengqiy)

  • Added a flag to kubectl version to detect whether a client/server version mismatch was outside the officially supported range. (#127365, @omerap12)

  • Added a new PreBindPreFlight function to the PreBindPlugin interface. All in-tree PreBind plugins have been updated to implement PreBindPreFlight function. (#132391, @sanposhiho) [SIG Node, Scheduling, Storage and Testing]

  • Added a warning when alpha metrics are used with emulated versions. (#132276, @michaelasp) [SIG API Machinery and Architecture]

  • Added alpha metrics for compatibility versioning (#131842, @michaelasp) [SIG API Machinery, Architecture, Instrumentation and Scheduling]

  • Added configurable flags to kube-apiserver for coordinated leader election. (#132433, @michaelasp) [SIG API Machinery and Testing]

  • Added machine readable output options (JSON & YAML) to kubectl api-resources. (#132604, @dharmit) [SIG Apps, CLI and Network]

  • Added memory tracking to scheduler performance tests to help detect memory leaks and monitored memory usage patterns while running scheduler_perf. (#132910, @utam0k) [SIG Scheduling and Testing]

  • Added support for CEL expressions with escaped names in the structured authentication config. Using [...] to access claims or user data was recommended when names contained characters that would otherwise need escaping. CEL optionals with ? could be used where has was not applicable — for example, claims[?"kubernetes.io"] or user.extra[?"domain.io/foo"]. (#131574, @enj) [SIG API Machinery and Auth]

  • Added support for --cpu, --memory flag to kubectl autoscale, started deprecating --cpu-precent. (#129373, @googs1025)

  • Added support for a new kubectl output format, kyaml. KYAML was a strict subset of YAML and should be accepted by any YAML processor. The formatting of KYAML was halfway between JSON and YAML. Because it was more explicit than the default YAML style, it was less error-prone. (#132942, @thockin) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Contributor Experience, Instrumentation, Network, Node, Scheduling, Storage and Testing]

  • Added the DetectCacheInconsistency feature gate, allowing the API server to periodically verify consistency between its cache and etcd. Detected inconsistencies reported via the apiserver_storage_consistency_checks_total metric and trigger purging of affected cache snapshots. (#132884, @serathius) [SIG API Machinery, Instrumentation and Testing]

  • Added the SizeBasedListCostEstimate feature gate (enabled by default), which changes how APF seats are assigned to LIST requests. With this feature, one seat is assigned per 100KB of data loaded into memory at once during a LIST operation. (#132932, @serathius)

  • Added useful endpoints for kube-apiserver. (#132581, @itssimrank) [SIG API Machinery, Architecture, Instrumentation, Network, Node, Scheduling and Testing]

  • Built Kubernetes using Go 1.24.3. (#131934, @cpanato) [SIG Release and Testing]

  • Built Kubernetes using Go 1.24.4. (#132222, @cpanato) [SIG Release and Testing]

  • Bumped DRA API version to v1 in deviceattribute package in k8s.io/dynamic-resource-allocation. (#133164,

These notes run past the length kept in the archive. The rest is on the publisher’s page.