1.33.0

Changelog since v1.32.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Added the ability to reduce both the initial delay and the maximum delay accrued between container restarts for a node for containers in CrashLoopBackOff across the cluster to the recommended values of 1s initial delay and 60s maximum delay. To set this for a node, turn on the feature gate ReduceDefaultCrashLoopBackOffDecay. If you are also using the feature gate KubeletCrashLoopBackOffMax with a configured per-node CrashLoopBackOff.MaxContainerRestartPeriod, the effective kubelet configuration will follow the conflict resolution policy described further in the documentation here. (#130711, @lauralorenz) [SIG Node and Testing]
• [Action Required] CSI drivers that call IsLikelyNotMountPoint should not assume false means that the path is a mount point. Each CSI driver needs to make sure correct usage of return value of IsLikelyNotMountPoint because if the file is an irregular file but not a mount point is acceptable (#129370, @andyzhangx) [SIG Storage and Windows]
• Fixed the behavior of the KUBE_PROXY_NFTABLES_SKIP_KERNEL_VERSION_CHECK environment variable in the nftables proxier. The kernel version check is now skipped only when this variable is explicitly set to a non-empty value. To skip the check, set the KUBE_PROXY_NFTABLES_SKIP_KERNEL_VERSION_CHECK environment variable. (#130401, @ryota-sakamoto)
• Renamed UpdatePodTolerations action type to UpdatePodToleration. Action required for custom plugin developers to update their code to follow the rename. (#129023, @zhifei92) [SIG Scheduling and Testing]

Changes by Kind

Deprecation

• The EndpointSlice hints field has graduated to GA. The beta annotation service.kubernetes.io/topology-mode is now considered deprecated and will not graduate to GA. It remains operational for backward compatibility. Users are encouraged to use the spec.trafficDistribution field in the Service API for topology-aware routing configuration. (#130742, @gauravkghildiyal) [SIG Network]
• The StorageCapacityScoring feature gate was added to score nodes by available storage capacity. It's in alpha and disabled by default. The VolumeCapacityPriority alpha feature was replaced with this, and the default behavior was changed. The VolumeCapacityPriority preferred a node with the least allocatable, but the StorageCapacityScoring preferred a node with the maximum allocatable. See KEP-4049 for details. (#128184, @cupnes) [SIG Scheduling, Storage and Testing]
• The WatchFromStorageWithoutResourceVersion feature was deprecated and can no longer be enabled. (#129930, @serathius)
• The pod status.resize field is now deprecated and will no longer be set. The status of a pod resize will be exposed under two new conditions: PodResizeInProgress and PodResizePending instead. (#130733, @natasha41575) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
• The v1 Endpoints API is now officially deprecated (though still fully supported). The API will not be removed, but all users should use the EndpointSlice API instead. (#130098, @danwinship) [SIG API Machinery and Network]

API Change

• A new alpha feature gate, MutableCSINodeAllocatableCount, has been introduced.

  When this feature gate is enabled, the CSINode.Spec.Drivers[*].Allocatable.Count field becomes mutable, and a new field, NodeAllocatableUpdatePeriodSeconds, is available in the CSIDriver object. This allows periodic updates to a node's reported allocatable volume capacity, preventing stateful pods from becoming stuck due to outdated information that kube-scheduler relies on. (#130007, @torredil) [SIG Apps, Node, Scheduling and Storage]

• Added feature gate DRAPartitionableDevices, when enabled, Dynamic Resource Allocation support partitionable devices allocation. (#130764, @cici37) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Scheduling, Storage and Testing]

• Added DRA support for a "one-of" prioritized list of selection criteria to satisfy a device request in a resource claim. (#128586, @mortent) [SIG API Machinery, Apps, Etcd, Node, Scheduling and Testing]

• Added a /flagz endpoint for kubelet endpoint (#128857, @zhifei92) [SIG Architecture, Instrumentation and Node]

• Added a new tolerance field to HorizontalPodAutoscaler, overriding the cluster-wide default. Enabled via the HPAConfigurableTolerance alpha feature gate. (#130797, @jm-franc) [SIG API Machinery, Apps, Autoscaling, Etcd, Node, Scheduling and Testing]

• Added support for configuring custom stop signals with a new StopSignal container lifecycle (#130556, @sreeram-venkitesh) [SIG API Machinery, Apps, Node and Testing]

• Added support for in-place vertical scaling of Pods with sidecars (containers defined within initContainers where the restartPolicy is set to Always). (#128367, @vivzbansal) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]

• CPUManager Policy Options support is GA (#130535, @ffromani) [SIG API Machinery, Node and Testing]

• Changed the Pod API to support hugepage resources at spec level for pod-level resources. (#130577, @KevinTMtz) [SIG Apps, CLI, Node, Scheduling, Storage and Testing]

• DRA API: The maximum number of pods that can use the same ResourceClaim is now 256 instead of 32. Downgrading a cluster where this relaxed limit is in use to Kubernetes 1.32.0 is not supported, as version 1.32.0 would refuse to update ResourceClaims with more than 32 entries in the status.reservedFor field. (#129543, @pohly) [SIG API Machinery, Node and Testing]

• DRA: CEL expressions using attribute strings exceeded the cost limit because their cost estimation was incomplete. (#129661, @pohly) [SIG Node]

• DRA: Device taints enable DRA drivers or admins to mark device as unusable, which prevents allocating them. Pods may also get evicted at runtime if a device becomes unusable, depending on the severity of the taint and whether the claim tolerates the taint. (#130447, @pohly) [SIG API Machinery, Apps, Architecture, Auth, Etcd, Instrumentation, Node, Scheduling and Testing]

• DRA: Starting Kubernetes 1.33, only users with access to an admin namespace with the kubernetes.io/dra-admin-access label are authorized to create ResourceClaim or ResourceClaimTemplate objects with the adminAccess field in this admin namespace if they want to and only they can reference these ResourceClaims or ResourceClaimTemplates in their pod or deployment specs. (#130225, @ritazh) [SIG API Machinery, Apps, Auth, Node and Testing]

• DRA: when asking for "All" devices on a node, Kubernetes <= 1.32 proceeded to schedule pods onto nodes with no devices by not allocating any devices for those pods. Kubernetes 1.33 changes that to only picking nodes which have at least one device. Users who want the "proceed with scheduling also without devices" semantic can use the upcoming prioritized list feature with one sub-request for "all" devices and a second alternative with "count: 0". (#129560, @bart0sh) [SIG API Machinery and Node]

• Expanded the on-disk kubelet credential provider configuration to allow an optional tokenAttribute field to be configured. When it is set, the kubelet will provision a token with the given audience bound to the current pod and its service account. This KSA token along with required annotations on the KSA defined in configuration will be sent to the credential provider plugin via its standard input (along with the image information that is already sent today). The KSA annotations to be sent are configurable in the kubelet credential provider configuration. (#128372, @aramase) [SIG API Machinery, Auth, Node and Testing]

• Fixed the example validation rule in godoc:

  When configuring a JWT authenticator:

  If username.expression uses 'claims.email', then 'claims.email_verified' must be used in username.expression or extra[].valueExpression or claimValidationRules[].expression. An example claim validation rule expression that matches the validation automatically applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true) == true'. By explicitly comparing the value to true, we let type-checking see the result will be a boolean, and to make sure a non-boolean email_verified claim will be caught at runtime. (#130875, @aramase) [SIG Auth and Release]

• For the InPlacePodVerticalScaling feature, the API server will no longer set the resize status to Proposed upon receiving a resize request. (#130574, @natasha41575) [SIG Apps, Node and Testing]

• Graduate the MatchLabelKeys (MismatchLabelKeys) feature in PodAffinity (PodAntiAffinity) to GA (#130463, @sanposhiho) [SIG API Machinery, Apps, Node, Scheduling and Testing]

• Graduated image volume sources to beta:

  • Allowed subPath/subPathExpr for image volumes
  • Added kubelet metrics kubelet_image_volume_requested_total, kubelet_image_volume_mounted_succeed_total and kubelet_image_volume_mounted_errors_total (#130135, @saschagrunert) [SIG API Machinery, Apps, Node and Testing]

• Implemented a new status field, .status.terminatingReplicas, for Deployments and ReplicaSets to track terminating pods. The new field is present when the DeploymentPodReplacementPolicy feature gate is enabled. (#128546, @atiratree) [SIG API Machinery, Apps and Testing]

• Implemented validation for NodeSelectorRequirement values in Kubernetes when creating pods. (#128212, @AxeZhan) [SIG Apps and Scheduling]

• Improved how the API server responds to list requests where the response format negotiates to Protobuf. List responses in Protobuf are marshalled one element at the time, drastically reducing memory needed to serve large collections. Streaming list responses can be disabled via the StreamingCollectionEncodingToProtobuf feature gate. (#129407, @serathius) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Network, Node, Release, Scheduling, Storage and Testing]

• InPlacePodVerticalScaling: Memory limits cannot be decreased unless the memory resize restart policy is set to RestartContainer. Container resizePolicy is no longer mutable. (#130183, @tallclair) [SIG Apps and Node]

• Introduced API type coordination.k8s.io/v1beta1/LeaseCandidate CoordinatedLeaderElection feature moves to Beta (#130751, @Jefftree) [SIG API Machinery, Etcd and Testing]

• Introduced API type coordination.k8s.io/v1beta1/LeaseCandidate (#130291, @Jefftree) [SIG API Machinery, Etcd and Testing]

• It introduces a new scope name VolumeAttributesClass.

  It matches all PVC objects that have the volume attributes class mentioned.

  If you want to limit the count of PVCs that have a specific volume attributes class. In that case, you can create a quota object with the scope name VolumeAttributesClass and a matchExpressions that match the volume attributes class. (#124360, @carlory) [SIG API Machinery, Apps and Testing]

• KEP-3857: Recursive Read-only (RRO) mounts: promote to GA (#130116, @AkihiroSuda) [SIG Apps, Node and Testing]

• kubectl: Added alpha support for customizing kubectl behavior using preferences from a kuberc file, separate from kubeconfig. (#125230, @ardaguclu) [SIG API Machinery, CLI and Testing]

• kubelet: added KubeletConfiguration.subidsPerPod. (#130028, @AkihiroSuda) [SIG API Machinery and Node]

• Kubernetes components that accepted X.509 client certificate authentication now read the user UID from a certificate subject name RDN with object ID 1.3.6.1.4.1.57683.2. An RDN with this object ID had to contain a string value and appear no more than once in the certificate subject. Reading the user UID from this RDN could be disabled by setting the beta feature gate AllowParsingUserUIDFromCertAuth to false(until the feature gate graduated to GA). (#127897, @modulitos) [SIG API Machinery, Auth and Testing]

• MergeDefaultEvictionSettings indicates that defaults for the evictionHard, evictionSoft, evictionSoftGracePeriod, and evictionMinimumReclaim fields should be merged into values specified for those fields in this configuration. Signals specified in this configuration take precedence. Signals not specified in this configuration inherit their defaults. (#127577, @vaibhav2107) [SIG API Machinery and Node]

• New configuration is introduced to the kubelet that allows it to track container images and the list of authentication information that leads to their successful pulls. This data is persisted across reboots of the host and restarts of the kubelet.

  The kubelet ensures any image requiring credential verification is always pulled if authentication information from an image pull is not yet present, thus enforcing authentication / re-authentication. This means an image pull might be attempted even in cases where a pod requests the IfNotPresent image pull policy, and might lead to the pod not starting if its pull policy is Never and is unable to present authentication information that led to a previous successful pull of the image it is requesting. (#128152, @stlaz) [SIG API Machinery, Architecture, Auth, Node and Testing]

• Promoted JobSuccessPolicy E2E to Conformance (#130658, @tenzen-y) [SIG API Machinery, Apps, Architecture and Testing]

• Promoted NodeInclusionPolicyInPodTopologySpread to Stable in v1.33 (#130920, @kerthcet) [SIG Apps, Node, Scheduling and Testing]

• Promoted the JobSuccessPolicy to Stable. (#130536, @tenzen-y) [SIG API Machinery, Apps, Architecture and Testing]

• Promoted the Job's JobBackoffLimitPerIndex feature-gate to stable. (#130061, @mimowo) [SIG API Machinery, Apps, Architecture and Testing]

• Promoted the feature gate AnyVolumeDataSource to GA. (#129770, @sunnylovestiramisu) [SIG Apps, Storage and Testing]

• Removed general available feature gate CPUManager. (#129296, @carlory) [SIG API Machinery, Node and Testing]

• Removed general available feature-gate PDBUnhealthyPodEvictionPolicy. (#129500, @carlory) [SIG API Machinery, Apps and Auth]

• Start reporting swap capacity as part of node.status.nodeSystemInfo. (#129954, @iholder101) [SIG API Machinery, Apps and Node]

• Graduated the MultiCIDRServiceAllocator feature gate to stable, and the DisableAllocatorDualWrite feature gate to beta (disabled by default). Action required for Kubernetes cluster administrators and for distributions that manage the cluster Service CIDR. Kubernetes now allows users to define the cluster Service CIDR via an API object: ServiceCIDR. Distributions or administrators of Kubernetes may want to control that new Service CIDRs added to the cluster do not overlap with other networks on the cluster, that only belong to a specific range of IPs. Administrators may also prefer to retain the existing behavior of only having one ServiceCIDR per cluster. You can use ValidatingAdmissionPolicy to achieve this. (#128971, @aojea) [SIG Apps, Architecture, Auth, CLI, Etcd, Network, Release and Testing]

• The ClusterTrustBundle API is moving to v1beta1. In order for the ClusterTrustBundleProjection feature to work on the kubelet side, the ClusterTrustBundle API must be available at v1beta1 version and the ClusterTrustBundleProjection feature gate must be enabled. If the API becomes later after kubelet started running, restart the kubelet to enable the feature. (#128499, @stlaz) [SIG API Machinery, Apps, Auth, Etcd, Node, Storage and Testing]

• The Service trafficDistribution field, including the PreferClose option, has graduated to GA. Services that do not have the field configured will continue to operate with their existing behavior. Refer to the documentation https://kubernetes.io/docs/concepts/services-networking/service/#traffic-distribution for more details. (#130673, @gauravkghildiyal) [SIG Apps, Network and Testing]

• The feature gate InPlacePodVerticalScalingAllocatedStatus is deprecated and no longer used. The AllocatedResources field in ContainerStatus is now guarded by the InPlacePodVerticalScaling feature gate. (#130880, @tallclair) [SIG CLI, Node and Scheduling]

• The kube-controller-manager will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (#130650, @natasha41575) [SIG API Machinery, Apps, Node, Scheduling, Storage, Testing and Windows]

• The kube-scheduler will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (#130649, @natasha41575) [SIG Node, Scheduling and Testing]

• The kubelet will set the observedGeneration field on pod conditions when the PodObservedGenerationTracking feature gate is set. (#130573, @natasha41575) [SIG Apps, Node, Scheduling, Storage, Testing and Windows]

• The minimum value validation of ReplicationController's replicas and minReadySeconds fields have been migrated to declarative validation. The requiredness of both fields is also declaratively validated. If the DeclarativeValidation feature gate is enabled, mismatches with existing validation are reported via metrics. If the DeclarativeValidationTakeover feature gate is enabled, declarative validation is the primary source of errors for migrated fields. (#130725, @jpbetz) [SIG API Machinery, Apps, Architecture, CLI, Cluster Lifecycle, Instrumentation, Network, Node and Storage]

• The resource.k8s.io/v1beta1 API is deprecated and will be removed in 1.36. Use v1beta2 instead. (#129970, @mortent) [SIG API Machinery, Apps, Auth, Etcd, Node, Scheduling and Testing]

• Validation now requires new StatefulSets with a .spec.serviceName field value to pass DNS1123 validation. Previously created StatefulSets with an invalid .spec.serviceName field value could not create any pods, and should be deleted.

  • Published OpenAPI for the StatefulSet schema is corrected to indicate the .spec.serviceName is optional. (#130233, @soltysh) [SIG API Machinery, Apps and Testing]

• When the PreferSameTrafficDistribution feature gate is enabled, a new trafficDistribution value PreferSameNode is available, which attempts to always route Service connections to an endpoint on the same node as the client. Additionally, PreferSameZone is introduced as an alias for PreferClose. (#130844, @danwinship) [SIG API Machinery, Apps, Network and Windows]

• When the PodObservedGenerationTracking feature gate was set, the kubelet populated status.observedGeneration to reflect the latest metadata.generation it observed for the pod. (#130352, @natasha41575) [SIG API Machinery, Apps, CLI, Node, Release, Scheduling, Storage, Testing and Windows]

• When the StrictIPCIDRValidation feature gate is enabled, Kubernetes will be slightly stricter about what values will be accepted as IP addresses and network address ranges (“CIDR blocks”).

  In particular, octets within IPv4 addresses are not allowed to have any leading 0s, and IPv4-mapped IPv6 values (e.g. ::ffff:192.168.0.1) are forbidden. These sorts of values can potentially cause security problems when different components interpret the same string as referring to different IP addresses (as in CVE-2021-29923).

  This tightening applies only to fields in built-in API kinds, and not to custom resource kinds, values in Kubernetes configuration files, or command-line arguments.

  (When the feature gate is disabled, creating an object with such an invalid IP or CIDR value will result in a warning from the API server about the fact that it will be rejected in the future.) (#122550, #128786, @danwinship) [SIG API Machinery, Apps, Network, Node, Scheduling and Testing]

• apidiscovery.k8s.io/v2beta1 API group is disabled by default (#130347, @Jefftree) [SIG API Machinery and Testing]

• kubectl apply now coerces null values for labels and annotations in manifests to empty string values, consistent with typed JSON metadata decoding, rather than dropping all labels and annotations (#129257, @liggitt) [SIG API Machinery]

Feature

• Added ListFromCacheSnapshot feature gate that allows apiserver to serve LISTs with exact RV and continuations from cache (#130423, @serathius) [SIG API Machinery, Etcd and Testing]
• Added Pressure Stall Information (PSI) metrics to node metrics. (#130701, @roycaihw) [SIG Node and Testing]
• Added Windows Server, Version 2025 for windows-servercore-cache test image (#130935, @aramase) [SIG Testing and Windows]
• Added metrics to expose the main known reasons for resource alignment errors (#129950, @ffromani) [SIG Node and Testing]
• Added SchedulerPopFromBackoffQ feature gate that is in beta and enabled by default. Improved scheduling queue behavior by popping pods from the backoffQ when the activeQ is empty. This allows to process potentially schedulable pods ASAP, eliminating a penalty effect of the backoff queue. (#130772, @macsko) [SIG Scheduling and Testing]
• Added apiserver.latency.k8s.io/authentication annotation to the audit log to record the time spent authenticating slow requests. Also added apiserver.latency.k8s.io/authorization annotation to record the time spent authorizing slow requests. (#130571, @hakuna-matatah)
• Added a /flagz endpoint for kube-proxy (#128985, @yongruilin) [SIG Instrumentation and Network]
• Added a /status endpoint for kube-proxy (#128989, @Henrywu573) [SIG Instrumentation and Network]
• Added a /statusz HTTP endpoint to the kube-scheduler. (#128818, @yongruilin) [SIG Architecture, Instrumentation, Scheduling and Testing]
• Added a /statusz HTTP endpoint to the kubelet. (#128811, @zhifei92) [SIG Architecture, Instrumentation and Node]
• Added a /statusz endpoint for kube-controller-manager (#128991, @Henrywu573) [SIG API Machinery, Cloud Provider, Instrumentation and Testing]
• Added a /statusz endpoint for kube-scheduler (#128987, @Henrywu573) [SIG Instrumentation, Scheduling and Testing]
• Added a mechanism that calculates a digest of etcd and the watch cache every 5 minutes and exposes it as the apiserver_storage_digest metric. (#130475, @serathius) [SIG API Machinery, Instrumentation and Testing]
• Added a new CLI flag --emulation-forward-compatible Added a new CLI --runtime-config-emulation-forward-compatible (#130354, @siyuanfoundation) [SIG API Machinery, Etcd and Testing]
• Added a new option strict-cpu-reservation for CPU Manager static policy. When this option is enabled, CPU cores in reservedSystemCPUs will be strictly used for system daemons and interrupt processing no longer available for any workload. (#130290, @psasnal) [SIG Node and Testing]
• Added an alpha feature gate OrderedNamespaceDeletion. When enabled, the pods resources are deleted before all other resources during namespace deletion. (#130035, @cici37) [SIG API Machinery, Apps and Testing]
• Added e2e tests for volume group snapshots. (#128972, @manishym) [SIG Cloud Provider, Storage and Testing]
• Added unit test helpers to validate CEL and patterns in CustomResourceDefinitions. (#129028, @sttts)
• Added validation of containerLogMaxFiles within kubelet configuration files. (#129072, @kannon92)
• Adding resource completion in kubectl debug command (#130033, @ardaguclu) [SIG CLI]
• Adds a /flagz endpoint for kube-controller-manager endpoint (#128824, @yongruilin) [SIG API Machinery and Instrumentation]
• Allowed ImageVolume for Restricted PSA profiles. (#130394, @Barakmor1)
• Allowed dynamic configuration of the service account name and audience that the kubelet could request a token for, as part of the node audience restriction feature. (#130485, @aramase) [SIG Auth and Testing]
• Automatically copy topology.k8s.io/zone, topology.k8s.io/region and kubernetes.io/hostname labels from Node objects to Pods when they are scheduled to a node (via the pods/binding endpoint) to allow applications that need to be explicitly aware of their assigned node topology to access this information via the downward API, rather than requiring permission to get node objects (exposing the entire API surface of the Node object to otherwise unprivileged workloads). (#127092, @munnerz) [SIG API Machinery, Node and Testing]
• Bumped ProcMountType feature to on by default beta (#130798, @haircommander) [SIG Node]
• Calculated pod resources are now cached when adding pods to NodeInfo in the scheduler framework, improving performance when processing unschedulable pods. (#129635, @macsko) [SIG Scheduling]
• cel-go has been bumped to v0.23.2. (#129844, @cici37) [SIG API Machinery, Auth, Cloud Provider and Node]
• Changed metadata management for Pods to populate .metadata.generation on writes. New pods will have a metadata.generation of 1; updates to mutable fields in the Pod .spec will result in metadata.generation being incremented by 1. (#130181, @natasha41575) [SIG Apps, Node and Testing]
• DRA: Starting Kubernetes 1.33, regular users with namespaced cluster edit role assigned have read permission to resourceclaims, resourceclaims/status,resourceclaimtemplates. And write permission for resourceclaims, resourceclaimtemplates. (#130738, @ritazh) [SIG Auth]
• DRAResourceClaimDeviceStatus is now turned on by default allowing DRA-Drivers to report device status data for each allocated device. (#130814, @LionelJouin) [SIG Network and Node]
• DistributeCPUsAcrossNUMA policy option is promoted to Beta. (#130541, @swatisehgal) [SIG Node]
• Enabled the OrderedNamespaceDeletion feature gate by default. (#130507, @cici37) [SIG API Machinery and Apps]
• Enabled user namespaces support (feature gate UserNamespacesSupport) by default. (#130138, @rata) [SIG Node and Testing]
• Endpoints resources created by the Endpoints controller now include a label indicating this. Users who manually create Endpoints can also add this label, but they should consider using