Feature/test-k8s-1.36.5 by riadelhajjaji-scality · Pull Request #5146 · scality/metalk8s

riadelhajjaji-scality · 2026-10-01T14:15:53Z

Component:

kubernetes, salt, docs

Context:

MetalK8s 134.0 moves from Kubernetes 1.34 to 1.36.5

1.35 and 1.36 bring changes that break MetalK8s as-is:

    - the kubelet `--pod-infra-container-image` flag was removed, and the kubelet                                                                                                                                                    
      refuses to start with it                                                                                                                                                                                                       
    - the kubelet refuses to start on cgroup v1 hosts by default, and EL8 still                                                                                                                                                      
      uses cgroup v1                                                                                                                                                                                                                 
    - `kubectl exec` now requires `--` before the command                                                                                                                                                                            
    - some feature gates were removed or locked, and kube-apiserver refuses to                                                                                                                                                       
      start if they are set

Summary:

  - Bump Kubernetes to 1.36.5, CoreDNS to 1.14.2 and pause to 3.10.2                                                                                                                                                               
  - Drop `--pod-infra-container-image` from the kubelet flags (containerd                                                                                                                                                          
    already sets the sandbox image)                                                                                                                                                                                                
  - Set `failCgroupV1: false` in the kubelet config so EL8 nodes keep working                                                                                                                                                      
  - Add `--` to the `kubectl exec` commands in the docs                                                                                                                                                                            
  - Add an upgrade precheck that refuses to start when                                                                                                                                                                             
    `kubernetes.apiServer.featureGates` contains a gate kube-apiserver 1.36                                                                                                                                                        
    would reject, and names the gates to remove from the Bootstrap config.                                                                                                                                                         
    Without it, the upgrade fails halfway, with the API server down.                                                                                                                                                               
  - Add `tools/kube-feature-gates/generate.py` to build the gate lists from                                                                                                                                                        
    upstream data, and document refreshing them on each bump in BUMPING.md

Acceptance criteria:

  - Fresh install works on EL8 (cgroup v1) and EL9 (cgroup v2)                                                                                                                                                                     
  - Upgrade from 134.x works, single node and multi-node

Closes: ARTESCA-17840

-->

bert-e · 2026-10-01T14:15:58Z

Hello riadelhajjaji-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/bypass_source_branch_lineage`	Bypass the cross-branch contamination check	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request.
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-10-01T14:16:06Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

claude · 2026-10-01T14:18:26Z

Upgrading 133.x (Kubernetes 1.33.7) to 134.0 (1.36.5) moves kube-apiserver up 3 minors in one step. Upstream doesn't support this, and it breaks the apiserver skew limit during a multi-node rolling upgrade.
- Upgrade through 1.34 and 1.35 first (in the orchestration or with an intermediate MetalK8s release)
- Add an upgrade precheck that enforces this
- No changelog entry. .changes/unreleased/ still says "Bump Kubernetes version to 1.34.7" (PR feat: bump Kubernetes to 1.34.7 (and etcd to 3.6.11) #4928)
- Update that entry, or add a changie entry, for Kubernetes 1.36.5, CoreDNS 1.14.2 and pause 3.10.2
- Add a Breaking/Enhanced entry for the new upgrade precheck that rejects removed or locked kubernetes.apiServer.featureGates, since users may need to edit their Bootstrap config before upgrading

Review by Claude Code

riadelhajjaji-scality · 2026-10-01T14:32:39Z

/changie Enhanced Bump Kubernetes version to 1.36.5 (https://github.com/kubernetes/kubernetes/releases/tag/v1.36.5)

claude · 2026-10-01T14:38:49Z

LGTM

Review by Claude Code

eg-ayoub

good work! I left some comments of things to consider for this kind of PR 👍🏼 🚀

bert-e · 2026-10-05T08:22:54Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-10-06T12:15:44Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

claude · 2026-10-06T14:36:04Z

Changelog body in .changes/unreleased/Enhanced-20261001-143303-842567940.yaml contains literal double quotes
- Edit the /changie comment so the body is unquoted, e.g. Bump Kubernetes version to [1.36.5](https://github.com/kubernetes/kubernetes/releases/tag/v1.36.5)
- The existing thread on the 1.33/1.34 to 1.36 multi-minor control plane jump (versions.py:22) is still open and unaddressed
- Add intermediate Kubernetes steps or require an intermediate MetalK8s release, enforced by a precheck

Review by Claude Code

claude · 2026-10-06T15:55:27Z

LGTM

Review by Claude Code

bert-e · 2026-10-06T18:41:20Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

bert-e · 2026-10-07T16:27:56Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-10-08T08:18:45Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

claude · 2026-10-08T08:20:26Z

The Kubernetes control plane jumps two minors (1.34 to 1.36) in one upgrade, and kube-apiservers end up 2 minors apart during the rolling apiserver upgrade (replied in the existing thread on buildchain/buildchain/versions.py)
- Ship 1.35 in an intermediate MetalK8s release, or add a 1.35 step to the upgrade orchestration
- Add a precheck that refuses an upgrade crossing more than one Kubernetes minor

Review by Claude Code

- Update kubernetes components and hashes. - Drop --pod-infra-container-image, removed from kubelet and fatal as an unknown flag. containerd already sets the sandbox image. - Set failCgroupV1: false, since kubelet refuses to start on cgroup v1 hosts by default since 1.35 and EL8 still uses cgroup v1.

…valid Some feature gates set in the Bootstrap config are removed or locked in Kubernetes 1.36, and kube-apiserver refuses to start with them. The upgrade precheck now catches these gates and tells the user which ones to remove, before anything is changed on the cluster. The gate lists are generated by tools/kube-feature-gates/generate.py and must be refreshed on each Kubernetes bump (see BUMPING.md).

…rnetes/kubernetes/releases/tag/v1.36.5) Auto-generated-by: changie-add-changelog

github-actions · 2026-10-08T11:23:51Z

✅ Changelog check passed.

claude · 2026-10-08T11:25:20Z

Upgrade still moves the control plane from 1.34 to 1.36 in one step, skipping 1.35. During the rolling upgrade a 1.36 kube-apiserver runs next to 1.34 ones, and upstream skew policy allows only 1 minor between apiservers. The existing thread on buildchain/buildchain/versions.py was resolved without a reply.
- Add an intermediate 1.35 step to the upgrade orchestrate, or require an intermediate MetalK8s release (enforced by a precheck)
- Or reply in the thread with why this skew is safe here

Review by Claude Code

riadelhajjaji-scality requested a review from a team as a code owner October 1, 2026 14:15

riadelhajjaji-scality requested a review from ezekiel-alexrod October 1, 2026 14:16

claude Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread buildchain/buildchain/versions.py

riadelhajjaji-scality changed the title ~~Feature/test k8s 136.5~~ Feature/test-k8s-1.36.5 Oct 1, 2026