feat(ci): validate & self-heal playbook prerequisites on runners before tests by harkgill-amd · Pull Request #858 · amd/playbooks

harkgill-amd · 2026-09-02T16:04:18Z

What

Adds a pre-flight prerequisite gate that runs on the runner before each playbook's tests. For every in-scope @require dependency it runs a validate → (if missing) install → re-validate loop:

validate passes → no-op (provisioned runner, hot path)
validate fails + install fixes it → self-healed, tests proceed
validate fails + can't fix → fails fast with a "provisioning issue, NOT a playbook bug" banner

Why

We're seeing alot of issues when implementing new playbooks (#615) or testing existing playbooks (#838) that relate to test system setup. The current expectation is that the test system already has the dependencies pre-installed but we fail to document this or provide a simple mechanism to ensure this is done.

This simple validation script will only install dependencies in the case where a system is missing them, better handling the current status quo where this case simply fails without indicating missing deps. It'll also help automatically setup all necessary dependencies on fresh runners for our shift to OrchestrAI.

Coverage (uneven by design; extensible)

validate+install: lemonade, podman, models, nodejs, n8n
validate-only: lmstudio (until CI safe install steps determined)
unchecked no-op: driver, pytorch, comfyui, rocm, etc. — pass through until commands are added to the registry

We can either expand on each dependency (giving it tested validate and install commands in playbooks/dependencies/registry.json) in this PR or in a followup.

adamlam2-amd

In theory it makes sense to me - although I didn't review every single change made.

adamlam2-amd · 2026-09-02T16:51:54Z

seems like much of the CI is failing though... do we know why?

harkgill-amd · 2026-09-02T17:31:52Z

seems like much of the CI is failing though... do we know why?

While the skeleton framework of "look at @requires" then validate prereqs is correct, the actual validate and install step needs to be tuned for each dependency. Working through the failing ones here but they'll need to be enabled on a rolling basis.

harkgill-amd · 2026-09-02T17:55:57Z

Seeing unrelated failures such as

Exit code: 1
Duration: 0.01s
STDOUT (last 10000 chars):
OK: Lemonade server is responding
Model Qwen3-Coder-30B-A3B-Instruct-GGUF is not present in Lemonade /api/v1/models.

on Test gaia-agents (linux/r9700)

This is exactly what the PR aims to resolve, will bundle in @requires checks and dependency validate + install steps for other playbooks.

@require

Add a pre-flight gate (.github/scripts/prereq_validate.py) that runs before each playbook's tests, driven by validate/install fields in dependencies/registry.json. For every @require'd dep it runs a validate -> (if missing) install -> re-validate loop: a provisioned runner does nothing, a cold runner self-heals, and only genuinely unresolvable prereqs fail the job with a clear "provisioning issue, not a playbook bug" message. Covers the missing-model / missing-binary gaps behind the recent P0/P1 playbook failures (n8n, openclaw, hermes, ollama, gaia, lemonade-getting-started, deepseek-v4-flash-ds4, lmstudio, vscode): lemonade + ollama models, and the nodejs/n8n/openclaw/hermes/ollama/distrobox/ds4-cockpit/ds4-toolbox binaries. nodejs self-heals via NodeSource 24 on Linux (purging conflicting distro packages first), accepts pre-provisioned Homebrew node, and is skipped on Windows. LM Studio model checks are marked optional (warn, don't hard-fail) since their CLI needs a running backend our check can't always reach.

… surface broken lms backend on r9700

lucbruni-amd

LGTM. In favour of landing this quickly to enable self-healing on our runners which are frequently plagued, and opting to fix-forward any issues that arise.

…(CodeQL)

harkgill-amd · 2026-09-08T18:24:13Z

@sreeram-11, could you please review when you get a chance?

sreeram-11 · 2026-09-09T18:28:27Z

@sreeram-11, could you please review when you get a chance?

@harkgill-amd, thank you for this PR!

As @adamlam2-amd mentioned, based on the PR description this looks good to me. That said, I'll need a bit more time to review the code thoroughly before I can approve.

A couple of scenarios I wanted to raise:

Prereqs not declared via @require: There are cases where prerequisites aren't always specified through @require. They might just be mentioned informally in the playbook instructions or the README.md. How would we handle that scenario?
Custom runner images (e.g., a Halo box as a runner with its own image): If we add a Halo box as a runner with its own image, and there's something wrong with its dependencies, we wouldn't want CI to try to fix it there, right? Based on this PR, it looks like we'd be attempting to modify that custom image, which may not be desirable.

harkgill-amd · 2026-09-10T14:41:28Z

A couple of scenarios I wanted to raise:

Prereqs not declared via @require: There are cases where prerequisites aren't always specified through @require. They might just be mentioned informally in the playbook instructions or the README.md. How would we handle that scenario?

Custom runner images (e.g., a Halo box as a runner with its own image): If we add a Halo box as a runner with its own image, and there's something wrong with its dependencies, we wouldn't want CI to try to fix it there, right? Based on this PR, it looks like we'd be attempting to modify that custom image, which may not be desirable.

We should try to maintain a standard where all deps are marked with a require. It'll help keep both the CI and user experience consistent across playbooks. If there are cases where this isn't possible, we can look at them specifically and determine how to go forward (maybe branch requires into seperate user facing and CI tags).
Yeah, this is something we were running into. My POV is if a dependency is pre-installed on a HaloBox image, we don't try to self-heal it and only validate. We'd be losing out on the ability to self heal that same dep on regular runners. The alternative is adding in gating checks for specific deps not to self heal if a system is deemed as using a custom runner image.

…r openhands

sreeram-11 · 2026-09-10T22:24:01Z


 <!-- @os:linux -->
-<!-- @require:lemonade,podman -->
+<!-- @require:lemonade,podman,nodejs,n8n -->


@adamlam2-amd,

Why was nodejs not within @require previously for Linux? Can you please confirm if this addition is correct?

On main Linux was @require:lemonade,podman and Windows lemonade,nodejs. Linux was intended to run n8n containerized via Podman, so host Node wasn't needed.

Added nodejs,n8n because the playbook's own Linux test (README:213) runs host n8n --version

@adamlam2-amd, to confirm which is correct.

…ring is unchanged

…openhands Windows model

…-validate # Conflicts: # playbooks/supplemental/github-slack-development-digest/README.md

…led models per owner feedback

…not live server health

…ls by starting LemonadeServer

harkgill-amd · 2026-09-30T19:05:36Z

@sreeram-11, I'm for merging this one as is.

The n8n failure seems to be a flaky test and the open-web-ui-chat is due to the runner missing a dependency (exactly what this PR fixes). I don't think we should be chasing after marking every dep with @prereq in this PR - we should merge and follow up with a PR that

Correctly tags all prereqs in all playbooks w/ @prereq
Aligns the install + validate commands for all prereqs with Luca's master script.

@bconsolvo for awareness.

bconsolvo · 2026-10-06T01:18:35Z

Force merging. Will need to address CI issues on a future PR.

Registry steps may now name a script under .github/scripts/prereqs/ instead of an inline cmd: bash <file> on Linux, powershell -File with -ExecutionPolicy Bypass on Windows. Recipes with control flow or a background process moved there, including #858's, verbatim; single commands stay inline. common.ps1 holds the Windows helpers (Start-Detached, Wait-Port) so the rule against children inheriting the step's output pipe lives in one place. test_prereq_validate.py gains script-step and registry-integrity tests (exactly one of cmd or script, referenced scripts exist and follow <dep>.<validate|install>.<ext>, no orphans). It now runs in the selector-tests job; CI never ran it before, so a broken script reference would only have surfaced as prereq failures on every runner.

@prereq

* registry: fill the prereq install/validate matrix Port the remaining install/validate recipes from runner provisioning so prereq_validate can check and self-heal every declared dependency. Fix podman, whose single apt call failed entirely without Docker's apt repo, and Linux lemonade, whose install never started lemond. Add install for lmstudio and two LM Studio models, and validate plus install for vscode and for Windows ollama, ollama models, hermes and openclaw. * prereqs: move multi-step recipes into script files Registry steps may now name a script under .github/scripts/prereqs/ instead of an inline cmd: bash <file> on Linux, powershell -File with -ExecutionPolicy Bypass on Windows. Recipes with control flow or a background process moved there, including #858's, verbatim; single commands stay inline. common.ps1 holds the Windows helpers (Start-Detached, Wait-Port) so the rule against children inheriting the step's output pipe lives in one place. test_prereq_validate.py gains script-step and registry-integrity tests (exactly one of cmd or script, referenced scripts exist and follow <dep>.<validate|install>.<ext>, no orphans). It now runs in the selector-tests job; CI never ran it before, so a broken script reference would only have surfaced as prereq failures on every runner. * registry: use the provisioning ids for the LM Studio Qwen installs qwen3.5-9b and qwen3-coder-30b are the ids runner provisioning uses, and they have a real download on a freshly reimaged runner behind them. They resolve to the same artifacts as the qwen/... forms. * prereqs: pull models from the internal mirror, with upstream fallback mirror/get.py, adapted from the Ryzen AI docs mirror tooling, puts a mirrored file in place: mirror first (base URL from the ARTIFACTORY_BASE secret, anonymous read), then the upstream URL from artifacts.json when the secret is unset, the mirror lacks the file, or the transfer fails. Every download is checked against the SHA-256 pinned in artifacts.json, and each request is recorded in an optional ledger so a run can show what the mirror served. First consumer: comfyui-models, whose hidden tests fail outright when the z_image_turbo files are missing from the runner's model cache. Its validate checks those same paths and its install fetches them through get.py. The pinned hashes match upstream byte for byte. * prereqs: seed Lemonade, LM Studio and Ollama models from the mirror An install step can name an artifacts.json group under "mirror". On a miss the validator runs `get.py seed <group>`, which puts the group's files where the tool's own install looks, re-validates, and runs the install only if still needed. Lemonade's pull then skips the seeded files. Pins 22 more mirrored files for the 11 model dependencies. get.py now gives up on an unreachable mirror after one 10 s connect attempt. * playbooks: declare the prereqs CI tests use but never checked open-webui-chat's Windows NPU smoke test needs Qwen3-4B-Hybrid, which had no registry entry; add lemonade-models-qwen3-4b-hybrid (mirror-seeded, like the other Lemonade models) and require it there. hermes-lemonade-server and deepseek-v4-flash-ds4 drive podman on Linux without declaring it. * prereqs: check Docker and Windows Node.js; name Lemonade models by what they pull docker: validate the engine on both OSes; on Windows, start an installed but stopped Docker Desktop. Required by openclaw, and on Windows by openhands and github-slack-development-digest. nodejs: validate native Node.js >= 24 and npm on Windows too, installing the LTS with winget. openclaw's nodejs prereq is Linux-only now: on Windows its Node.js lives in WSL, which the openclaw prereq covers. github-slack-development-digest's Windows CI uses host npm. Rename lemonade-models-qwen3-4b to -qwen3-5-4b and -qwen3-35b-a3b to -qwen3-6-35b-a3b; the old ids hid that they pull Qwen3.5-4B and Qwen3.6-35B-A3B. * mirror: add a maintainer tool to push and check the pinned files mirror.py, adapted from the Ryzen AI docs mirror's push and check: push uploads the artifacts.json entries the mirror lacks, verified against their pins and deployed with X-Checksum-Sha256 so Artifactory checks them again, and never replaces a file already there; check reports missing, different, and upstream drift. Run by maintainers with their own token, never by CI. * prereqs: install Docker and Node.js the way the provisioning scripts do docker: Linux installs Docker CE from get.docker.com and adds the runner user to the docker group, which takes effect when the runner service restarts; a runner on Docker Desktop for Linux only gets it started. Windows installs Docker Desktop with winget, then starts it. nodejs: presence-only checks on both OSes, as in the provisioning scripts, since upgrading a working Node.js can break npm globals such as n8n. Windows installs OpenJS.NodeJS with winget when it is absent. agent-canvas installs the fleet-validated 1.5.2. * prereqs: align with the provisioning scripts on major differences docker (Windows): the runners install Docker Desktop per user, so look there first; start it with `docker desktop start`, which also recovers from `docker desktop stop`, where launching the app did not. lemonade (Windows): install with winget when absent, as provisioning does. podman: write provisioning's cgroupfs config when none exists (the runner's system service has no session bus), and check and install podman inside WSL on Windows, where hermes runs it. driver (Windows): check for AMD Software: Adrenalin Edition, which provisioning treats as a manual install, so there is no install step. ollama-models-gpt-oss-20b: fall back to the same MXFP4 GGUF on Hugging Face, copied to gpt-oss:20b, when the Ollama registry fails. * prereqs: LM Studio and the Linux driver as the provisioning scripts set them up lmstudio: check for the CLI plus what provisioning adds to start the server, the lmstudio service on Linux and the "LM Studio Server" logon task on Windows, and install them the same way: Linux reuses an existing CLI, Windows installs ElementLabs.LMStudio with winget and lets its first launch add lms. driver (Linux): check that amdgpu is loaded and that /dev/kfd and a render node are usable. No install step: a driver change needs a reboot. After a Windows install, the validator adds PATH entries the installer wrote to the registry, to its own PATH for the re-check and to GITHUB_PATH for the tests, so a self-heal works in the same job instead of after a runner restart. Seeding LM Studio models falls back to each file's pinned Hugging Face URL; `lms get` downloads through LM Studio's proxy. mirror.py links a file the mirror already stores under another path by checksum instead of uploading it, and `check --upstream` compares commit-pinned URLs against the current main. * prereqs: check the exit code of piped commands in Windows validates After `cmd | Out-Null`, $? reports Out-Null, so a failing lms or npm read as present. * mirror: pin every model and dataset CI downloads artifacts.json now lists all 191 files CI runs on, English and zh-CN, each by SHA-256 with its upstream URL at a fixed commit, and all of them are on the mirror. Groups name the registry dependency that seeds them, or `hf:`, `hf-dataset:`, `modelscope:`, `modelscope-dataset:` and `zh-CN/` for files nothing seeds yet. lemonade-models-gpt-oss-20b gains the two EAGLE3 drafts that `lemonade pull` fetches with the repo. * prereqs: validate and self-heal localized runs too The localized test job had no prereq step. It now runs prereq_validate.py with --locale, which reads the localized README and the English registry overlaid field by field with the locale's: a localized entry translates the docs and may add its own steps, and the steps it leaves out come from English, so a shared dependency is checked the same way in every language. zh-CN gets steps for the models only it uses (Lemonade Qwen3.5-35B-A3B and Qwen3.5-9B, LM Studio qwen3.5-35b-a3b), seeded from the mirror, plus the n8n prereq its English twin declares. The zh-CN ComfyUI README required "comfyui_models", which no tag pattern matches, so its model section rendered empty and nothing checked the models; it now requires comfyui-models. * prereqs: seed the Hugging Face models and datasets the tests download One optional dependency per model or dataset the pytorch-style tests fetch on first use. Validate checks that every pinned file is in the Hugging Face cache at refs/main (get.py present); install seeds the current commit's snapshot from the mirror (get.py seed), so the libraries skip the download. A miss only warns, since the tests still download whatever is missing. Pins now cover just the files CI fetches. * zh-CN pytorch-finetuning: load the ModelScope dataset on offline runners MsDataset is built on Hugging Face datasets, which refuses to fetch when HF_HUB_OFFLINE is set, even from ModelScope. The two quick-train tests that use it turn it off for their own process; they only contact ModelScope (checked with Hugging Face unreachable and by tracing connections). The QLoRA test reads english_quotes from Hugging Face and stays offline. * mirror: seed Hugging Face groups at their pinned commit when offline Runners that set HF_HUB_OFFLINE are never asked about Hugging Face, and a failed API read no longer skips the group: either way the files go into the snapshot of the commit they were pinned at, checked by their SHA-256. Tests on those runners cannot download a missing model, so the mirror is their only way back. * tests: give the steps that run apt update 300 s apt update alone has run past 120 s on runners where the Ubuntu archive answers slowly. 300 s is what the English create-venv steps already use; the zh-CN copies had drifted to 60 and 120 s, and lemonade-update-linux and one unsloth create-venv were at 120 s. * prereqs: seed the ModelScope models the zh-CN tests download One optional zh-CN dependency per model the tests fetch with ModelScope's snapshot_download: gpt-oss-20b on halo, Qwen3.5-4B elsewhere, gemma-3-4b-it for fine-tuning. Validate checks every pinned file is where that client reads the model (the master snapshot, or a pre-1.38 folder); install seeds it from the mirror. The client still checks each file's SHA-256 against ModelScope, and skips them all once seeded. The mirror path names each file, since large files are pinned to identical Hugging Face ones. The 13 MB dolly dataset is not seeded: MsDataset keeps it in a datasets cache keyed by url and ETag, so its group is now a label. zh-CN fine-tuning also declares english_quotes, which its QLoRA test reads from Hugging Face. * tests: show what a timed-out test printed A timeout discarded the test's output, so a hung step (apt update waiting on a lock or a source, say) left no clue. Print the tail of what it wrote before it was killed and save it with the other results, as the normal path does. * zh-CN: drop the dependency docs nothing reads The five zh-CN dependencies this PR added are only @prereq'd, and only @require inlines a dependency's doc, on the website and in tests. Their docs were never shown or run, and the translation pipeline never touches localized-playbooks, so they would only be unreviewed Chinese to maintain. * mirror, prereqs: trim Plans no longer return the skipped entries nothing read, TOOLS names its fields instead of indexing them, seed and present share one helper, the integrity tests read each registry once, and the docstrings say the same in fewer words. * registry: podman runs on Windows too Its Windows steps, inside WSL where hermes runs it, came without windows in platforms. An integrity test now keeps each dependency's steps within its declared platforms. * prereqs: fixes from review - Ollama seeding no longer places a manifest over a missing blob, so a half-seeded model never lists. - Only a complete seed can stand in for the install: a tool may list a model with a part missing (a split GGUF, say), so a failed seed now always runs the install. - A fresh Docker CE install passed its own check with sudo, then failed the re-validate: the runner is not in the docker group until it restarts. An ACL on the socket now lets this job use it. - nodejs: apt-get remove gave up on unknown package names, so the distro packages were never removed; remove only the installed ones, and stop if the NodeSource setup fails. --------- Co-authored-by: Harkirat Gill <harkirat.gill@amd.com>

harkgill-amd requested review from adamlam2-amd, danielholanda, lucbruni-amd and sreeram-11 as code owners September 2, 2026 16:04

github-advanced-security Bot found potential problems Sep 2, 2026

View reviewed changes

Comment thread .github/scripts/test_prereq_validate.py Fixed

harkgill-amd requested a review from bconsolvo September 2, 2026 16:06

adamlam2-amd approved these changes Sep 2, 2026

View reviewed changes

harkgill-amd force-pushed the users/harkgill/prereq-validate branch from 1c4b2a3 to 2c31603 Compare September 3, 2026 14:49

fix(ci): make lmstudio model checks gating again (revert optional) to…

50d6c4d

… surface broken lms backend on r9700

lucbruni-amd approved these changes Sep 8, 2026

View reviewed changes

test(ci): replace deprecated tempfile.mktemp with TemporaryDirectory …

644f003

…(CodeQL)

Merge branch 'main' into users/harkgill/prereq-validate

e179e50

sreeram-11 reviewed Sep 9, 2026

View reviewed changes

Comment thread playbooks/core/n8n-automation-gpt-oss/README.md

docs(n8n): remove redundant npm install step now covered by @require:n8n

c8aaa5e

harkgill-amd mentioned this pull request Sep 10, 2026

[Bug]: lemonade-model-load (134.28s) fail when run Test hermes-lemonade-server (linux/halo) on Gorgon Halo 495W 128G #846

Closed

Merge origin/main into prereq-validate; add uv + agent-canvas deps fo…

cff4f57

…r openhands

sreeram-11 reviewed Sep 10, 2026

View reviewed changes

Comment thread playbooks/core/n8n-automation-gpt-oss/README.md

sreeram-11 reviewed Sep 10, 2026

View reviewed changes

Comment thread playbooks/dependencies/registry.json Outdated

sreeram-11 reviewed Sep 10, 2026

View reviewed changes

Comment thread playbooks/dependencies/registry.json

sreeram-11 reviewed Sep 10, 2026

View reviewed changes

Comment thread playbooks/dependencies/registry.json

Merge branch 'main' into users/harkgill/prereq-validate

9967029

bconsolvo removed their request for review September 14, 2026 16:53

harkgill-amd added 3 commits September 14, 2026 15:06

feat(ci): declare CI-only prerequisites with @prereq so website rende…

2ae7562

…ring is unchanged

fix(ci): add @prereq deps for github-slack digest (sync to main) and …

d46e58d

…openhands Windows model

Merge remote-tracking branch 'origin/main' into users/harkgill/prereq…

7d97e58

…-validate # Conflicts: # playbooks/supplemental/github-slack-development-digest/README.md

lucbruni-amd mentioned this pull request Sep 17, 2026

CI prerequisites all embedded within each playbook #829

Open

harkgill-amd and others added 5 commits September 21, 2026 17:13

fix(n8n): Linux runs n8n via Podman (no Node); mark halobox-preinstal…

f1fcde4

…led models per owner feedback

fix(ci): lemonade Windows prereq checks installed binary (where.exe) …

728d4a5

…not live server health

Merge branch 'main' into users/harkgill/prereq-validate

5c07d50

Merge branch 'main' into users/harkgill/prereq-validate

d6dcd55

fix(ci): lemonade Windows prereq validates server health and self-hea…

e959c5f

…ls by starting LemonadeServer

harkgill-amd mentioned this pull request Sep 28, 2026

fix(ci): skip merge commits in selection replay test, combined diff hides inherited changes #1042

Merged

Merge branch 'main' into users/harkgill/prereq-validate

f95f6c6

harkgill-amd mentioned this pull request Oct 1, 2026

Add authors and publication dates to every playbook #1046

Merged

2 of 4 tasks

Merge branch 'main' into users/harkgill/prereq-validate

a32100b

sreeram-11 requested a review from bconsolvo as a code owner October 1, 2026 22:22

sreeram-11 approved these changes Oct 5, 2026

View reviewed changes

sreeram-11 and others added 2 commits October 5, 2026 11:55

Merge branch 'main' into users/harkgill/prereq-validate

3237048

Merge branch 'main' into users/harkgill/prereq-validate

cddd7ed

bconsolvo merged commit ddd7003 into main Oct 6, 2026
251 of 284 checks passed

lucbruni-amd mentioned this pull request Oct 6, 2026

registry: fill the prereq install/validate matrix #1075

Merged

This was referenced Oct 9, 2026

l10n: update translations #1066

Closed

Automated translation failed on 2026-10-06 #1067

Closed

Conversation

harkgill-amd commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Coverage (uneven by design; extensible)

Uh oh!

Uh oh!

adamlam2-amd left a comment

Choose a reason for hiding this comment

Uh oh!

adamlam2-amd commented Sep 2, 2026

Uh oh!

harkgill-amd commented Sep 2, 2026

Uh oh!

harkgill-amd commented Sep 2, 2026

Uh oh!

lucbruni-amd left a comment

Choose a reason for hiding this comment

Uh oh!

harkgill-amd commented Sep 8, 2026

Uh oh!

Uh oh!

sreeram-11 commented Sep 9, 2026

Uh oh!

harkgill-amd commented Sep 10, 2026

Uh oh!

sreeram-11 Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

harkgill-amd Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harkgill-amd commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bconsolvo commented Oct 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

harkgill-amd commented Sep 2, 2026 •

edited

Loading

harkgill-amd commented Sep 30, 2026 •

edited

Loading