Problem
The screenshot tools (screenshot-game-view, screenshot-camera) capture the camera's render target, so ScreenSpaceOverlay canvases are invisible to them — overlay UI composites after the camera renders. For games whose entire HUD / menus / input fields live on an overlay canvas (the common default), an agent literally cannot see or verify the UI it just changed. World-space canvases capture fine; overlay ones don't.
Today the only workaround is hand-writing script-execute snippets that walk the hierarchy and read TMP_Text.text / Graphic.color / RectTransform geometry — re-derived every session, easy to get wrong, and not discoverable as a tool.
(Related but distinct: #619 added screenshot-isolated for renderers; #391 was about serialization errors. Neither gives a way to read overlay UI state.)
Proposal
A built-in read-only tool — e.g. ui-probe — that, given a hierarchy path or a bare name, returns the element's:
- existence +
activeInHierarchy / activeSelf (inactive-inclusive lookup),
- text content (
TMP_Text or UnityEngine.UI.Text),
Graphic.color,RectTransform geometry (anchoredPosition, sizeDelta, world corners).
This makes overlay-UI verification deterministic and token-cheap, and complements the screenshot tools rather than replacing them.
Prior art / offer
I built a project-local version as a custom [AiToolType] tool (one file, ~150 lines, pure Unity APIs, EditMode-tested, registers and runs against the live editor on 0.81.0). Happy to open a PR generalizing it into the package if you're open to the feature — just let me know the preferred home (a core tool vs. an optional module like the existing extension packages).
Environment: plugin/CLI 0.81.0, Unity 6000.2.10f1, macOS.
Problem
The screenshot tools (
screenshot-game-view,screenshot-camera) capture the camera's render target, soScreenSpaceOverlaycanvases are invisible to them — overlay UI composites after the camera renders. For games whose entire HUD / menus / input fields live on an overlay canvas (the common default), an agent literally cannot see or verify the UI it just changed. World-space canvases capture fine; overlay ones don't.Today the only workaround is hand-writing
script-executesnippets that walk the hierarchy and readTMP_Text.text/Graphic.color/RectTransformgeometry — re-derived every session, easy to get wrong, and not discoverable as a tool.(Related but distinct: #619 added
screenshot-isolatedfor renderers; #391 was about serialization errors. Neither gives a way to read overlay UI state.)Proposal
A built-in read-only tool — e.g.
ui-probe— that, given a hierarchy path or a bare name, returns the element's:activeInHierarchy/activeSelf(inactive-inclusive lookup),TMP_TextorUnityEngine.UI.Text),Graphic.color,RectTransformgeometry (anchoredPosition, sizeDelta, world corners).This makes overlay-UI verification deterministic and token-cheap, and complements the screenshot tools rather than replacing them.
Prior art / offer
I built a project-local version as a custom
[AiToolType]tool (one file, ~150 lines, pure Unity APIs, EditMode-tested, registers and runs against the live editor on 0.81.0). Happy to open a PR generalizing it into the package if you're open to the feature — just let me know the preferred home (a core tool vs. an optional module like the existing extension packages).Environment: plugin/CLI 0.81.0, Unity 6000.2.10f1, macOS.