feat(agent): per-command agent schema with response shapes - #883
platinummonkey wants to merge 5 commits into
Conversation
`pup agent schema` was all-or-nothing (~185 KB compact) and described inputs only, so agents paid ~50k tokens per lookup and still guessed response shapes. - `pup agent schema <path>` accepts `logs aggregate`, `logs.aggregate`, or a domain (`logs`); aliases resolve like clap. Unknown paths exit non-zero listing valid subcommands. `auth token` stays hidden. - `--search TERM` lists matching leaf commands (path + short description), optionally scoped to a path. - Filtered output adds a shared `envelope` contract and a `returns` entry on every leaf; hand-verified shapes for logs aggregate/search, traces aggregate/search, and metrics query, with `jq_root`, notes, and a worked example on single-command lookups. - Filtered output is minified: logs aggregate ~3.8 KB, logs ~16.8 KB. - No-argument and `--compact` output are unchanged. - Conformance test feeds recorded API bodies through `output::build_agent_envelope` and validates them against `returns`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In agent mode, `pup logs aggregate --help` resolved only the top-level token and returned the whole logs domain (~32 KB), with every entry's full_path built from an empty parent (so read_only could be wrong for nested commands) and no response shape. - Resolve the deepest command named on the command line (aliases map to canonical names; positional values and global flag values are skipped; `auth token` is never descended into). - Build the entry with its real parent path and attach `envelope` and per-leaf `returns`, as `pup agent schema <path>` does. - Keep global_flags, script_authoring and anti_patterns so a lone `--help` still carries the --no-agent guidance. Root `--help` is unchanged. - Replace `find_subcommand` with `help_command_path` and port its tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`pup security findings schema` downloads the whole findings reference
(~200 KB, ~50k tokens) on every call with no way to narrow it.
- Parse the reference's attribute tables into {path, type, section,
description} rows (all 618 rows in the current page).
- `--search TERM` keeps fields whose path or description match;
`--section NAME` keeps one namespace (nested headings included).
Filtered output goes through the formatter, so it honors agent mode,
--jq and --output. `--section advisory` is ~2 KB.
- An unknown section errors, listing the available ones; a page that
yields no parseable rows errors rather than returning an empty list.
- No flags: output unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without filters the command prints markdown, not an enveloped JSON value, so the generic contract misdescribed it. Document both modes, add a conformance fixture, and note in the envelope that some commands print plain text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Three distinct issues were found: a timestamp unit mismatch in the metrics query contract (milliseconds vs. seconds), a missing blank-input guard in the section filter logic, and incorrect skipping of parent-local option values during help resolution.
🤖 Bits Code Review · Commit d1e2550 · @DataDog review to ask questions
| "from_date": {"type": "integer", "description": "ms since epoch"}, | ||
| "to_date": {"type": "integer", "description": "ms since epoch"}, |
There was a problem hiding this comment.
Document metric query bounds in seconds
The published contract tells agents that from_date and to_date are milliseconds, but Datadog's v1 metrics query response expresses these bounds in Unix seconds. Consumers following the contract will interpret valid dates as 1970-era timestamps.
| "from_date": {"type": "integer", "description": "ms since epoch"}, | |
| "to_date": {"type": "integer", "description": "ms since epoch"}, | |
| "from_date": {"type": "integer", "description": "seconds since epoch"}, | |
| "to_date": {"type": "integer", "description": "seconds since epoch"}, |
Was this helpful? React 👍 or 👎
🤖 Bits Code Review · @DataDog review to ask questions · Open Bits AI session
| search: Option<&str>, | ||
| section: Option<&str>, | ||
| ) -> Result<Vec<SchemaField>> { | ||
| let section = section.map(|s| s.trim().to_lowercase()); |
There was a problem hiding this comment.
For --section='' or a whitespace-only value, trimming produces an empty string and contains("") matches every section. Instead of rejecting the invalid filter, the command returns the entire structured schema, defeating the scoped lookup and producing unexpectedly large output.
| let section = section.map(|s| s.trim().to_lowercase()); | |
| let section = section.map(|s| s.trim().to_lowercase()); | |
| if section.as_deref() == Some("") { | |
| anyhow::bail!("--section term must not be empty"); | |
| } |
Was this helpful? React 👍 or 👎
🤖 Bits Code Review · @DataDog review to ask questions · Open Bits AI session
| const VALUE_GLOBALS: &[&str] = &["-o", "--output", "--org", "--jq"]; | ||
| let mut prev_consumes_value = false; | ||
| for arg in args.iter().skip(1) { | ||
| args.iter().skip(1).filter_map(move |arg| { |
There was a problem hiding this comment.
Handle parent-option values during help resolution
For valid invocations such as pup profiling --header "test-drive-hummer-aurora: 1" services list --help, the header value is treated as a positional token because only global option values are skipped. Resolution stops at profiling, returning the broad parent schema instead of the requested leaf schema; any parent-local value option can break targeted agent help similarly.
Was this helpful? React 👍 or 👎
🤖 Bits Code Review · @DataDog review to ask questions · Open Bits AI session


Summary
pup agent schemareturns everything at once (~185 KB) and only describes what each command takes, not what it returns, so agents guess the output format. This adds lookups for a single command or domain that also show what the command returns.Changes
pup agent schema logs aggregate(orlogs.aggregate, or a domain likelogs) returns just that part, with areturnsshape for each command.--search TERMfinds commands by name or description.pup <cmd> --helpnow covers the exact command asked for, not its whole domain.pup security findings schema --search/--sectionreturns only the matching fields instead of the full ~200 KB reference.Testing
returnsshapes match real command output.Follow-ups
--jqnote in agent-mode output still conflicts with the published jq paths.🤖 Generated with Claude Code