Skip to content

Backport: feat(google): support agentic and static video processing on generateContent - #21926

Draft
http-samc wants to merge 1 commit into
release-v6.0from
backport-pr-21925-to-release-v6.0
Draft

http-samc wants to merge 1 commit into
release-v6.0from
backport-pr-21925-to-release-v6.0

Conversation

@http-samc

@http-samc http-samc commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

This is a backport of #21925 to the release-v6.0 branch. It was ported manually because the automated cherry-pick conflicts (see below). FYI @http-samc

Background

Same as #21925. Gemini agentic video understanding (mediaProcessing) and static clipping/frame rate (videoMetadata) were unreachable from google(...) / generateContent, and the option was silently dropped.

Summary

This is a port of #21925 to the @ai-sdk/google 3.0.x layout. It changes convert-to-google-generative-ai-messages.ts and google-generative-ai-prompt.ts, and resolves namespaces via getGoogleProviderOptions.

  • providerOptions.google.processing on user video/* file parts maps as follows:
    • 'agentic' becomes mediaProcessing: 'AGENTIC'.
    • 'static' becomes mediaProcessing: 'STATIC'.
    • { type: 'static', startOffset?, endOffset?, fps? } becomes STATIC plus videoMetadata.
    • Invalid values produce a warning and are dropped.
  • Docs are in 15-google-generative-ai.mdx. Unlike on main, there is no GoogleInteractionsVideoOptions type and no Interactions cross-link, because this branch's Interactions provider doesn't support agentic video.
  • Example: examples/ai-functions/src/generate-text/google/agentic-video.ts.
  • Patch changeset.

Why manual: git cherry-pick of #21925 onto release-v6.0 conflicts on convert-to-google-generative-ai-messages.ts, google-prompt.ts (absent here) and the docs file. This branch is named backport-pr-21925-to-release-v6.0, so if the backport label is applied to #21925 the workflow detects this PR and skips instead of opening a conflict-marker PR.

End-to-End Verification

Same live check as #21925, run against this branch's source: gemini-3.7-flash with a 10s clip that has 8 flashes in total (4 in the first 5s).

processing release-v6.0 today this branch
(unset) "2", 646 input tokens "2", 646 input tokens
'agentic' "2", 646 tokens (ignored) "8", 108 input tokens*
{ type: 'static', startOffset: 0, endOffset: 5, fps: 5 } "2", 646 tokens (ignored) "4", 1,591 input tokens

* On this branch, usage.inputTokens excludes toolUsePromptTokenCount (Google reported about 35k tool-use tokens for this request). The fix is #21033 on main, which cherry-picks cleanly onto release-v6.0. Please apply the backport label to #21033 so the workflow carries it over.

Package checks: vitest (node: 647 passed; edge for the changed suites: 236 passed), tsc --build, oxfmt, oxlint.

Checklist

  • All commits are signed (PRs with unsigned commits cannot be merged)
  • Tests have been added / updated (for bug fixes / features)
  • Documentation has been added / updated (for bug fixes / features)
  • A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
  • I have reviewed this pull request (self-review)

Related Issues

Backport of #21925. Continues to release-v5.0 in #21927.

…Content (#21925)

Co-Authored-By: Samarth Chitgopekar <67826352+http-samc@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant