# Agentic video understanding

> Source: https://aiwiki.ai/wiki/agentic_video_understanding
> Updated: 2026-09-03
> Fact-checked: 2026-09-03
> Categories: Computer Vision, Machine Learning, Multimodal AI
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Agentic video understanding." aiwiki.ai, 3 Sept 2026. https://aiwiki.ai/wiki/agentic_video_understanding
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Agentic video understanding is an approach to video analysis in which a reasoning model chooses which parts of a video to inspect for a particular question. Instead of receiving only a predetermined sample of frames, the model can request selected time ranges, examine visual frames at different sampling rates or resolutions, and use audio or transcript evidence. It may repeat this process before answering. The term describes a family of methods rather than a single standard architecture.[1][7][8]

The approach addresses a basic constraint in long-form [computer vision](https://aiwiki.ai/wiki/computer_vision): a video can contain far more visual and audio information than is relevant to one query. Loading a dense representation of the entire recording consumes context and computation, while sparse fixed sampling can miss a brief event. Agentic systems try to allocate processing to evidence that appears relevant. This can reduce token use, but it introduces a different failure mode because the system may search the wrong interval or stop before finding necessary evidence.[8][9][10]

## Relationship to adaptive video analysis

Agentic video understanding grew from research on query-directed retrieval, adaptive frame selection, and tool-using language models. It was not introduced by one product or company. VideoAgent, first submitted in March 2024, used a large language model as a central agent that iteratively identified evidence while vision-language models acted as retrieval and translation tools. Its authors reported zero-shot accuracies of 54.1% on EgoSchema and 71.3% on NExT-QA while using an average of 8.4 and 8.2 frames, respectively.[7]

Other work explored related ways to avoid uniform sampling. The 2024 Adaptive Video Understanding Agent used query-adaptive frame sampling and feedback-driven reasoning. Adaptive Keyframe Sampling, published at CVPR 2025, balanced relevance to the prompt against coverage of the source video under a fixed visual-token budget.[8][9] These systems differ in their tools and stopping rules, but they share the idea that the question should influence which video evidence reaches the reasoning model.

Google introduced an API feature under the name "agentic video understanding" on September 1, 2026. That release is one commercial implementation of the broader approach, not the origin of adaptive or agent-based video analysis.[1][3]

## How it works

Implementations vary, but an agentic video analysis generally separates evidence selection from answer generation. A typical process has the following stages:

1. The system receives a video and a question, instruction, or target event.
2. It uses a coarse view of the timeline, a transcript, metadata, or an index to identify candidate intervals.
3. A planner requests a selected interval and modality. The request may target frames, audio, or transcript text and may specify a denser temporal sample or higher visual resolution.
4. The system assesses the returned evidence. If it is incomplete or ambiguous, it can inspect another interval or revisit the same interval at a different density.
5. The model composes an answer from the evidence it selected.

This sequence is descriptive, not a required design. Some research systems retrieve captions or keyframes, while others expose a media-processing operation inside a model's reasoning loop.[7][8] Public documentation for Google's implementation describes internal video tools but does not disclose the underlying model architecture or training procedure.[1][2]

### Contrast with fixed sampling

Fixed sampling extracts frames according to a schedule decided before the question is answered. A one-frame-per-second sampler, for example, gives regular coverage and predictable input size. It can nevertheless omit a state that appears between sampled frames. It also spends the same visual budget on a relevant ten-second interval and on a long period that does not help answer the query.[2][9]

| Property | Fixed or static processing | Agentic processing |
|---|---|---|
| Frame choice | Predetermined rate or keyframe schedule | Chosen in response to the query and gathered evidence |
| Modalities | Usually loaded in a predefined pass | Frames, audio, and transcript can be requested selectively |
| Token use | More predictable and generally proportional to duration and sampling rate | Variable because the number and type of evidence requests can change |
| Brief events | May be missed when they fall between sampled frames | Can be revisited at a higher frame rate if the search identifies the interval |
| Latency | Often one media-processing pass | Can require several reasoning and retrieval round trips |
| Main risk | Important detail omitted by the fixed schedule | Important detail omitted because the planner searched or selected poorly |

Neither mode guarantees that all relevant evidence will be captured. A denser fixed sample can improve temporal coverage at greater cost. An agentic system can spend more of its budget on a promising interval, but only after locating that interval correctly. MINERVA, a benchmark for complex video reasoning, found that temporal localization and visual perception were leading failure modes among evaluated multimodal models.[10]

## Google Gemini implementation

[Google DeepMind](https://aiwiki.ai/wiki/google_deepmind) launched agentic video processing for Gemini 3.7 Flash, Gemini 3.6 Flash, and Gemini 3.5 Flash-Lite on September 1, 2026. The launch covered both the Interactions API and the GenerateContent API. It accepted uploaded video files and public YouTube video URLs through the Gemini API in Google AI Studio and Gemini Enterprise Agent Platform.[1][3]

Gemini 3.8 Flash became generally available on September 2, one day after the feature's initial release. Google's current guide lists [Gemini](https://aiwiki.ai/wiki/gemini) 3.8 Flash alongside the three launch models as supporting agentic video. This later documentation should not be read as evidence that Gemini 3.8 Flash was part of the September 1 launch.[2][3]

Google's static mode remains the default. It extracts video frames at 1 FPS and places them in context in one pass. In agentic mode, the model can navigate the timeline and load transcript segments, frames, or audio on demand. It can also adjust the requested frame rate and resolution. Google describes this as an internal processing operation rather than a client-defined sequence of retrieval calls.[1][2]

### Input and processing flow

The Gemini API accepts a video uploaded through its Files API or supplied as a public YouTube URL. For an uploaded file, the documented flow is to upload the media, poll until its state becomes `ACTIVE`, and then pass its URI and MIME type with the text prompt. A `FAILED` processing state must be handled before a generation request is made. Google's YouTube input support is limited to public videos; private and unlisted videos are not supported.[2]

| Interface | Agentic control | Static control | Evidence trace |
|---|---|---|---|
| Interactions API | `processing: "agentic"` on a video input | `processing: "static"`, or a static processing object | `processing_call` and `processing_result` steps before `model_output` |
| GenerateContent, Python | `media_processing="AGENTIC"` on a video part | `media_processing="STATIC"` | `tool_call` and `tool_response` parts with tool type `MEDIA_PROCESSING` |
| GenerateContent, JavaScript | `mediaProcessing: "AGENTIC"` | `mediaProcessing: "STATIC"` | The same `MEDIA_PROCESSING` call and response parts |
| GenerateContent, REST | `media_processing: "AGENTIC"` | `media_processing: "STATIC"` | The same `MEDIA_PROCESSING` call and response parts |

Google documents these controls separately for the Interactions and GenerateContent interfaces.[2][5]

The API reference defines three media-processing enum values: `MEDIA_PROCESSING_UNSPECIFIED`, `STATIC`, and `AGENTIC`.[6] In the Interactions API, a `processing_call` identifies a request and its following `processing_result` refers back to that request. Optional `thought` entries may appear between processing operations, followed by the final `model_output`. Clients may use these entries as a progress trace, but they do not need to respond to each processing call.[2]

Static processing offers controls that are not equivalent to agentic navigation. In the Interactions API, a static processing object can include `start_offset`, `end_offset`, or `fps`. These let a developer clip a known interval or choose a fixed frame rate. Google's guide says these custom clipping and frame-rate options are supported only in static mode.[2]

### Tokens, price, and latency

Google's token guide estimates static video at about 100 tokens per second at low media resolution and about 300 tokens per second at high media resolution. The amount is predictable because the default samples at 1 FPS, and it scales with duration and a custom static frame rate. Agentic token use is variable because the model loads only the transcript, frames, and audio it decides are needed.[4]

The Interactions API separates the usage record into `total_input_tokens` for the initial reference and prompt, `total_thought_tokens` for navigation reasoning, `total_tool_use_tokens` for retrieved transcript, frame, and audio data, and `total_output_tokens` for the final answer.[4] Google said at launch that the feature carries no additional feature fee and uses standard Gemini API token pricing. This does not make a request free: the relevant input, thought, media-processing, and output tokens remain part of usage.[1][4]

Selective processing can add latency before generation begins. Google's guide warns that reasoning and internal processing round trips may slightly increase time to first token for videos under five minutes. The company recommends static processing for latency-sensitive questions about short clips and for cases that require frame-level precision across the whole clip.[2]

## Reported evaluation results

Google's September 1 launch post compared static and agentic processing with Gemini 3.7 Flash. The chart specifies high thinking, low media resolution, and a static baseline sampled at 1 FPS. Google reported the following results:[1]

| Benchmark | Static tokens | Agentic tokens | Token reduction | Static accuracy | Agentic accuracy | Accuracy change |
|---|---:|---:|---:|---:|---:|---:|
| MINERVA | 80.9K | 33.6K | 58.4% | 73.7% | 79.0% | +5.3 percentage points, +7.2% relative |
| 1H-VideoQA | 397.6K | 47.7K | 88.0% | 87.5% | 88.5% | +1.0 point, +1.1% relative |
| LVBench | 300.3K | 36.0K | 88.0% | 85.1% | 88.6% | +3.5 points, +4.1% relative |

The launch headline summarized these results as up to 88% fewer tokens, up to 66% lower analysis cost, and up to 7% higher accuracy.[1] The accuracy figure is a rounded relative improvement on MINERVA, not a seven-percentage-point increase. The largest absolute increase in the displayed table was 5.3 points.

These are vendor-reported measurements, not an independent evaluation. The launch material names the processing settings but does not publish the complete prompts, repeated-run results, confidence intervals, or raw outputs needed to reproduce the comparison. It also does not establish that the maximum reduction applies to every video, prompt, supported model, or pricing tier. Agentic token consumption depends on the question and on how much evidence the model chooses to retrieve.[1][2][4]

The three benchmarks cover different long-video demands. MINERVA uses complex, multi-step questions with hand-written reasoning traces.[10] The current public 1H-VideoQA repository contains 101 five-choice questions about 21 YouTube videos, each 40 to 90 minutes long.[11] LVBench was designed to test extremely long-video understanding across tasks including key-information retrieval, temporal grounding, event understanding, reasoning, recognition, and summarization.[12] Google's chart does not identify the exact repository revision, prompt set, or scoring script used for each result.

## Uses and limitations

Agentic processing is most suited to long videos and questions whose answer depends on a small part of the timeline. Examples include finding a short state change, searching a lecture or meeting for a particular claim, reviewing a suspected anomaly at a higher sampling rate, counting repeated actions, and combining spoken and visual evidence. These tasks benefit from selecting evidence before filling the model's context window.[1][2]

It is less suitable when the entire recording must be inspected under a fixed, auditable procedure. Static or specialized processing can be preferable for short latency-sensitive clips, exhaustive frame-by-frame review, deterministic measurements, and cases where a missed interval would carry a high cost. Agentic selection also does not remove ordinary model errors. A system can misread a frame, misinterpret audio, choose the wrong interval, or produce an answer not supported by the retrieved evidence.[2][10]

Results need stronger verification in medical, legal, security, surveillance, transportation, and other consequential settings. Google's own developer guidance states that generative models can produce inaccurate, biased, or offensive output and recommends post-processing and human evaluation to reduce harm.[2] An operational system should retain enough evidence, timestamps, and usage traces for reviewers to check how an answer was formed. A processing trace shows that retrieval calls occurred, but it does not by itself prove that the answer is complete or correct.

## References

1. Google, ["Introducing agentic video understanding with Gemini"](https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-agentic-video-in-gemini/), September 1, 2026.
2. Google AI for Developers, ["Video understanding"](https://ai.google.dev/gemini-api/docs/video-understanding), last updated September 2, 2026.
3. Google AI for Developers, ["Release notes"](https://ai.google.dev/gemini-api/docs/changelog), September 1-2, 2026 entries.
4. Google AI for Developers, ["Understand and count tokens"](https://ai.google.dev/gemini-api/docs/tokens), accessed September 3, 2026.
5. Google AI for Developers, ["Video understanding, Gemini Generate Content API (Legacy)"](https://ai.google.dev/gemini-api/docs/generate-content/video-understanding), accessed September 3, 2026.
6. Google AI for Developers, ["Generating content", MediaProcessing reference](https://ai.google.dev/api/generate-content), accessed September 3, 2026.
7. Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy, ["VideoAgent: Long-form Video Understanding with Large Language Model as Agent"](https://arxiv.org/abs/2403.10517), arXiv:2403.10517, submitted March 15, 2024.
8. Sullam Jeoung, Goeric Huybrechts, Bhavana Ganesh, Aram Galstyan, and Sravan Bodapati, ["Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning"](https://arxiv.org/abs/2410.20252), arXiv:2410.20252, submitted October 26, 2024.
9. Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye, ["Adaptive Keyframe Sampling for Long Video Understanding"](https://openaccess.thecvf.com/content/CVPR2025/html/Tang_Adaptive_Keyframe_Sampling_for_Long_Video_Understanding_CVPR_2025_paper.html), Proceedings of CVPR 2025, pp. 29118-29128.
10. Arsha Nagrani et al., ["MINERVA: Evaluating Complex Video Reasoning"](https://arxiv.org/abs/2505.00681), arXiv:2505.00681, submitted May 1, 2025.
11. Google DeepMind, ["1H-VideoQA"](https://github.com/google-deepmind/1h-videoqa), repository documentation, accessed September 3, 2026.
12. Weihan Wang et al., ["LVBench: An Extreme Long Video Understanding Benchmark"](https://arxiv.org/abs/2406.08035), arXiv:2406.08035, submitted June 12, 2024.