A multimodal model is a machine learning model, or a model-based system, that processes, relates, or produces information across more than one kind of data. Each kind is called a modality. Common examples include language, images, audio, video, depth measurements, thermal measurements, and motion-sensor readings. The defining property is not that every input passes through one monolithic network. A system with separate encoders, an alignment objective, a connector, and a decoder can also be multimodal if its prediction depends on learned relations between modalities.

Multimodal learning addresses several different problems. It can align an image with a caption, combine speech with lip motion, answer a written question about a diagram, generate an image conditioned on text, or use camera observations and instructions to predict robot actions. A foundational taxonomy separates representation, translation, alignment, fusion, and co-learning, while later work emphasizes the heterogeneity of modalities, the connections between them, and the ways they interact.[1][2] These distinctions matter because "multimodal" does not identify one architecture, one training objective, or one measure of capability.

The dominant engineering pattern has changed at least once. Systems built between roughly 2021 and 2023 usually attached a separately trained visual encoder to a separately trained language model and learned only the connector between them. Since late 2023 the largest systems have instead been pretrained on interleaved text, images, audio, and video from the beginning, and several of them generate images themselves rather than delegating to a separate image model. As of July 2026 both patterns remain in production, the labels applied to them are not standardized across vendors, and for the closed systems the public evidence is often a model card or a safety document rather than a technical report.

Scope and terminology

What counts as a modality

A modality is an operational category, not merely a filename extension. Text and speech can carry related linguistic content, yet their signals, noise, timing, and preprocessing are different. A color image and a depth map can be stored as arrays of similar shape while measuring different physical properties. Video may be treated as one spatiotemporal modality, or its visual and audio streams may be modeled separately. Whether two sources count as distinct modalities therefore depends on what they represent and how the learning problem uses them.[1]

This task-dependent definition prevents several common category errors:

Multiple camera views do not automatically make a system multimodal. They may be several observations within one visual modality.
A model with several task heads is multitask, but it is not necessarily multimodal.
An ensemble of text-only models remains unimodal unless it also relates text to another modality.
Converting an image from JPEG to PNG changes the encoding, not the modality.
Metadata such as time, location, or device state can be a modality when it is represented as a distinct information stream and affects the learned prediction.

The boundary can also depend on the unit of analysis. An audio encoder is unimodal as a component. A larger audiovisual speech system that combines the encoder with mouth-motion features is multimodal. A speech recognizer maps audio to text, so the task crosses representational domains, but a deployed recognizer that consumes only audio need not contain a joint audio-text representation. Good technical descriptions state whether "multimodal" refers to the training data, architecture, input interface, output space, or complete application.

Model, component, and system

In practice, multimodal systems are often modular. A modality-specific encoder converts raw input into features. A projection, shared latent space, or attention block makes features compatible. A fusion module or decoder then uses them for prediction or generation. Training may update every component, only the connector, or some intermediate subset. Calling the whole assembly a multimodal model is conventional, but architecture descriptions should still identify which parts are frozen, shared, or separately trained.

This distinction is especially important for systems built around a large language model. Adding a visual encoder and learned connector can make the overall system accept images without changing every parameter of the language model. Flamingo, BLIP-2, and LLaVA illustrate different versions of this modular pattern.[18][19][20] The public evidence for one system should not be used to infer how an undisclosed system works.

Related terms

A vision language model is a multimodal model focused on visual and linguistic information. A multimodal language model usually produces or interprets language while conditioning on one or more additional modalities. A vision-language-action model further connects perception and language to action prediction or control. These are overlapping subfamilies, not synonyms for every multimodal model.

Multimodal learning also overlaps with representation learning, data fusion, cross-modal retrieval, and conditioned generation. None of those terms alone implies the others. A dual encoder may learn coordinated representations without fusing token streams. A decision-fusion system may combine outputs without learning a common embedding. A text-conditioned image generator translates between modalities without supporting general visual question answering.

Several newer labels appear mainly in product announcements and have no agreed technical definition. "Natively multimodal" usually means the model was pretrained on more than one modality rather than adapted afterward, but vendors apply it to designs that differ substantially, including ones that still contain a separately trained visual encoder. "Omni" and "any-to-any" describe the input and output interface, not the internals. None of these terms says where modality-specific processing ends, whether any component is frozen, or how each modality is tokenized. Those questions have to be asked separately, and for closed systems they often go unanswered.

Plain-language explanation

Imagine trying to understand a cooking video. The pictures show what is happening, the soundtrack contains speech and kitchen sounds, and subtitles provide words. Any one stream can be incomplete. A multimodal model tries to connect the streams: which spoken instruction matches which action, whether a sound supports what the image shows, and what answer follows from the evidence together. It may use a separate specialist for each stream and a learned mechanism for combining them. The important part is the learned relationship, not whether the specialists are physically one network.

Historical development

Before large-scale vision-language pretraining

Multimodal machine learning did not begin with web-scale image-text models. Earlier work addressed audiovisual speech recognition, multimedia retrieval, human interaction analysis, and relationships between images and language. Many systems extracted features independently and concatenated them or combined their decisions. This made the quality and timing of hand-designed features central constraints. The historical survey by Baltrusaitis and colleagues traces these strands and uses them to motivate a taxonomy broader than early and late fusion.[1]

Deep neural networks made it possible to learn cross-modal features rather than specifying all of them manually. In 2011, Ngiam and colleagues studied audio and video for speech classification, cross-modal feature learning, and cases in which only one modality was available at test time.[3] In 2012, Srivastava and Salakhutdinov used a multimodal Deep Boltzmann Machine to learn a generative image-text representation and to infer missing modalities.[4] These papers already contained themes that remain important: shared structure, asymmetric information, missing inputs, and the difference between joint training and joint availability at inference.

Image captioning and visual question answering then provided influential image-language tasks. Show and Tell used a convolutional image encoder and recurrent decoder to generate captions.[5] The original Visual Question Answering work formalized open-ended questions about images and provided human answers for evaluation.[6] These systems typically relied on separately initialized visual and textual pipelines with a learned bridge or fusion layer. They established cross-modal generation and reasoning tasks before large-scale contrastive pretraining.

Cross-modal Transformers

The Transformer introduced attention-based sequence modeling without recurrence.[7] Multimodal researchers subsequently adapted attention to interactions between visual regions and language tokens. ViLBERT used two streams connected by co-attention.[8] LXMERT separated object, language, and cross-modality encoders and combined several pretraining tasks.[9] UNITER placed image and text features in a joint Transformer and trained with masking, matching, and word-region alignment objectives.[10] ViLT later demonstrated a lighter image-text Transformer that operated on image patches and word embeddings without depending on a heavy region-feature extractor.[11]

These models did more than replace recurrent layers. They made the location and direction of interaction explicit. Visual features could attend to language, language could attend to vision, or a shared sequence could be processed by the same blocks. The resulting designs remain useful reference points for distinguishing two-stream, single-stream, and cross-attention architectures.

Large paired datasets and coordinated embeddings

CLIP and ALIGN showed how dual encoders can learn from large collections of image-text pairs. CLIP trained separate visual and text encoders on 400 million pairs. Within a batch, normalized image and text embeddings were compared by cosine similarity, and a symmetric cross-entropy objective increased similarity for matched pairs relative to mismatched ones.[12] At inference, natural-language class descriptions could act as a classifier without fitting a new conventional classification head.

ALIGN used more than one billion noisy image-alt-text pairs and a related dual-encoder contrastive design.[13] The work emphasized scale and tolerance of noisy pairing rather than requiring a manually curated label set. SigLIP later replaced the global softmax-normalized pairing objective with a pairwise sigmoid loss.[14] These methods learn a coordinated embedding space: image and text are encoded independently, but the training objective makes their outputs comparable. They are not examples of late decision fusion unless a downstream system actually combines decisions.

Alignment and fusion were also combined in the same model. ALBEF first aligned unimodal image and text representations with a contrastive objective, then fused them through cross-attention; it also used momentum distillation to improve learning from noisy web data.[15] BLIP supported both understanding and generation and introduced a captioner and filter for constructing cleaner training pairs from noisy images and text.[16] These designs show that architectural categories are often stages in one pipeline rather than mutually exclusive labels.

Connecting pretrained components

Another line of work connects strong pretrained unimodal components instead of training the whole system from scratch. The Perceiver architecture uses asymmetric attention to map large structured inputs into a fixed-size latent bottleneck and was demonstrated on images, point clouds, audio, and audiovisual data.[17] A bottleneck can control the number of features passed to later computation even when the raw inputs have very different sizes.

Flamingo uses a frozen visual encoder and frozen language model. A Perceiver Resampler converts a variable-size visual feature grid into a fixed set of visual tokens, and newly trained gated cross-attention layers condition language-model blocks on those tokens. Its input can interleave text with images or videos.[18] BLIP-2 instead uses a Querying Transformer between a frozen image encoder and a frozen language model and trains that connector in two stages.[19] LLaVA connects a visual encoder to a language model with a learned projection and then applies visual instruction tuning.[20]

The three systems illustrate a design space, not a linear ranking. A projection is simple and inexpensive but compresses the interface into the language model's embedding format. A querying connector can select a bounded set of visual features. Repeated cross-attention can expose visual evidence at multiple language-model depths. Their relative value depends on the task, input length, training data, optimization budget, and whether the base components remain frozen.

More than image and text

ImageBind aligns six types of data: images, text, audio, depth, thermal measurements, and inertial measurements. It uses images as a binding modality, so every non-image modality need not be directly paired with every other one.[21] This is an example of transitive coordination through a common anchor, not proof that all modality pairs become semantically interchangeable.

Audiovisual speech provides a different setting because the streams are synchronized in time. AV-HuBERT predicts masked cluster assignments from audio and lip-motion input to learn audiovisual speech representations.[22] u-HuBERT uses modality dropout and a unified training approach so that a model can operate on audio, video, or both.[23] These systems make temporal alignment and missing-modality behavior part of the central problem rather than treating them as afterthoughts.

Conditioned generation and embodied control further broaden the field. Latent diffusion models perform denoising in a compressed latent space and can use cross-attention for conditioning on inputs such as text.[24] RT-2 represents robot actions as tokens and co-fine-tunes robot trajectories with vision-language tasks, providing a documented example of connecting multimodal representations to physical action.[25] The full task, rather than any isolated encoder, determines whether these systems are being used multimodally.

Core technical problems

The five-part taxonomy of representation, translation, alignment, fusion, and co-learning remains a practical map of the field.[1] It should not be read as five disjoint model types. One captioning system may face all five.

Problem	Central question	Example
Representation	How should heterogeneous signals be encoded so their useful structure is retained?	Encode an image and a sentence as comparable vectors or as interacting token sequences
Translation	How can information in one modality be expressed in another?	Generate a caption from an image or an image from text
Alignment	Which elements, moments, or concepts correspond across modalities?	Match a spoken word to lip motion or a phrase to an image region
Fusion	How should evidence from several modalities affect one prediction?	Combine audio and video when recognizing speech in noise
Co-learning	How can knowledge from one modality improve learning or inference in another?	Use paired image-text data to transfer semantic supervision to an image encoder

Later taxonomies add reasoning, generation, transference, and quantification, and organize the field around heterogeneity, connections, and interactions.[2] The newer framing helps distinguish simply placing data in one model from measuring whether the model actually uses cross-modal evidence.

Representation

Modalities differ in structure. Text is usually discrete and sequential. Images are spatial arrays. Audio is sampled over time. Video adds both time and space. Depth, thermal, and inertial signals have different units, noise processes, and sampling rates. A representation must preserve task-relevant information while putting these sources in a form that later modules can compare or combine.

Foundational work distinguishes joint representations from coordinated representations.[1] A joint representation is computed from multiple modalities together. Concatenating audio and visual features and passing them through a network is a simple example. A coordinated representation encodes each modality separately but constrains the outputs to have a defined relationship, such as high similarity for matching pairs. CLIP is a coordinated dual-encoder system; UNITER is closer to a joint token representation. ALBEF uses both coordination and fusion at different stages.

A representation may be global or local. One vector per image and one vector per caption are efficient for retrieval, but they do not directly identify which phrase corresponds to which region. Token-level or region-level features can express finer relationships at greater computational cost. Temporal systems must also decide whether to retain frame-level or sample-level timing, compress it into segments, or summarize the whole sequence.

Translation

Cross-modal translation maps one modality to another. Image captioning maps visual input to language, text-conditioned image generation maps language to visual output, and speech recognition maps an acoustic signal to text. Translation is often one-to-many: several captions may accurately describe one image, and several images may satisfy one prompt. Evaluation therefore cannot assume a single exact target.

Translation systems can be autoregressive, masked, diffusion-based, or otherwise generative. The output decoder may use a joint representation or may be conditioned on coordinated features through attention. The design should preserve the asymmetry of the task. A model that retrieves an existing caption and a model that generates new text may use similar aligned encoders but have different output spaces, risks, and evaluation methods.

Alignment

Alignment identifies correspondences between modalities. At the global level, the question may be whether an image and sentence belong together. At a finer level, alignment may match words to regions, speech segments to mouth motion, video moments to transcript spans, or sensor events to actions. Some correspondence is explicit in labels; some comes from co-occurrence, temporal synchronization, or contrastive pairing.

Alignment can be ambiguous. A caption may mention only one object in a crowded image. Alt text may describe a webpage's purpose rather than the pixels. Speech and lip motion can be slightly offset. An image may illustrate a paragraph without depicting every statement in it. Treating all paired elements as perfectly aligned creates noisy supervision, while treating every unpaired sample as a true negative can create false negatives.

Fusion

Fusion determines how evidence from multiple streams influences a prediction. The common early, intermediate, and late terminology describes where interaction occurs:

Early or input-level fusion combines raw or lightly processed features before most task modeling. It can expose low-level interactions but requires compatible scales and timing.
Intermediate fusion lets each modality develop some specialized features before they interact. Token concatenation, co-attention, cross-attention, gated exchange, and latent bottlenecks are different intermediate mechanisms.
Late or decision-level fusion combines independently produced predictions or high-level features. It is modular and can tolerate replacement of one component, but it may miss fine-grained cross-modal relationships.
Hybrid fusion uses more than one interaction point, such as contrastive alignment followed by a cross-attention decoder.

This location taxonomy is orthogonal to the joint-versus-coordinated distinction. A dual encoder can produce coordinated embeddings and then feed a late decision module. A joint model can still delay interaction until after modality-specific encoders. Describing both axes is more informative than assigning one label.

There is no universally optimal fusion point. A study of multimodal Transformers under missing modalities found that sensitivity and the best fusion configuration varied across datasets.[35] Architecture selection should therefore be tested under the intended noise, missingness, and latency conditions rather than inferred from an aggregate leaderboard.

The same three words are also used at a completely different scale to describe whole pretraining strategies, where "early fusion" means one model trained on all modalities at once and "late fusion" means separately pretrained components joined by a connector. That usage is discussed below and should not be confused with the layer-level sense here.

Co-learning and transference

Co-learning uses information from one modality to improve representation or prediction in another. Cross-modal supervision can supply labels that would be expensive to collect directly. Natural-language descriptions can supervise visual categories; visual context can disambiguate speech; a shared anchor can relate modalities that are not directly paired. ImageBind's image-centered alignment is one example.[21]

Transfer can be asymmetric. A high-resource modality may act as a teacher, while a low-resource modality receives the benefit. Training may require paired inputs even if inference later uses only one. Conversely, the system may have all modalities at training time but need to survive a missing input at deployment. Reports should state these availability assumptions explicitly.

Architecture patterns

Modality-specific encoders

Most systems begin by converting each input into a sequence or set of features. An image encoder may emit patch or region tokens. A text encoder emits word or subword tokens. Audio may be represented as waveform segments, spectrogram patches, or learned codec units. Video may produce frame, tube, or event tokens. Sensor signals may be grouped by timestamp or channel.

Separate encoders allow each modality to use appropriate inductive biases and pretrained parameters. They also create an interface problem: feature widths, scales, position conventions, and semantic granularity may not match. Projection layers, normalization, learned queries, and attention blocks are common ways to define that interface. A technical description should not call the resulting representations "shared" merely because they have the same vector width.

Dual encoders

A dual encoder maps two modalities independently into a coordinated space. Similarity between embeddings supports retrieval, matching, and zero-shot classification. Because each side can be encoded separately, large collections can be indexed in advance. The architecture is efficient when the desired output is a similarity score.

The same independence limits fine-grained interaction. A global image vector cannot, by itself, show which visual token supports a particular word. Adding region features, token-level losses, reranking, or a fusion model can address that need, but changes the computation and evaluation. CLIP, ALIGN, and SigLIP illustrate different training choices within the dual-encoder family.[12][13][14]

Single-stream joint encoders

A single-stream design converts modality-specific inputs into compatible tokens, adds modality or position information, and processes the combined sequence with shared layers. UNITER and ViLT provide image-text examples.[10][11] In principle, every token can interact through attention. In practice, sequence length, attention masks, positional encodings, and preprocessing determine which interactions are computationally feasible.

Shared layers do not guarantee equal use of every modality. A model may still solve a benchmark from language priors or visual shortcuts. Input ablations and counterfactual examples are needed to establish cross-modal dependence.

Two-stream and cross-attention models

Two-stream models retain separate modality pathways and add directed exchanges. In cross-attention, queries from one stream attend to keys and values from another. The direction matters: language-conditioned vision and vision-conditioned language are not identical computations. ViLBERT uses co-attentional streams, while LXMERT explicitly separates unimodal and cross-modality encoders.[8][9]

Cross-attention can occur once, repeatedly, or only in a decoder. Repeated interaction can support finer reasoning but increases memory and compute. Gating can control how newly introduced multimodal information perturbs pretrained representations, as in Flamingo.[18] The connector's location is therefore as important as its name.

Latent bottlenecks and learned queries

Inputs such as images and video can create many more tokens than text. A latent bottleneck uses a fixed or bounded set of learned vectors to attend to a larger input and summarize it. Perceiver applies this idea broadly, Flamingo uses a Perceiver Resampler for visual features, and BLIP-2 uses learned queries in its connector.[17][18][19]

The bottleneck reduces downstream sequence length, but it also sets an information budget. Too few latent tokens may omit small text, rare events, or spatial detail. More tokens improve capacity at additional cost. The appropriate size depends on resolution, duration, task, and the amount of fine-grained grounding required.

Generative conditioning

In a generative system, one modality often conditions a decoder for another. A language decoder can attend to image features while predicting the next text token. A diffusion denoiser can attend to a text representation while predicting a denoising update in image latent space.[24] Conditioning can be applied at every layer or selected layers, and it can use a global vector or a sequence of features.

Generation does not by itself prove understanding. A system can produce plausible output from strong priors while ignoring part of the conditioning input. Prompt-image consistency tests, counterfactual prompts, region grounding, and controlled input ablations are needed to measure whether the output depends on the specified evidence.

Decision fusion

Decision fusion keeps models largely independent and combines their scores or predictions. Examples include averaging class probabilities, using a learned gate, or selecting a specialist based on input quality. This pattern can be robust when one sensor fails and can support modular deployment. It cannot recover interactions that were discarded before the decision stage. If a lip-reading model and audio model each commit to the wrong phoneme independently, a simple average has limited access to the timing evidence that might have resolved the conflict.

Representative designs

System	Published design	What it illustrates
Multimodal deep learning	Shared audio-video representations and cross-modal learning	Learned fusion and transfer before modern Transformers[3]
Multimodal Deep Boltzmann Machine	Generative joint image-text representation	Modeling missing modalities[4]
Show and Tell	Convolutional image encoder plus recurrent language decoder	Image-to-text translation[5]
ViLBERT	Two streams connected by co-attention	Directed cross-modal interaction[8]
LXMERT	Object, language, and cross-modality encoders	Multiple pretraining objectives and staged fusion[9]
UNITER	Joint image-text Transformer	Masking, matching, and fine-grained alignment in one encoder[10]
ViLT	Patch and word tokens in a relatively simple joint Transformer	Reducing dependence on heavy visual feature extraction[11]
CLIP	Independent visual and text encoders with contrastive alignment	Coordinated embeddings and retrieval-style inference[12]
ALIGN	Dual encoders trained on more than one billion noisy pairs	Scaling weakly paired image-text supervision[13]
ALBEF	Contrastive alignment followed by cross-attention fusion	Combining coordinated and joint stages[15]
BLIP	Multimodal encoder-decoder with caption generation and filtering	Joint support for understanding, generation, and data bootstrapping[16]
Flamingo	Frozen vision and language components plus resampling and gated cross-attention	Interleaved visual-language conditioning[18]
BLIP-2	Frozen image encoder and language model connected by a Querying Transformer	Parameter-conscious adaptation of pretrained components[19]
LLaVA	Visual encoder projected into a language model, followed by visual instruction tuning	A projection-based language-assistant design[20]
ImageBind	Multiple encoders coordinated through image-paired data	One shared space spanning six modalities[21]

The table is architectural, not a performance ranking. The systems use different data, objectives, parameter counts, tasks, and evaluation protocols. It also stops in 2023, because the designs after that point are better described by how they were pretrained than by their connector.

Native multimodal pretraining

A multimodal system can be assembled from parts that were trained separately, or trained on several modalities at once. The first approach dominated published work between roughly 2021 and 2023 and is still widely used. The second is what recent releases usually mean by "natively multimodal." The distinction is real, but the vocabulary is loose enough that two organizations can apply the same phrase to different designs.

The adapter pattern and what it assumes

Flamingo, BLIP-2, and LLaVA all start from components that already work and learn an interface between them. The shared premise is that a visual encoder trained with its own objective already produces features carrying what the language model needs, and that the missing piece is only a translation layer. The premise is attractive because it is cheap: the connector is small, the expensive components are reused, and comparatively little paired data is required. A 2025 scaling study calls this family late fusion and describes its sample efficiency as remarkable.[42]

The premise is only partly right. Meta's Perception Encoder work reports that contrastive vision-language training alone can produce embeddings general enough for zero-shot classification and retrieval, document and video question answering, and dense prediction such as detection and depth estimation, but that those embeddings are hidden in the network's intermediate layers rather than sitting at its output. The authors introduce two separate procedures to draw them out, language alignment for multimodal language modeling and spatial alignment for dense prediction.[61] If the most useful representation is not at the layer a connector conventionally reads from, then which layer to tap is a design decision with measurable consequences, not an implementation detail. It is one reason a vision encoder is better described by its training objective, its output layer, and its alignment procedure than by its name.

Training on interleaved modalities from the start

Google's first Gemini report describes models trained jointly on image, audio, video, and text data from the start, accepting text interleaved with a wide variety of visual and audio input including natural images, charts, screenshots, PDFs, and video, and able to produce image output using discrete image tokens.[39] The Gemini 1.5 report describes a sparse mixture-of-experts Transformer that is natively multimodal and supports mixing audio, visual, text, and code within one input sequence.[40] The Gemini 3 Pro model card, whose model release is dated November 2025 and which was last updated in May 2026, describes a sparse mixture-of-experts Transformer with native multimodal support for text, vision, and audio inputs. It accepts text, images, audio, and video files within a context window of up to one million tokens and produces text output of up to 64,000 tokens.[47]

OpenAI describes GPT-4o as an autoregressive omni model that accepts any combination of text, audio, image, and video and generates any combination of text, audio, and image outputs, and states that it is trained end to end across text, vision, and audio so that all inputs and outputs are processed by the same neural network.[41]

Meta's Llama 4 announcement uses a third formulation. It says the models are designed with native multimodality, incorporating early fusion to integrate text and vision tokens into a unified model backbone, and that early fusion is what allows joint pre-training on large amounts of unlabeled text, image, and video data. The same announcement says the vision encoder is based on MetaCLIP and was trained separately alongside a frozen Llama model to adapt it to the language model.[43] A system can therefore be described by its developer as early fusion and still contain a separately trained encoder.

That matters because the 2025 scaling study defines early fusion as a design that does not rely on image encoders or tokenizers at all.[42] Under that definition Llama 4 would not qualify. Neither usage is wrong; they draw the boundary in different places. A description that says where modality-specific processing ends and shared computation begins carries more information than either label.

Early fusion versus late fusion at pretraining scale

Shukor and colleagues trained 457 models across architectures and training mixtures to compare the two families under matched compute. They report no inherent advantage for late fusion. Early fusion performed better at lower parameter counts, was more efficient to train, and was easier to deploy. At equal compute budgets the two families reached similar validation loss, but through a different balance between parameters and training tokens, with early fusion needing fewer parameters. Adding mixture-of-experts layers allowed models to learn modality-specific weights, which improved results further.[42]

Read at the level of its evidence, that result concerns training native multimodal models from scratch on the study's own data mixtures and measuring validation loss. It does not say that an organization already holding a strong pretrained language model should discard it. Sample efficiency, which the same paper attributes to late fusion, is exactly what matters when paired multimodal data is scarce relative to text. The two findings are compatible: late fusion buys data efficiency by importing representations, and early fusion buys parameter efficiency by learning them jointly.

Question	Adapter or late-fusion assembly	Native or early-fusion pretraining
What is reused	A pretrained encoder and a pretrained language model	A tokenizer at most, sometimes nothing
What is trained	The connector, sometimes with light tuning of the base models	The whole model
Multimodal data required	Comparatively little	Large interleaved corpora
Where modality-specific structure lives	In the frozen encoder	In the shared weights, or in routed experts
Interface risk to test	Information lost or unreachable at the connector	One modality dominating the optimization
Replacing a component later	Possible without retraining the rest	Generally requires retraining

The table summarizes tradeoffs described in the sources above rather than measured results, and it is not a ranking. Production systems mix the patterns: Llama 4 pretrains jointly while keeping a separately trained encoder, and several native models retain a discrete image tokenizer, which is itself a separately trained component.

One model that understands and generates

Until about 2024, a system that both answered questions about images and produced images usually did so by calling a separate generator. Several designs since then put both capabilities in one set of weights, and they disagree about how.

Chameleon (Meta, May 2024) is a family of early-fusion token-based mixed-modal models that understand and generate images and text in any arbitrary sequence. The paper describes a stable training approach from inception, an alignment recipe, and an architectural parameterization specific to the early-fusion token-based setting, and evaluates one model on visual question answering, captioning, text generation, image generation, and long-form mixed-modal generation.[38] Because images are quantized into discrete tokens, the model can predict them with the same next-token objective it uses for text.

Transfusion (Meta, August 2024) declines the quantization step. It trains one Transformer with two objectives at once: next-token prediction on discrete text tokens, and a diffusion objective on continuous image patches processed in parallel. The authors pretrained models up to 7 billion parameters on 2 trillion multimodal tokens and report that the recipe scales better than quantizing images into discrete tokens and training a language model over them.[44]

Janus (DeepSeek, October 2024) locates the conflict in the visual encoder rather than the backbone. Understanding and generation need different levels of information granularity, so Janus decouples visual encoding into separate pathways while keeping a single unified transformer for processing, which lets each task select its own encoding method. The paper explicitly contrasts this with the single visual encoder used by Chameleon.[45] Janus-Pro later scaled the same design at 1 billion and 7 billion parameters.

OpenAI's March 2025 addendum to the GPT-4o system card describes that model's native image generation as an autoregressive model natively embedded within ChatGPT, in contrast to DALL-E, which the addendum describes as a diffusion model, and says the capability is embedded natively, deep in the architecture of the omnimodal GPT-4o model. No further architectural detail is given; the document's subject is safety, and it describes a layered mitigation stack of chat-model refusals, prompt blocking by text and image classifiers, and output blocking that includes a custom-trained multimodal reasoning monitor.[46]

Later image and video systems have generally been announced with less architectural information rather than more. Google's Nano Banana Pro, released in November 2025, is the image model of the Gemini 3 family, which the Gemini 3 Pro model card lists as Gemini 3 Pro Image.[47][69] Meta's Muse Image, announced on 7 July 2026 as the first released image model from Meta Superintelligence Labs, takes text prompts and reference images and produces images. Meta's announcement describes behavior rather than structure: the model invokes search and coding tools to improve accuracy, self-refines its own generations, and improves with additional test-time compute, and Meta states that self-refinement emerged during reinforcement learning because it produced higher-reward images. No model card or technical report accompanied the launch.[68] Google's Gemini Omni, announced at Google I/O in May 2026, combines images, audio, video, and text as input and generates video, with image and audio outputs described as future work; the first released member is Gemini Omni Flash. The announcement contains no architecture statement.[70]

For systems like these the accurate description is functional: what goes in, what comes out, what the vendor claims, and what an independent party has measured. Inferring an architecture from a sibling system's paper is not evidence, and neither is inferring one from a demonstration.

What has been disclosed

System	Announced	Primary source type	Architecture stated in that source	Not stated
Chameleon	May 2024	Paper and weights	Early-fusion token-based mixed-modal Transformer over interleaved image and text tokens[38]	Nothing material for the described models
Transfusion	August 2024	Paper	One Transformer trained with a language-modeling loss on text and a diffusion loss on image patches, up to 7B parameters[44]	Nothing material for the described models
Janus	October 2024	Paper and weights	Decoupled visual encoding pathways feeding one unified autoregressive transformer[45]	Nothing material for the described models
GPT-4o	May 2024	System card	Autoregressive omni model, trained end to end, all inputs and outputs through the same network[41]	Parameter count, layer structure, audio and image tokenization
4o image generation	March 2025	System card addendum	Autoregressive, natively embedded in GPT-4o, unlike diffusion-based DALL-E[46]	Everything beyond that sentence
Llama 4	April 2025	Blog post and weights	Early fusion of text and vision tokens into a unified backbone, mixture-of-experts, MetaCLIP-derived vision encoder[43]	Full training mixture, image tokenization detail
Qwen3-Omni	September 2025	Technical report and weights	Thinker-Talker mixture-of-experts, multi-codebook speech codec, causal ConvNet decoder[48]	Nothing material for the described models
Gemini 3 Pro	November 2025	Model card	Sparse mixture-of-experts Transformer with native multimodal support for text, vision, and audio[47]	Parameter and expert counts, tokenization, training compute
Qwen3.5-Omni	April 2026	Technical report	Hybrid-attention mixture-of-experts Thinker and Talker, 256k context[49]	Exact parameter count, stated only as hundreds of billions
Gemini Omni	May 2026	Blog post	None	Architecture, training, evaluation protocol
Muse Image	July 2026	Blog post	None beyond agentic tool use and RL-induced self-refinement[68]	Architecture, training, parameter count

Two patterns are visible. Open-weight releases have generally come with a paper that names the objective and the fusion point, because the weights are inspectable anyway. Closed consumer releases have generally come with a capability description, a safety document, or neither. Encyclopedic writing about these systems should preserve that split rather than smoothing it over, and should say "not disclosed" where nothing was disclosed.

Audio, speech, and video as first-class modalities

Much of the vocabulary in this article was developed for image and text. Audio and video are not simply more inputs of the same kind. They are continuous, their timing carries meaning, they often arrive while the user is still producing them, and they cost far more tokens per second of content than text does.

From cascades to end-to-end audio

The conventional way to give a language model a voice interface is a cascade: a speech recognizer produces a transcript, the language model consumes the transcript, and a text-to-speech system reads the answer aloud. The SeamlessM4T authors describe the same pattern in translation and its cost, noting that conventional speech-to-speech translation relies on cascaded systems that translate progressively, which puts high-performing unified systems out of reach.[64] A cascade also discards whatever the transcript does not carry: prosody, speaker identity, overlap, and non-speech sound.

SeamlessM4T (Meta, August 2023) is a single model supporting speech-to-speech, speech-to-text, text-to-speech, and text-to-text translation as well as automatic speech recognition for up to 100 languages, trained with 1 million hours of open speech audio to learn self-supervised speech representations and then with a corpus of automatically aligned speech translations combined with human-labeled and pseudo-labeled data.[64]

Qwen2-Audio (Alibaba, July 2024) has two interaction modes, voice chat and audio analysis, and switches between them without a system prompt to select one. Its report gives a compact illustration of what end-to-end audio buys: given a segment containing environmental sounds, a multi-speaker conversation, and a spoken command, the model can follow the command and interpret the surrounding audio in the same pass.[63] A transcript would flatten all of that into words.

Streaming input and real-time interaction

Batch multimodal inference assumes the whole input exists before the model runs. Live interaction does not. The input arrives incrementally, the user may interrupt, and a response that is correct but late is a worse product than one that is slightly worse and prompt.

OpenAI reports that GPT-4o can respond to audio input in as little as 232 milliseconds with an average of 320 milliseconds, which the system card compares to human response time in conversation.[41] Google's Live API documentation describes the streaming contract concretely: it accepts raw 16-bit PCM audio at 16 kHz, images at up to one frame per second, and text, returns raw 16-bit PCM audio at 24 kHz, and supports barge-in so a user can interrupt the model mid-response.[50]

Qwen3-Omni (September 2025) is one of the few systems in this class with a published architecture. It uses a Thinker-Talker mixture-of-experts design in which the Talker autoregressively predicts discrete speech codecs with a multi-codebook scheme, replacing block-wise diffusion with a lightweight causal convolutional network so that streaming can begin from the first codec frame. The report gives a theoretical cold-start end-to-end first-packet latency of 234 milliseconds, and states support for text in 119 languages, speech understanding in 19, and speech generation in 10.[48] Qwen3.5-Omni (April 2026) scales the same family to what its report describes as hundreds of billions of parameters with a 256,000-token context, applies a hybrid-attention mixture-of-experts framework to both Thinker and Talker, and adds a mechanism called ARIA that dynamically aligns text and speech units. The team attributes streaming instability to differing encoding efficiency between the text and speech tokenizers, and presents ARIA as the fix.[49] Both sets of figures are the developers' own.

Streaming changes evaluation as much as it changes architecture. A system that must begin answering before the utterance ends cannot condition on the complete input, so accuracy measured on complete recordings is an upper bound rather than an estimate. Turn-taking, endpoint detection, and interruption handling become measurable behaviors rather than interface details. Latency becomes a reported metric alongside accuracy, and the two trade against each other in ways an aggregate score hides.

Long video: frames, tokens, and context

Video is the most token-expensive common modality, and the arithmetic is public for at least one system. Google's Gemini API documentation, as of July 2026, states that video is sampled at one frame per second by default, that each frame costs 258 tokens at default media resolution or 66 tokens at low media resolution, and that accompanying audio costs 32 tokens per second. That works out to roughly 300 tokens per second of video at default resolution and roughly 100 at low resolution, so a one-million-token context holds about one hour of video at default resolution or about three hours at low resolution.[51] The Gemini 1.5 report describes research settings extending to at least 10 million tokens, corresponding to about 10.5 hours of video at one frame per second or roughly 107 hours of audio.[40] Alibaba reports that Qwen3.5-Omni supports more than 10 hours of audio understanding and 400 seconds of 720p video at one frame per second.[49]

Three constraints follow, and they trade against one another rather than being solved together.

Temporal sampling. One frame per second discards most frames. Anything shorter than the sampling interval, a gesture, a single frame of on-screen text, a brief collision, may never reach the model at all. Raising the frame rate raises token count proportionally.
Spatial detail. Dropping from 258 to 66 tokens per frame cuts cost by roughly four and removes the resolution needed for small text, distant objects, and fine structure. Frames that are themselves documents, such as slides and screens, lose the most.
Attention cost. Self-attention is quadratic in sequence length, so token count is not only a context-window question. It sets memory use and latency as well.

Token compression is the standard response. A 2025 survey of the area organizes methods by the redundancy they exploit, spatial for images, spatiotemporal for video, and temporal and spectral for audio, and by mechanism, dividing them into transformation-based, similarity-based, attention-based, and query-based approaches.[53] Every one of these is lossy by construction. What a scheme discards should be stated and tested rather than left implicit, because the discarded content is often exactly what a fine-grained video question asks about.

The problem overlaps with the one facing long-context language models, but video adds a dimension text lacks: tokens have a fixed relationship to wall-clock time, so a context limit is also a duration limit. A system advertised with a large context window may still be unable to watch a feature-length film at full resolution, and the tokenization policy, not the advertised window, determines what it can actually see.

Evaluating video and audio understanding

Video-MME (May 2024) is built around this spread of durations. It contains 900 videos totalling 254 hours with 2,700 manually annotated question-answer pairs, spans six visual domains and 30 subfields, and covers clips from 11 seconds to one hour. It supplies subtitles and audio alongside the frames, which makes it possible to measure how much of a score comes from each track instead of assuming that the video was used at all.[52]

EgoSchema (August 2023) attacks long-form video from a different direction. Derived from Ego4D, it has more than 5,000 human-curated multiple-choice questions over three-minute clips drawn from more than 250 hours of video, with five options per question. Its authors argue that clip length does not by itself measure temporal difficulty, and introduce temporal certificate sets to capture the intrinsic length of video a task actually requires; on that measure they report EgoSchema's certificate length as more than 5.7 times that of the next closest dataset and 10 to 100 times that of others. In the original evaluation, models with several billion parameters scored under 33% where random choice gives 20%, against roughly 76% for humans.[71]

Both benchmarks make the same methodological point in different ways. A video score is uninterpretable without knowing the frame rate, the resolution, whether audio and subtitles were supplied, and how many frames the model actually received.

Learning objectives

Supervised task losses

The simplest objective trains a model for a labeled outcome: answer classification, caption likelihood, speech recognition, sentiment, event detection, or action prediction. Supervision may attach to a fused representation or to modality-specific and joint heads. Auxiliary unimodal losses can preserve useful specialist features, but they can also let the model optimize unimodal shortcuts unless the cross-modal objective requires interaction.

Contrastive alignment

Contrastive learning makes matched cross-modal examples more similar than mismatched ones. In a batch of image-text pairs, CLIP computes all pairwise similarities and applies image-to-text and text-to-image cross-entropy losses.[12] SigLIP instead treats pairs with a sigmoid loss and does not require a global softmax across the batch.[14]

The construction of positives and negatives is part of the method. Two captions for the same scene may be semantically compatible even if only one pairing is labeled positive. Two images from the same event may be false negatives. Hard-negative mining can focus learning on subtle differences but can also amplify annotation errors. Batch size, temperature, sampling, and duplicate handling affect what the embedding geometry represents.

Matching and alignment losses

A binary matching loss asks whether two inputs belong together. Token-region or word-patch losses target finer correspondences. UNITER combines image-text matching with masked and alignment objectives.[10] ALBEF separates an initial contrastive stage from later fusion and matching.[15] These objectives can improve grounding, but a supplied pair may still lack exact local annotations. A global caption does not state that every word maps to one visible region.

Masked modeling and reconstruction

Self-supervised learning can mask part of one or more modalities and train the model to reconstruct or predict the hidden content. LXMERT and UNITER use masked language and visual objectives.[9][10] AV-HuBERT predicts masked audiovisual cluster targets.[22] The difficulty depends on the mask pattern. If nearby frames or another modality reveal the answer directly, low loss may measure redundancy rather than abstract reasoning.

Reconstruction objectives also differ in target. A system may reconstruct pixels, acoustic features, discrete codes, visual-region labels, or representations from a teacher. These targets encourage different invariances. Reporting only "masked multimodal pretraining" hides important design choices.

Autoregressive generation

An autoregressive decoder predicts the next token conditioned on previous tokens and multimodal context. Captioning, question answering, transcription, and action-token prediction can all be expressed this way. Show and Tell, Flamingo, and RT-2 use this general pattern in different domains.[5][18][25]

The same objective covers image output once images are quantized into discrete tokens, which is how Chameleon generates them and how Gemini 1.0 described its native image output.[38][39] Quantization is not free: it imposes a codebook, and the Transfusion authors report that keeping image data continuous and applying a diffusion loss to it scaled better than tokenizing images and predicting them autoregressively.[44]

Teacher forcing during training supplies previous ground-truth tokens, while inference conditions on the model's own outputs. This can compound early errors. A fluent decoder can also rely heavily on language priors, so likelihood alone is not a sufficient measure of grounding.

Diffusion and denoising

Diffusion objectives corrupt data and train a model to reverse the corruption. In latent diffusion, an encoder compresses images, a denoising model operates in latent space, and cross-attention can introduce text conditioning.[24] The objective is not the same as contrastive alignment even if both systems use text and images. A text encoder, denoiser, and image decoder may each have separate training histories.

The two objectives are not mutually exclusive within one network. Transfusion applies a language-modeling loss to text tokens and a diffusion loss to image patches inside the same Transformer, processing the image patches in parallel rather than one at a time.[44] Describing such a system as "autoregressive" or "diffusion-based" is incomplete; the useful statement names which loss applies to which part of the sequence.

Distillation, bootstrapping, and instruction tuning

Multimodal systems often reuse supervision produced by other models. ALBEF uses momentum distillation, while BLIP generates and filters captions for noisy web images.[15][16] Such bootstrapping can increase usable training data, but it can reproduce teacher errors and narrow linguistic diversity.

Visual instruction tuning formats image-language examples as user instructions and responses so a language-model-based system learns conversational task behavior. LLaVA is an early documented example.[20] Instruction tuning changes how a model follows requests; it should not be treated as independent proof of visual grounding or factual accuracy.

Data and alignment

Pairing structures

Multimodal data can be organized in several ways:

Explicit pairs, such as an image with a human-written caption.
Weak pairs, such as an image and nearby web text whose relationship is uncertain.
Interleaved sequences, such as a webpage containing several images and paragraphs.
Temporally synchronized streams, such as audio and video recorded together.
Grouped observations, such as camera, depth, thermal, and inertial readings from the same event.
Task demonstrations, in which multimodal inputs are paired with instructions and responses.

The pairing structure constrains what the model can learn. A global image-caption pair supports broad semantic alignment but supplies weak evidence for exact region-word correspondence. Timestamped audio and video support temporal alignment, but recording delays and different sample rates must still be handled. Interleaved documents provide context beyond one pair while introducing uncertainty about which paragraph refers to which image.

Interleaved data has become more consequential than it once was, because native multimodal pretraining consumes it directly. Meta describes early fusion as the property that lets Llama 4 be pretrained jointly on large amounts of unlabeled text, image, and video data, and Google describes Gemini as trained from the start on interleaved sequences of text, images, video, and audio.[43][39] A corpus of clean caption pairs is not a substitute: the interleaved format is what teaches a model that an image can refer backward to a paragraph it has already read.

Granularity

Alignment has levels. Dataset-level alignment means sources cover similar domains. Example-level alignment means two records correspond. Segment-level alignment links moments or regions. Token-level alignment links fine elements. A model trained only with example-level contrastive loss should not be assumed to have learned token-level grounding.

Granularity also affects negative examples. A caption can be wrong for the whole image yet correct for one region. A sentence can describe an event that occurs elsewhere in a video. Evaluation should match the resolution of the claimed capability.

Noise and incomplete descriptions

Web-derived captions and alt text can be duplicated, promotional, inaccessible, or unrelated to the visible content. Image-text pairs can omit people, objects, text, and context. Filtering can remove obvious noise, while caption generation can replace or supplement weak text. BLIP's captioning-and-filtering approach is one documented strategy.[16] The result is still model-mediated data, not ground truth.

Noisy labels interact with scale. ALIGN demonstrates that useful representations can be learned from more than one billion noisy pairs under its reported setup.[13] That result does not imply that more noise is harmless or that data curation never matters. Accuracy, demographic coverage, licensing, privacy, duplication, and source traceability are separate dimensions of data quality.

Missing and conflicting modalities

Real systems encounter blank images, silent audio, failed sensors, truncated documents, or a modality that arrives late. Training only on complete examples can make a model brittle. Modality dropout, learned masks, uncertainty-aware gating, and unimodal fallback paths are possible responses. u-HuBERT applies modality dropout for audio and visual speech input.[23]

Missingness is not always random. A camera may fail in darkness, audio may be suppressed for privacy, or metadata may be absent in one population. Performance on uniformly random masks can therefore overestimate robustness. Evaluation should reproduce plausible missingness mechanisms.

Modalities can also conflict. A caption may contradict an image, a transcript may mishear speech, or two sensors may disagree. Always trusting one modality can be dangerous, while naive averaging can hide the conflict. A system intended for consequential decisions should detect disagreement, preserve provenance, and expose uncertainty rather than converting every conflict into a confident fused answer.

Tasks and applications

Retrieval and classification

Cross-modal retrieval searches one modality using another, such as finding images from a text query or finding descriptions for an image. Dual encoders are well suited because candidate embeddings can be computed and indexed before the query. Retrieval evaluation commonly measures how highly a matching item is ranked, but the result depends on the candidate pool and the definition of relevance.

Zero-shot classification uses natural-language class descriptions as comparison targets. CLIP demonstrated this pattern across multiple image datasets.[12] The result depends on prompt wording, class vocabulary, dataset shift, and the relationship between pretraining data and the evaluation set. It is not equivalent to learning a new category from no prior semantic exposure.

Captioning and visual question answering

Image captioning generates a textual description of visual input. It tests translation and grounding, but valid outputs are open-ended. A concise caption may omit details; a detailed caption has more opportunities to introduce an unsupported object. Reference overlap and image compatibility therefore measure different properties.

Visual question answering conditions an answer on an image and a question. Some questions can be answered from language priors without looking at the image. VQA v2 introduced complementary image pairs associated with similar questions but different answers to reduce this shortcut.[26] Even then, performance can vary by question form, answer vocabulary, visual detail, and annotation convention.

Document, chart, and diagram questions add layout, optical character recognition, symbolic notation, and spatial relations. A model can correctly extract text yet fail to connect it to the relevant mark, or reason from a chart while misreading its scale. Evaluation should separate perception, alignment, and reasoning errors where possible.

Documents, screens, and interfaces

Donut (2021) is an early demonstration that the optical character recognition stage can be removed. Its authors identify three costs of outsourcing reading to an off-the-shelf engine: the computation it consumes, its inflexibility across languages and document types, and the propagation of its errors into everything downstream. Donut maps the document image directly to structured output using a Transformer trained with a cross-entropy objective, and the paper reports state-of-the-art results on visual document understanding tasks in both speed and accuracy at the time, along with a synthetic data generator for pretraining across languages and domains.[66]

A rendered screen is a related modality with its own conventions, and the task is control rather than extraction. UI-TARS (January 2025) is a graphical interface agent that takes only screenshots as input and emits keyboard and mouse actions, trained end to end rather than assembled as prompt scaffolding around a commercial model. Its authors report 24.6 on OSWorld with a 50-step budget and 22.7 with 15 steps, and 46.6 on AndroidWorld, in their own evaluation.[62] Screen understanding combines the failure modes of document reading, which include small text and dense layout, with those of control, since a misgrounded click has consequences that a wrong caption does not. See computer-use agent for the wider category.

These tasks also illustrate why "image input" is too coarse a description. A photograph, a scanned invoice, a chart, and a screenshot are all images, but they reward different resolutions, different token budgets, and different error analyses.

Conditioned media generation

Text-to-Image Models generate visual output conditioned on language. Related tasks include text-guided editing, image-conditioned variation, audio generation from text, and video generation from prompts. These tasks are multimodal because the condition and output occupy different representational domains, even if the generator is built from modular encoders and decoders.

Useful evaluation includes condition adherence, output quality, diversity, safety, provenance, and human judgment for the intended use. A photorealistic sample can still violate the prompt, while a prompt-aligned sample can contain distorted text or geometry. One scalar metric cannot represent all of these properties.

Provenance has moved from a research topic to a shipped feature. Meta states that images from Muse Image carry an invisible watermark it calls Content Seal, and OpenAI describes classifier-based output blocking applied after generation.[68][46] Neither is a claim that generated media can be reliably detected downstream; both are claims about what the producer attaches or blocks at generation time, and the distinction matters when writing about provenance.

Audiovisual and temporal understanding

Audiovisual speech recognition can use mouth motion when audio is noisy. Audio and video must be synchronized closely enough for the relationship to help. AV-HuBERT and u-HuBERT illustrate masked pretraining and missing-modality handling for this setting.[22][23]

Video-language tasks include temporal retrieval, video question answering, event description, and following interleaved demonstrations. A model must choose which frames or segments matter and distinguish temporal order from visual co-occurrence. Sampling too sparsely may miss brief evidence; sampling densely increases sequence length and cost.

Orchestration instead of one model

Not every multimodal capability lives inside one network. HuggingGPT (March 2023) uses a language model as a controller: it plans the task from the user's request, selects models from a hub according to their published function descriptions, executes each subtask with the selected model, and summarizes the results.[65] The system is multimodal; no single component in it is.

This pattern makes substitution easy and error attribution comparatively clear, and it inherits the limits described earlier for decision-level designs. Information the controller does not pass along cannot be recovered downstream, each hop adds latency, and the language description of a subtask becomes a bottleneck through which all cross-modal evidence must fit. It remains a reasonable design when mature specialists exist for each modality and the task decomposes cleanly, and a poor one when the answer depends on relationships that no single specialist can see.

Sensor fusion and robotics

Sensor-fusion models combine signals such as cameras, depth, thermal sensing, position, and inertial measurements. Each sensor has its own range, calibration, latency, and failure conditions. Fusion can improve coverage when streams are complementary, but only if the training and deployment pipelines preserve calibration and timing.

In robotics, multimodal input can be connected to action. RT-2 expresses actions as tokens and co-fine-tunes robot trajectory data with vision-language tasks.[25] This does not make language-model evaluation sufficient for a robot. Physical deployment also requires control stability, latency, recovery behavior, and safety testing under distribution shift.

Later systems keep the same basic shape while changing what the model consumes and emits. The model card for Google DeepMind's on-device member of Gemini Robotics 2, published on 30 July 2026, describes a vision-language-action model built on the on-device Gemma models that takes text instructions, images from the robot's own viewpoint, and numerical proprioception data, and outputs numerical robot action values.[67] Generalist AI announced GEN-1 on 2 April 2026 and describes it as a large multimodal model for real-time robot operation, pretrained largely on human activity recordings collected with low-cost wearable devices and then adapted to new robots and tasks with roughly an hour of robot data per task.[73] Mistral AI announced Robostral Navigate in July 2026, an 8-billion-parameter vision language model that takes images from one RGB camera plus a natural-language instruction and produces navigation decisions, positioned as a way to avoid multi-sensor hardware.[74] These are the developers' own descriptions, and the disclosure caveats set out above apply to them as much as to consumer models.

Proprioception deserves to be named as a modality in the sense used here: a distinct information stream with its own units, sampling rate, and failure behavior, which the model must relate to vision and language.

Evaluation

Start with the claimed capability

Evaluation should begin by specifying the input modalities, output, allowed context, adaptation method, and success criterion. "Multimodal accuracy" is not a single property. Retrieval recall, caption consensus, question-answer accuracy, denoising quality, speech error rate, and robot success rate measure different things.

The protocol should also distinguish frozen evaluation, linear probing, task-specific fine-tuning, few-shot prompting, and full training. Comparing scores across these settings without qualification can attribute gains to architecture when they come from additional labels, data, or adaptation.

For streaming and video systems, two more parameters belong in the protocol: how much of the input the model was allowed to see before answering, and at what frame rate and resolution it saw the visual track. Two reported scores on the same benchmark are not comparable if one model received 16 frames and the other received 600.

Test whether every modality matters

A multimodal model can score well while ignoring one input. Useful ablations include:

Remove each modality separately.
Replace it with noise, a blank input, or an unrelated sample.
Shuffle pairings while preserving marginal distributions.
Create counterfactual pairs in which only one modality changes.
Present contradictory modalities and inspect which source dominates.
Evaluate complete, partially missing, and naturally missing inputs.
Compare against strong unimodal baselines with similar capacity.

If performance is unchanged after shuffling images, an image-question benchmark is not demonstrating visual dependence. If the model fails only when a particular modality disappears, the deployment plan needs a fallback or a requirement that the modality is always available.

Benchmarks that can be answered without the image

The ablation above is not hypothetical hygiene. Chen and colleagues applied it to the multimodal benchmarks in common use in early 2024 and found two distinct problems.[54]

The first is that visual content is unnecessary for many samples: answers can be inferred from the question, from the answer options, or from world knowledge already inside the language model. Their headline example is that Gemini Pro reached 42.9% on MMMU with no visual input at all, and that text-only evaluation beat the random-choice baseline by more than 24 points on average across six benchmarks. The second is unintentional training-data leakage, in which a model answers a genuinely vision-dependent question without the image because it has memorized the item. Sphinx-X-MoE scored 43.6% on MMMU without access to images, 17.9 points above its own language-model backbone. In response the authors released MMStar, 1,500 samples selected first by an automated pipeline and then by human review to be visually dependent, minimally leaked, and demanding of real multimodal skill, covering six core capabilities and 18 detailed axes, together with two metrics for measuring leakage and actual multimodal gain.[54]

MMMU-Pro (September 2024) rebuilt MMMU around the same concern in three steps: filter out questions that text-only models can answer, augment the candidate options so that guessing is harder, and add a vision-only setting in which the question itself is rendered inside the image. Reported scores fell by 16.8 to 26.9 points relative to the original MMMU.[55]

Both results point the same way. A text-only baseline is not an optional robustness check on a multimodal benchmark; without it, a reported score cannot be attributed to multimodal ability at all. The same logic extends to other modalities: a video benchmark needs a frames-removed baseline, and an audiovisual benchmark needs audio-only and video-only baselines.

Task-specific metrics

Captioning illustrates why several metrics may be needed. CIDEr measures consensus with human references using weighted n-gram similarity.[30] CLIPScore measures compatibility between an image and a candidate caption without requiring a reference caption.[31] Neither directly proves that every mentioned object exists or that the caption is useful for a particular reader.

Question-answer benchmarks may use exact match, multiple-choice accuracy, or a consensus rule across annotators. Retrieval commonly reports recall at a rank or median rank. Generation may use distributional, perceptual, preference, or condition-consistency measures. Robotics requires task completion and safety outcomes. Metrics should be reported with the dataset version, preprocessing, prompts, decoding, and confidence intervals or repeated-run variation when applicable.

Diagnostic benchmarks

Benchmark or test	Design	What it can reveal	Important limit
VQA v2	Similar questions paired with complementary images and different answers	Reliance on image evidence rather than answer priors[26]	Still samples a bounded answer and image distribution
Winoground	Two images and two captions use the same words in different orders	Visio-linguistic compositionality and relation sensitivity[27]	Small diagnostic set, not a broad capability average
SugarCrepe	Hard negative captions created and adversarially refined	Whether retrieval models exploit benchmark artifacts[28]	Focuses on image-text compositionality
MMMU	Image-and-text questions across disciplines and subject areas	Perception and reasoning on varied expert-style problems[29]	Aggregate accuracy combines different skills and item types
MMStar	1,500 human-vetted items screened for visual dependence and leakage	Whether a score reflects multimodal ability rather than text priors[54]	Deliberately small; skewed toward vision-critical items
MMMU-Pro	MMMU with text-answerable items removed, more options, and a vision-only setting	How much of an MMMU score survives harder conditions[55]	Inherits MMMU's subject distribution
Video-MME	900 videos, 254 hours, 11 seconds to 1 hour, with subtitles and audio supplied	Whether video, subtitle, and audio tracks each contribute[52]	Scores depend heavily on frame budget and resolution
EgoSchema	Long-form egocentric multiple choice with temporal certificate sets	Temporal understanding length rather than clip length[71]	Egocentric domain only
BlindTest	Seven low-level geometric tasks trivial for humans	Fine-grained spatial perception and counting[56]	Synthetic shapes, not natural images
ZeroBench	Adversarially filtered to be unsolvable at release	Headroom that survives model progress[72]	Says nothing about mid-range capability
Missing-modality tests	Remove one or more inputs under defined patterns	Robustness and fallback behavior[35]	Results depend on whether missingness matches deployment

Winoground holds caption vocabulary constant while changing word order and visual relations; evaluated models in the original paper performed near chance.[27] SugarCrepe was motivated by evidence that models could exploit artifacts in prior compositionality benchmarks and uses adversarial refinement to reduce such shortcuts.[28] MMMU contains 11,500 questions across 30 subjects and many image types.[29] These details describe the original benchmark designs, not permanent leaderboards.

Benchmarks also lose usefulness by being solved. ZeroBench (February 2025) was curated by adversarial filtering to be impossible for the frontier models of its release, and its authors report initial state-of-the-art scores of 0% on both pass@1 and pass^5. Tracking progress over the following year, they report state of the art reaching 6% pass^5 and 19% pass@5 across 46 evaluated models.[72] A benchmark built this way stays informative about the top of the field for longer, at the cost of telling you nothing about the middle of it.

Hallucination and grounding

Multimodal hallucination should be defined relative to the task. In captioning, object hallucination means mentioning an object not supported by the image. CHAIR was introduced to measure this failure and showed that standard caption metrics could miss it.[32] POPE evaluates object hallucination in large vision-language models using questions about whether objects are present.[33]

Other grounding failures include invented text in a document, incorrect spatial relationships, unsupported temporal events, and answers copied from language priors. Open-ended generation needs claim-level checks against the input, not only fluency or answer preference. Human evaluators should have access to the original modalities and a rubric that separates unsupported content from harmless omission.

Longer inputs add error classes rather than more of the same one. In an hour of video sampled at one frame per second, a model can attribute an event to the wrong minute, merge two visually similar scenes, or answer about a segment the sampling never delivered. The last case is not hallucination in the usual sense: the evidence was removed by the preprocessing pipeline before the model saw it, which is a reason to report the frame budget alongside the error rate.

Robustness, calibration, and efficiency

Robustness should be evaluated separately for each modality and for their interactions. Image corruption, text perturbation, audio noise, timing offsets, and sensor dropout are not interchangeable. A study of 11 adaptation methods across four vision-language datasets, using 96 visual and 87 textual corruptions, found greater sensitivity to text corruptions in its tests and found that more adaptation examples or trainable parameters did not guarantee greater robustness.[37]

Confidence should be tested when evidence is absent or conflicting. Calibration measured on clean complete examples may not hold when one stream is corrupted. Selective prediction, abstention, or routing to a specialist can be more useful than forcing an answer.

Efficiency reports should separate parameter count, trainable parameter count, input token count, operations, peak memory, throughput, latency, and energy. A frozen language model can dominate total parameters while a small connector dominates the newly trained portion. A latent bottleneck may reduce downstream attention cost while the visual encoder remains expensive. For streaming systems, first-token or first-packet latency is a separate number from total generation time, and the two can move in opposite directions. Comparisons are meaningful only with compatible hardware, precision, batch size, resolution, duration, and decoding settings.

Limitations and failure modes

Modality dominance and shortcuts

A model may learn that one modality usually predicts the label and ignore the others. Language priors in visual question answering are a well-documented example; VQA v2 was specifically designed to counter some of them.[26] Dominance can also reverse by dataset: a prominent object may make the question irrelevant, or clean audio may make video unnecessary.

Shortcut use is not always visible in aggregate accuracy. Per-example attribution, modality ablation, balanced counterfactuals, and conflict tests can reveal it. Architectural symmetry does not guarantee behavioral balance, because signal quality and supervision can remain asymmetric.

Modality imbalance during training

The problem also appears during optimization, before any inference-time shortcut. Wang, Tran, and Feiszli found that a multimodal network can underperform its own best unimodal counterpart despite receiving strictly more information, and reported this across different modality combinations and several video classification benchmarks. They attribute it to two causes: multimodal networks have more capacity and so overfit more readily, and different modalities overfit and generalize at different rates, which makes a single joint optimization schedule suboptimal. Their proposed remedy, Gradient-Blending, computes a blending of modalities from their measured overfitting behavior.[57]

This is a training-dynamics problem rather than an architecture problem, which is why it survives changes of architecture. It is one reason the scaling study on native multimodal models reports a benefit from mixture-of-experts layers: routing lets the model learn modality-specific weights rather than forcing every parameter to serve every modality.[42] It is also a reason to report per-modality validation curves rather than only a combined loss, since a combined curve can look healthy while one branch has already stopped improving.

Fine-grained spatial perception and counting

Large multimodal models remain weak on low-level visual tasks that people find trivial. Rahmanzadehgervi and colleagues built BlindTest, a suite of seven such tasks including deciding whether two circles overlap and counting how many times two lines intersect. Four then-current models averaged 58.07%, and the best of them, Claude 3.5 Sonnet, reached 77.84%, against an expected human accuracy of 100%. Performance stayed poor across image resolutions and line widths whenever the geometric primitives overlapped or came close together. The paper's linear-probing analysis indicated that the vision encoders contained enough information to solve the tasks and that the failure lay in translating that information into an answer, which is why the revised title describes the models as failing to translate detailed visual features into words.[56]

Those specific numbers date from mid-2024 and apply to the models tested then; the failure class, counting, precise spatial relations, and fine geometric discrimination, has proved more durable than any individual score. The result also bears on the architectural argument above. If the encoder holds the evidence and the language model cannot retrieve it, the bottleneck is the interface between them, which is precisely what the adapter pattern fixes in place and native pretraining leaves learnable.

Modality gap

Contrastively aligned image and text embeddings need not occupy the same distribution. Research on the "modality gap" found that image and text embeddings in tested contrastive models occupied separated narrow regions and connected the effect to initialization and contrastive optimization.[34] Similarity can still work for retrieval, but the geometry can affect interpolation, transfer, fairness, and methods that assume complete distributional overlap.

Closing the gap is not automatically beneficial. Some separation may preserve modality-specific information. The desired geometry depends on whether the downstream task needs comparison, fusion, generation, or modality-specific discrimination.

Missingness and distribution shift

Multimodal models are often evaluated with every expected input present. In deployment, sensors fail and user-provided inputs vary. Controlled experiments found that multimodal Transformers could be sensitive to missing inputs and that the best fusion design varied by dataset.[35] Training with modality dropout can help, but it does not cover every structured failure.

Distribution shift can affect only one modality or the relationship between them. A new camera may change image statistics while text stays stable. A new captioning convention may change language without changing images. Audio and video can remain individually familiar but become misaligned. Monitoring each stream and their correspondence is therefore more informative than monitoring only final outputs.

Noisy alignment and false confidence

Weakly paired data teaches correlations, not guaranteed entailment. A caption associated with an image may mention context outside the frame. A webpage may place unrelated text near an image. A transcript may summarize rather than quote. Models trained on this data can learn plausible associations and present them with unwarranted confidence.

Filtering, re-captioning, and stronger matching losses can reduce some noise but introduce their own selection effects. Data documentation should record the source, pairing process, filters, deduplication, synthetic contributions, and excluded content.

Bias and uneven performance

Bias can enter through every modality, the pairing process, and the task labels. MMBias contains about 3,800 images and phrases covering 14 population subgroups and was used to assess bias in CLIP, ALBEF, and ViLT; the study reported meaningful bias favoring certain groups.[36] A balanced text vocabulary does not ensure balanced images, and balanced marginal datasets do not ensure balanced image-text combinations.

Evaluation should report relevant subgroups and intersections, inspect stereotyped associations, and test whether missing or low-quality inputs affect groups differently. Bias mitigation in one encoder may not survive fusion or generation. The intended use determines which subgroup and harm analyses are necessary.

Hallucination and conflict resolution

Language decoders are optimized to produce likely sequences, so they can continue fluently after losing contact with visual or acoustic evidence. Additional multimodal context can also introduce new failure modes: the system may trust an incorrect caption over an image, combine mutually inconsistent sources, or cite a detail from an earlier image in an interleaved sequence.

Grounding losses, retrieval support, constrained decoding, verification models, and preference tuning are possible interventions, but none removes the need for task-specific evaluation. A claim that a system "does not hallucinate" is too broad unless it is tied to a defined error class and test distribution.

Safety exposure introduced by image and audio inputs

Accepting images and audio widens the attack surface as well as the capability surface, because both are channels through which text can reach the model without passing a text filter.

The simplest version needs no optimization at all. The Distill study of multimodal neurons in CLIP showed that attaching a handwritten label reading "iPod" to an apple was enough to make the model classify the apple as an iPod, and that photographs of handwritten text were often sufficient. The authors named these typographic attacks and noted that they require no more technology than pen and paper.[58] The mechanism is the model's own strength: neurons responding to a concept across formats do not distinguish a depicted object from a written word naming a different one.

A stronger version turns the image into a carrier for instructions. Bagdasaryan and colleagues generated adversarial perturbations corresponding to a chosen prompt and blended them into images and audio recordings. When a user asks the model about the perturbed file, the perturbation steers the model to output attacker-chosen text and to follow the attacker's instructions for the remainder of the dialogue. They demonstrated the attack against LLaVA and PandaGPT.[59]

The attack does not require perturbations either. Nagaraja and colleagues describe a black-box image-based prompt injection pipeline that embeds instructions as rendered text inside ordinary photographs, using segmentation-based region selection, adaptive font scaling, and background-aware rendering to keep the text inconspicuous to a human while still legible to the model. Evaluating 12 prompt strategies and several embedding configurations on COCO images against GPT-4-turbo, they report up to 64% attack success under their stealth constraints.[60]

Three properties make this class structurally different from text prompt injection. The payload arrives through a channel a human reviewer often does not read closely, or at all. There is no canonical string to filter, because the attack is defined by what the model reads out of the pixels rather than by any byte sequence in the file. And an agent that browses the web or operates a screen consumes attacker-influenced pixels by default, so the exposure grows with autonomy rather than with model size. Practical mitigations, such as treating all model-read content as untrusted, constraining what an agent can do without confirmation, and separating instruction channels from data channels, are the same mitigations used for text injection, and they are no more complete here.

Generation-side safety is a separate problem with its own machinery. OpenAI describes a layered stack for 4o image generation consisting of chat-model refusals, prompt blocking by text and image classifiers, and post-generation output blocking that includes child sexual abuse material classifiers and a safety-focused reasoning monitor custom-trained to reason about content policies.[46] Input safety and output safety should be evaluated and reported separately, since a system can be strong at one and weak at the other.

Computational and information imbalance

Modalities create different token loads. A short question may accompany a high-resolution image or long video. Full attention over all tokens can be expensive, while aggressive pooling can remove the evidence needed for small objects, text, or brief events. Latent bottlenecks, frame sampling, region selection, and token pruning trade information for cost.

The imbalance also affects optimization. One stream can produce larger gradients or easier losses. Loss weighting, normalization, sampling, and curriculum design may change which modality the model uses. These implementation choices should be documented because the high-level architecture alone does not determine behavior.

Interpretability

Attention maps and embedding similarity can help inspect a model, but they do not by themselves establish causal reliance. A highlighted region may correlate with an output without being necessary for it. Removing, replacing, or counterfactually changing evidence provides stronger behavioral evidence than visualization alone.

For modular systems, debugging should inspect intermediate interfaces. Failures can arise in preprocessing, an encoder, temporal alignment, a connector, retrieval, decoding, or postprocessing. Treating the system as one opaque "multimodal model" makes corrective action harder. Native multimodal models make this harder still: with no connector to probe and no frozen encoder to test in isolation, the available interfaces are the input tokenizers and the output distribution.

Design and reporting practice

Define the contract first

Before selecting an architecture, specify:

required and optional input modalities;
their sampling rate, resolution, length, and synchronization;
the output and acceptable latency;
what should happen when an input is missing, corrupt, or contradictory;
whether examples are paired globally, locally, or in time;
whether inference requires all modalities seen during training;
whether input arrives complete or as a stream, and what the system may do before it ends;
the privacy, provenance, and retention rules for each source;
the errors that matter most for the intended use.

This contract determines whether a dual encoder, joint encoder, cross-attention connector, latent bottleneck, decision-fusion system, or jointly pretrained model is appropriate. It also defines the tests required before deployment.

Establish unimodal and simple baselines

A strong evaluation includes each unimodal component, simple feature concatenation or score fusion, and a capacity-matched multimodal alternative. If a complex model does not improve on the best unimodal baseline under realistic conditions, its cross-modal machinery may not be helping. If it improves only on complete clean inputs, the deployment benefit may be narrower than the headline score.

The published record supports treating this as mandatory rather than advisable. A multimodal network can lose to its own unimodal branch under joint training,[57] and a text-only model can score well above chance on benchmarks intended to test vision.[54] Both results are invisible without the baseline.

Document preprocessing and interfaces

Report tokenization, image transforms, audio features, frame sampling, sensor calibration, position encodings, and connector shapes. State which parameters were initialized from pre-training, which were frozen, and which were updated. For instruction-following systems, describe the prompt template and how images, audio, or other inputs are delimited.

Interfaces deserve version control. Changing an image crop, audio normalization, tokenizer, or projection checkpoint can alter the distribution seen by later modules even when the core model is unchanged. For video and audio, the sampling rate and per-frame token budget belong in the same category: they are preprocessing parameters that silently determine what the model can perceive.

Report data composition

Describe data sources, licenses where known, collection dates, languages, geography, modality combinations, missingness, synthetic data, filters, and deduplication. Give separate counts for complete pairs and unpaired or weakly paired examples. If images act as an anchor between modalities, state which direct pairs were actually observed. For jointly pretrained models, state the proportion of interleaved documents relative to isolated pairs, since the two teach different things.

Train, validation, and test splits should avoid near duplicates and source overlap where feasible. Diagnostic examples used to tune prompts or select checkpoints should not be presented as untouched evaluation.

Use a multi-axis evaluation

At minimum, report primary task performance, strong unimodal baselines, modality ablations, missing-input behavior, conflict tests, subgroup results relevant to the use, calibration, and resource cost. For generation, add grounding and human evaluation. For temporal systems, vary synchronization and sampling. For robotics or other consequential control, evaluate recovery and safety separately from average task success.

An overall mean can be useful for summary, but it should not replace per-task and per-condition results. A model that gains on easy complete examples and fails under one critical sensor fault may have a higher mean and still be unsuitable.

Keep claims at the level of evidence

An architecture paper supports claims about the reported architecture and experiments. It does not establish that every implementation with the same label has the same components. A benchmark supports performance on its sampled items under a stated protocol. It does not prove general perception, reasoning, or safety. A commercial interface demonstrates exposed behavior, not undisclosed internals.

The document type is itself evidence about how much is known. A technical report with released weights supports claims about objectives, fusion points, and training data. A model card supports whatever it states and nothing beyond it, which for recent frontier models has often meant one sentence of architecture and no parameter count.[47] A system card scoped to safety supports claims about mitigations and measured risks, not about structure.[41][46] A product blog post supports claims about behavior and availability only, and several significant 2026 releases have shipped with nothing more.[68][70] Writing "not disclosed" is more accurate than inferring the architecture from a competitor's paper, and it stays correct when the inference would not have.

This discipline is particularly important in a rapidly changing field. Durable descriptions focus on mechanisms, data assumptions, failure modes, and reproducible protocols rather than model-release chronology.

See also

References

Baltrusaitis, T., Ahuja, C., and Morency, L.-P. (2019). "Multimodal Machine Learning: A Survey and Taxonomy." IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423-443. https://arxiv.org/abs/1705.09406
Liang, P. P., Zadeh, A., and Morency, L.-P. (2024). "Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions." ACM Computing Surveys, 56(10). https://arxiv.org/abs/2209.03430
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Y. (2011). "Multimodal Deep Learning." Proceedings of the 28th International Conference on Machine Learning. https://ai.stanford.edu/~jngiam/papers/NgiamKhoslaKimNamLeeNg2011.pdf
Srivastava, N., and Salakhutdinov, R. R. (2012). "Multimodal Learning with Deep Boltzmann Machines." Advances in Neural Information Processing Systems 25. https://proceedings.neurips.cc/paper_files/paper/2012/hash/af21d0c97db2e27e13572cbf59eb343d-Abstract.html
Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015). "Show and Tell: A Neural Image Caption Generator." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content_cvpr_2015/html/Vinyals_Show_and_Tell_2015_CVPR_paper.html
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015). "VQA: Visual Question Answering." Proceedings of the IEEE International Conference on Computer Vision. https://openaccess.thecvf.com/content_iccv_2015/html/Antol_VQA_Visual_Question_ICCV_2015_paper.html
Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). "Attention Is All You Need." Advances in Neural Information Processing Systems 30. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
Lu, J., Batra, D., Parikh, D., and Lee, S. (2019). "ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks." Advances in Neural Information Processing Systems 32. https://proceedings.neurips.cc/paper_files/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html
Tan, H., and Bansal, M. (2019). "LXMERT: Learning Cross-Modality Encoder Representations from Transformers." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. https://aclanthology.org/D19-1514/
Chen, Y.-C., Li, L., Yu, L., et al. (2020). "UNITER: Universal Image-Text Representation Learning." Computer Vision - ECCV 2020. https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/7093_ECCV_2020_paper.php
Kim, W., Son, B., and Kim, I. (2021). "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision." Proceedings of the 38th International Conference on Machine Learning. https://proceedings.mlr.press/v139/kim21k.html
Radford, A., Kim, J. W., Hallacy, C., et al. (2021). "Learning Transferable Visual Models From Natural Language Supervision." Proceedings of the 38th International Conference on Machine Learning. https://proceedings.mlr.press/v139/radford21a.html
Jia, C., Yang, Y., Xia, Y., et al. (2021). "Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision." Proceedings of the 38th International Conference on Machine Learning. https://proceedings.mlr.press/v139/jia21b.html
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. (2023). "Sigmoid Loss for Language Image Pre-Training." Proceedings of the IEEE/CVF International Conference on Computer Vision. https://openaccess.thecvf.com/content/ICCV2023/html/Zhai_Sigmoid_Loss_for_Language_Image_Pre-Training_ICCV_2023_paper.html
Li, J., Selvaraju, R. R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. (2021). "Align before Fuse: Vision and Language Representation Learning with Momentum Distillation." Advances in Neural Information Processing Systems 34. https://proceedings.neurips.cc/paper_files/paper/2021/hash/505259756244493872b7709a8a01b536-Abstract.html
Li, J., Li, D., Xiong, C., and Hoi, S. (2022). "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation." Proceedings of the 39th International Conference on Machine Learning. https://proceedings.mlr.press/v162/li22n.html
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. (2021). "Perceiver: General Perception with Iterative Attention." Proceedings of the 38th International Conference on Machine Learning. https://proceedings.mlr.press/v139/jaegle21a.html
Alayrac, J.-B., Donahue, J., Luc, P., et al. (2022). "Flamingo: a Visual Language Model for Few-Shot Learning." Advances in Neural Information Processing Systems 35. https://proceedings.neurips.cc/paper_files/paper/2022/hash/960a172bc7fbf0177ccccbb411a7d800-Abstract-Conference.html
Li, J., Li, D., Savarese, S., and Hoi, S. (2023). "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." Proceedings of the 40th International Conference on Machine Learning. https://proceedings.mlr.press/v202/li23q.html
Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023). "Visual Instruction Tuning." Advances in Neural Information Processing Systems 36. https://proceedings.neurips.cc/paper_files/paper/2023/hash/6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. (2023). "ImageBind: One Embedding Space To Bind Them All." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content/CVPR2023/html/Girdhar_ImageBind_One_Embedding_Space_To_Bind_Them_All_CVPR_2023_paper.html
Shi, B., Hsu, W.-N., Lakhotia, K., and Mohamed, A. (2022). "Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction." International Conference on Learning Representations. https://arxiv.org/abs/2201.02184
Hsu, W.-N., and Shi, B. (2022). "u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality." Advances in Neural Information Processing Systems 35. https://proceedings.neurips.cc/paper_files/paper/2022/hash/853e781cb2af58956ed5c89aa59da3fc-Abstract-Conference.html
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). "High-Resolution Image Synthesis With Latent Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content/CVPR2022/html/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.html
Zitkovich, B., Yu, T., Xu, S., et al. (2023). "RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control." Proceedings of the 7th Conference on Robot Learning. https://proceedings.mlr.press/v229/zitkovich23a.html
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. (2017). "Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content_cvpr_2017/html/Goyal_Making_the_v_CVPR_2017_paper.html
Thrush, T., Jiang, R., Bartolo, M., et al. (2022). "Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content/CVPR2022/html/Thrush_Winoground_Probing_Vision_and_Language_Models_for_Visio-Linguistic_Compositionality_CVPR_2022_paper.html
Hsieh, C.-Y., Zhang, J., Ma, Z., Kembhavi, A., and Krishna, R. (2023). "SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality." Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track. https://proceedings.neurips.cc/paper_files/paper/2023/hash/63461de0b4cb760fc498e85b18a7fe81-Abstract-Datasets_and_Benchmarks.html
Yue, X., Ni, Y., Zhang, K., et al. (2024). "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content/CVPR2024/html/Yue_MMMU_A_Massive_Multi-discipline_Multimodal_Understanding_and_Reasoning_Benchmark_for_CVPR_2024_paper.html
Vedantam, R., Zitnick, C. L., and Parikh, D. (2015). "CIDEr: Consensus-based Image Description Evaluation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content_cvpr_2015/html/Vedantam_CIDEr_Consensus-Based_Image_2015_CVPR_paper.html
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. (2021). "CLIPScore: A Reference-free Evaluation Metric for Image Captioning." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. https://aclanthology.org/2021.emnlp-main.595/
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. (2018). "Object Hallucination in Image Captioning." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. https://aclanthology.org/D18-1437/
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. (2023). "Evaluating Object Hallucination in Large Vision-Language Models." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. https://aclanthology.org/2023.emnlp-main.20/
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. (2022). "Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning." Advances in Neural Information Processing Systems 35. https://proceedings.neurips.cc/paper_files/paper/2022/hash/702f4db7543a7432431df588d57bc7c9-Abstract-Conference.html
Ma, M., Ren, J., Zhao, L., Testuggine, D., and Peng, X. (2022). "Are Multimodal Transformers Robust to Missing Modality?" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content/CVPR2022/html/Ma_Are_Multimodal_Transformers_Robust_to_Missing_Modality_CVPR_2022_paper.html
Janghorbani, S., and De Melo, G. (2023). "Multi-Modal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision-Language Models." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. https://aclanthology.org/2023.eacl-main.126/
Chen, S., Gu, J., Han, Z., Ma, Y., Torr, P. H. S., and Tresp, V. (2023). "Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models." Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track. https://proceedings.neurips.cc/paper_files/paper/2023/hash/a2a544e43acb8b954dc5846ff0d77ad5-Abstract-Datasets_and_Benchmarks.html
Chameleon Team (2024). "Chameleon: Mixed-Modal Early-Fusion Foundation Models." arXiv preprint, 16 May 2024. https://arxiv.org/abs/2405.09818
Gemini Team, Google (2023). "Gemini: A Family of Highly Capable Multimodal Models." arXiv preprint, 19 December 2023. https://arxiv.org/abs/2312.11805
Gemini Team, Google (2024). "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context." arXiv preprint. https://arxiv.org/abs/2403.05530
OpenAI (2024). "GPT-4o System Card." arXiv preprint, 25 October 2024. https://arxiv.org/abs/2410.21276
Shukor, M., Fini, E., Turrisi da Costa, V. G., Cord, M., Susskind, J., and El-Nouby, A. (2025). "Scaling Laws for Native Multimodal Models." Proceedings of the IEEE/CVF International Conference on Computer Vision. arXiv preprint, 10 April 2025. https://arxiv.org/abs/2504.07951
Meta AI (2025). "The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation." 5 April 2025. https://ai.meta.com/blog/llama-4-multimodal-intelligence/
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. (2024). "Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model." arXiv preprint, 20 August 2024. https://arxiv.org/abs/2408.11039
Wu, C., Chen, X., Wu, Z., et al. (2024). "Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation." arXiv preprint, 17 October 2024. https://arxiv.org/abs/2410.13848
OpenAI (2025). "Addendum to GPT-4o System Card: Native image generation." 25 March 2025. https://cdn.openai.com/11998be9-5319-4302-bfbf-1167e093f1fb/Native_Image_Generation_System_Card.pdf
Google DeepMind (2026). "Gemini 3 Pro Model Card." Model release November 2025, last updated May 2026. https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf
Qwen Team (2025). "Qwen3-Omni Technical Report." arXiv preprint, 22 September 2025. https://arxiv.org/abs/2509.17765
Qwen Team (2026). "Qwen3.5-Omni Technical Report." arXiv preprint, 17 April 2026. https://arxiv.org/abs/2604.15804
Google (2026). "Live API." Gemini API documentation. https://ai.google.dev/gemini-api/docs/live
Google (2026). "Video understanding." Gemini API documentation. https://ai.google.dev/gemini-api/docs/video-understanding
Fu, C., Dai, Y., Luo, Y., et al. (2024). "Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis." arXiv preprint, 31 May 2024. https://arxiv.org/abs/2405.21075
Shao, K., Tao, K., Zhang, K., et al. (2025). "A Survey of Token Compression for Efficient Multimodal Large Language Models." arXiv preprint, 27 July 2025. https://arxiv.org/abs/2507.20198
Chen, L., Li, J., Dong, X., et al. (2024). "Are We on the Right Way for Evaluating Large Vision-Language Models?" Advances in Neural Information Processing Systems 37. arXiv preprint, 29 March 2024. https://arxiv.org/abs/2403.20330
Yue, X., Zheng, T., Ni, Y., et al. (2024). "MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark." arXiv preprint, 4 September 2024. https://arxiv.org/abs/2409.02813
Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., and Nguyen, A. T. (2024). "Vision language models are blind: Failing to translate detailed visual features into words." arXiv preprint, 9 July 2024. https://arxiv.org/abs/2407.06581
Wang, W., Tran, D., and Feiszli, M. (2020). "What Makes Training Multi-Modal Classification Networks Hard?" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content_CVPR_2020/html/Wang_What_Makes_Training_Multi-Modal_Classification_Networks_Hard_CVPR_2020_paper.html
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., and Olah, C. (2021). "Multimodal Neurons in Artificial Neural Networks." Distill, 6(3). https://distill.pub/2021/multimodal-neurons/
Bagdasaryan, E., Hsieh, T.-Y., Nassi, B., and Shmatikov, V. (2023). "Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs." arXiv preprint, 19 July 2023. https://arxiv.org/abs/2307.10490
Nagaraja, N., Zhang, L., Wang, Z., Zhang, B., and Patil, P. (2026). "Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions." arXiv preprint, 4 March 2026. https://arxiv.org/abs/2603.03637
Bolya, D., Huang, P.-Y., Sun, P., et al. (2025). "Perception Encoder: The best visual embeddings are not at the output of the network." arXiv preprint, 17 April 2025. https://arxiv.org/abs/2504.13181
Qin, Y., Ye, Y., Fang, J., et al. (2025). "UI-TARS: Pioneering Automated GUI Interaction with Native Agents." arXiv preprint, 21 January 2025. https://arxiv.org/abs/2501.12326
Chu, Y., Xu, J., Yang, Q., et al. (2024). "Qwen2-Audio Technical Report." arXiv preprint, 15 July 2024. https://arxiv.org/abs/2407.10759
Seamless Communication team, Meta AI (2023). "SeamlessM4T: Massively Multilingual and Multimodal Machine Translation." arXiv preprint, 22 August 2023. https://arxiv.org/abs/2308.11596
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. (2023). "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face." arXiv preprint, 30 March 2023. https://arxiv.org/abs/2303.17580
Kim, G., Hong, T., Yim, M., et al. (2022). "OCR-free Document Understanding Transformer." Computer Vision - ECCV 2022. arXiv preprint, 30 November 2021. https://arxiv.org/abs/2111.15664
Google DeepMind (2026). "Gemini Robotics On-Device 2: Model Card." Published 30 July 2026. https://deepmind.google/models/model-cards/gemini-robotics-on-device-2/
Meta (2026). "Introducing Muse Image and Muse Video." 7 July 2026. https://ai.meta.com/blog/introducing-muse-image-muse-video-msl/
Google (2025). "Nano Banana Pro: Gemini 3 Pro Image model from Google DeepMind." November 2025. https://blog.google/innovation-and-ai/products/nano-banana-pro/
Google (2026). "Introducing Gemini Omni." May 2026. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-omni/
Mangalam, K., Akshulakov, R., and Malik, J. (2023). "EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding." arXiv preprint, 17 August 2023. https://arxiv.org/abs/2308.09126
Roberts, J., Taesiri, M. R., Sharma, A., et al. (2025). "ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models." arXiv preprint, 13 February 2025. https://arxiv.org/abs/2502.09696
Generalist AI (2026). "GEN-1: Scaling Embodied Foundation Models to Mastery." 2 April 2026. https://generalistai.com/blog/gen-1
Bloomberg (2026). "Mistral AI Releases Robotics Model to Support Physical AI Push." 8 July 2026. https://www.bloomberg.com/news/articles/2026-07-08/mistral-ai-releases-robotics-model-to-support-physical-ai-push

Editing: Multimodal Model