Citation and evidence

ImageBind

7 min full readUpdated 8 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

AI ModelsMeta AIMultimodal AI

Cite this article

ImageBind is a multimodal model from Meta AI (its Fundamental AI Research lab) that learns a single joint embedding space across six different modalities: images and video, text, audio, depth, thermal (infrared) data, and inertial measurement unit (IMU) signals. It was announced on 9 May 2023, alongside open-source code and model weights, and the accompanying paper, "ImageBind: One Embedding Space To Bind Them All" by Rohit Girdhar and colleagues, was published as a highlight at CVPR 2023 [1][2][3].

Related: ModalityEgo4DCLIP (Contrastive Language-Image Pre-training)Make-A-Scene

Meta logo
Meta logo. Source: Wikimedia Commons, Public domain.

The central idea is that you do not need training data in which all six modalities appear together. Instead, each modality is aligned only to images, and that shared anchor is enough for the rest of the modalities to line up with one another. Meta calls this an "emergent" alignment: pairs the model never saw together, such as audio and text, end up close in the embedding space because each was independently bound to images [2][3]. This lets a single model perform zero-shot tasks like retrieving an image from a sound, and even combine embeddings arithmetically across modalities.

The binding idea

Most prior work on joint embeddings needed paired examples for each combination of modalities it wanted to relate. Collecting such data for every pair (audio with depth, thermal with text, and so on) does not scale. ImageBind sidesteps this by treating images as a hub. Images naturally co-occur with many other signals: a video frame comes with its soundtrack, a photo from an RGB-D camera comes with a depth map, a scene from a thermal camera comes with an infrared frame, and footage from a head-mounted camera comes with motion sensor readings. By binding each non-image modality to images, all of the modalities become mutually comparable through that common reference [2][3].

Training uses the standard InfoNCE contrastive objective. For each modality, the model is given naturally paired (image-or-video, modality) examples and learns to pull matching pairs together and push non-matching pairs apart in the embedding space [4][5]. Only image-paired data is used; no (audio, depth) or (text, thermal) pairs are ever shown, yet those relationships emerge. The strength of the emergent behavior scales with the quality of the image encoder: a stronger vision backbone produces better zero-shot results on the non-vision modalities [1][6].

Architecture and backbone

ImageBind reuses large-scale pretrained image-text models rather than training vision and language understanding from scratch. The image and text encoders are initialized from a pretrained CLIP-style model (OpenCLIP) and kept frozen; the image encoder is a ViT-H with roughly 630M parameters and the text encoder has on the order of 300M parameters. The encoders for the remaining modalities are trained to map their inputs into the same space as these frozen image features [4][5].

Each modality has a dedicated encoder, but they share the Transformer family of architectures. Depth and thermal data are treated as single-channel images and passed through a Vision Transformer. Audio is converted to a mel-spectrogram and encoded with a Vision Transformer over spectrogram patches. IMU data (a multi-channel time series of accelerometer and gyroscope readings) is split into fixed-length clips, projected with a 1D convolution, and encoded with a Transformer [4][5].

ModalityEncoderPaired data source for binding
Image / videoViT-H, frozen OpenCLIP weights (~630M params)Web-scale image-text (with text encoder)
TextFrozen OpenCLIP text encoder (~300M params)Web-scale image-text
AudioVision Transformer over mel-spectrogram patchesAudioSet (video paired with its audio)
DepthVision Transformer (depth as 1-channel image)SUN RGB-D (images paired with depth)
ThermalVision Transformer (thermal as 1-channel image)LLVIP (visible images paired with infrared)
IMU1D convolution then TransformerEgo4D (egocentric video paired with IMU)

Expanded article table

The naturally occurring pairings come from existing datasets: video and audio from AudioSet, image and depth from SUN RGB-D, image and thermal from LLVIP, and video and IMU from Ego4D, in addition to large web image-text collections for the vision-language side [4][5]. Because depth, thermal, and audio are recast into image-like or spectrogram representations, the same contrastive recipe applies uniformly across them.

Emergent zero-shot capabilities

Once trained, ImageBind supports several behaviors without any task-specific fine-tuning:

  • Cross-modal retrieval. Given a query in one modality, retrieve matching items in another. The model can fetch images from audio clips, retrieve audio for a text prompt, or find depth maps for an image, including pairs it never saw together during training [2][3].
  • Embedding-space arithmetic. Because all modalities share one space, their embeddings can be added to compose meaning. Combining an image of fruits on a table with the sound of chirping birds, for example, retrieves an image of fruits on trees with birds [1].
  • Audio-driven generation. Feeding ImageBind audio embeddings into the decoder of a pretrained image generator (Meta demonstrated this with a DALLE-2-style decoder) produces images from sounds such as rain, waves, fire, or an engine [1].
  • Cross-modal detection and few-shot recognition. The paper also shows the embeddings can upgrade detectors to new modalities and serve as strong few-shot recognizers [1][6].

On standard benchmarks, ImageBind set new results for emergent zero-shot recognition across modalities, in several cases matching or surpassing specialist models trained with direct supervision on the target modality. Meta's released checkpoint (imagebind_huge) reports the following zero-shot classification accuracies [3][6]:

BenchmarkModalityTop-1 accuracy
ImageNet-1kImage77.7%
Kinetics-400Video50.0%
NYU Depth v2Depth54.0%
ESC-50Audio66.9%
LLVIPThermal63.4%
Ego4DIMU25.0%

Expanded article table

On audio specifically, ImageBind's emergent zero-shot classification and retrieval matched or beat prior specialist audio-text models on tasks built from datasets such as ESC, Clotho, and AudioCaps, despite never being trained on paired audio-text data [1][6].

Release, license, and use in later work

Meta released the code and the imagebind_huge weights publicly on GitHub (facebookresearch/ImageBind), under a non-commercial Creative Commons license, which permits research use but not commercial use. The repository is inconsistent about which one: the README names CC-BY-NC 4.0, while the LICENSE file it points to is the text of CC BY-NC-SA 4.0, which adds a share-alike condition [3]. A project page with interactive demos was published at imagebind.metademolab.com [2].

Meta framed ImageBind partly as a foundation for generative and search systems. The announcement noted that the embeddings could let a text-to-image system such as Make-A-Scene generate images from audio (for example, the sounds of a market or a rainforest), and more broadly enable richer multimodal search and content understanding [2]. The work sits in a line of Meta research on aligning many modalities to a single representation and on extending pretrained vision-language models to senses beyond sight and language.

Relationship to other multimodal binders

ImageBind should not be confused with Meta-Transformer, a separate framework introduced in July 2023 by a different group (Yiyuan Zhang, Kaixiong Gong and colleagues, arXiv 2307.10802). Meta-Transformer targets twelve modalities, including point clouds, hyperspectral and X-ray images, graphs, tabular and time-series data, and uses a modality-shared encoder with frozen parameters and a unified tokenizer; it does not rely on a pretrained large language model (its shared encoder is a ViT pretrained contrastively on LAION-2B) and reports results on several tasks where it outperforms ImageBind [7][8]. The naming is coincidental: Meta-Transformer is not a Meta AI product [8]. ImageBind's distinguishing contribution is specifically the image-as-hub binding that produces emergent alignment from image-paired data alone.

Continue reading

More to read

Topics:AI ModelsMeta AIMultimodal AI

References

  1. ^1 ^2 ^3 ^4 ^5 ^6Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., Misra, I. "ImageBind: One Embedding Space To Bind Them All." arXiv:2305.05665, May 2023. arxiv.org/...2305.05665
  2. ^1 ^2 ^3 ^4 ^5 ^6Meta AI. "ImageBind: Holistic AI learning across six modalities." Meta AI Blog, 9 May 2023. ai.meta.com/...imagebind-six-modalities-binding-ai
  3. ^1 ^2 ^3 ^4 ^5 ^6facebookresearch/ImageBind, GitHub repository (README, checkpoints, license, benchmark table). github.com/...ImageBind
  4. ^1 ^2 ^3 ^4"ImageBind: One Embedding Space To Bind Them All." CREATIS MYRIAD paper review (encoder and dataset details). creatis-myriad.github.io/...ImageBind
  5. ^1 ^2 ^3 ^4"ImageBind MultiJoint Embedding Model from Meta Explained." Encord Blog. encord.com/...imagebind-embedding-model-explained
  6. ^1 ^2 ^3 ^4Girdhar et al. "ImageBind: One Embedding Space To Bind Them All." CVPR 2023 Open Access. openaccess.thecvf.com/..._Them_All_CVPR_2023_paper
  7. ^Zhang, Y., Gong, K., et al. "Meta-Transformer: A Unified Framework for Multimodal Learning." arXiv:2307.10802, July 2023. arxiv.org/...2307.10802
  8. ^1 ^2"Meta-Transformer: A Unified Framework for Multimodal Learning." Encord Blog. encord.com/...meta-transformer-explained

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

7 revisions · v7 · 1,409 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check of the full article against its cited sources (45 claims, 9 sources fetched); 0 material + 5 minor fixed

Cite this page: AI Wiki. "ImageBind." aiwiki.ai, updated 17 Nov 2026, fact-checked 5 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/imagebind

Edit article