WeMM-Embedding

WeMM-Embedding (WeChat Multi-Modal Embedding) is a family of open-weight universal multimodal embedding models built by the WeChat Vision team at Tencent. Released on August 25, 2026 in 2B, 4B and 9B sizes, the models map text, images, videos, visual documents and arbitrarily interleaved combinations of those inputs into one vector space, so a text query can be scored against a video, a product photo against an article, or a screenshot against a tool description with a single model.^[1][2] All three are fine-tuned from the matching Qwen3.5 checkpoints and return L2-normalized vectors of 2,048, 2,560 or 4,096 dimensions that can be truncated down to 64 dimensions without retraining, using Matryoshka representation learning.^[2][4] Audio input is not supported.^[2]

Tencent's technical report (arXiv:2608.24053) states that the 9B model scored 80.6 overall on the MMEB-v2 benchmark, which the report describes as first on the official leaderboard as of August 24, 2026, and that the 2B model already outscores the previously leading 8B open-source baseline.^[1] The same report says the models were serving recommendation and search inside WeChat Channels, Official Accounts, Moments and e-commerce before the public release, with gains in 14 online A/B tests.^[1] Weights are on Hugging Face; the code, the MMEB-v3 evaluation harness and the license file are on GitHub.^[2][3]

Facts

Scroll sideways for more →

Background

Most open releases from Tencent AI come from the Hunyuan foundation-model group. WeMM-Embedding comes instead from a product organization, the WeChat Vision team inside the Weixin business, which the Chinese README names as 微信视觉团队.^[1][10] The report lists six authors: Junjie Zhou, Ke Mei and Lei Li are marked as core contributors, Ke Mei as project leader, and Fengyun Rao as corresponding author, with Tianyi Wang and Jing Lyu completing the list.^[1]

The models exist to serve information retrieval and recommender systems on a platform where a single item may be a short video with a cover image and a transcript, an article with inline pictures, or a product listing. The report's related-work section sets out why the authors prefer a multimodal-LLM backbone to CLIP-style dual encoders: separate text and image towers cannot naturally represent inputs that mix modalities, such as interleaved text-image documents, composed multimodal queries, or videos paired with transcripts.^[1] The alternative, which VLM2Vec, GME, mmE5 and Qwen3-VL-Embedding had already explored, is to fine-tune a multimodal large language model so that one of its hidden states becomes the embedding.^[1] WeMM-Embedding follows that line and builds on the natively multimodal Qwen3.5 backbones; Qwen3-VL-Embedding, the Qwen team's own embedding model, is the strongest open-weight baseline in every comparison table of the report.^[1]

Release timeline

Scroll sideways for more →

Model family

Scroll sideways for more →

The size labels follow the Qwen3.5 checkpoints the models were fine-tuned from; the "stored parameters" column is the safetensors total reported by the Hugging Face Hub API for each repository, all in BF16.^[2][8] Every repository ships with custom modeling code (modeling_wemm_embedding.py and a Sentence Transformers wrapper), a chat template for embedding inputs, a patch_sglang_video.py helper and a config.json whose architectures entry is Qwen3_5ForConditionalGeneration with a matryoshka_dimensions list.^[4][8] The full-dimension figure for each size equals the text hidden size in that model's config.json.^[4][5][6][8]

Architecture and embedding extraction

WeMM-Embedding keeps the Qwen3.5 architecture intact and changes how the output is read. A dedicated <embedding> token is appended to the input sequence, and the final-layer hidden state at that position, after L2 normalization, is the embedding.^[1][2] Because the backbone uses a causal attention mask, the token attends to everything before it, whether text, image patches or video frames, which is what lets a single forward pass encode an interleaved sequence.^[1]

The same causal formulation allows more than one <embedding> token in a sequence. The report gives the example of a video followed by its speech-recognition transcript: one token placed after the video tokens and a second at the end of the sequence yield a video-only embedding and a joint video-plus-text embedding from one pass, for downstream systems with different modality requirements.^[1]

Matryoshka representation learning, introduced by Kusupati and colleagues in 2022, is applied during training so that any prefix of the vector at a supported length is itself a usable embedding.^[1][18] At inference, an embedding of dimension d is obtained by keeping the first d coordinates and normalizing again; the README shows this as a one-line torch.nn.functional.normalize(embedding[..., :d], dim=-1).^[2] Similarity between embeddings is cosine similarity, which for normalized vectors is a dot product.^[1]

Training data

The training corpus is described only at the level of families and scale. Stage 1 draws on "several hundred million" source-target pairs collected from public datasets, web-scale weakly supervised sources, task-oriented synthetic data and in-house collections, spanning text, images, videos and their interleaved combinations.^[1] Every example is written in one pair-based format: an optional task instruction, a source, a paired target, an optional set of explicit hard negatives, and an optional graded relevance score.^[1] The report lists six data families:^[1]

Scroll sideways for more →

The Stage 2 corpus is a curated set roughly one tenth the size of the large collection.^[1] Three steps produce it. First, a Semantic-ID-guided resampling: the longer side of each pair is encoded with an intermediate WeMM-Embedding checkpoint, a three-level residual k-means quantizer is fitted to those vectors, and each pair receives a three-element Semantic ID; pairs whose codes are crowded are sampled less often, so that frequent patterns stop dominating without forcing a uniform distribution.^[1] Second, a multimodal LLM checks whether each pair actually reflects the intended matching relation, drops mismatches, and rewrites noisy alt-text-style descriptions to fix factual errors while keeping their style.^[1] Third, a subset is enriched with explicit hard negatives: for text targets an MLLM writes plausible but wrong candidates, and for image and video targets intermediate checkpoints retrieve similar candidates from task-specific pools, a smaller fraction of which are then scored by reranking models.^[1]

Training strategy

Stage 1: large-scale multimodal alignment

The first stage trains on the full collection with batches that each come from a single data source, interleaved across tasks over the course of training.^[1] Standard pairs use an InfoNCE contrastive learning objective with in-batch negatives plus any explicit hard negatives, and a learnable temperature.^[1] Two details matter for the numbers reported later. Task-consistent batching keeps the task definition and candidate space the same within a batch, which the authors argue produces more informative negatives than mixed sampling.^[1] Duplicate-aware masking removes, from the negative pool, any candidate whose source or target is a near-duplicate of the current pair's (measured by a similarity threshold), which avoids the false negatives that appear when many examples share a label, as in classification.^[1]

Examples carrying graded relevance labels use a different loss: a score-gap-weighted CoSENT-style ranking objective that pushes higher-labelled pairs above lower-labelled ones and weights each comparison by the size of the label gap, with no constraint between pairs of equal label.^[1] Both losses are evaluated at every supported Matryoshka dimension and summed with per-dimension weights, so each batch is optimized in its multi-dimensional form.^[1]

Stage 2: curated fine-tuning and distillation

The second stage continues on the curated corpus with the same two losses and adds two sources of supervision. Dedicated multimodal rerankers, trained by the team, score query-specific candidate sets (one annotated positive plus mined hard negatives), and those scores replace the manual relevance levels in the same ranking loss, with comparisons restricted to candidates of the same query.^[1] The authors note that reranking candidates retrieved by their own embedding models "does not consistently improve performance across multimodal tasks", so reranker supervision is used only where it gave stable gains.^[1]

The second addition is embedding distillation, a form of knowledge distillation in which a larger WeMM-Embedding model acts as teacher. For each batch, teacher and student compute source-to-target and target-to-source similarity matrices, each row is turned into a softmax distribution with its own temperature, and the student minimizes the KL divergence to the teacher's distributions in both directions.^[1] Unlike one-hot contrastive targets, the teacher's distributions preserve relative similarities among candidates; the report says this "contributes substantially" to the smaller models' gains.^[1] For the 2B and 4B models the frozen 9B model is the teacher. No larger teacher exists for the 9B model, so the team trained several specialized Stage 2 variants on complementary data mixtures and combined them through model merging, citing TIES-Merging.^[1]

Benchmark results

MMEB-v2

MMEB (Massive Multimodal Embedding Benchmark) was introduced with the VLM2Vec paper in October 2024 as 36 datasets across four meta-tasks: classification, visual question answering, multimodal retrieval and visual grounding.^[14] MMEB-V2, released with VLM2Vec-V2 in July 2025, added visual-document retrieval, video retrieval, temporal grounding, video classification and video question answering, for 78 datasets over text, image, video and visual-document inputs.^[15] The leaderboard is a Hugging Face Space maintained by TIGER-Lab, separate from the MTEB text leaderboard.^[17] In the report's protocol, image and video tasks are scored with Hit@1 and visual-document tasks with NDCG@5.^[1][2]

The table below reproduces the summary version of Table 1 from the report as published in the GitHub README; baseline rows are the leaderboard's figures as of August 24, 2026.^[1][2]

Scroll sideways for more →

"Closed" marks what the report calls a closed-source leaderboard submission without public weights or a public inference endpoint; DME is the Douyin Multimodal Embedding model in the report's reference list.^[1][2] The full Table 1 also lists proprietary models with undisclosed parameter counts: Seed-1.6-Embedding at 76.9, QQMM-embed-v4 at 78.3, Octen-VL-Large at 80.1 and DME-Large at 80.2, all below the 9B model's 80.6.^[1] The finer breakdown for the three WeMM models, from the same table, is:^[1]

Scroll sideways for more →

Numbers in parentheses are dataset counts per group. The report's own summary of the table: the 2B model beats Qwen3-VL-Embedding-2B by 4.7 points and DME-Small by 3.1, and "slightly" passes Qwen3-VL-Embedding-8B; the 4B model outscores every 8B and 9B baseline; and the 9B model ranks first on the leaderboard among open and proprietary entries listed as of August 24, 2026.^[1]

MMEB-v3

MMEB-V3, published in April 2026 and accepted at COLM 2026, extends MMEB-V2's 78 tasks to 190 by adding 53 text tasks, 47 agent tasks, 11 audio tasks and the MCMR image-retrieval task: audio tasks (classification, cross-modal retrieval, temporal grounding), text retrieval (instruction-following, reasoning, long-context, multi-condition and general retrieval) and agent tasks (tool retrieval, GUI control and agent memory retrieval), plus a diagnostic set of semantically equivalent tuples across modalities.^[16][22] The V3-All score averages all 190 tasks and assigns zero to tasks a model cannot run; because WeMM-Embedding has no audio input, its 11 audio tasks count as zero.^[1][2] The report recomputed V3-All for VLM2Vec, VLM2Vec-V2, GME and Qwen3-VL-Embedding by combining their published MMEB-v2 results with their scores on the new tasks.^[1] The table below is Table 2 as reproduced in the README.^[1][2]

Scroll sideways for more →

Text results use NDCG@5; agent, MCMR and audio results use Hit@1.^[2] Within the text group, the 9B model's weakest sub-score is reasoning retrieval (31.8) and its strongest is long-context retrieval (64.5); on multi-condition retrieval (58.3) it trails Omni-Embed-Nemotron-3B (69.7), and on general retrieval (61.1) it trails GME-8B (62.5) and Tianmu-Emb-Uni (62.0).^[1] The audio-capable models in the table (Omni-Embed-Nemotron, E5-Omni, WAVE, LCO-Embedding-Omni, Tianmu-Emb-Uni) are the only ones with non-zero audio scores, so the V3-All comparison mixes models of different modality coverage by design.^[1]

Cross-modal retrieval suite

The report also runs the 12-dataset cross-modal retrieval suite used in Google's Gemini Embedding 2 report, taking the proprietary models' figures from that report and evaluating Qwen3-VL-Embedding-8B and WeMM-Embedding itself.^[1] The suite has four text-to-image and four image-to-text datasets (MSCOCO, Flickr30k, DOCCI, TextCaps; Recall@1), three text-to-video datasets (VATEX, MSR-VTT, YouCook2; NDCG@10) and ViDoRe V2 for document retrieval (NDCG@10).^[1]

Scroll sideways for more →

The average favors WeMM-Embedding at every size, but the pattern is uneven. Gemini Embedding 2 has the best text-to-image mean of the seven systems (83.8 against the 9B model's 83.6) and beats every WeMM model except the 4B on image-to-text (91.2 against 91.4 for the 4B and 90.3 for the 9B); the WeMM lead comes largely from text-to-video, where the 9B model scores 72.9 against Gemini's 63.1.^[1] On ViDoRe V2 the 2B and 4B models (61.4 and 62.1) trail Gemini Embedding 2 (64.9) and Voyage Multimodal 3.5 (65.5); only the 9B model (66.3) leads that column.^[1]

Ablations

Two studies on the 2B model explain where the score comes from. A small-scale Stage 1 ablation (run at reduced scale, so its baseline of 71.9 is below the full Stage 1 checkpoint) shows that task-consistent batching is the most important of the three design choices tested: replacing it with mixed sampling costs 3.4 points, against 0.8 for removing task instructions and 0.5 for removing duplicate-aware masking.^[1]

Scroll sideways for more →

The cumulative Stage 2 study starts from the full Stage 1 checkpoint and adds each strategy in turn. The four additions together add 2.2 points; curated data and distillation from the 9B teacher contribute 0.9 points each, reranker supervision 0.1, and the larger visual input budget 0.3.^[1]

Scroll sideways for more →

The last row raises image resolution and samples video frames more densely, which helps video most.^[1] The evaluation code in the repository samples 64 frames per video.^[2]

Matryoshka retention

The report evaluates the 2B model at every supported dimension from 64 to 2,048 on MMEB-v2 and reports the share of the 2,048-dimensional score retained. At 256 dimensions the model keeps 98.7% of its full performance on both the image and the video subsets; at 512 dimensions the figures are 99.2% for images and 98.8% for video.^[1][2] Visual-document tasks lose more as the dimension shrinks, which the authors attribute to the density of text in such documents.^[1] By task type, classification is the least sensitive, question answering degrades moderately, and retrieval suffers most at 64 and 128 dimensions; from 256 dimensions upward all three groups keep more than 97% of full performance.^[1] The authors' practical recommendation is 256 or 512 dimensions for efficiency-sensitive uses.^[1] Tencent's X post put the same idea as "truncate to 64 dims for fast recall or 2048 for fine ranking, no retraining".^[13] The report does not publish equivalent retention curves for the 4B and 9B models.

Deployment inside WeChat

Beyond public benchmarks, the report evaluates an in-house benchmark of 26 tasks drawn from WeChat applications in five categories: classification (4 tasks), search (7), cross-domain content matching (4), article relevance (4) and video relevance (7).^[1] Only the 2B model and one baseline are reported.^[1]

Scroll sideways for more →

The report describes production use in two places. In recommendation, the embeddings, which combine text, cover images and video frames, feed candidate retrieval, ranking features, user-sequence modeling and cross-domain content understanding across WeChat Official Accounts, WeChat Channels and e-commerce content; Semantic IDs derived from the embeddings serve as compact discrete features for indexing and sequence modeling.^[1] The team reports "consistent gains in 14 online A/B tests" that were subsequently rolled out, with the largest benefit for long-tail and newly published content.^[1] In WeChat search, the model supports semantic search across Channels videos, Official Accounts articles and Moments, for both unimodal and cross-modal queries.^[1] All of these are company-reported results; the report gives no absolute traffic, latency or lift figures.

Tencent's X account (@TencentAI_News) went further on September 4, 2026, opening its post with "1B requests a day in Weixin" and describing the model as "built for real traffic and verified in real use".^[13] That daily-request figure appears only in the post; the technical report says the model is "deployed at scale" without quantifying it.^[1][13]

Availability and tooling

The GitHub repository, created on August 25, 2026 and pushed to most recently on September 3, holds the inference examples, serving scripts, the technical report PDF and the mmeb_v3_eval/ directory; it had roughly 1,250 stars and 90 forks during September 5, 2026 (the counts moved through the day), and carries the topics embedding-models, multimodal and multimodal-llm.^[2][9] Weights for all three sizes are in a Hugging Face collection under the tencent organization, in BF16 safetensors, with trust_remote_code required for the custom modeling classes.^[4][8][11]

The README pins transformers==5.2.0, "as newer versions may differ in preprocessing behavior", and the model cards add qwen-vl-utils[decord]==0.0.14, sentence-transformers>=5.7.0 and accelerate>=1.1.0.^[2][4] Three inference paths are documented. Plain Transformers uses AutoModel and AutoProcessor with a chat-style message list mixing image, video and text items and calls model.embedding(**inputs). Sentence Transformers loads the model by Hub id and exposes encode_query and encode_document, accepting strings, image or video paths and dicts that mix image, video and text, with truncate_dim for Matryoshka truncation.^[4] For serving, the tested versions are vLLM 0.27.0 (--runner pooling with the shipped embedding_chat_template.jinja) and SGLang 0.5.9, the latter after running patch_sglang_video.py and launching with --is-embedding --enable-precise-embedding-interpolation.^[2][4] The Sentence Transformers chat template shipped in each repository carries a comment explaining that video inputs render as a bare <|video_pad|> token because the processor expands it into one block per frame and, from Transformers 5.3, no longer consumes a surrounding pair of vision-boundary tokens.^[8]

The evaluation directory is the official TIGER-AI-Lab VLM2Vec pipeline "with a minimal diff": multi-node inference through torchrun, a wemm_embedding backbone, task instructions aligned with the released model, and 64-frame video sampling, so the published MMEB-v3 numbers can be reproduced from the repository.^[2]

Licensing

The license situation has two readings that agree in substance and differ in scope. The GitHub LICENSE file states that Tencent releases WeMM-Embedding under the Apache License 2.0 and adds that "WeMM-Embedding refers solely to code made publicly available by Tencent in this repository"; the README repeats that "Tencent-authored code in this repository" is Apache 2.0 and third-party components keep their own licenses.^[2][3] The LICENSE file in each Hugging Face repository is broader: "WeMM-Embedding-9B is licensed under Apache-2.0, except for the third-party components listed below", and the model card states that the license covers "the code, model parameters, and weights made publicly available by Tencent".^[4][7] The Hugging Face metadata nevertheless sets license: other with license_name: apache-2.0 and a link to that file, which is why the Hub's license filter and the GitHub API both report the license as "other" or "NOASSERTION".^[8][9] The Hub API also marks the repositories as not gated.^[8]

Community ports

Within two weeks of release, third parties had produced conversions that Tencent has not endorsed or published. A GGUF conversion of the 2B model appeared on Hugging Face on August 26, 2026, with BF16, Q8_0 and Q4_K_M files and a separate multimodal projector file for llama.cpp-style loaders, and had 4,322 downloads by September 5; GGUF files for all three sizes from another user followed on September 1.^[20][23] A 4-bit MLX conversion of the 2B model for Apple silicon was uploaded on August 30, alongside a GitHub script for producing it.^[21] On September 4, a GitHub project published an ONNX port of the 2B model that exports the text, image and video branches separately for ONNX Runtime, states that it does not retrain or modify the model, and reports retrieval results identical to the official demo with similarity deviations of about 0.02 from BF16-versus-FP32 and batching differences.^[19] GitHub search on September 5 also turned up small applications built on the model, including a local image-library search tool and a text-to-video-moment search over HLS streams.^[9][19]

Coverage

TechNode reported the release on August 27, 2026, describing WeMM-Embedding as "a family of multimodal embedding models that represents and matches text, images, videos and other content types" and noting that Tencent's published results "place the 9B model first among the listed models on both the MMEB-v2 and MMEB-v3 benchmarks".^[12] The Tencent AI post on X of September 4, 2026 had 300 likes and 41 reposts when fetched the following day.^[13] The paper's own claims are framed carefully: "first on the official MMEB-v2 leaderboard" is footnoted to leaderboard status as of August 24, 2026, and every comparison to proprietary models on the cross-modal suite reuses numbers from the Gemini Embedding 2 report rather than rerunning those APIs.^[1]

Limitations

Audio is the obvious gap. The models cannot embed audio, MMEB-v3 audio tasks score zero, and the report lists "omni-modal inputs" as future work together with larger model sizes and better data curation and relevance supervision.^[1][2] The Matryoshka analysis shows visual documents losing more than images and video at low dimensions, so document-retrieval deployments have less room to truncate.^[1] Reranker supervision helped on only a subset of tasks, and the report does not identify which.^[1] The training data is described by family, not by named dataset or count per family, and the in-house benchmark is not released, so the WeChat results cannot be reproduced outside Tencent.^[1] The README also warns that Transformers versions other than 5.2.0 may change preprocessing and therefore the embeddings.^[2]

See also

• Embeddings
• Vector embeddings
• Matryoshka representation learning
• MTEB
• MMTEB
• Qwen3.5
• Qwen3 Embedding
• Tencent AI
• Vision language model
• Vector database

References

1. ^WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report (arXiv:2608.24053) - arXiv (Junjie Zhou, Ke Mei, Lei Li, Tianyi Wang, Fengyun Rao, Jing Lyu; WeChat Vision, Tencent), August 25, 2026.
2. ^Tencent/WeMM-Embedding README - GitHub (Tencent), accessed September 5, 2026.
3. ^Tencent/WeMM-Embedding LICENSE - GitHub (Tencent), accessed September 5, 2026.
4. ^tencent/WeMM-Embedding-9B model card - Hugging Face (Tencent), accessed September 5, 2026.
5. ^tencent/WeMM-Embedding-2B model card - Hugging Face (Tencent), accessed September 5, 2026.
6. ^tencent/WeMM-Embedding-4B model card - Hugging Face (Tencent), accessed September 5, 2026.
7. ^tencent/WeMM-Embedding-9B LICENSE file - Hugging Face (Tencent), accessed September 5, 2026.
8. ^Hugging Face Hub API records for tencent/WeMM-Embedding-2B, -4B and -9B - Hugging Face, fetched September 5, 2026 (createdAt, lastModified, safetensors totals, downloads, likes, config and chat templates).
9. ^GitHub API record for Tencent/WeMM-Embedding and repository search for "WeMM-Embedding" - GitHub, fetched September 5, 2026.
10. ^Tencent/WeMM-Embedding README_zh - GitHub (Tencent), accessed September 5, 2026.
11. ^WeMM-Embedding collection - Hugging Face (Tencent), last updated August 26, 2026.
12. ^WeChat team open-sources WeMM-Embedding for multimodal search and recommendation - TechNode (TechNode Feed), August 27, 2026.
13. ^Tencent AI (@TencentAI_News) post on X - X, September 4, 2026, 13:55 UTC.
14. ^VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (arXiv:2410.05160) - arXiv (Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, Wenhu Chen), October 2024.
15. ^VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents (arXiv:2507.04590) - arXiv (Rui Meng et al.), July 2025.
16. ^MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models (arXiv:2604.23321) - arXiv (Haohang Huang et al.), April 25, 2026; accepted at COLM 2026.
17. ^MMEB Leaderboard - Hugging Face Space (TIGER-Lab), last modified September 3, 2026.
18. ^Matryoshka Representation Learning (arXiv:2205.13147) - arXiv (Aditya Kusupati et al.), 2022; NeurIPS 2022.
19. ^Choise-ieee/WeMM-Embedding-ONNX - GitHub, created September 4, 2026.
20. ^DreamBlooms/WeMM-Embedding-2B-GGUF - Hugging Face, created August 26, 2026.
21. ^hfadam/WeMM-Embedding-2B-MLX-4bit - Hugging Face, created August 30, 2026; conversion script at metaswang/voxstudio-wemm-embedding-mlx-4bit, GitHub, created August 30, 2026.
22. ^TIGER-AI-Lab/VLM2Vec README (MMEB-V3) - GitHub (TIGER-AI-Lab), accessed September 5, 2026.
23. ^TuTuCSF/WeMM-Embedding-9B-GGUF - Hugging Face, created September 1, 2026.