GEM (Meta)

GEM (Generative Ads Recommendation Model) is a proprietary foundation model for advertising recommendation developed by Meta Platforms. Meta describes GEM as "the central recommendations foundation model behind Meta's ads system," powering the ranking stage of advertising delivery across Facebook and Instagram ^[1][2][4]. First disclosed on Meta's first-quarter 2025 earnings call on April 30, 2025 ^[3], GEM combines trillions of sparse embedding parameters with billions of dense parameters and is trained on several thousand GPUs, a scale Meta compares to large language model training ^[2]. Meta calls it "the largest foundation model for recommendation systems (RecSys) in the industry" ^[1], a company claim that has not been independently verified. Unlike a conventional recommender system model, GEM is not served directly to users: Meta says the model is too large and complex for cost-effective inference, so its knowledge is transferred to smaller runtime models through post-training techniques ^[1][5]. Meta detailed GEM's architecture in a November 10, 2025 engineering blog post ^[1] and its training infrastructure in an August 3, 2026 follow-up, which reported that end-to-end training efficiency had doubled to 20 to 25 percent model FLOPs utilization (MFU) while total training FLOPs grew 4x over 12 months ^[2][7].

Key facts

Scroll sideways for more →

Role in Meta's ads system

Meta's ads delivery pipeline separates retrieval, which selects candidate ads, from ranking, which decides which of those candidates to show. On the company's Q2 2025 earnings call, CFO Susan Li said GEM "powers the ranking stage of our ads system, which is the part of the process after ads retrieval where we determine which ads to show someone from candidates suggested by our retrieval engine" ^[4]. Meta discusses GEM alongside two other ads model programs: Andromeda, its machine-learning retrieval system, and Lattice, a model-consolidation architecture for ranking ^[3][6].

GEM functions as a teacher model rather than a serving model. Li told analysts on the Q3 2025 call that Meta does not "use our larger model architectures like GEM for inference because their size and complexity would make it too cost prohibitive," and that Meta instead drives performance "by using them to transfer knowledge to smaller, lightweight models that are used at runtime" ^[5]. In its November 2025 post, Meta says GEM propagates its learnings to hundreds of user-facing "vertical models" across the ads stack through post-training knowledge transfer ^[1].

Architecture

According to Meta, GEM has a hybrid architecture with trillions of sparse embedding parameters and billions of dense parameters. It is trained on ad content and user engagement data organized into two feature groups: sequence features such as user activity history, and non-sequence features such as user location, age, ad format, and ad creative representation. Customized attention mechanisms are applied to each group independently while also enabling cross-feature learning ^[1][2]. Meta says this design made GEM 4x more efficient at converting a given amount of data and compute into ad performance gains than its previous generation of ads ranking models ^[1].

Meta's November 2025 post describes four architectural components:

• Non-sequence feature interaction modeling. GEM enhances Meta's Wukong architecture, using stackable factorization machines with cross-layer attention connections to learn which combinations of user and ad attributes matter most. Each Wukong block can scale vertically for deeper interactions or horizontally for broader feature coverage ^[1].
• Offline sequence feature modeling. A "pyramid-parallel" structure stacks parallel interaction modules to process long user behavior sequences, with offline feature infrastructure that Meta says handles sequences of up to thousands of events with minimal storage cost ^[1].
• Cross-feature learning. A design Meta calls InterFormer alternates between sequence-learning layers and cross-feature interaction layers using parallel summarization with an interleaving structure, preserving full sequence information rather than compressing user history into compact vectors ^[1].
• Multi-domain learning. GEM learns from cross-surface interactions spanning Facebook, Instagram, and Business Messaging while tailoring predictions to each surface's objective, so that, for example, Instagram video ad engagement can inform Facebook Feed ad predictions ^[1].

Post-training and knowledge transfer

Because GEM itself is not used at serving time, Meta treats knowledge transfer as a core part of the system. The company uses direct transfer, where GEM teaches major vertical models trained in the same data spaces, and hierarchical transfer, where GEM distills into domain-specific foundation models that in turn teach vertical models. The techniques include knowledge distillation with a "Student Adapter" that refines the teacher's outputs against recent ground-truth data, representation learning to generate semantically aligned features, and parameter sharing that lets latency-sensitive models reuse foundation-model components. Meta says the combined framework is twice as effective as standard knowledge distillation ^[1]. Unlike large language models, GEM is also continuously trained online to keep the foundation model fresh ^[1].

Training infrastructure

Meta says its LLM-oriented training stack did not transfer directly to GEM because recommendation workloads differ from language model workloads: training samples are "jagged" (user histories vary from hundreds to tens of thousands of events, so padding to a fixed length would waste up to 50 percent of compute), attention shapes are asymmetric, many operations are memory bound, and click-through-rate and conversion prediction are highly sensitive to numerical precision ^[2]. The November 2025 post reported that a re-engineered training stack delivered a 23x increase in effective training FLOPs using 16x more GPUs, with a 1.43x improvement in MFU ^[1]. The August 2026 post reported a further doubling of end-to-end training efficiency, to 20 to 25 percent MFU, while total training FLOPs scaled 4x over the preceding 12 months ^[2].

Custom kernels

Meta built a kernel library specific to recommendation workloads ^[2]:

• Jagged Flash Attention (JFA) is a variant of FlashAttention that operates directly on variable-length jagged tensors, eliminating padding. Meta says the fourth generation, built on Triton Low-Level Extensions (TLX) with warp specialization and persistent kernels, achieves 40 to 140 percent higher TFLOPS than the second generation and contributed an 18.5 percent relative gain in local MFU ^[2].
• Generalized Dot-Product Attention (GDPA) unifies GEM's self-attention, cross-attention, and pooled multi-head attention modules, which replace softmax with activations such as GELU or SiLU. Meta reports the kernel reaches 1,145 BF16 TFLOPS (about 97 percent Tensor Core utilization) for a 2x forward speedup, up to 3.5x faster than Flash Attention 4 under short key/value production settings, and over 30 percent higher end-to-end training throughput applied across the full model ^[2].
• BlockAttention restructures long user-history self-attention into independent 64-token block computations after an earlier move to sliding-window attention, which had already cut long-sequence self-attention latency by up to 68 percent. Meta says the TLX block-attention kernel with a fused rotary position embedding backward pass improves self-attention layer MFU by roughly 44 percent over the sliding-window baseline ^[2].

Ultra-low-precision training

GEM training uses mixed ultra-low precision, including MXFP8 (a block-scaled FP8 format) for attention and MLP layers. Meta extended the Flash Attention 4 kernel with end-to-end MXFP8 block-scaled matrix multiplication in both forward and backward passes, reporting kernel speedups of more than 1.3x forward and 1.5x backward on representative GEM shapes ^[2]. To keep quantization overhead from erasing those gains, weights are quantized per shard before the FSDP all-gather, FSDP communication itself carries low-precision payloads, and activation quantization is fused into preceding normalization and projection kernels ^[2]. For numerical stability, Meta applies random Hadamard transforms to spread outliers, stochastic rounding to remove deterministic rounding bias, selectively computes weight gradients at higher precision, and falls back to BF16 in precision-sensitive later layers ^[2].

5D parallelism and networking

Meta scales GEM with what it calls topology-aware 5D parallelism, matched to a training cluster with a three-tier network: eight GPUs per host on NVLink, hosts within an "AI zone" on RoCE Ethernet, and AI zones linked by oversubscribed RoCE ^[2]. Dense parameters use a two-dimensional FSDP scheme, sharding parameters within groups of roughly 128 to 256 GPUs while replica groups synchronize gradients by all-reduce, with expert parallelism layered on top so that large expert modules shard their all-gathers down to a single expert's parameters and place the extra communication on fast intra-node NVLink ^[2]. Sparse embedding tables use what Meta calls Fully Sharded 2D Model Parallelism, the third generation of its sparse scaling design, which shards each replica group's copy of the tables to nearly eliminate the memory overhead of earlier 2D approaches ^[2].

To stop communication from competing with compute, Meta uses NCCLX, its extension of NVIDIA's NCCL library, for SM-free collectives: copy engines and RDMA move data instead of streaming multiprocessors, cutting SM usage for all-gather from about 24 SMs to 1 and yielding a roughly 5 percent end-to-end throughput gain at full scale ^[2]. Memory pressure is managed with compiler-based automatic activation checkpointing in PyTorch using per-region recompute budgets, plus quantization of checkpointed activations, enabling local batch sizes above 1,000 samples ^[2]. Because jagged sequences skew per-GPU load (Meta measured the heaviest rank at about 15 percent above average each iteration), a technique Meta calls Base Batch Shuffling sorts and interleaves small sub-batches by sequence length, which the company says recovered 4 percent efficiency with zero cross-rank communication ^[2].

History and reported impact

All performance figures below are Meta's own, disclosed on earnings calls and company blogs; they have not been independently audited.

• Q1 2025. Meta introduced GEM for ads ranking, saying the new architecture was twice as efficient at improving ad performance per unit of data and compute and was trained on its largest ads-training cluster to date. Early tests on Facebook Reels showed up to a 5 percent increase in ad conversions ^[3].
• Q2 2025. Meta scaled training capacity, added organic and ads engagement data from Instagram, and doubled the length of user event sequences the system considers. The company attributed an approximately 5 percent increase in ad conversions on Instagram and 3 percent on Facebook Feed and Reels in the quarter to these improvements ^[4].
• Q3 2025. Meta said architecture improvements doubled the performance benefit obtained from a given amount of data and compute ^[1].
• November 10, 2025. Meta published its first detailed technical description of GEM on the Engineering at Meta blog ^[1].
• Q4 2025. Meta doubled the number of GPUs used to train GEM and began deploying a new sequence-learning architecture; together these drove a 3.5 percent lift in ad clicks on Facebook and a more than 1 percent gain in ad conversions on Instagram in the quarter. GEM's coverage was extended to Facebook Reels, which Meta said completed coverage of all major Facebook and Instagram surfaces. Li told analysts, "This is the first time we have found a recommendation model architecture that can scale with similar efficiency as LLMs" ^[6].
• August 3, 2026. Meta published the GEM training deep dive, reporting the doubling of end-to-end training efficiency to 20 to 25 percent MFU alongside the 4x scaling of training FLOPs over 12 months ^[2][7].

Meta has said it plans to scale GEM training to a still larger cluster in 2026, expand its training data, and evolve the model toward multimodal learning over text, images, audio, and video, with a long-term goal of a unified engagement model that ranks both organic content and ads ^[1][6].

See also

• Recommender system
• Foundation models
• FSDP
• Expert parallelism
• FP8
• Distributed training

References

1. ^Meta (Engineering at Meta). "Meta's Generative Ads Model (GEM): The Central Brain Accelerating Ads Recommendation AI Innovation." November 10, 2025. engineering.fb.com/...recommendation-ai-innovation
2. ^Meta (Engineering at Meta). "GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model." August 3, 2026. engineering.fb.com/...ommendation-foundation-model
3. ^Meta Platforms. "First Quarter 2025 Results Conference Call Transcript." April 30, 2025. web.archive.org/...25-Earnings-Call-Transcript.pdf
4. ^Meta Platforms. "Second Quarter 2025 Results Conference Call Transcript." July 30, 2025. s21.q4cdn.com/...2025-Earnings-Call-Transcript.pdf
5. ^Meta Platforms. "Third Quarter 2025 Results Conference Call Transcript." October 29, 2025. s21.q4cdn.com/...2025-Earnings-Call-Transcript.pdf
6. ^Meta Platforms. "Fourth Quarter 2025 Results Conference Call Transcript." January 28, 2026. s21.q4cdn.com/...2025-Earnings-Call-Transcript.pdf
7. ^Engineering at Meta (@Meta_Engineers). X post announcing the GEM training deep dive. August 4, 2026. x.com/...2084690539123982381