Attention-FFN disaggregation

Attention-FFN disaggregation (AFD), also written attention-feedforward disaggregation or A/F disaggregation, is a large language model serving architecture that runs the attention sublayers and the feed-forward (FFN) or mixture-of-experts sublayers of each transformer layer on separate pools of accelerators during the decode phase. The attention pool holds the KV cache and does the memory-bound work of reading it; the FFN pool holds the bulk of the model weights and does the compute-heavy matrix multiplications. Activations travel between the two pools twice per layer, for every generated token, so the design trades a large amount of tightly scheduled network traffic for the ability to size, batch, and choose hardware for the two operators independently.^[6][8][11]

AFD is the next step after prefill-decode disaggregation, which splits a request's prompt processing from its token generation across different machines. Where prefill-decode disaggregation moves a KV cache once per request, AFD moves intermediate activations on the critical path of every layer of every decode step, which is why it is usually discussed together with micro-batch "ping-pong" pipelining and specialised communication libraries.^[6][8] The idea appeared in research systems from 2024 onward (Lamina, Adrenaline, ByteDance's MegaScale-Infer, StepFun's Step-3, Huawei Cloud's xDeepServe) and became an industry talking point in March 2026, when NVIDIA described AFD as the operating mode that pairs its Vera Rubin NVL72 GPUs with the SRAM-based Groq 3 LPX rack.^{[2][3][4][5][6][8][10]} A run of 2026 analysis papers has since mapped where AFD helps and where it does not: it tends to win on per-user latency and on interconnect-rich "superpod" hardware, while aggregated serving with large-scale expert parallelism often keeps the raw throughput crown on ordinary clusters.^[12][13][14]

Why attention and FFN want different hardware

A decoder-only transformer layer is two operators run back to back. The attention operator computes each new token's interaction with every earlier token in the sequence by reading that request's KV cache. The FFN operator pushes the attention output through large weight matrices; in an MoE model, a router picks a few "experts" (independent FFNs) per token from a much larger bank.^[6][8]

During decode these two operators sit on opposite sides of the roofline. The MegaScale-Infer authors describe the split directly: because each request has its own KV cache, batching requests together does not change attention's memory access intensity, so attention stays memory-bound no matter how large the batch. FFN, by contrast, loads the same weights for every token in the batch, so its arithmetic intensity grows with batch size and it can be made compute-bound simply by batching enough tokens.^[6] The StepFun team put the same point in terms of state: attention is stateful (its cost grows with context length and with the number of concurrent requests), while FFN is stateless and depends only on how many tokens arrive.^[8]

MoE sparsity makes the mismatch worse. MegaScale-Infer works a concrete example: an A100 with 312 TFLOPS and 2 TB/s of memory bandwidth needs a batch of at least 156 tokens for a bfloat16 GEMM to become compute-bound. For Mixtral 8x22B with top-2 routing over 8 experts, a 156-token batch sends an average of only 39 tokens to each expert, so the theoretical FFN utilisation is 25 percent. Larger models with more, finer experts push this further: in DeepSeek-V3 each token activates 8 of 256 routed experts. Raising the batch size to compensate is capped by latency targets and by KV-cache memory, which is exactly the memory that a large MoE's weights are competing for.^[6][12]

On a single homogeneous device, then, the GPU's compute sits idle during attention and its memory sits idle when KV caches are small and FFN weights are large. Joe Fioti, co-founder of the inference-compiler startup Luminal, summarised the operational consequence in a September 2026 post: FFN storage needs are "fixed and determinable at compile time", whereas attention adds up to a "relatively low-compute, high-memory workload with unpredictable memory access patterns", and running both on the same device wastes one resource or the other at any given moment.^[1][32]

How the split works

AFD assigns attention to one set of devices (often called A-workers or attention instances) and FFN or MoE experts to another (F-workers or expert instances). In MegaScale-Infer's formulation, the attention parameters and the KV caches are replicated across attention nodes using data parallelism, while the experts are spread across expert nodes using expert parallelism, with tensor parallelism inside each node. Consolidating the token streams of several attention replicas into one expert pool gives each expert a large enough batch to become compute-bound, which is the cure for the sparsity problem described above.^[6]

The cost is communication. Within every layer, the attention nodes must send their outputs to the expert nodes (a dispatch or "A2F" transfer) and receive the expert outputs back (a combine or "F2A" transfer). MegaScale-Infer notes that arbitrary parallelism on each side turns the usual all-to-all exchange of expert parallelism into an M-to-N pattern between M attention GPUs and N expert GPUs, which existing collective libraries handled poorly; the paper built a dedicated M2N library on RDMA that removed GPU-to-CPU copies, group initialisation, and GPU synchronisation, and reported 4.2 times the throughput and 68.2 percent lower median latency than NCCL at the 256 KB message sizes typical of MoE dispatch.^[6] StepFun's StepMesh library, released with Step-3, executes its RDMA operations on CPU threads so that communication does not compete with computation for GPU streaming multiprocessors, and pre-registers tensors so that no serialisation or concatenation happens on the critical path.^[8][9]

Ping-pong pipelining

If one batch simply bounced between the two pools, each pool would idle while the other computed and while data was in flight. AFD systems therefore split the batch into micro-batches and stagger them so that attention on one micro-batch overlaps with FFN and communication on others. MegaScale-Infer calls this ping-pong pipeline parallelism and derives the condition for it to hide communication: with T_f the longer of the attention and FFN compute times per micro-batch and T_c the one-way communication time, the number of micro-batches m must satisfy m >= 2 x (1 + T_c / T_f). When communication is fast (T_c below half of T_f) three micro-batches suffice; slower links need four. The paper also warns that pipelining raises utilisation without lowering the per-token latency of any single micro-batch, and can add latency.^[6]

Step-3 targets a 50 ms time per output token with a three-stage pipeline in which attention, FFN, and communication each get 16.6 ms summed across all layers, which for its 61 layers works out to roughly 272 microseconds per layer per stage. Its authors describe this per-layer budget as the central engineering constraint on the communication library and on any weaker hardware used on either side.^[8] Baidu's 2026 analysis calls the same pattern three-batch overlap (3BO) and points out that AFD needs at least 3BO to be bubble-free, whereas aggregated large-scale expert-parallel decode commonly runs with no, single, or two-batch overlap. The paper's concern is fragility: 3BO raises the ceiling but lowers tolerance for jitter, because once a communication or compute stage overruns its slot, bubbles propagate in both directions between the two roles.^[12]

Sizing the two pools

Because the two pools scale independently, an operator must choose a ratio of attention instances to FFN instances. Researchers at HKUST and Huawei's Hong Kong Research Center formalised this as an r-A to 1-F bundle, in which r attention workers feed a shared FFN server, and derived a closed-form provisioning rule that decomposes into attention-bottleneck, communication-bottleneck, and FFN-bottleneck regimes. The rule depends on a single workload statistic that captures how per-slot token load evolves as KV caches grow and finished requests are replaced by new prompts of random length; a trace-calibrated simulator put the analytically predicted optimum within 10 percent of the simulation-optimal ratio.^[11] The Georgia Tech, Intel, and Google DeepMind design-space study describes the same choice as rate matching: AFD allocates attention GPUs only as far as needed to keep up with the FFN pool's output rate, so models with cheap, compact attention push most of a cluster into the FFN pool, while models with heavy full-context attention need a wider attention slice.^[13]

Fioti's post lists this rigidity as one of the two main disadvantages of the approach: the topology and device ratio have to be fixed ahead of time from a prediction of the load, and if context lengths shift, either the attention devices sit idle behind the FFN devices or the attention side runs out of room and forces cache evictions and re-prefill.^[1] Baidu's authors make a related point analytically: AFD scales in discrete node-sized steps, so it pays a higher penalty for data-parallel and expert-parallel imbalance than aggregated expert parallelism, which can adjust batch composition continuously.^[12]

Relationship to other forms of disaggregation

AFD sits at the fine-grained end of a progression that the 2026 design-space paper summarises as "from chunked-prefill aggregation, to prefill-decode (P/D) disaggregation, and most recently to operator-level Attention-FFN Disaggregation".^[13] The levels are not exclusive: MegaScale-Infer, Step-3, and xDeepServe all assume prefill-decode disaggregation is already in place and apply AFD only inside the decode pool, and the Imperial College and Cambridge HeteroPanacea study evaluates a four-way "PDAF" split that separates prefill-attention, prefill-FFN, decode-attention, and decode-FFN.^{[6][8][10][14]}

Scroll sideways for more →

The DistServe authors' November 2025 retrospective explains why AFD went from "considered impractical" to a live option. Each layer transfers activations twice, which looked prohibitive for dense models; but large MoE models already pay for an all-to-all exchange in every MoE layer, and "by aligning the attention-FFN split with the existing all-to-all patterns and fusing their communication, the additional data transfer from AFD becomes almost free". The same post notes that current AFD results are confined to MoE models and that "dense models remain an open challenge".^[15]

Fioti draws the contrast in terms of where the network sits: in prefill-decode disaggregation the transfer only affects time to the second token and can be hidden behind prefill compute, whereas in AFD "our comms are in the decode hot loop, on the critical path of every single layer", with activations crossing "hundreds of times per forward pass", so tolerable communication times are "on the order of a microsecond or two". That figure is his characterisation; published systems budget in the hundreds of microseconds per layer once pipelining is accounted for.^[1][8]

Research lineage

Scroll sideways for more →

Model-attention disaggregation (Lamina, 2024)

The earliest of these papers used the name model-attention disaggregation rather than AFD. Chen and colleagues at Tsinghua and ByteDance observed that in prefill-decode disaggregated systems the decode pool remained inefficient because attention's memory access pattern "clashes with the strengths of modern accelerators, especially for long context requests". Their answer was to use "a collection of cheap, memory-optimized devices for the attention operator while still utilizing high-end accelerators for other parts of the model". The Lamina prototype ran on a cluster of H100 nodes (compute-optimised) and H20 nodes (memory-optimised) connected by 400 Gbps RoCE, served LLaMA-33B, LLaMA-65B, and LLaMA3-70B, and reported that 200 or 400 Gbps data-center networks were enough to make the per-layer transfers manageable.^[4]

Adrenaline (2025)

Adrenaline, from Sun Yat-sen University and Huawei Cloud, took a different angle on the same observation. In a prefill-decode disaggregated deployment, prefill GPUs leave their memory under-used while decode GPUs leave their compute under-used. Adrenaline offloads part of the decode phase's attention computation to the prefill instances, which both lifts prefill memory utilisation and lets decode instances run larger batches. It was implemented on vLLM and evaluated with Llama-2 7B and 13B on ShareGPT traces.^[5]

MegaScale-Infer (ByteDance, 2025)

MegaScale-Infer is the paper most often credited with making AFD a serving architecture in its own right. Its authors evaluated Mixtral 8x22B (141B parameters), DBRX (132B), and an internal Scaled-MoE with 317B parameters on two clusters: a homogeneous cluster of 80 GB Ampere GPUs with 200 Gbps InfiniBand, and a heterogeneous cluster in which H20 GPUs hosted attention and L40S GPUs hosted experts. With a 150 ms time-between-tokens constraint, the system delivered 2.56x and 1.28x higher per-GPU decode throughput than vLLM and TensorRT-LLM on the two public models, rising to 7.11x and 1.90x on Scaled-MoE, where the baselines had to span nodes. On the heterogeneous cluster it reached up to 3.24x and 1.86x the per-cost decode throughput of the two baselines running on H20, 1.66x end to end including prefill, and 1.80x (decode) and 1.72x (end to end) higher throughput per unit of power. The paper states that the system had been deployed in ByteDance's inference services and "reduces the serving cost by 1.5-2.0x", a company-reported figure.^[6] The conference version appeared at ACM SIGCOMM 2025 under a slightly different title.^[7]

Step-3 and StepMesh (StepFun, 2025)

StepFun's Step-3 technical report is the earliest of these papers to design a model around AFD rather than only a serving system; the report itself describes its system as "one of the first production quality serving systems" built on AFD and credits MegaScale-Infer as the first AFD serving system. Step-3 is a 321B-parameter vision-language model that activates 38B parameters per token; its Multi-Matrix Factorization Attention (MFA) was tuned so that the attention side's arithmetic intensity (128 with 8-bit KV) sits near the rooflines of cheaper accelerators, and its MoE sparsity was kept no sparser than about 0.058 so that the FFN side stays compute-bound on H800 GPUs within the pipeline's latency budget. The report argues that AFD lets a decoding instance run at a much smaller scale than large-scale expert parallelism (32 GPUs for Step-3 against the 320 the authors cite for DeepSeek-V3's published deployment), that long contexts no longer starve the FFN because the two pools scale separately, and that attention and FFN can sit on different hardware. On the latest Hopper GPUs at 4,096-token average context, FP8 GEMMs, no multi-token prediction, and a 20 tokens-per-second-per-user service level, Step-3 reached 3,910 tokens per GPU per second on average and 4,039 at peak with a 2A2F layout on 32 GPUs, which the report compares with the 2,324 peak reported for DeepSeek-V3 in the same setup.^[8]

The accompanying StepMesh library, open-sourced under Apache-2.0 in July 2025, is described in its README as built on ByteDance's BytePS ps-lite codebase. It exposes AFTensorWorker and AFTensorServer APIs for attention and FFN nodes, runs RDMA operations from dedicated CPU threads with NUMA-aware core binding, and is designed with pluggable backends so that new accelerators can be added. StepFun's production deployment runs it over a rail-optimised RoCE network with congestion control disabled in favour of priority flow control.^[8][9]

xDeepServe (Huawei Cloud, 2025)

Huawei Cloud's xDeepServe report describes the production serving system behind its Model-as-a-Service offering on CloudMatrix384, a 48-server "SuperPod" with 384 Ascend 910C chips connected by a UB fabric with global shared memory. Its Transformerless architecture decomposes a transformer into attention, feed-forward, and MoE modules that run on separate NPUs, and supports both prefill-decode disaggregation and what the paper calls disaggregated MoE-Attention. The MoE-attention deployment runs DeepSeek-V3/R1 across a full pod of 768 NPU dies, with 288 dies running EP288 (256 routed and 32 shared experts) and 480 dies running MLA attention, organised as three data-parallel domains of 160 groups each so that only one domain talks to the MoE NPUs at a time. The paper credits FastDecode, Lamina, and InstAttention with pioneering the split and MegaScale-Infer with extending it to MoE, and introduces XCCL, a memory-semantic communication layer with microsecond-level point-to-point and all-to-all primitives, including a two-stage "trampoline" route for the asymmetric attention-to-expert exchange.^[10]

The 2026 analysis papers

The HKUST and Huawei provisioning paper (January 2026) supplies the queueing-style model described above and, notably, observed at the time that AFD remained "an emerging paradigm with no mature open-source implementations".^[11]

Baidu's "Revealing the Challenges" paper (February 2026) is the most sceptical of the group. Extending the roofline model to the communication level, its authors show a "dead zone" on standard clusters: adding FFN instances stops improving hardware FLOPS utilisation because the number of tokens that can reach the FFN pool is capped by scale-out bandwidth, so operator active time shrinks relative to a fixed latency budget. On a non-superpod H800 platform they put AFD's theoretical FFN-stage utilisation ceiling at 33.1 percent against roughly 60 percent achievable with large-scale expert parallelism. The conditions that reverse the picture are abundant scale-up bandwidth, such as GB200 or GB300 NVL72 systems, and coarser experts with lower sparsity; the paper contrasts Step-3's expert dimension of 5,120 with DeepSeek-V3's 2,048. Its conclusion positions AFD "as a promising approach for specific hardware-model combinations rather than a universal solution".^[12]

The Georgia Tech, Intel, and Google study "How Far Can Disaggregation Go?" (May 2026) built AIC++ by combining NVIDIA's AIConfigurator performance model with the ASTRA-sim network simulator, grounded in a customised vLLM AFD prototype, and swept parallelism, prefill-decode, and AFD choices for a cluster of 128 B200 GPUs with TensorRT-LLM as the backend. The models were DeepSeek-V3.2 (MLA with sparse attention), GPT-OSS-120B (alternating full and sliding-window GQA), Nemotron 3 Super 120B (Mamba-2 plus GQA hybrid), and Qwen3-235B (dense GQA), on chat, coding, and agentic-coding workloads, the last with a 524K-token prefix. Two findings stand out. Under strict service levels (time to first token under 50, 100, or 150 ms by workload and a 15 ms time-per-output-token cap) only AFD variants produced any feasible DeepSeek-V3.2 deployment, sustaining around 4,000 tokens per second of system throughput. And on the latency axis AFD won every panel, with splits that track each model's attention cost: 2 attention plus 126 FFN GPUs for DeepSeek-V3.2 agentic coding, 8 plus 120 for Qwen3-235B, a near-symmetric 16 plus 16 for GPT-OSS-120B chat, and an attention-heavy 96 plus 32 for the Nemotron hybrid, whose recurrent state must propagate across the long prefix. Aggregated serving with chunked prefill nevertheless won most throughput panels, because each AFD replica consumes more GPUs.^[13]

HeteroPanacea (August 2026), from Imperial College London and the University of Cambridge, asks the hardware question directly: if each serving stage could have its own accelerator, which stages should be split? Its simulator, validated component by component against an 8x B200 node, searches over non-disaggregated (ND), prefill-decode (PD), attention-FFN (AF), and four-way PDAF layouts under a power or cost budget. In a synthetic NPU design space at a prefill-to-output ratio of 100, PDAF beat ND for all eight models tested (1.05x to 1.92x) and was the best mode for six, while AF on its own never exceeded ND at any ratio. The transition into a disaggregation-favouring regime happened between ratios of 1 and 10. When the same search was repeated over rentable GPU instances, 31 of 32 PDAF stage assignments came back as H100s and the four-way split stopped paying off; the authors' explanation is that splitting attention from FFN "only helps when the two can be given genuinely different hardware", which fixed-ratio GPU catalogues do not allow. The paper confirms a throughput gain of up to 75 percent from prefill-decode disaggregation alone with current GPUs.^[14]

NVIDIA Vera Rubin and Groq 3 LPX

AFD reached a mainstream audience at GTC in March 2026, when NVIDIA presented Groq 3 LPX, a rack of 256 Groq 3 language processing units, as a decode engine to be deployed beside Vera Rubin NVL72. In the NVIDIA Technical Blog post "Inside NVIDIA Groq 3 LPX" (March 16, 2026, by Kyle Aubrey and Farshad Ghodsian), decode is described as "a two-engine loop": GPUs handle the decode work that benefits from throughput and memory capacity, "such as full-context attention over the accumulated KV cache", while LPX "accelerates latency-sensitive execution within decode, such as sparse MoE expert feed-forward networks (FFNs) and other pointwise operations". The post continues: "This split, often described as decode phase disaggregation or attention-FFN disaggregation (AFD), separates attention from FFN within decode and exchanges intermediate activations for each token, so each engine runs the part of the loop it is best suited to execute."^[2]

The hardware rationale is the LPU's memory. Each Groq 3 LPU carries 500 MB of compiler-managed on-chip SRAM at 150 TB/s, and the rack aggregates 128 GB of SRAM, 40 PB/s of on-chip bandwidth, 315 PFLOPS of inference compute, and 640 TB/s of scale-up bandwidth over 96 chip-to-chip links per LPU at 112 Gbps each. Fixed-size, weight-stationary FFN and expert layers suit a memory that is fast but small; a growing KV cache does not, and under AFD the KV cache never has to leave the GPUs' HBM. NVIDIA's product page describes the arrangement as Rubin GPUs and LPUs boosting decode "by jointly computing every layer of the AI model for every output token".^[2][16] The company's headline claims, which are its own "up to" figures rather than third-party measurements, are up to 35 times higher tokens per second per megawatt at 400 tokens per second per user relative to a GB200 NVL72, and up to 10 times more revenue opportunity for trillion-parameter models when NVL72 is paired with LPX.^[2]

NVIDIA Dynamo is named as the orchestration layer. According to the March post, Dynamo routes prefill to GPU workers to build the KV cache and then "orchestrates the AFD loop where GPUs run attention over the accumulated KV cache, intermediate activations are handed off to LPUs for FFN/MoE execution, and outputs return to the GPUs to continue token generation", using KV-aware routing and latency-target scheduling to keep tail latency stable. The post's Figure 7, titled "NVIDIA Dynamo Orchestrates Heterogeneous Compute", shows a Vera Rubin NVL72 with prefill GPUs and decode GPUs labelled ATTN, a Groq 3 LPX with decode LPUs labelled FFN, and an arrow labelled "Interim Decode Activations, Repeat per token" crossing Ethernet between the racks. The post also points readers to a GTC session, "The Future of AI Inference", for an explainer on AFD.^[2][33]

A follow-up post on August 24, 2026 ("How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin", by Seth Weidman, Kirthi Devleker, and Andrew Ling) presents AFD as one of three ways to pair the racks rather than the only one. In standard prefill-decode disaggregation, NVL72 handles prefill and hands the KV cache to LPX once per turn, and LPX then performs the entire decode step from weights held in SRAM. In attention-FFN disaggregation, "Vera Rubin NVL72 computes attention and holds the KV cache in DRAM, while Groq 3 LPX executes the FFN layers. Only intermediate tokens are sent between racks, once per full-attention layer." In external-drafter speculative decoding, LPX runs a small draft model ahead of the target model on NVL72, and only draft tokens cross the link. The same post reports that Artificial Analysis measured a median 3,431 output tokens per second for Gemma 4 31B on LPX at 100K context and that NVIDIA's own run of the SPEED-Bench coding set produced a median of 4,767 tokens per second with a P80 of 5,520.^[3] Those measurements are of a 31B model on the LPX rack; the 35x and 10x figures for trillion-parameter models paired with NVL72 remain NVIDIA's projected "up to" claims, and no independent measurement of the AFD loop at that scale had been published as of September 6, 2026. NVIDIA announced full production of LPX on August 24, 2026, with Nebius as the first cloud adopter, and ServeTheHome's report from NVIDIA's Hot Chips 2026 talk the next day described the design as offloading "the attention portion of the decode process back to the GPUs, making decode a disaggregated process".^[17][18]

Fioti's September 2026 post, which circulated NVIDIA's Figure 7, framed AFD as the technique "that will save SRAM-only chips", because it lets an SRAM accelerator hold weights and activations but no KV cache. He added that "we can keep the attention layers on GPUs, which can handle the much larger memory requirements of storing and accessing large KV caches". That is consistent with NVIDIA's description of LPX; his further statement that Cerebras "is targeting" the same approach through its AMD and AWS partnerships is discussed below.^[1]

Heterogeneous deployments that are not AFD

Several 2026 announcements pair GPUs or other accelerators with SRAM-based chips without splitting attention from FFN. They are prefill-decode or draft-target splits, and public materials describe them as such.

On March 13, 2026, AWS and Cerebras announced a collaboration to deploy AWS Trainium servers and Cerebras CS-3 systems in AWS data centers, connected by Elastic Fabric Adapter networking and accessed through Amazon Bedrock. The press release defines the split as "inference disaggregation", with "Trainium optimized for prefill and the Cerebras CS-3 optimized for decode", and says the CS-3 "will be fully dedicated to decoding acceleration". AWS vice president David Brown is quoted saying that "by splitting the inference workload across Trainium and CS-3 ... each system does what it's best at". The release describes the service as launching "in the next couple of months".^[27] Cerebras's own explainer, "The GPU Is Being Split in Half" (March 26, 2026, by Sarah Chieng), likewise frames the trend as prefill on compute-optimised systems and decode on memory-bandwidth-optimised systems, and puts NVIDIA and Groq in the same bucket.^[29]

On July 23, 2026, AMD and Cerebras announced a partnership, unveiled at Advancing AI 2026, in which AMD Helios rack-scale systems and the Cerebras Wafer-Scale Engine "will operate as a single disaggregated inference workflow", with Helios "processing prompts and large context windows" and the wafer-scale engine handling "the memory-bandwidth-intensive token generation". The companies said the pair is "expected to deliver up to 5x higher tokens per second per watt", a figure the release footnotes as based on July 2026 modelling by AMD Performance Labs and Cerebras against a WSE-only configuration on a Kimi 2.6 1T model, and said Cerebras plans to offer the solution through Cerebras Cloud in the second half of 2026.^[28]

A third pairing, announced March 12, 2026, has Gimlet Labs adding d-Matrix Corsair accelerators to its cloud alongside GPUs for latency-sensitive work "including speculative decoding", with availability to select customers planned for the second half of 2026.^[30] The paywalled Chip Log newsletter grouped all three March announcements together because "the bandwidth-hungry half of the split (decode, draft, FFN) runs on an SRAM-based chip".^[31]

Fioti's post states that AFD is "exactly the approach Nvidia is taking with their SRAM-only Groq LPX system, and Cerebras is targeting with their partnerships with AMD and AWS Trainium".^[1] The NVIDIA half of that sentence matches NVIDIA's own description. The Cerebras half does not match the public announcements: both the AWS and the AMD press releases describe prefill-decode splits in which the Cerebras system runs the whole decode step, KV cache included, and neither mentions attention-FFN disaggregation.^[27][28] No Cerebras statement describing an AFD deployment had been located as of September 6, 2026; readers should treat Fioti's characterisation as his interpretation of where those partnerships could lead rather than as a description of what was announced.

Software support

As of September 2026, AFD support in mainstream open-source serving frameworks is early and uneven, which matches the January 2026 observation that no mature open-source implementation existed.^[11] The table summarises what the public repositories show; entries are dated from the GitHub records themselves.

Scroll sideways for more →

Trade-offs and open questions

The published record supports a fairly consistent list of when AFD pays and what it costs.

• Interconnect is the deciding resource. Every analytic study finds that AFD's benefit is capped by the bandwidth available between the two pools. Baidu's dead-zone result on H800-class clusters, the design-space study's emphasis on placing high-traffic attention and FFN operators on tightly coupled nodes, and NVIDIA's decision to build a purpose-designed rack with 640 TB/s of internal scale-up bandwidth all point the same way.^[2][12][13]
• Latency budgets are tight and brittle. Step-3's roughly 272 microsecond per-layer stage budget and Baidu's warning about 3BO fragility describe the same constraint from two sides; Fioti's "microsecond or two" is the practitioner's shorthand for it.^[1][8][12]
• AFD wins interactivity more reliably than throughput. In the design-space study AFD took the latency frontier on every workload but the throughput frontier on only one, because each replica consumes more GPUs than an aggregated replica.^[13]
• Heterogeneity is where the payoff is largest. MegaScale-Infer's per-cost and per-watt gains came from mixing H20 and L40S GPUs, HeteroPanacea found the four-way split worthwhile only when stages can be given genuinely different hardware, and NVIDIA's LPX pairing is a heterogeneous design by construction.^[2][6][14]
• Model architecture matters. Coarser experts and lower sparsity (Step-3), compressed attention (MLA with sparse attention in DeepSeek-V3.2), and sliding-window attention (GPT-OSS) shift the optimal split and the achievable utilisation; hybrid state-space models can invert the usual ratio.^[8][12][13]
• Dense models remain open. The DistServe retrospective and the MoE-only scope of every production deployment described above leave AFD for dense models as an unsolved problem.^[15]
• Provisioning is static. Both Fioti and the Baidu authors note that a fixed attention-to-FFN ratio is a liability when context lengths or traffic mix drift; the elastic-AFD RFC in vLLM and Luminal's pitch of compiling several topologies and swapping between them are two proposed responses, neither of which has a published production result.^[1][12][35]

References

1. ^X post by @joefioti, "There's only one technique that will save SRAM-only chips: attention-feedforward disaggregation" - X (Joe Fioti), September 4, 2026.
2. ^Inside NVIDIA Groq 3 LPX: The Low-Latency Inference Accelerator for the NVIDIA Vera Rubin Platform - NVIDIA Technical Blog (Kyle Aubrey and Farshad Ghodsian), March 16, 2026.
3. ^How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin - NVIDIA Technical Blog (Seth Weidman, Kirthi Devleker, and Andrew Ling), August 24, 2026.
4. ^Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation - arXiv (Shaoyuan Chen, Wencong Xiao, Yutong Lin, Mingxing Zhang, Yingdi Shan, Jinlei Jiang, Kang Chen, Yongwei Wu), May 3, 2024 (v2 April 10, 2025).
5. ^Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation - arXiv (Yunkai Liang, Zhangyu Chen, Pengfei Zuo, Zhi Zhou, Xu Chen, Zhou Yu), March 26, 2025.
6. ^MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism - arXiv (Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, Xin Liu), April 3, 2025 (v4 July 26, 2025).
7. ^MegaScale-Infer: Efficient Mixture-of-Experts Model Serving with Disaggregated Expert Parallelism - Proceedings of ACM SIGCOMM 2025, pages 592-608 (Ruidong Zhu et al.), 2025.
8. ^Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding - arXiv (StepFun), July 25, 2025.
9. ^stepfun-ai/StepMesh: A High-Performance, Low-Latency Communication Library for Attention-FFN Disaggregation - GitHub (StepFun), repository created July 21, 2025, accessed September 6, 2026.
10. ^Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPod (xDeepServe) - arXiv (Ao Xiao et al., Huawei Cloud), August 4, 2025 (v6 March 1, 2026).
11. ^Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads - arXiv (Chendong Song, Meixuan Wang, Hang Zhou, Hong Liang, Yuan Lyu, Zixi Chen, Yuwei Fan, Zijie Zhou), January 29, 2026 (v4 August 14, 2026).
12. ^Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems - arXiv (Guowei Liu, Hongming Li, Yaning Guo, Yongxi Lyu, Mo Zhou, Yi Liu, Zhaogeng Li, Yanpeng Wang), February 10, 2026.
13. ^How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving - arXiv (Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee, Tuhin Khare, Sudarshan Srinivasan, Suvinay Subramanian, Souvik Kundu, Madhu Kumar, Midhilesh Elavazhagan, William Won, Amir Yazdanbakhsh, Tushar Krishna), May 27, 2026.
14. ^When Does Disaggregation Pay? Simulating Prefill-Decode-Attention-FFN Specialization for Agentic LLM Inference - arXiv (Przemyslaw Forys, Haoran Wu, Can Xiao, Jiayi Nie, Tony Liu, Rika Antonova, Timothy Jones, Robert Mullins, Wayne Luk, Aaron Zhao, George A. Constantinides), August 4, 2026.
15. ^Disaggregated Inference: 18 Months Later - Hao AI Lab, UC San Diego (Junda Chen, Yonghao Zhuang, Hao Zhang), November 3, 2025.
16. ^NVIDIA Groq 3 LPX: Inference Accelerator for Agentic AI - NVIDIA product page, accessed September 6, 2026.
17. ^NVIDIA Groq 3 LPX Now in Full Production With World-Class Speed for Agentic AI - NVIDIA Newsroom, August 24, 2026.
18. ^NVIDIA's Groq 3 LPU Accelerators for Heterogeneous AI Compute at Hot Chips 2026 - ServeTheHome (Ryan Smith), August 25, 2026.
19. ^RFC: Attention-FFN Disaggregation (AFD), pull request #10900 - GitHub, sgl-project/sglang, opened September 25, 2025, closed August 24, 2026.
20. ^NPU/AFD: Attention-FFN Disaggregation Experimental, adopt from #10900, pull request #36258 - GitHub, sgl-project/sglang, merged August 25, 2026.
21. ^RFC: ATTN-FFN Disaggregation for MoE Models, issue #22799 - GitHub, vllm-project/vllm, opened August 13, 2025, closed May 31, 2026.
22. ^Feature: AFD basic implemetation (sic), pull request #29772 - GitHub, vllm-project/vllm, opened December 1, 2025, closed July 24, 2026.
23. ^feat: Support AFD, issue #888 - GitHub, ai-dynamo/aiconfigurator, opened April 22, 2026.
24. ^feat(afd): add AFD estimate mode (pull request #1129) - GitHub, ai-dynamo/aiconfigurator, merged June 8, 2026.
25. ^feat(afd): add AFD default CLI mode built on the v2 Task architecture (pull request #1323) - GitHub, ai-dynamo/aiconfigurator, merged August 5, 2026.
26. ^feat(sglang): Add AFD (Attention-FFN Disaggregation) infrastructure (pull request #6570) - GitHub, ai-dynamo/dynamo, opened and closed February 25, 2026.
27. ^AWS and Cerebras Collaboration Aims to Set a New Standard for AI Inference Speed and Performance in the Cloud - Cerebras press release, March 13, 2026.
28. ^AMD and Cerebras Announce Industry-Leading Ultra-Low-Latency and High Throughput AI Inference Solution - Cerebras press release, July 23, 2026.
29. ^The GPU Is Being Split in Half - Cerebras blog (Sarah Chieng), March 2026 (page metadata shows a last-modified time of 2026-03-25T23:41Z).
30. ^d-Matrix and Gimlet Labs to Deliver 10x Speed Ups, Massive Power Efficiency for Frontier AI Workloads - d-Matrix press release, March 12, 2026.
31. ^Inside Attention-FFN disaggregation: What Groq LP40 will look like - Chip Log (Subbu), July 6, 2026 (free section only).
32. ^Luminal raises $5.3 million to build a better GPU code framework - TechCrunch (Russell Brandom), November 17, 2025.
33. ^The Future of AI Inference - NVIDIA On-Demand session listing, 2026, accessed September 6, 2026.
34. ^feat(afd): Rebase/afd onto upstream (pull request #1597) - GitHub, ai-dynamo/aiconfigurator, opened August 27, 2026.
35. ^RFC: Elastic Attn-FFN Disaggregation, issue #27584 - GitHub, vllm-project/vllm, opened October 27, 2025, closed June 4, 2026.
36. ^Roadmap: support Multi-Matrix Factorization Attention and Attention-FFN Disaggregation, issue #8360 - GitHub, sgl-project/sglang, opened July 25, 2025.