Revision History
Distributed training · 11 revisions
Sizes are character counts of the article source. The signed number is the change from the previous revision.
Recent edit summaries
Detailed summaries recorded by editors. Generic maintenance summaries are omitted here; every recorded revision remains below. Dates describe the edit, not necessarily the event it covers.
- Correction: Corrected Llama 3.1 405B training figures (30.84M not 39.3M GPU-hours; 54 days was an interruption snapshot, not training length) and the cost estimate derived from them, Llama 3 405B's network (RoCE, not InfiniBand), DeepSeek-V3's cluster size in the overview and the reason it skipped tensor parallelism, a non-existent ZeRO trillion-parameter demonstration, MegaScale's authorship (ByteDance, not Meta) and a fabricated 90%-communication claim, the Goyal warmup length, Llama 3's optimizer, Grok 4's year, Decoupled DiLoCo's date and scale, and a stale largest-cluster claim (plus 27 minor fixes)
Version 11 · Nov 1, 2026, 11:24 AM
- Correction: TorchTitan row omitted context parallelism, which the TorchTitan paper (arXiv 2410.06511, section 2.1.5) covers; added the paper as a reference
Version 10 · Oct 16, 2026, 11:26 PM
- Linked TorchTitan at its first mention in the frameworks table
Version 9 · Oct 16, 2026, 11:07 PM
- Link [TorchTitan] to the new torchtitan article
Version 8 · Oct 16, 2026, 10:57 PM
- Improve internal links (2026-10-13)
Version 7 · Oct 13, 2026, 10:07 AM