Revision History
Data Parallelism · 13 revisions
Sizes are character counts of the article source. The signed number is the change from the previous revision.
Recent edit summaries
Detailed summaries recorded by editors. Generic maintenance summaries are omitted here; every recorded revision remains below. Dates describe the edit, not necessarily the event it covers.
- Correction: Corrected ZeRO stage 1 and 2 per-device memory figures (about 4P and 2P bytes, not 6P and 4P), the Horovod benchmark scale (88% efficiency on up to 128 GPUs, not near-linear on 512), the date PyTorch DDP's bucketing and overlap first shipped (v1.1, 2019, not 2020), the claim that PaLM used 3D parallelism (it used no pipeline parallelism), and Horovod's status (repository archived September 2026) (plus 18 minor fixes)
Version 13 · Nov 2, 2026, 07:12 AM
- Improve internal links (2026-10-13)
Version 11 · Oct 13, 2026, 10:02 AM
- Added 6 contextual internal links
Version 10 · Jul 23, 2026, 03:00 PM
- Render formulas with LaTeX math notation
Version 9 · Jul 13, 2026, 11:06 PM