Revision History
SwiGLU · 10 revisions
Sizes are character counts of the article source. The signed number is the change from the previous revision.
Recent edit summaries
Detailed summaries recorded by editors. Generic maintenance summaries are omitted here; every recorded revision remains below. Dates describe the edit, not necessarily the event it covers.
- Correction: Corrected SwiGLU's GLUE rank (second to ReGLU), the claim that GELU's 2016 paper first described SiLU (that passage dates from a 2018 revision), Gemma's GeGLU (approximated, not exact GELU), Qwen2's FFN widths, a misread of DeepSeek-V3's SwiGLU recomputation, an unsupported Transformer Engine attribution, the original T5 activation (ReLU), a false 'within noise' statistic, and stale claims that only Gemma and OpenAI avoid SwiGLU (Nemotron-4 uses squared ReLU; gpt-oss uses SwiGLU) (plus 17 minor fixes)
Version 10 · Nov 2, 2026, 09:01 AM
- Update: added a video.
Version 9 · Nov 1, 2026, 08:17 AM
- Improve internal links (2026-10-13)
Version 7 · Oct 13, 2026, 10:20 AM
- Added 6 contextual internal links
Version 6 · Jul 23, 2026, 03:29 PM
- Render formulas with LaTeX math notation
Version 5 · Jul 11, 2026, 06:09 PM