Extropic Z1T

Z1T is a family of sparse, transformer-like language models that Extropic designed to run partly on its Z1 probabilistic chip, described in a research post dated September 4, 2026 by Guillaume Verdon, Alexander Neagoe, Owen Lockwood and Seth Morton ^[1]. The company calls it "our first family of transformer-like models for sparse probabilistic chips like Z1" ^[1]. The post replaces the dense pieces of a standard transformer block with operations that fit Z1's fixed, degree-16 connection graph: RMS normalization becomes a Dynamic Tanh, softmax attention becomes a gated convolutional attention derived from the Attention Free Transformer, and every linear layer becomes a 4-sparse "tanh-linear" unit realized by averaging the output of a probabilistic bit; the remaining digital work (embeddings, residual additions, the vocabulary readout) runs on an FPGA coprocessor ^[1]. Alongside the architecture, Extropic reports an empirical scaling law for sparse transformers, released the training code as the Apache-2.0 repository extropic-ai/sparse-transformers, and published open weights for one model, Extropic-AI/Z1T-0, on Hugging Face ^[1][3][4].

The headline efficiency numbers are company estimates, not measurements of Z1 silicon. The post states that its projections are "based on theoretical chip energy consumption of Z1 based on our best estimates, which are anchored to reality from our experiments with similar pbits in X0", that the FPGA accounts for more than 95 percent of the estimated energy, and that the comparison against an NVIDIA H100 excludes the dense logit readout and the data movement between the chips ^[1]. On those terms Extropic estimates that a small Z1T model would use about 139 times less energy per decoded token than an H100 at 10 percent utilization, about 28 times less at 50 percent and about 14 times less at 100 percent; the company's X post rounded the first figure to "up to 140x energy efficiency gains over GPUs" ^[1][5]. Z1 itself is described by the company as taped out, with systems built on it due for early access in 2027 ^[2].

Background

Extropic builds what it calls thermodynamic sampling units (TSUs): arrays of probabilistic bits (pbits) that use transistor noise to draw samples from a programmable energy-based model, an approach covered more broadly at thermodynamic computing. The company's earlier public work centered on probabilistic graphical models and diffusion-like models; the Z1T post cites five of its own papers and posts before turning to transformers ^[1]. Its stated reason for the turn is that transformer training and inference "account for an enormous share of the workloads being executed on modern digital hardware accelerators", and that the resulting buildout has driven "a sharp spike in energy demand", for which the post cites an International Energy Agency report and a Congressional Research Service report ^[1]. The company's long-term position is that its hardware will eventually give rise to models "native to this new paradigm", but it describes the energy efficiency of transformer inference as "a very pressing need" in the meantime ^[1].

The practical framing is disaggregated inference. Extropic notes that transformer serving is already being split across substrates (the compute-bound prefill step and the memory-bound decode step on separate hardware, and attention-feedforward splits within a single decode pass) and proposes its own version: run Z1T "model parallel across Z1s and pipeline parallel across the heterogeneous network of FPGAs and Z1s" ^[1]. The first-generation Z1 systems ship with an FPGA on the same board, which is why the post's estimates use an FPGA as the companion processor; the company says a GPU or another accelerator "could in principle" fill the same role and that it "may explore" this later ^[1]. The post calls the work "an initial study in sparse neural-network and hardware co-design", adds that "the design of Z1 predates these considerations", and says the findings are being used "to inform future chip designs" ^[1].

Hardware substrate

Z1 is presented in the post as a graphical model of pbits: binary stochastic CMOS circuits whose probability of being +1 or -1 is set by a bias induced by neighboring pbits ^[1]. The chip samples from a programmable Ising model, the same family of model treated in software by Markov chain Monte Carlo methods, using Gibbs sampling at a 50 MHz internal update clock: each pbit redraws its value conditioned on its neighbors, in a "chromatic" schedule that updates non-adjacent pbits in parallel, "in-memory" ^[1]. The energy function is written as E(z) = sum over nodes j of h_j z_j plus sum over edges {j,k} of J_jk z_j z_k, with each z_j in {-1, +1}; h is the vector of biases and J the coupling matrix, and J_jk can only be nonzero where the silicon actually has a wire between pbits j and k ^[1].

Scroll sideways for more →

The two coupling figures differ by roughly a factor of ten. The die caption ("Eight cores, 269,568 pbits, 215,904 coupling parameters, >50 MHz sampling rate, <1 W") appears in the August Z1 announcement and again in the Z1T post, while the Z1T post's body text and its model-details box both give 2,135,904 ^[1][2]. Neither document reconciles the two, and this article reports both as published. A simple check favors the larger figure: 269,568 pbits with 16 couplings each gives at most 269,568 x 16 / 2 = 2,156,544 distinct edges if every pbit had a full set of neighbors, so 2,135,904 is consistent with a degree-16 graph whose edge pbits have fewer neighbors, while 215,904 would imply fewer than two couplings per pbit (this arithmetic is the article's own, not the company's).

Fixed degree is the property that shapes everything else in Z1T. Because each pbit has exactly 16 couplings, any matrix embedded in the chip's couplings is a subgraph of a degree-16 pattern and therefore very sparse; the post illustrates this by drawing a patch of the Z1 graph next to its adjacency matrix ^[1]. Extropic contrasts this with GPUs, which reach all-to-all connectivity through a shared cache hierarchy and are therefore built for large, regular, dense matrix multiplications; the post argues that sparsity buys little on a GPU because irregular memory access removes most of the theoretical saving, so "a matrix multiplication on a GPU between two 99% sparsity matrices is not 100x faster" ^[1]. Sparse connectivity is described as "key to its power efficiency and ... a hard constraint of the hardware" ^[1].

Encoding numbers in pbits

Neural-network activations and weights are continuous, while a pbit is binary. The Z1T post follows what it calls "the Boltzmann-machine lineage of neural computation" (citing the 1985 Boltzmann machine paper by Ackley, Hinton and Sejnowski) and uses averages of pbit statistics, rather than individual samples, as the classical numbers the network operates on ^[1].

A continuous value x is quantized into four pbits with power-of-two weights: x is approximated by the sum over i from 1 to 4 of 2^(-i-1) s_i, where each s_i is a spin in {-1, +1}, so the four streams are weighted 1/4, 1/8, 1/16 and 1/32 and give one of 16 levels per sample ^[1]. Extropic calls this "dy4p" (dyadic, 4 bits of precision) and describes it as the spin analogue of a 4-pbit probabilistic integer, "pInt4" ^[1]. Because the pbits are stochastic, a pInt is really a distribution over quantized numbers; the value used downstream is an empirical mean over N parallel samples from each pbit ^[1].

The post derives what that averaging buys. If the empirical variance is sigma squared, then requiring the standard error sigma / sqrt(N) to fit inside one bin of a k-bit fixed-point representation gives k at most (1/2) log2 N minus log2 sigma ^[1]. Effective precision therefore grows with the number of samples, can exceed the four physical bits, includes "what is effectively fractional bits of precision", and can be tuned at runtime by changing N ^[1]. Extropic presents this as a natural fit for the low-precision regime of modern deep learning, comparable in spirit to digital quantization but with the precision knob moved from the number format to the sample count ^[1].

Tanh-linear units

The building block of every Z1T layer is a "tanh-linear unit" made from one visible pbit and its 16 neighbors. With energy E(v, h) = -(b_v v + sum over j of J_j v h_j + sum over j of b_j h_j), the visible spin conditioned on its neighbors is a Bernoulli variable whose expectation is tanh(b_v + sum over j of J_j h_j) ^[1]. A weighted activation tanh(w x + b) is obtained by setting the couplings to w times the dy4p weights, and the vector form tanh(W x + b) is obtained by encoding W into the couplings over the flattened spin vector, which the post writes as tanh((W kron a) s + b) with a the vector of dyadic weights ^[1]. The hardware therefore computes a sparse matrix-vector product fused with a tanh as its native operation, with one unit per output node running in parallel ^[1].

Sixteen couplings per output pbit divide as four input values times four dy4p pbits each, so every projection in Z1T has fan-in 4; the post's Figure 4 shows "each output pbit spends its 16 couplings on 4 input values x 4 dy4p pbits" and notes that the illustrated 12-input matrix is schematic while "real models use D = 512" ^[1]. Two caveats are stated: the study assumes enough Z1 chips run side by side to place all tanh-linear units in a model-parallel layout, and the scaling results in Figure 1 do not include quantization of the activations ^[1].

Architecture

A standard transformer block combines self-attention, a normalization layer and a feed-forward network, all built from dense matrix operations ^[1]. Z1T substitutes a Z1-compatible primitive for each and assigns the leftover digital work to the FPGA.

Scroll sideways for more →

Dynamic Tanh in place of normalization

RMSNorm divides each activation vector by its root-mean-square, a reduction across the feature axis that has no direct analogue in a locally connected chip ^[1][11]. Extropic relies on the observation from "Transformers without Normalization" (Zhu, Chen, He, LeCun and Liu, 2025) that each component of a normalization layer behaves like a scaled tanh and can be replaced by DyT(x) = gamma tanh(alpha x) + beta, with alpha learnable ^[1][9]. The released code documents the reason it matters for Z1: "Nothing is reduced across the feature axis, so it's pure elementwise, which is what keeps the model in Z1-compilable form", and DyT is "the model's only normalisation" ^[12].

Gated convolutional attention

Instead of the O(T^2 D) query-key score matrix of softmax attention, Z1T adapts the Attention Free Transformer of Zhai et al. (2021) to sparse hardware ^[1][8]. For head i at position t, the output is Y = tanh(Q) elementwise-times N/D, where the numerator N is a causal one-dimensional convolution of exp(K) elementwise-times V with kernel exp(w) - 1, plus a running sum over all earlier positions of exp(K) elementwise-times V, and the denominator D is the same construction applied to exp(K) alone ^[1]. The gates Q, K and V come from 4-sparse projection matrices ^[1]. The effect is a per-channel weighted average of past values with a learned local stencil on top of a global pool; the post's Figure 5 walks a token through the mechanism, and its illustration shows a four-token kernel window plus one running sum covering everything earlier ^[1]. Unlike the windowed sparse attention used in the connectivity sweep below, the running sum lets every position see the whole prefix; the structure is closer to linear attention, with a per-channel rather than per-pair weighting.

The released implementation, class AFTConv in the z1t package, follows "AFT-conv (Eq. 6, causal)" of the paper: the K projection has one output per head rather than per channel, both K and Q pass through a tanh before the exponential and the gate respectively, the local kernel is initialized at zero so that the global pool dominates at the start of training, and the module is "genuinely causal" (the code comments explain that a non-causal variant would leak future tokens under teacher forcing and produce "fake-perfect" training loss) ^[12]. Two other AFT formulations are in the same file (a learned position-bias "full" variant and a per-channel decaying recurrence), but the released model and the published configs use the convolutional one ^[3][12].

Feed-forward and everything else

An MLP layer is written as tanh(W x + b); with sparse W it is compiled, in the post's words, "by laying out d_out many tanh-linear units on Z1 and stream[ing] the samples between the cores" ^[1]. In the code the MLP expansion ratio is 4, the hidden activation is SiLU by default and switches to tanh when the tanh_mlp option is set ("the RBM-native nonlinearity"), and a tanh_linear option wraps every linear layer except the logit head in a tanh so that the model matches "the tanh-linear form the z1nn Z1 compiler ingests" ^[12]. The logit head is exempt because a tanh would bound the logits to (-1, 1) and "wreck softmax cross-entropy" ^[12]. Positional information is an additive sinusoidal encoding held as a frozen buffer, and the post lists "pooling, positional-encoding, and residual work" among the FPGA's duties ^[1][12].

Extropic notes that the FPGA was chosen "to get concrete estimates quickly" and that the dense vocabulary matmul and the rest of the digital work "could run on other accelerators, such as GPUs or dedicated accelerators from Cerebras, Etched, etc." ^[1].

Scaling experiments

Extropic frames the study against the dense-model scaling laws of Hestness et al. (2017) and Hoffmann et al. (2022), which it says have "held for 15 orders of magnitude", and adds one variable: connectivity ^[1]. The stated questions are how loss changes with sparsity and connectivity at fixed compute and data, how sparse scaling compares with dense scaling, and "which design levers matter most for our next chips" ^[1]. Two experiments address them.

Z1T models

The first sweeps model size for the Z1-matched architecture (4-bit weights, four incoming edges per output node, GCA attention) trained on OpenWebText with the GPT-2 byte-pair-encoding tokenizer; the FPGA handles the non-Z1 operations ^[1]. The post says the featured split between Z1 and FPGA "was our best-performing configuration for Z1 from our fairly broad agentically-driven explorations" ^[1]. Figure 1 plots validation loss against training FLOPs, colored by "body" parameter count (excluding the final embedding-to-vocabulary matmul), with GPT-2-small and GPT-2-XL marked at 85 million and 1,476 million parameters under the same convention ^[1]. The company's extrapolation is that Z1T needs "about an order of magnitude more FLOPs" than a GPT-2 model to reach the same loss, and the dashed log-log fit to the frontier puts the cost of matching GPT-2-small at about 9.5e19 FLOPs ^[1]. The figure's published description has the GCA compute-optimal frontier falling from a loss of about 6.5 near 2e14 FLOPs to about 4.3 near 2e18 FLOPs, against a GPT-2-small reference loss of about 3.4 ^[1].

Extropic's argument for accepting that penalty is stated in the same paragraph: the operations that run on Z1 are estimated to be about three orders of magnitude more energy efficient than on a GPU, so a model that spends ten times the FLOPs can still come out about two orders of magnitude ahead on energy per token ^[1]. Whether that holds depends entirely on the energy model in the next section.

The connectivity sweep

The second experiment removes the Z1-specific operations and asks what fixed connectivity does to an ordinary transformer. The model is a GPT-2-style decoder with masked dot-product self-attention, RMS normalization and GELU feed-forward blocks, trained on byte-tokenized OpenWebText ^[1]. A single knob, connectivity c, controls both the linear layers (each row of a weight matrix keeps c nonzero inputs) and the attention (scores are windowed to the last c tokens, in the manner of sliding window attention) ^[1].

Scroll sideways for more →

Sources: ^[1][13].

Extropic stresses that this is "a fixed connectivity pattern, not a percentage sparsity": because the degree is fixed in silicon, the fraction of zeros rises toward 100 percent as width grows, which differs from the usual sparse-model setup that holds, say, 95 percent sparsity constant ^[1]. It also trains with the fixed connectivity from the start and organizes the analysis around iso-FLOP budgets, which it contrasts with the sparsely-connected foundation-model scaling laws of Frantar et al. (2023) ^[1][10]. The reported result is that "to achieve the same number of parameters with lower connectivity, the models must be (sometimes substantially) deeper and wider than the dense model", and that the curves "do provide some indication that dense FLOPs are more efficient than sparser FLOPs given the same parameter count" ^[1]. The company's rejoinder is the same as above: FLOPs "do not pay the same price across substrates", so a sparse model on Z1 "can still reach the same performance at a fraction of the power" ^[1].

The released code for this experiment (the st package, script experiments/sparse_gpt_scaling.py) implements the sparse layers as dense weights under a fixed random mask, counts FLOPs per example from the configuration, and logs runs to Weights & Biases; its sample configuration trains a 10-layer, 1,536-wide, 16-head model with attention window 4 and fan-in 4 at batch size 512 for 173,439 steps ^[4][13].

Energy and latency estimates

Every number in this section is an Extropic estimate produced from a model of Z1's energy per sample plus assumptions about the FPGA, compared against an H100 running the same small model; none of it is a measurement on a Z1 chip. The post says so directly ("theoretical chip energy consumption of Z1 based on our best estimates ... anchored to reality from our experiments with similar pbits in X0"), and the scope exclusions below are the post's own ^[1].

Energy per token

Scroll sideways for more →

Company estimates from simulation and stated FPGA assumptions, not measured Z1 silicon; the dense logit readout and FPGA-to-Z1 data movement are excluded. Source: ^[1].

The first column is model FLOPs utilization, the fraction of the H100's theoretical peak actually reached; the post says MFU is "around 40%" for LLMs such as Llama 3 and "often much lower" for very sparse models that GPUs are not optimized for ^[1]. The third column compares the whole Z1 system (Z1 fabric plus the connected FPGA, excluding data movement between them and excluding the final dense logit computation) against the H100; the last column compares only the layers that run on Z1, the sparse tanh layers ^[1].

The Z1T estimate is 294.52 nanojoules per token, of which 8.74 nJ is Z1 sampling and 285.78 nJ is the FPGA's share ^[1]. Including the final logit readout on the FPGA raises the total to roughly 136.4 microjoules per token, several hundred times the body estimate, which is why that layer is excluded from the ratios ^[1]. The post's model-details box lists the assumptions:

Scroll sideways for more →

Source: ^[1].

Several features of this setup limit what the ratios say. The reference model is small (11.55 million body parameters), so the H100 comparison is against a workload far below the GPU's design point. The H100 figure is a derived energy (peak fp32 energy per FLOP divided by an assumed utilization) rather than a measured power draw, and the higher multipliers come from assuming low utilization. The count of Z1 chips needed to host the model is explicitly "not modeled". And the post itself reports that the FPGA, not Z1, consumes more than 95 percent of the estimated energy, which makes the system-level ratio mostly a statement about the FPGA assumptions ^[1].

Throughput and latency

Scroll sideways for more →

The Z1T row is an estimate; the H100 rows are the company's measurements of a dense-equivalent model at batch size 1. Source: ^[1].

The Z1T figure assumes a single serial stream with no advanced pipelining: 25 sequential sampling layers, 8 FPGA-to-Z1 writes and 5 Z1-to-FPGA reads, with the FPGA doing the rest ^[1]. The time breakdown is 38.0 microseconds of FPGA orchestration (38 serial operations at 1 microsecond each), 16.0 microseconds of Z1 sampling (25 layers x 32 samples at 50 MHz), 3.2 microseconds of data reading, 1.3 microseconds of data writing, 0.13 microseconds of FPGA sample averaging and 0.16 microseconds of pooling, positional and residual work, at an assumed Z1 I/O rate of 25.6 Gbit/s ^[1]. Extropic calls the estimate conservative because parallel samples are not counted: drawing samples in parallel would cost the same energy but reach a given precision sooner ^[1].

The H100 baseline is an H100 80GB HBM3 running torch 2.7.0 on a dense-equivalent model with D = 512, L = 4 and 11.55 million body parameters in fp16, measured on August 12, 2026, with batch-1 sequential decoding and the vocabulary logits excluded on both sides ^[1]. The post concedes that this workload "achieves 0.006% MFU" on the GPU, that kernel launch times dominate at that size, and that "if we were to batch on the GPU, H100s would be substantially more efficient"; its recommendation is to use "our Z1 + XPU setup for decode rather than prefill, which is better done on GPU for now" ^[1].

Outlook stated by the company

Extropic's closing section is titled "One small step for co-design. One giant leap for the densification of intelligence." It describes the exercise as putting transformer-like operations onto "a joint probabilistic/FPGA fabric that was not originally designed for this set of operations" and finding "compelling energy efficiency gains" even so ^[1]. Because the FPGA takes more than 95 percent of the estimated energy, the company argues that a chip designed with Z1T-style models in mind, moving more operations onto sub-threshold CMOS, could reach "up to 1000x greater energy efficiency than GPUs", in line with the Z1-only-layer ratios in the table; it leaves this "for future algorithmic and hardware explorations" ^[1]. This is a projection about hardware that does not yet exist, and the post presents it as such.

Open release

Weights

The Hugging Face repository Extropic-AI/Z1T-0 was created on September 4, 2026 at 18:06 UTC and last modified at 20:43 UTC the same day ^[3]. It holds five files: a README, config.json, load_model.py, model.eqx (4,968,300,072 bytes) and .gitattributes ^[3]. The model card reads, in full, "Open weight for the first Z1T-0" with links to the GitHub repository and the research post; it declares library_name z1t and pipeline tag text-generation and carries no license tag as of September 5, 2026 ^[3]. The loader downloads the config and weights, builds the model with the z1t package's create_model and deserializes the Equinox weight tree into it, so the code repository is required to use the weights ^[3].

The configuration file specifies a vocabulary of 50,257 (the GPT-2 tokenizer), sequence length 256, 4 layers, embedding width 12,288, convolutional AFT attention with 4 heads and kernel size 4, DyT alpha 0.5, fan-in 4 on every linear layer, tanh wrapping on the linears and a tanh MLP ^[3]. The card does not state a parameter count. Applying the released module definitions to that configuration (this article's arithmetic, not a company figure) gives roughly 617.6 million values in the token embedding, 617.6 million in the dense logit head, and about 2.2 million in the four sparse blocks, about 1.24 billion in total, which at 32-bit precision accounts for the size of the weights file ^[3][12]. The released model is therefore much wider (D = 12,288) than the D = 512 model used for the energy and latency estimates, and its parameter count is dominated by the two dense vocabulary matrices that the efficiency analysis excludes.

Code

The GitHub repository extropic-ai/sparse-transformers ("Sparse Transformers in JAX") was created on August 31, 2026, is licensed under the Apache License 2.0, and had about a dozen stars and its most recent push on September 3, 2026 as of September 5 ^[4]. Its commit history runs from August 31 to September 3, with Owen Lockwood as the most frequent committer ^[4]. The top-level README describes "JAX research projects for sparse and attention variant transformer models" in two packages ^[4]:

Scroll sideways for more →

Sources: ^[4][12][14].

Both packages are written in JAX with the Equinox module library and Optax optimizers and require Python 3.12 or later; the z1t trainer uses AdamW with a warmup-cosine schedule, and the Shakespeare and OpenWebText configs use a peak learning rate of 3e-4 and weight decay 0.1, with the trainer defaulting to betas 0.9 and 0.95 and gradient clipping at 1.0 (the smoke and addition configs use 1e-3 and 5e-4) ^[4][12][14]. The OpenWebText loader is described as a "faithful port of nanoGPT/data/openwebtext/prepare.py" that tokenizes the Skylion007/openwebtext dataset with tiktoken's GPT-2 encoding into uint16 memmaps ^[14]. The z1t package's own README calls it "a JAX-based library for building and training decoder models with attention-free (AFT) attention, Dynamic Tanh norms, and fixed-sparsity linear layers" and points to the smoke config as the entry point ^[14].

What the repository does not contain is also worth stating. The checked-in z1t configurations train on character-level Shakespeare or a synthetic addition task; no configuration reproducing the OpenWebText Z1T runs of Figure 1 or the exact Z1T-0 training run is included, although the loader and the scaling script needed to run one are ^[4][14]. The z1t scaling script also carries no FLOPs accounting, because the repository's FLOPs formula is derived for windowed softmax attention and "has no AFT counterpart" ^[4]. The Z1 compiler that the code refers to as "z1nn" is not part of the release ^[12].

Reception

Extropic announced the post on X on September 4, 2026 with the wording "Achieving up to 140x energy efficiency gains over GPUs and revealing a new scaling law for sparse transformers"; the post's own table gives about 139x at 10 percent H100 utilization and the post's subtitle says "over 100x" ^[1][5]. On September 5 the X account MTS published a clip of Verdon in which he attributes the gap to GPUs being poor at sparse matrix multiplication ("It's like shipping a big freight ship across the ocean with a bunch of empty containers"), says that placing TSUs like Z1 next to GPUs "can get 100X energy efficiency gain at a data center level", and argues that computing more efficiently should come before proposals to build more power generation ^[6]. The post was submitted to Hacker News the same day it appeared and had drawn two points and one comment (a link to the weights and code) by September 5 ^[7]. A search of Bing News on September 5, 2026 for "Extropic Z1T" and "Extropic sparse transformers" returned no press coverage of the release; the most recent items were the July 2026 letter-of-intent stories.

References

1. ^Z1T: Sparse Transformer-Like Models for Probabilistic Hardware - Extropic (Guillaume Verdon, Alexander Neagoe, Owen Lockwood, Seth Morton), September 4, 2026.
2. ^From One to One Billion: Torx, Thermalizers, and Z1 - Extropic, August 3, 2026.
3. ^Extropic-AI/Z1T-0 - Hugging Face (Extropic), created September 4, 2026, accessed September 5, 2026 (README, config.json and load_model.py).
4. ^extropic-ai/sparse-transformers: Sparse Transformers in JAX - GitHub (Extropic), repository created August 31, 2026, accessed September 5, 2026.
5. ^Introducing Z1T (post by @extropic) - X (Extropic), September 4, 2026.
6. ^Extropic founder @beffjezos explains how Z1 chips sitting alongside GPUs could make AI data centers 100X more energy efficient (post by @MTSlive) - X (MTS), September 5, 2026.
7. ^Z1T: Sparse Transformer-Like Models for Probabilistic Hardware (Hacker News submission) - Hacker News, September 4, 2026, accessed September 5, 2026.
8. ^An Attention Free Transformer (arXiv:2105.14103) - arXiv (Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, Josh Susskind), May 28, 2021.
9. ^Transformers without Normalization (arXiv:2503.10622) - arXiv (Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, Zhuang Liu), March 13, 2025.
10. ^Scaling Laws for Sparsely-Connected Foundation Models (arXiv:2309.08520) - arXiv (Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, Utku Evci), September 15, 2023.
11. ^Root Mean Square Layer Normalization (arXiv:1910.07467) - arXiv (Biao Zhang, Rico Sennrich), October 16, 2019.
12. ^research/z1t/components.py - GitHub (Extropic), sparse-transformers repository, accessed September 5, 2026.
13. ^configs/st_owt.yaml - GitHub (Extropic), sparse-transformers repository, accessed September 5, 2026.
14. ^research/z1t/README.md, research/z1t/dataset.py and configs - GitHub (Extropic), sparse-transformers repository, accessed September 5, 2026.