# GPT-NeoX

> Source: https://aiwiki.ai/wiki/gpt_neox
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: Large Language Models, Open Source AI, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "GPT-NeoX." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/gpt_neox
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**GPT-NeoX** is an open-source library from [EleutherAI](https://aiwiki.ai/wiki/eleutherai) for training large autoregressive [language models](https://aiwiki.ai/wiki/language_model) on GPUs. The name is also used for **GPT-NeoX-20B**, the 20-billion-parameter model the group trained with the library and released in February 2022. [1][2] The library is written in [PyTorch](https://aiwiki.ai/wiki/pytorch), builds on NVIDIA's [Megatron-LM](https://aiwiki.ai/wiki/megatron_lm) and Microsoft's [DeepSpeed](https://aiwiki.ai/wiki/deepspeed), and is distributed under the [Apache License 2.0](https://aiwiki.ai/wiki/apache_license). [1] GPT-NeoX-20B was trained on [The Pile](https://aiwiki.ai/wiki/the_pile) using [NVIDIA A100](https://aiwiki.ai/wiki/nvidia_a100) GPUs provided by [CoreWeave](https://aiwiki.ai/wiki/coreweave), and its weights were released under the same Apache 2.0 license. [2][3] Its technical report described it as, to the authors' knowledge, "the largest dense autoregressive model that has publicly available weights at the time of submission." [2]

The name also covers two things that outlived the 20B model itself. The model's tokenizer, a BPE vocabulary trained on the Pile, was reused or adapted by later projects including [Pythia](https://aiwiki.ai/wiki/pythia), [OLMo](https://aiwiki.ai/wiki/olmo), [Dolma](https://aiwiki.ai/wiki/dolma) and [DCLM](https://aiwiki.ai/wiki/dclm). [4][5][6] The `gpt_neox` model type that Hugging Face added to its [Transformers](https://aiwiki.ai/wiki/transformers_library) library is also the format in which Pythia, StableLM-Base-Alpha and RedPajama-INCITE checkpoints load. [7][8][9][38] The library is still maintained: its most recent commit as of mid-November 2026 was made on November 7, 2026, and EleutherAI says it has been used by more than 40 organizations to train more than 50 models. [11][1]

## History

### Origins (2020-2021)

EleutherAI's earlier GPT-Neo models were trained on Google TPUs with Mesh TensorFlow. [12] The GPT-NeoX repository was created on GitHub on December 22, 2020. [11] According to the group's one-year retrospective, the preprint of the Pile went live on New Year's Day 2021, and the next day EleutherAI announced a collaboration with CoreWeave, which pledged GPUs for a GPT-3-scale project; work on the new GPU codebase, GPT-NeoX, "began in earnest soon after." [12]

By mid-2021 the retrospective said the team "finally had code that could scale all the way to 175B, and beyond," but that a global GPU shortage was delaying the hardware. [12] While it waited, EleutherAI used spare TPUs to train the 6-billion-parameter [GPT-J](https://aiwiki.ai/wiki/gpt_j) with a separate codebase, Mesh Transformer JAX. [12]

### GPT-NeoX-20B (2022)

[Connor Leahy](https://aiwiki.ai/wiki/connor_leahy) announced GPT-NeoX-20B on EleutherAI's blog on February 2, 2022, after what the post called "a year-long odyssey through months of chip shortage-induced shipping delays." [3] As a thank-you to the compute donors, the public download was held back for seven days; on February 9, 2022 the weights became available from The Eye under Apache 2.0, and a hosted demo ran on GooseAI, an inference service from CoreWeave and Anlatan. [3] The same post called both the library and the model "very much research artifacts" and advised against deploying either in production without careful consideration. [3]

The technical report, "GPT-NeoX-20B: An Open-Source Autoregressive Language Model," appeared on arXiv on April 14, 2022 and was published in the proceedings of the ACL 2022 workshop on Challenges & Perspectives in Creating Large Language Models (BigScience Episode #5). [13][2] It lists 17 authors; Sid Black was the lead developer and overall point person, and Stella Biderman was the lead scientist and project manager. [2] [Hugging Face](https://aiwiki.ai/wiki/hugging_face) added the architecture to its Transformers library in version 4.20.0, released on June 16, 2022. [7]

Within months, larger open models followed. Meta AI's OPT paper (May 2022) named EleutherAI, which had "released dense models up to 20B in size," among the exceptions to large models whose full weights were not available, and Meta itself released [OPT](https://aiwiki.ai/wiki/opt) models up to 66B parameters, with research access to the 175B model on request. [14]

### Library versions and later development

Until March 9, 2023 the library depended on DeeperSpeed, EleutherAI's fork of an old DeepSpeed release (0.3.15). [1] The team then tagged two versioned releases: version 1.0 preserves the stack that trained GPT-NeoX-20B and the Pythia suite, and version 2.0, paired with a DeeperSpeed 2.0 built on the then-latest upstream DeepSpeed, is the maintained line. [1][15] The README's setup instructions still go through DeeperSpeed and recommend an isolated environment so other DeepSpeed-based projects do not break. [1]

Later additions broadened the library beyond dense transformers:

| Date merged | Addition | Pull request |
|---|---|---|
| 2024-03-10 | [Mamba](https://aiwiki.ai/wiki/mamba) architecture | #1157 [16] |
| 2024-05-04 | Dropless [mixture of experts](https://aiwiki.ai/wiki/mixture_of_experts) via the megablocks library | #1192 [17] |
| 2024-05-06 | [RWKV](https://aiwiki.ai/wiki/rwkv) support | #1198 [18] |
| 2024-09-14 | Post-training examples for SFT, [DPO](https://aiwiki.ai/wiki/dpo) and reward-model training | #1277 [19] |
| 2024-09-26 | [KTO](https://aiwiki.ai/wiki/kto) post-training example | #1294 [20] |
| 2024-12-19 | NVIDIA Transformer Engine integration | #1282 [21] |
| 2025-05-09 | Dropless MoE using megablocks' grouped GEMM, with experts sharded across the model-parallel group | #1210 [22] |
| 2026-09-04 | Training configurations for OLMo dense models | #1407 [23] |

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/JcK1wUyIdxA" title="NLP Seminar, March 16, 2022. Stella Biderman (EleutherAI) Training GPT-NeoX 20B"></iframe>
<figcaption>"NLP Seminar, March 16, 2022. Stella Biderman (EleutherAI) Training GPT-NeoX 20B", video from AI Sweden, published March 31, 2022. <a href="https://www.youtube.com/watch?v=JcK1wUyIdxA">Watch on YouTube</a>.</figcaption>
</figure>

## The GPT-NeoX library

### Design and features

The repository describes GPT-NeoX as EleutherAI's library for training large-scale language models on GPUs, based on Megatron-LM and "augmented with techniques from DeepSpeed as well as some novel optimizations." [1] Its main features, as listed by the maintainers, are:

- distributed training with ZeRO and 3D parallelism (data, [tensor](https://aiwiki.ai/wiki/tensor_parallelism) and [pipeline](https://aiwiki.ai/wiki/pipeline_parallelism) parallelism);
- launching through [Slurm](https://aiwiki.ai/wiki/slurm), MPI or the IBM Job Step Manager, with runs at scale on AWS, CoreWeave, Oak Ridge's Summit and Frontier, Argonne's Polaris, LUMI and other systems;
- architectural options including rotary and ALiBi positional embeddings, parallel attention and feed-forward layers, and [FlashAttention](https://aiwiki.ai/wiki/flash_attention);
- predefined configurations for Pythia, PaLM, Falcon and [LLaMA](https://aiwiki.ai/wiki/llama) 1 and 2;
- curriculum learning;
- integration with Hugging Face's tokenizers and Transformers libraries, logging to Weights & Biases, Comet or TensorBoard, and evaluation through EleutherAI's [LM Evaluation Harness](https://aiwiki.ai/wiki/lm_evaluation_harness). [1]

Models are defined in YAML configuration files passed to a launcher script, `deepy.py`, which wraps DeepSpeed's launcher and starts one process per GPU across the nodes listed in a hostfile. [1] Fused kernels are compiled just in time, which lets the same code run on AMD MI100 and MI250X GPUs as well as NVIDIA hardware. [1] Checkpoints can be exported to Hugging Face formats as `GPTNeoXForCausalLM`, `LlamaForCausalLM` or `MistralForCausalLM`, and conversion tools import Llama 1, Llama 2, Code Llama and Mistral-7B-v0.1 checkpoints for further training. [1]

The library is aimed at pretraining, not serving. Its README warns: "If you are not looking to train models with billions of parameters from scratch, this is likely the wrong library to use," and recommends Hugging Face Transformers for inference. [1]

### Licensing and citation

The code is licensed under Apache 2.0. Files derived from NVIDIA's Apache-licensed Megatron code keep NVIDIA's copyright headers, and the README asks derivative works to preserve them. [1] EleutherAI asks users to cite the software as "GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch" (Andonian et al.), archived on Zenodo with DOI 10.5281/zenodo.5879544; the archived version 2.0 is dated March 10, 2023. [1][24] As of November 2026 the GitHub repository had about 7,500 stars and 1,100 forks. [11]

## GPT-NeoX-20B

### Architecture

GPT-NeoX-20B is a decoder-only [autoregressive](https://aiwiki.ai/wiki/autoregressive_model) transformer whose architecture "largely follows that of GPT-3." [2] The authors wrote that it is almost identical to GPT-J but used [GPT-3](https://aiwiki.ai/wiki/gpt-3) as the point of reference because GPT-J had no canonical published design description. [2]

| Hyperparameter | Value |
|---|---|
| Parameters | 20,554,567,680 (about 19.9 billion non-embedding) |
| Layers | 44 |
| Hidden size ($d_{model}$) | 6,144 |
| Attention heads | 64 (head size 96) |
| Feed-forward size | 24,576 |
| Context length | 2,048 tokens |
| Tokenizer vocabulary | 50,257 (embedding matrix sized 50,432 in the released config) |
| Positional encoding | Rotary, on the first 25% of each head's dimensions |
| Input and output embeddings | Untied |

Sources for the table: the paper, the model card and the released configuration file. [2][25][26]

The main departures from GPT-3 were:

- **[Rotary position embeddings](https://aiwiki.ai/wiki/rotary_position_embedding)** instead of learned positional embeddings. Following GPT-J, rotation is applied only to the first 25% of embedding dimensions, which the authors' early experiments found to be the best balance of performance and computational cost. [2]
- **Parallel attention and feed-forward layers.** Each block computes attention and the MLP from the same input and sums them, instead of running them in sequence. The motivation was efficiency: with tensor parallelism, the two results can be reduced locally before a single all-reduce. In Mesh Transformer JAX this had given a 15% throughput increase with comparable early loss curves. [2]
- **Untied layer norms, by accident.** The intended block was $x + \mathrm{Attn}(\mathrm{LN}_1(x)) + \mathrm{FF}(\mathrm{LN}_1(x))$, but a code oversight gave $x + \mathrm{Attn}(\mathrm{LN}_1(x)) + \mathrm{FF}(\mathrm{LN}_2(x))$, with two independent [layer norms](https://aiwiki.ai/wiki/layer_normalization). The bug was found too late to restart; later small-scale experiments showed no performance difference. [2]
- **Initialization.** Feed-forward output layers before the residual use $\frac{2}{L\sqrt{d}}$ and other layers use the "small init" scheme $\sqrt{\frac{2}{d + 4d}}$, where $L$ is the number of layers and $d$ the hidden size. [2]
- **Dense layers only.** GPT-3 alternated dense and sparse attention layers; GPT-NeoX-20B uses dense layers throughout to keep the implementation simple. [2]

### Training

The model was trained on the Pile, an 825 GiB English corpus drawn from 22 sources grouped as academic writing, web scrapes, prose, dialogue and miscellaneous text (GitHub, DeepMind Mathematics, Enron Emails). [2] The Pile was not deduplicated. The run continued past one epoch because validation loss kept falling into the start of the second. [2][25]

Hyperparameters were taken largely from the GPT-3 paper, since a sweep at 20B scale was out of reach; the learning rate of 0.97E-5 was interpolated between GPT-3's 13B and 175B values. [2] Training used a batch of about 3.15 million tokens (1,538 sequences of 2,048 tokens) for 150,000 steps (which works out to about 472 billion tokens), with a cosine decay to 10% of the peak learning rate, [AdamW](https://aiwiki.ai/wiki/adamw) ($\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$), weight decay 0.01 and ZeRO optimizer-state sharding. [2] The published configuration trains in fp16. [2][27] Asked about random seeds on Twitter, Leahy said there had been compute for only one 20B run, so seeds were not compared. [28]

The hardware was twelve Supermicro AS-4124GO-NART servers, each with eight NVIDIA A100-SXM4-40GB GPUs and two AMD EPYC 7532 CPUs, connected over an [InfiniBand](https://aiwiki.ai/wiki/infiniband) fabric with GPUDirect RDMA. [2] The model was split with a tensor-parallel size of 2 and a pipeline-parallel size of 4, keeping the most communication-heavy traffic inside a node and data-parallel traffic between nodes; this sustained 117 teraFLOPS per GPU. [2] The software stack was PyTorch 1.10.0 built with [CUDA](https://aiwiki.ai/wiki/cuda) 11.1 and [NCCL](https://aiwiki.ai/wiki/nccl) 2.10.3. [2]

### Tokenizer

GPT-NeoX-20B uses a new [byte pair encoding](https://aiwiki.ai/wiki/byte_pair_encoding) tokenizer with the same vocabulary size as [GPT-2](https://aiwiki.ai/wiki/gpt-2) (50,257), trained on the Pile. [2] It differs from the GPT-2 tokenizer in three ways: it is trained on more varied data, it applies prefix spaces consistently at the start of a string, and it has dedicated tokens for runs of 1 to 24 spaces. [2] On the Pile validation set it produced 342,887,807 tokens against GPT-2's 383,111,734, a ratio of 0.895. The largest savings, more than 20%, were on the EuroParl, GitHub and PubMed Central components, and code-heavy subsets such as arXiv, GitHub and StackExchange also needed noticeably fewer tokens. [2] In the paper's example, a short recursive Python function takes 55 GPT-2 tokens and 39 GPT-NeoX-20B tokens. [2] The longest tokens in the new vocabulary are mostly scientific terms, which the authors attribute to the Pile's PubMed data. [2] Hugging Face's documentation notes that the extra whitespace tokens make the model more suitable for tasks such as code generation. [27]

### Evaluation

EleutherAI evaluated the model with its LM Evaluation Harness against GPT-J-6B, the FairSeq dense models and the GPT-3 models on OpenAI's API. [2] Results reported in the announcement post:

| Task | GPT-3 Curie | GPT-J-6B | FairSeq-13B | GPT-NeoX-20B | GPT-3 DaVinci |
|---|---|---|---|---|---|
| [LAMBADA](https://aiwiki.ai/wiki/lambada) | 69.51% | 68.29% | 70.95% | 72.00% | 75.16% |
| ANLI R1 | 32.80% | 32.40% | 34.00% | 34.00% | 36.30% |
| ANLI R2 | 33.50% | 34.00% | 33.00% | 34.40% | 37.00% |
| ANLI R3 | 35.50% | 35.50% | 34.75% | 35.40% | 36.83% |
| WSC | 49.54% | 49.54% | 55.44% | 50.00% | 59.18% |
| [WinoGrande](https://aiwiki.ai/wiki/winogrande) | 64.56% | 64.01% | 67.40% | 66.10% | 69.93% |
| [HellaSwag](https://aiwiki.ai/wiki/hellaswag) | 54.81% | 36.53% | 57.69% | 53.50% | 63.46% |
| Average of the above | 48.60% | 45.75% | 50.43% | 49.34% | 53.98% |
| HendrycksTest ([MMLU](https://aiwiki.ai/wiki/mmlu)), zero-shot, average of subject groups | 27.20% | 27.62% | 26.79% | 28.98% | 33.41% |

Source: EleutherAI's announcement post. [3] The GPT-NeoX-20B figures match the zero-shot results in the technical report, but the report's appendix gives different zero-shot scores for several of the baselines, for example 51.8% on HellaSwag for GPT-J-6B and 55.4% for FairSeq 13B. [2]

Across 32 evaluations against FairSeq 13B, the paper reports that GPT-NeoX-20B did better on 22, worse on 4 and within the margin of error on 6. [2] Its weakest result was HellaSwag, four standard deviations below FairSeq 13B in both zero- and five-shot settings, which the authors called "largely inexplicable." [2] On arithmetic tasks GPT-J and GPT-NeoX consistently beat the GPT-3 and FairSeq models; the authors suggested this may come from the many equations in the Pile and warned against reading it as better out-of-distribution arithmetic reasoning. [2] The paper's headline finding was that GPT-J-6B and GPT-NeoX-20B gained much more from five-shot prompting than the FairSeq models: going from zero-shot to five-shot, GPT-NeoX-20B improved by 0.0598, against 0.0183 for FairSeq 13B. [2] The team did not run coding benchmarks, although many design choices had targeted code, because it underestimated their cost and difficulty. [2] InfoQ summarized the results as roughly a linear interpolation between OpenAI's Curie and DaVinci models. [28]

Independent evaluations came later. Stanford's [HELM](https://aiwiki.ai/wiki/helm) benchmark included GPT-NeoX (20B) among 30 models and listed it as one of the open models whose full weights could be downloaded. [29] HELM found it among the models with the lowest bits-per-byte on The Pile, TwitterAAE and ICE, and noted that its training data, unlike that of most evaluated models, was documented. [29] On the Hugging Face Open LLM Leaderboard the model card reports an average of 36.02, including 25.0 on five-shot MMLU and 2.43 on five-shot [GSM8K](https://aiwiki.ai/wiki/gsm8k). [25]

### Release rationale and intended use

The paper argued that open access to large models was "critical to advancing research" in [AI safety](https://aiwiki.ai/wiki/ai_safety), [mechanistic interpretability](https://aiwiki.ai/wiki/mechanistic_interpretability) and the study of how capabilities scale, and called the public release "the most important contribution of this paper." [2] The authors acknowledged that release could speed up capabilities research but judged the benefits greater, reasoning that the labs pushing capabilities hardest already had stronger models. [2] The paper also promised weights from every 1,000 training steps to support research on training dynamics; the repository says the 150 intermediate checkpoints can be obtained by emailing EleutherAI while it works out how to serve them at scale. [2][1]

The paper estimated that inference was most economical on a pair of RTX 3090 Ti cards or a single A6000 GPU. [2] The repository lists "slim" weights of 39 GB without optimizer states and full weights of 268 GB with them, by direct download or BitTorrent. [1] The Hugging Face model card says the model was developed primarily for research and is "not intended for deployment as-is": it is English-only, was not fine-tuned with methods such as [RLHF](https://aiwiki.ai/wiki/rlhf), and may produce offensive text because the Pile contains such material. [25]

### Energy use and emissions

Because widely quoted estimates of NLP emissions had drawn attention, the team tracked its energy use. [2] Training consumed 43.92 MWh over 1,830 hours; scaling, testing and evaluation added the equivalent of another 920 hours, for 66.24 MWh in total. With an Illinois grid mix averaging 0.47905 tCO2/MWh, the paper reports 31.73 metric tons of CO2, about 10% of the figure Strubell et al. (2019) had estimated for a neural architecture search paper. [2] A later sentence in the same appendix instead gives the total as "just under 35 metric tons of CO2"; the 31.73-ton figure is the one consistent with 66.24 MWh at the stated emission factor. [2]

## Adoption and influence

### Models trained with the library

EleutherAI maintains a list of users in the repository; as of November 2026 it lists more than 50 models and more than 60 publications from more than 40 organizations, trained on NVIDIA and AMD GPUs on systems including Frontier, Summit, LUMI, CINECA Leonardo, AWS and CoreWeave. [30] Examples confirmed by the projects' own papers or model cards include:

| Model | Developer | Sizes | Notes |
|---|---|---|---|
| Pythia | EleutherAI | 70M to 12B (16 models) | Trained with the GPT-NeoX library and the GPT-NeoX-20B tokenizer; 154 checkpoints per model [4] |
| Polyglot-Ko | EleutherAI | 1.3B, 3.8B, 5.8B, 12.8B | Korean models trained with the GPT-NeoX codebase on compute from [Stability AI](https://aiwiki.ai/wiki/stability_ai) [31] |
| StableLM-Base-Alpha | Stability AI | 3B, 7B | Model card lists GPT-NeoX as the training library and the NeoX tokenizer [32][10] |
| RedPajama-INCITE | [Together](https://aiwiki.ai/wiki/together_ai) and partners | 3B, 7B | Pythia architecture, trained with EleutherAI's DeeperSpeed on 3,072 V100 GPUs of the Summit supercomputer [33][34] |
| Llemma | Princeton, EleutherAI and others | 7B, 34B | Math models trained in bfloat16 with GPT-NeoX on 256 A100 40GB GPUs [35] |
| TildeOpen | Tilde | 30B | Trained with Tilde's branch of GPT-NeoX on 768 AMD MI250X GPUs on LUMI [36] |

Pythia was built for research on [interpretability](https://aiwiki.ai/wiki/interpretability) and training dynamics: its 16 models saw the same public data in the same order, and its paper credits GPT-NeoX's ability to scale linearly with the number of GPUs for large wall-clock speed-ups on the smaller models. [1][4] The repository also lists frameworks derived from GPT-NeoX, among them Savanna, which the Arc Institute used for the [Evo 2](https://aiwiki.ai/wiki/evo_2) genomic models. [1][30]

The 20B model itself served as a base for fine-tunes. Together's OpenChatKit, announced on March 10, 2023, was built on GPT-NeoXT-Chat-Base-20B, a chat model fine-tuned from GPT-NeoX-20B on more than 43 million instructions. [37]

### The tokenizer and architecture class

The GPT-NeoX-20B tokenizer has also been reused outside EleutherAI's own models. Pythia uses the GPT-NeoX-20B tokenizer, and its paper notes that the standard Pile comes to 334 billion tokens under it. [4] The Allen Institute for Artificial Intelligence's OLMo used a modified version of the tokenizer with extra tokens for masking personally identifiable information (vocabulary 50,280), and token counts for its Dolma corpus are given in GPT-NeoX tokens. [5] DCLM, a 2024 testbed for training-data experiments, measured its DCLM-Pool corpus as 240 trillion GPT-NeoX tokens and used that tokenizer for most experiments. [6]

In Hugging Face Transformers, checkpoints of Pythia, StableLM-Base-Alpha and RedPajama-INCITE all declare the `gpt_neox` model type and load as `GPTNeoXForCausalLM`. [8][9][38]

## Timeline

| Date | Event |
|---|---|
| 2020-12-22 | GPT-NeoX repository created on GitHub [11] |
| 2021-01-01 | The Pile preprint released; CoreWeave collaboration announced the next day [12] |
| 2022-02-02 | GPT-NeoX-20B announced [3] |
| 2022-02-09 | GPT-NeoX-20B weights released under Apache 2.0 [3] |
| 2022-04-14 | Technical report posted to arXiv [13] |
| 2022-05-27 | Paper published in the BigScience Episode #5 workshop proceedings [2] |
| 2022-06-16 | Hugging Face Transformers 4.20.0 adds GPT-NeoX [7] |
| 2023-03-09 | Library version 1.0 tagged (version 2.0, on upstream DeepSpeed, followed on March 10) [15][1] |
| 2023-04-03 | Pythia paper released [4] |
| 2024-03-10 | Mamba support merged [16] |
| 2024-12-19 | Transformer Engine integration merged [21] |
| 2026-09-04 | OLMo training configurations merged [23] |

## See also

- [EleutherAI](https://aiwiki.ai/wiki/eleutherai)
- [GPT-J](https://aiwiki.ai/wiki/gpt_j)
- [The Pile](https://aiwiki.ai/wiki/the_pile)
- [DeepSpeed](https://aiwiki.ai/wiki/deepspeed)
- [Megatron-LM](https://aiwiki.ai/wiki/megatron_lm)
- [Open weights](https://aiwiki.ai/wiki/open_weights)

## References

1. EleutherAI. "GPT-NeoX" (repository README). GitHub, accessed November 2026. https://github.com/EleutherAI/gpt-neox
2. Black, Sid; Biderman, Stella; Hallahan, Eric; Anthony, Quentin; Gao, Leo; Golding, Laurence; He, Horace; Leahy, Connor; McDonell, Kyle; Phang, Jason; Pieler, Michael; Prashanth, USVSN Sai; Purohit, Shivanshu; Reynolds, Laria; Tow, Jonathan; Wang, Ben; Weinbach, Samuel. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model". Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-136, Association for Computational Linguistics, May 2022. https://aclanthology.org/2022.bigscience-1.9/
3. Leahy, Connor. "Announcing GPT-NeoX-20B". EleutherAI Blog, February 2, 2022. https://blog.eleuther.ai/announcing-20b/
4. Biderman, Stella; Schoelkopf, Hailey; Anthony, Quentin; et al. "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling". Proceedings of the 40th International Conference on Machine Learning, PMLR 202:2397-2430, 2023 (arXiv:2304.01373, April 3, 2023). https://arxiv.org/abs/2304.01373
5. Groeneveld, Dirk; Beltagy, Iz; Walsh, Pete; et al. "OLMo: Accelerating the Science of Language Models". arXiv:2402.00838, February 2024. https://arxiv.org/abs/2402.00838
6. Li, Jeffrey; Fang, Alex; et al. "DataComp-LM: In search of the next generation of training sets for language models". arXiv:2406.11794, June 2024. https://arxiv.org/abs/2406.11794
7. Hugging Face. "v4.20.0: Big Model inference, BLOOM, CvT, GPT Neo-X, LayoutLMv3, LeViT, LongT5, M-CTC-T, Trajectory Transformer and Wav2Vec2-Conformer". GitHub release, June 16, 2022. https://github.com/huggingface/transformers/releases/tag/v4.20.0
8. EleutherAI. "config.json" for EleutherAI/pythia-12b. Hugging Face. https://huggingface.co/EleutherAI/pythia-12b/blob/main/config.json
9. Together. "config.json" for togethercomputer/RedPajama-INCITE-7B-Base. Hugging Face. https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blob/main/config.json
10. Stability AI. "stabilityai/stablelm-base-alpha-7b" (model card). Hugging Face. https://huggingface.co/stabilityai/stablelm-base-alpha-7b
11. GitHub. "EleutherAI/gpt-neox" repository metadata (creation date, last push, stars, forks). GitHub REST API, accessed November 18, 2026. https://api.github.com/repos/EleutherAI/gpt-neox
12. Leahy, Connor; Hallahan, Eric; Gao, Leo; Biderman, Stella. "What A Long, Strange Trip It's Been: EleutherAI One Year Retrospective". EleutherAI Blog, July 7, 2021. https://blog.eleuther.ai/year-one/
13. Black, Sid; Biderman, Stella; Hallahan, Eric; et al. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model". arXiv:2204.06745, April 14, 2022. https://arxiv.org/abs/2204.06745
14. Zhang, Susan; et al. "OPT: Open Pre-trained Transformer Language Models". arXiv:2205.01068, May 2, 2022. https://arxiv.org/abs/2205.01068
15. EleutherAI. "Releases: GPT-NeoX 1.0 and GPT-NeoX 2.0". GitHub, March 2023. https://github.com/EleutherAI/gpt-neox/releases
16. EleutherAI/gpt-neox. "Add Mamba Architecture" (pull request #1157). GitHub, merged March 10, 2024. https://github.com/EleutherAI/gpt-neox/pull/1157
17. EleutherAI/gpt-neox. "Add megablocks dropless MoE" (pull request #1192). GitHub, merged May 4, 2024. https://github.com/EleutherAI/gpt-neox/pull/1192
18. EleutherAI/gpt-neox. "add rwkv support" (pull request #1198). GitHub, merged May 6, 2024. https://github.com/EleutherAI/gpt-neox/pull/1198
19. EleutherAI/gpt-neox. "Post training examples" (pull request #1277). GitHub, merged September 14, 2024. https://github.com/EleutherAI/gpt-neox/pull/1277
20. EleutherAI/gpt-neox. "Add KTO Post-training example" (pull request #1294). GitHub, merged September 26, 2024. https://github.com/EleutherAI/gpt-neox/pull/1294
21. EleutherAI/gpt-neox. "TransformerEngine Integration" (pull request #1282). GitHub, merged December 19, 2024. https://github.com/EleutherAI/gpt-neox/pull/1282
22. EleutherAI/gpt-neox. "Dmoe integration" (pull request #1210). GitHub, merged May 9, 2025. https://github.com/EleutherAI/gpt-neox/pull/1210
23. EleutherAI/gpt-neox. "OLMo Dense Model Training Configs" (pull request #1407). GitHub, merged September 4, 2026. https://github.com/EleutherAI/gpt-neox/pull/1407
24. Andonian, Alex; et al. "GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch" (version 2.0). Zenodo, March 10, 2023. https://doi.org/10.5281/zenodo.5879544
25. EleutherAI. "EleutherAI/gpt-neox-20b" (model card). Hugging Face. https://huggingface.co/EleutherAI/gpt-neox-20b
26. EleutherAI. "config.json" for EleutherAI/gpt-neox-20b. Hugging Face. https://huggingface.co/EleutherAI/gpt-neox-20b/blob/main/config.json
27. Hugging Face. "GPT-NeoX". Transformers documentation. https://huggingface.co/docs/transformers/model_doc/gpt_neox
28. Alford, Anthony. "EleutherAI Open-Sources 20 Billion Parameter AI Language Model GPT-NeoX-20B". InfoQ, April 5, 2022. https://www.infoq.com/news/2022/04/eleutherai-gpt-neox
29. Liang, Percy; Bommasani, Rishi; Lee, Tony; et al. "Holistic Evaluation of Language Models". Transactions on Machine Learning Research, August 2023 (arXiv:2211.09110). https://arxiv.org/abs/2211.09110
30. EleutherAI. "Models and Publications Using GPT-NeoX" (USERS.md). GitHub, accessed November 2026. https://github.com/EleutherAI/gpt-neox/blob/main/USERS.md
31. Ko, Hyunwoong; Yang, Kichang; Ryu, Minho; et al. "A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models". arXiv:2306.02254, June 2023. https://arxiv.org/abs/2306.02254
32. Stability AI. "StableLM: Stability AI Language Models" (repository README). GitHub. https://github.com/Stability-AI/StableLM
33. Together. "Releasing 3B and 7B RedPajama-INCITE family of models including base, instruction-tuned & chat models". Together AI blog, May 5, 2023. https://www.together.ai/blog/redpajama-models-v1
34. Together. "togethercomputer/RedPajama-INCITE-7B-Base" (model card). Hugging Face. https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base
35. Azerbayev, Zhangir; Schoelkopf, Hailey; Paster, Keiran; et al. "Llemma: An Open Language Model For Mathematics". ICLR 2024 (arXiv:2310.10631). https://arxiv.org/abs/2310.10631
36. Tilde. "TildeAI/TildeOpen-30b" (model card). Hugging Face. https://huggingface.co/TildeAI/TildeOpen-30b
37. Together. "Announcing OpenChatKit". Together AI blog, March 10, 2023. https://www.together.ai/blog/openchatkit
38. Stability AI. "config.json" for stabilityai/stablelm-base-alpha-7b. Hugging Face. https://huggingface.co/stabilityai/stablelm-base-alpha-7b/blob/main/config.json

