GPT-NeoX
GPT-NeoX is an open-source library from EleutherAI for training large autoregressive language models on GPUs. The name is also used for GPT-NeoX-20B, the 20-billion-parameter model the group trained with the library and released in February 2022. [1][2] The library is written in PyTorch, builds on NVIDIA's Megatron-LM and Microsoft's DeepSpeed, and is distributed under the Apache License 2.0. [1] GPT-NeoX-20B was trained on The Pile using NVIDIA A100 GPUs provided by CoreWeave, and its weights were released under the same Apache 2.0 license. [2][3] Its technical report described it as, to the authors' knowledge, "the largest dense autoregressive model that has publicly available weights at the time of submission." [2]
The name also covers two things that outlived the 20B model itself. The model's tokenizer, a BPE vocabulary trained on the Pile, was reused or adapted by later projects including Pythia, OLMo, Dolma and DCLM. [4][5][6] The gpt_neox model type that Hugging Face added to its Transformers library is also the format in which Pythia, StableLM-Base-Alpha and RedPajama-INCITE checkpoints load. [7][8][9][38] The library is still maintained: its most recent commit as of mid-November 2026 was made on November 7, 2026, and EleutherAI says it has been used by more than 40 organizations to train more than 50 models. [11][1]
History
Origins (2020-2021)
EleutherAI's earlier GPT-Neo models were trained on Google TPUs with Mesh TensorFlow. [12] The GPT-NeoX repository was created on GitHub on December 22, 2020. [11] According to the group's one-year retrospective, the preprint of the Pile went live on New Year's Day 2021, and the next day EleutherAI announced a collaboration with CoreWeave, which pledged GPUs for a GPT-3-scale project; work on the new GPU codebase, GPT-NeoX, "began in earnest soon after." [12]
By mid-2021 the retrospective said the team "finally had code that could scale all the way to 175B, and beyond," but that a global GPU shortage was delaying the hardware. [12] While it waited, EleutherAI used spare TPUs to train the 6-billion-parameter GPT-J with a separate codebase, Mesh Transformer JAX. [12]
GPT-NeoX-20B (2022)
Connor Leahy announced GPT-NeoX-20B on EleutherAI's blog on February 2, 2022, after what the post called "a year-long odyssey through months of chip shortage-induced shipping delays." [3] As a thank-you to the compute donors, the public download was held back for seven days; on February 9, 2022 the weights became available from The Eye under Apache 2.0, and a hosted demo ran on GooseAI, an inference service from CoreWeave and Anlatan. [3] The same post called both the library and the model "very much research artifacts" and advised against deploying either in production without careful consideration. [3]
The technical report, "GPT-NeoX-20B: An Open-Source Autoregressive Language Model," appeared on arXiv on April 14, 2022 and was published in the proceedings of the ACL 2022 workshop on Challenges & Perspectives in Creating Large Language Models (BigScience Episode #5). [13][2] It lists 17 authors; Sid Black was the lead developer and overall point person, and Stella Biderman was the lead scientist and project manager. [2] Hugging Face added the architecture to its Transformers library in version 4.20.0, released on June 16, 2022. [7]
Within months, larger open models followed. Meta AI's OPT paper (May 2022) named EleutherAI, which had "released dense models up to 20B in size," among the exceptions to large models whose full weights were not available, and Meta itself released OPT models up to 66B parameters, with research access to the 175B model on request. [14]
Library versions and later development
Until March 9, 2023 the library depended on DeeperSpeed, EleutherAI's fork of an old DeepSpeed release (0.3.15). [1] The team then tagged two versioned releases: version 1.0 preserves the stack that trained GPT-NeoX-20B and the Pythia suite, and version 2.0, paired with a DeeperSpeed 2.0 built on the then-latest upstream DeepSpeed, is the maintained line. [1][15] The README's setup instructions still go through DeeperSpeed and recommend an isolated environment so other DeepSpeed-based projects do not break. [1]
Later additions broadened the library beyond dense transformers:
| Date merged | Addition | Pull request |
|---|---|---|
| 2024-03-10 | Mamba architecture | #1157 [16] |
| 2024-05-04 | Dropless mixture of experts via the megablocks library | #1192 [17] |
| 2024-05-06 | RWKV support | #1198 [18] |
| 2024-09-14 | Post-training examples for SFT, DPO and reward-model training | #1277 [19] |
| 2024-09-26 | KTO post-training example | #1294 [20] |
| 2024-12-19 | NVIDIA Transformer Engine integration | #1282 [21] |
| 2025-05-09 | Dropless MoE using megablocks' grouped GEMM, with experts sharded across the model-parallel group | #1210 [22] |
| 2026-09-04 | Training configurations for OLMo dense models | #1407 [23] |
The GPT-NeoX library
Design and features
The repository describes GPT-NeoX as EleutherAI's library for training large-scale language models on GPUs, based on Megatron-LM and "augmented with techniques from DeepSpeed as well as some novel optimizations." [1] Its main features, as listed by the maintainers, are:
- distributed training with ZeRO and 3D parallelism (data, tensor and pipeline parallelism);
- launching through Slurm, MPI or the IBM Job Step Manager, with runs at scale on AWS, CoreWeave, Oak Ridge's Summit and Frontier, Argonne's Polaris, LUMI and other systems;
- architectural options including rotary and ALiBi positional embeddings, parallel attention and feed-forward layers, and FlashAttention;
- predefined configurations for Pythia, PaLM, Falcon and LLaMA 1 and 2;
- curriculum learning;
- integration with Hugging Face's tokenizers and Transformers libraries, logging to Weights & Biases, Comet or TensorBoard, and evaluation through EleutherAI's LM Evaluation Harness. [1]
Models are defined in YAML configuration files passed to a launcher script, deepy.py, which wraps DeepSpeed's launcher and starts one process per GPU across the nodes listed in a hostfile. [1] Fused kernels are compiled just in time, which lets the same code run on AMD MI100 and MI250X GPUs as well as NVIDIA hardware. [1] Checkpoints can be exported to Hugging Face formats as GPTNeoXForCausalLM, LlamaForCausalLM or MistralForCausalLM, and conversion tools import Llama 1, Llama 2, Code Llama and Mistral-7B-v0.1 checkpoints for further training. [1]
The library is aimed at pretraining, not serving. Its README warns: "If you are not looking to train models with billions of parameters from scratch, this is likely the wrong library to use," and recommends Hugging Face Transformers for inference. [1]
Licensing and citation
The code is licensed under Apache 2.0. Files derived from NVIDIA's Apache-licensed Megatron code keep NVIDIA's copyright headers, and the README asks derivative works to preserve them. [1] EleutherAI asks users to cite the software as "GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch" (Andonian et al.), archived on Zenodo with DOI 10.5281/zenodo.5879544; the archived version 2.0 is dated March 10, 2023. [1][24] As of November 2026 the GitHub repository had about 7,500 stars and 1,100 forks. [11]
GPT-NeoX-20B
Architecture
GPT-NeoX-20B is a decoder-only autoregressive transformer whose architecture "largely follows that of GPT-3." [2] The authors wrote that it is almost identical to GPT-J but used GPT-3 as the point of reference because GPT-J had no canonical published design description. [2]
| Hyperparameter | Value |
|---|---|
| Parameters | 20,554,567,680 (about 19.9 billion non-embedding) |
| Layers | 44 |
| Hidden size ($d_{model}$) | 6,144 |
| Attention heads | 64 (head size 96) |
| Feed-forward size | 24,576 |
| Context length | 2,048 tokens |
| Tokenizer vocabulary | 50,257 (embedding matrix sized 50,432 in the released config) |
| Positional encoding | Rotary, on the first 25% of each head's dimensions |
| Input and output embeddings | Untied |
Sources for the table: the paper, the model card and the released configuration file. [2][25][26]
The main departures from GPT-3 were:
- Rotary position embeddings instead of learned positional embeddings. Following GPT-J, rotation is applied only to the first 25% of embedding dimensions, which the authors' early experiments found to be the best balance of performance and computational cost. [2]
- Parallel attention and feed-forward layers. Each block computes attention and the MLP from the same input and sums them, instead of running them in sequence. The motivation was efficiency: with tensor parallelism, the two results can be reduced locally before a single all-reduce. In Mesh Transformer JAX this had given a 15% throughput increase with comparable early loss curves. [2]
- Untied layer norms, by accident. The intended block was $x + \mathrm{Attn}(\mathrm{LN}_1(x)) + \mathrm{FF}(\mathrm{LN}_1(x))$, but a code oversight gave $x + \mathrm{Attn}(\mathrm{LN}_1(x)) + \mathrm{FF}(\mathrm{LN}_2(x))$, with two independent layer norms. The bug was found too late to restart; later small-scale experiments showed no performance difference. [2]
- Initialization. Feed-forward output layers before the residual use $\frac{2}{L\sqrt{d}}$ and other layers use the "small init" scheme $\sqrt{\frac{2}{d + 4d}}$, where $L$ is the number of layers and $d$ the hidden size. [2]
- Dense layers only. GPT-3 alternated dense and sparse attention layers; GPT-NeoX-20B uses dense layers throughout to keep the implementation simple. [2]
Training
The model was trained on the Pile, an 825 GiB English corpus drawn from 22 sources grouped as academic writing, web scrapes, prose, dialogue and miscellaneous text (GitHub, DeepMind Mathematics, Enron Emails). [2] The Pile was not deduplicated. The run continued past one epoch because validation loss kept falling into the start of the second. [2][25]
Hyperparameters were taken largely from the GPT-3 paper, since a sweep at 20B scale was out of reach; the learning rate of 0.97E-5 was interpolated between GPT-3's 13B and 175B values. [2] Training used a batch of about 3.15 million tokens (1,538 sequences of 2,048 tokens) for 150,000 steps (which works out to about 472 billion tokens), with a cosine decay to 10% of the peak learning rate, AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$), weight decay 0.01 and ZeRO optimizer-state sharding. [2] The published configuration trains in fp16. [2][27] Asked about random seeds on Twitter, Leahy said there had been compute for only one 20B run, so seeds were not compared. [28]
The hardware was twelve Supermicro AS-4124GO-NART servers, each with eight NVIDIA A100-SXM4-40GB GPUs and two AMD EPYC 7532 CPUs, connected over an InfiniBand fabric with GPUDirect RDMA. [2] The model was split with a tensor-parallel size of 2 and a pipeline-parallel size of 4, keeping the most communication-heavy traffic inside a node and data-parallel traffic between nodes; this sustained 117 teraFLOPS per GPU. [2] The software stack was PyTorch 1.10.0 built with CUDA 11.1 and NCCL 2.10.3. [2]
Tokenizer
GPT-NeoX-20B uses a new byte pair encoding tokenizer with the same vocabulary size as GPT-2 (50,257), trained on the Pile. [2] It differs from the GPT-2 tokenizer in three ways: it is trained on more varied data, it applies prefix spaces consistently at the start of a string, and it has dedicated tokens for runs of 1 to 24 spaces. [2] On the Pile validation set it produced 342,887,807 tokens against GPT-2's 383,111,734, a ratio of 0.895. The largest savings, more than 20%, were on the EuroParl, GitHub and PubMed Central components, and code-heavy subsets such as arXiv, GitHub and StackExchange also needed noticeably fewer tokens. [2] In the paper's example, a short recursive Python function takes 55 GPT-2 tokens and 39 GPT-NeoX-20B tokens. [2] The longest tokens in the new vocabulary are mostly scientific terms, which the authors attribute to the Pile's PubMed data. [2] Hugging Face's documentation notes that the extra whitespace tokens make the model more suitable for tasks such as code generation. [27]
Evaluation
EleutherAI evaluated the model with its LM Evaluation Harness against GPT-J-6B, the FairSeq dense models and the GPT-3 models on OpenAI's API. [2] Results reported in the announcement post:
| Task | GPT-3 Curie | GPT-J-6B | FairSeq-13B | GPT-NeoX-20B | GPT-3 DaVinci |
|---|---|---|---|---|---|
| LAMBADA | 69.51% | 68.29% | 70.95% | 72.00% | 75.16% |
| ANLI R1 | 32.80% | 32.40% | 34.00% | 34.00% | 36.30% |
| ANLI R2 | 33.50% | 34.00% | 33.00% | 34.40% | 37.00% |
| ANLI R3 | 35.50% | 35.50% | 34.75% | 35.40% | 36.83% |
| WSC | 49.54% | 49.54% | 55.44% | 50.00% | 59.18% |
| WinoGrande | 64.56% | 64.01% | 67.40% | 66.10% | 69.93% |
| HellaSwag | 54.81% | 36.53% | 57.69% | 53.50% | 63.46% |
| Average of the above | 48.60% | 45.75% | 50.43% | 49.34% | 53.98% |
| HendrycksTest (MMLU), zero-shot, average of subject groups | 27.20% | 27.62% | 26.79% | 28.98% | 33.41% |
Source: EleutherAI's announcement post. [3] The GPT-NeoX-20B figures match the zero-shot results in the technical report, but the report's appendix gives different zero-shot scores for several of the baselines, for example 51.8% on HellaSwag for GPT-J-6B and 55.4% for FairSeq 13B. [2]
Across 32 evaluations against FairSeq 13B, the paper reports that GPT-NeoX-20B did better on 22, worse on 4 and within the margin of error on 6. [2] Its weakest result was HellaSwag, four standard deviations below FairSeq 13B in both zero- and five-shot settings, which the authors called "largely inexplicable." [2] On arithmetic tasks GPT-J and GPT-NeoX consistently beat the GPT-3 and FairSeq models; the authors suggested this may come from the many equations in the Pile and warned against reading it as better out-of-distribution arithmetic reasoning. [2] The paper's headline finding was that GPT-J-6B and GPT-NeoX-20B gained much more from five-shot prompting than the FairSeq models: going from zero-shot to five-shot, GPT-NeoX-20B improved by 0.0598, against 0.0183 for FairSeq 13B. [2] The team did not run coding benchmarks, although many design choices had targeted code, because it underestimated their cost and difficulty. [2] InfoQ summarized the results as roughly a linear interpolation between OpenAI's Curie and DaVinci models. [28]
Independent evaluations came later. Stanford's HELM benchmark included GPT-NeoX (20B) among 30 models and listed it as one of the open models whose full weights could be downloaded. [29] HELM found it among the models with the lowest bits-per-byte on The Pile, TwitterAAE and ICE, and noted that its training data, unlike that of most evaluated models, was documented. [29] On the Hugging Face Open LLM Leaderboard the model card reports an average of 36.02, including 25.0 on five-shot MMLU and 2.43 on five-shot GSM8K. [25]
Release rationale and intended use
The paper argued that open access to large models was "critical to advancing research" in AI safety, mechanistic interpretability and the study of how capabilities scale, and called the public release "the most important contribution of this paper." [2] The authors acknowledged that release could speed up capabilities research but judged the benefits greater, reasoning that the labs pushing capabilities hardest already had stronger models. [2] The paper also promised weights from every 1,000 training steps to support research on training dynamics; the repository says the 150 intermediate checkpoints can be obtained by emailing EleutherAI while it works out how to serve them at scale. [2][1]
The paper estimated that inference was most economical on a pair of RTX 3090 Ti cards or a single A6000 GPU. [2] The repository lists "slim" weights of 39 GB without optimizer states and full weights of 268 GB with them, by direct download or BitTorrent. [1] The Hugging Face model card says the model was developed primarily for research and is "not intended for deployment as-is": it is English-only, was not fine-tuned with methods such as RLHF, and may produce offensive text because the Pile contains such material. [25]
Energy use and emissions
Because widely quoted estimates of NLP emissions had drawn attention, the team tracked its energy use. [2] Training consumed 43.92 MWh over 1,830 hours; scaling, testing and evaluation added the equivalent of another 920 hours, for 66.24 MWh in total. With an Illinois grid mix averaging 0.47905 tCO2/MWh, the paper reports 31.73 metric tons of CO2, about 10% of the figure Strubell et al. (2019) had estimated for a neural architecture search paper. [2] A later sentence in the same appendix instead gives the total as "just under 35 metric tons of CO2"; the 31.73-ton figure is the one consistent with 66.24 MWh at the stated emission factor. [2]
Adoption and influence
Models trained with the library
EleutherAI maintains a list of users in the repository; as of November 2026 it lists more than 50 models and more than 60 publications from more than 40 organizations, trained on NVIDIA and AMD GPUs on systems including Frontier, Summit, LUMI, CINECA Leonardo, AWS and CoreWeave. [30] Examples confirmed by the projects' own papers or model cards include:
| Model | Developer | Sizes | Notes |
|---|---|---|---|
| Pythia | EleutherAI | 70M to 12B (16 models) | Trained with the GPT-NeoX library and the GPT-NeoX-20B tokenizer; 154 checkpoints per model [4] |
| Polyglot-Ko | EleutherAI | 1.3B, 3.8B, 5.8B, 12.8B | Korean models trained with the GPT-NeoX codebase on compute from Stability AI [31] |
| StableLM-Base-Alpha | Stability AI | 3B, 7B | Model card lists GPT-NeoX as the training library and the NeoX tokenizer [32][10] |
| RedPajama-INCITE | Together and partners | 3B, 7B | Pythia architecture, trained with EleutherAI's DeeperSpeed on 3,072 V100 GPUs of the Summit supercomputer [33][34] |
| Llemma | Princeton, EleutherAI and others | 7B, 34B | Math models trained in bfloat16 with GPT-NeoX on 256 A100 40GB GPUs [35] |
| TildeOpen | Tilde | 30B | Trained with Tilde's branch of GPT-NeoX on 768 AMD MI250X GPUs on LUMI [36] |
Pythia was built for research on interpretability and training dynamics: its 16 models saw the same public data in the same order, and its paper credits GPT-NeoX's ability to scale linearly with the number of GPUs for large wall-clock speed-ups on the smaller models. [1][4] The repository also lists frameworks derived from GPT-NeoX, among them Savanna, which the Arc Institute used for the Evo 2 genomic models. [1][30]
The 20B model itself served as a base for fine-tunes. Together's OpenChatKit, announced on March 10, 2023, was built on GPT-NeoXT-Chat-Base-20B, a chat model fine-tuned from GPT-NeoX-20B on more than 43 million instructions. [37]
The tokenizer and architecture class
The GPT-NeoX-20B tokenizer has also been reused outside EleutherAI's own models. Pythia uses the GPT-NeoX-20B tokenizer, and its paper notes that the standard Pile comes to 334 billion tokens under it. [4] The Allen Institute for Artificial Intelligence's OLMo used a modified version of the tokenizer with extra tokens for masking personally identifiable information (vocabulary 50,280), and token counts for its Dolma corpus are given in GPT-NeoX tokens. [5] DCLM, a 2024 testbed for training-data experiments, measured its DCLM-Pool corpus as 240 trillion GPT-NeoX tokens and used that tokenizer for most experiments. [6]
In Hugging Face Transformers, checkpoints of Pythia, StableLM-Base-Alpha and RedPajama-INCITE all declare the gpt_neox model type and load as GPTNeoXForCausalLM. [8][9][38]
Timeline
| Date | Event |
|---|---|
| 2020-12-22 | GPT-NeoX repository created on GitHub [11] |
| 2021-01-01 | The Pile preprint released; CoreWeave collaboration announced the next day [12] |
| 2022-02-02 | GPT-NeoX-20B announced [3] |
| 2022-02-09 | GPT-NeoX-20B weights released under Apache 2.0 [3] |
| 2022-04-14 | Technical report posted to arXiv [13] |
| 2022-05-27 | Paper published in the BigScience Episode #5 workshop proceedings [2] |
| 2022-06-16 | Hugging Face Transformers 4.20.0 adds GPT-NeoX [7] |
| 2023-03-09 | Library version 1.0 tagged (version 2.0, on upstream DeepSpeed, followed on March 10) [15][1] |
| 2023-04-03 | Pythia paper released [4] |
| 2024-03-10 | Mamba support merged [16] |
| 2024-12-19 | Transformer Engine integration merged [21] |
| 2026-09-04 | OLMo training configurations merged [23] |
Continue reading
- EleutherAIEleutherAI is a non-profit artificial intelligence research institute that builds and openly releases large language models, datasets, and evaluation tools, and studies their interpretability and alignment.Mentioned 20 times above
- The Pile (dataset)The Pile is an 825.18 GiB (approximately 886 GB) English text corpus designed for training large language models, assembled from 22 diverse, high-quality subsets spanning academic, professional, internet…Mentioned 12 times above
- GPT-JGPT-J (full release name GPT-J-6B) is a 6-billion-parameter autoregressive transformer language model released by the EleutherAI collective in June 2021Mentioned 9 times above
- DeepSpeedDeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.Mentioned 7 times above
- Megatron-LMMegatron-LM is NVIDIA's open-source framework for training very large transformer language models across GPU clustersMentioned twice above
- Open weightsOpen weights is the practice of publishing the trained parameter tensors of a neural network so that anyone can download, run, fine-tune, and redistribute the model without obtaining a separate inference…Listed in See also
More to read
Topics:Large Language ModelsOpen Source AITraining & Optimization
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19EleutherAI. "GPT-NeoX" (repository README). GitHub, accessed November 2026. github.com/...gpt-neox
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41Black, Sid; Biderman, Stella; Hallahan, Eric; Anthony, Quentin; Gao, Leo; Golding, Laurence; He, Horace; Leahy, Connor; McDonell, Kyle; Phang, Jason; Pieler, Michael; Prashanth, USVSN Sai; Purohit, Shivanshu; Reynolds, Laria; Tow, Jonathan; Wang, Ben; Weinbach, Samuel. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model". Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-136, Association for Computational Linguistics, May 2022. aclanthology.org/2022.bigscience-1.9
- ^1 ^2 ^3 ^4 ^5 ^6 ^7Leahy, Connor. "Announcing GPT-NeoX-20B". EleutherAI Blog, February 2, 2022. blog.eleuther.ai/announcing-20b
- ^1 ^2 ^3 ^4 ^5Biderman, Stella; Schoelkopf, Hailey; Anthony, Quentin; et al. "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling". Proceedings of the 40th International Conference on Machine Learning, PMLR 202:2397-2430, 2023 (arXiv:2304.01373, April 3, 2023). arxiv.org/...2304.01373
- ^1 ^2Groeneveld, Dirk; Beltagy, Iz; Walsh, Pete; et al. "OLMo: Accelerating the Science of Language Models". arXiv:2402.00838, February 2024. arxiv.org/...2402.00838
- ^1 ^2Li, Jeffrey; Fang, Alex; et al. "DataComp-LM: In search of the next generation of training sets for language models". arXiv:2406.11794, June 2024. arxiv.org/...2406.11794
- ^1 ^2 ^3Hugging Face. "v4.20.0: Big Model inference, BLOOM, CvT, GPT Neo-X, LayoutLMv3, LeViT, LongT5, M-CTC-T, Trajectory Transformer and Wav2Vec2-Conformer". GitHub release, June 16, 2022. github.com/...v4.20.0
- ^1 ^2EleutherAI. "config.json" for EleutherAI/pythia-12b. Hugging Face. huggingface.co/...config.json
- ^1 ^2Together. "config.json" for togethercomputer/RedPajama-INCITE-7B-Base. Hugging Face. huggingface.co/...config.json
- ^Stability AI. "stabilityai/stablelm-base-alpha-7b" (model card). Hugging Face. huggingface.co/...stablelm-base-alpha-7b
- ^1 ^2 ^3 ^4GitHub. "EleutherAI/gpt-neox" repository metadata (creation date, last push, stars, forks). GitHub REST API, accessed November 18, 2026. api.github.com/...gpt-neox
- ^1 ^2 ^3 ^4 ^5Leahy, Connor; Hallahan, Eric; Gao, Leo; Biderman, Stella. "What A Long, Strange Trip It's Been: EleutherAI One Year Retrospective". EleutherAI Blog, July 7, 2021. blog.eleuther.ai/year-one
- ^1 ^2Black, Sid; Biderman, Stella; Hallahan, Eric; et al. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model". arXiv:2204.06745, April 14, 2022. arxiv.org/...2204.06745
- ^Zhang, Susan; et al. "OPT: Open Pre-trained Transformer Language Models". arXiv:2205.01068, May 2, 2022. arxiv.org/...2205.01068
- ^1 ^2EleutherAI. "Releases: GPT-NeoX 1.0 and GPT-NeoX 2.0". GitHub, March 2023. github.com/...releases
- ^1 ^2EleutherAI/gpt-neox. "Add Mamba Architecture" (pull request #1157). GitHub, merged March 10, 2024. github.com/...1157
- ^EleutherAI/gpt-neox. "Add megablocks dropless MoE" (pull request #1192). GitHub, merged May 4, 2024. github.com/...1192
- ^EleutherAI/gpt-neox. "add rwkv support" (pull request #1198). GitHub, merged May 6, 2024. github.com/...1198
- ^EleutherAI/gpt-neox. "Post training examples" (pull request #1277). GitHub, merged September 14, 2024. github.com/...1277
- ^EleutherAI/gpt-neox. "Add KTO Post-training example" (pull request #1294). GitHub, merged September 26, 2024. github.com/...1294
- ^1 ^2EleutherAI/gpt-neox. "TransformerEngine Integration" (pull request #1282). GitHub, merged December 19, 2024. github.com/...1282
- ^EleutherAI/gpt-neox. "Dmoe integration" (pull request #1210). GitHub, merged May 9, 2025. github.com/...1210
- ^1 ^2EleutherAI/gpt-neox. "OLMo Dense Model Training Configs" (pull request #1407). GitHub, merged September 4, 2026. github.com/...1407
- ^Andonian, Alex; et al. "GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch" (version 2.0). Zenodo, March 10, 2023. doi.org/...zenodo.5879544
- ^1 ^2 ^3 ^4EleutherAI. "EleutherAI/gpt-neox-20b" (model card). Hugging Face. huggingface.co/...gpt-neox-20b
- ^EleutherAI. "config.json" for EleutherAI/gpt-neox-20b. Hugging Face. huggingface.co/...config.json
- ^1 ^2Hugging Face. "GPT-NeoX". Transformers documentation. huggingface.co/...gpt_neox
- ^1 ^2Alford, Anthony. "EleutherAI Open-Sources 20 Billion Parameter AI Language Model GPT-NeoX-20B". InfoQ, April 5, 2022. infoq.com/...eleutherai-gpt-neox
- ^1 ^2Liang, Percy; Bommasani, Rishi; Lee, Tony; et al. "Holistic Evaluation of Language Models". Transactions on Machine Learning Research, August 2023 (arXiv:2211.09110). arxiv.org/...2211.09110
- ^1 ^2EleutherAI. "Models and Publications Using GPT-NeoX" (USERS.md). GitHub, accessed November 2026. github.com/...USERS.md
- ^Ko, Hyunwoong; Yang, Kichang; Ryu, Minho; et al. "A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models". arXiv:2306.02254, June 2023. arxiv.org/...2306.02254
- ^Stability AI. "StableLM: Stability AI Language Models" (repository README). GitHub. github.com/...StableLM
- ^Together. "Releasing 3B and 7B RedPajama-INCITE family of models including base, instruction-tuned & chat models". Together AI blog, May 5, 2023. together.ai/...redpajama-models-v1
- ^Together. "togethercomputer/RedPajama-INCITE-7B-Base" (model card). Hugging Face. huggingface.co/...RedPajama-INCITE-7B-Base
- ^Azerbayev, Zhangir; Schoelkopf, Hailey; Paster, Keiran; et al. "Llemma: An Open Language Model For Mathematics". ICLR 2024 (arXiv:2310.10631). arxiv.org/...2310.10631
- ^Tilde. "TildeAI/TildeOpen-30b" (model card). Hugging Face. huggingface.co/...TildeOpen-30b
- ^Together. "Announcing OpenChatKit". Together AI blog, March 10, 2023. together.ai/...openchatkit
- ^1 ^2Stability AI. "config.json" for stabilityai/stablelm-base-alpha-7b. Hugging Face. huggingface.co/...config.json
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
1 revision · v1 · 3,928 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check before publication on November 18, 2026: 128 claims checked against 40 cited and independent sources; 1 material and 6 minor corrections applied before publication.
Cite this page: AI Wiki. "GPT-NeoX." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/gpt_neox

