Citation and evidence

GPT-NeoX

20 min full readUpdated 38 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

GPT-NeoX is an open-source library from EleutherAI for training large autoregressive language models on GPUs. The name is also used for GPT-NeoX-20B, the 20-billion-parameter model the group trained with the library and released in February 2022. [1][2] The library is written in PyTorch, builds on NVIDIA's Megatron-LM and Microsoft's DeepSpeed, and is distributed under the Apache License 2.0. [1] GPT-NeoX-20B was trained on The Pile using NVIDIA A100 GPUs provided by CoreWeave, and its weights were released under the same Apache 2.0 license. [2][3] Its technical report described it as, to the authors' knowledge, "the largest dense autoregressive model that has publicly available weights at the time of submission." [2]

The name also covers two things that outlived the 20B model itself. The model's tokenizer, a BPE vocabulary trained on the Pile, was reused or adapted by later projects including Pythia, OLMo, Dolma and DCLM. [4][5][6] The gpt_neox model type that Hugging Face added to its Transformers library is also the format in which Pythia, StableLM-Base-Alpha and RedPajama-INCITE checkpoints load. [7][8][9][38] The library is still maintained: its most recent commit as of mid-November 2026 was made on November 7, 2026, and EleutherAI says it has been used by more than 40 organizations to train more than 50 models. [11][1]

History

Origins (2020-2021)

EleutherAI's earlier GPT-Neo models were trained on Google TPUs with Mesh TensorFlow. [12] The GPT-NeoX repository was created on GitHub on December 22, 2020. [11] According to the group's one-year retrospective, the preprint of the Pile went live on New Year's Day 2021, and the next day EleutherAI announced a collaboration with CoreWeave, which pledged GPUs for a GPT-3-scale project; work on the new GPU codebase, GPT-NeoX, "began in earnest soon after." [12]

By mid-2021 the retrospective said the team "finally had code that could scale all the way to 175B, and beyond," but that a global GPU shortage was delaying the hardware. [12] While it waited, EleutherAI used spare TPUs to train the 6-billion-parameter GPT-J with a separate codebase, Mesh Transformer JAX. [12]

GPT-NeoX-20B (2022)

Connor Leahy announced GPT-NeoX-20B on EleutherAI's blog on February 2, 2022, after what the post called "a year-long odyssey through months of chip shortage-induced shipping delays." [3] As a thank-you to the compute donors, the public download was held back for seven days; on February 9, 2022 the weights became available from The Eye under Apache 2.0, and a hosted demo ran on GooseAI, an inference service from CoreWeave and Anlatan. [3] The same post called both the library and the model "very much research artifacts" and advised against deploying either in production without careful consideration. [3]

The technical report, "GPT-NeoX-20B: An Open-Source Autoregressive Language Model," appeared on arXiv on April 14, 2022 and was published in the proceedings of the ACL 2022 workshop on Challenges & Perspectives in Creating Large Language Models (BigScience Episode #5). [13][2] It lists 17 authors; Sid Black was the lead developer and overall point person, and Stella Biderman was the lead scientist and project manager. [2] Hugging Face added the architecture to its Transformers library in version 4.20.0, released on June 16, 2022. [7]

Within months, larger open models followed. Meta AI's OPT paper (May 2022) named EleutherAI, which had "released dense models up to 20B in size," among the exceptions to large models whose full weights were not available, and Meta itself released OPT models up to 66B parameters, with research access to the 175B model on request. [14]

Library versions and later development

Until March 9, 2023 the library depended on DeeperSpeed, EleutherAI's fork of an old DeepSpeed release (0.3.15). [1] The team then tagged two versioned releases: version 1.0 preserves the stack that trained GPT-NeoX-20B and the Pythia suite, and version 2.0, paired with a DeeperSpeed 2.0 built on the then-latest upstream DeepSpeed, is the maintained line. [1][15] The README's setup instructions still go through DeeperSpeed and recommend an isolated environment so other DeepSpeed-based projects do not break. [1]

Later additions broadened the library beyond dense transformers:

Date mergedAdditionPull request
2024-03-10Mamba architecture#1157 [16]
2024-05-04Dropless mixture of experts via the megablocks library#1192 [17]
2024-05-06RWKV support#1198 [18]
2024-09-14Post-training examples for SFT, DPO and reward-model training#1277 [19]
2024-09-26KTO post-training example#1294 [20]
2024-12-19NVIDIA Transformer Engine integration#1282 [21]
2025-05-09Dropless MoE using megablocks' grouped GEMM, with experts sharded across the model-parallel group#1210 [22]
2026-09-04Training configurations for OLMo dense models#1407 [23]

Expanded article table

The GPT-NeoX library

Design and features

The repository describes GPT-NeoX as EleutherAI's library for training large-scale language models on GPUs, based on Megatron-LM and "augmented with techniques from DeepSpeed as well as some novel optimizations." [1] Its main features, as listed by the maintainers, are:

  • distributed training with ZeRO and 3D parallelism (data, tensor and pipeline parallelism);
  • launching through Slurm, MPI or the IBM Job Step Manager, with runs at scale on AWS, CoreWeave, Oak Ridge's Summit and Frontier, Argonne's Polaris, LUMI and other systems;
  • architectural options including rotary and ALiBi positional embeddings, parallel attention and feed-forward layers, and FlashAttention;
  • predefined configurations for Pythia, PaLM, Falcon and LLaMA 1 and 2;
  • curriculum learning;
  • integration with Hugging Face's tokenizers and Transformers libraries, logging to Weights & Biases, Comet or TensorBoard, and evaluation through EleutherAI's LM Evaluation Harness. [1]

Models are defined in YAML configuration files passed to a launcher script, deepy.py, which wraps DeepSpeed's launcher and starts one process per GPU across the nodes listed in a hostfile. [1] Fused kernels are compiled just in time, which lets the same code run on AMD MI100 and MI250X GPUs as well as NVIDIA hardware. [1] Checkpoints can be exported to Hugging Face formats as GPTNeoXForCausalLM, LlamaForCausalLM or MistralForCausalLM, and conversion tools import Llama 1, Llama 2, Code Llama and Mistral-7B-v0.1 checkpoints for further training. [1]

The library is aimed at pretraining, not serving. Its README warns: "If you are not looking to train models with billions of parameters from scratch, this is likely the wrong library to use," and recommends Hugging Face Transformers for inference. [1]

Licensing and citation

The code is licensed under Apache 2.0. Files derived from NVIDIA's Apache-licensed Megatron code keep NVIDIA's copyright headers, and the README asks derivative works to preserve them. [1] EleutherAI asks users to cite the software as "GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch" (Andonian et al.), archived on Zenodo with DOI 10.5281/zenodo.5879544; the archived version 2.0 is dated March 10, 2023. [1][24] As of November 2026 the GitHub repository had about 7,500 stars and 1,100 forks. [11]

GPT-NeoX-20B

Architecture

GPT-NeoX-20B is a decoder-only autoregressive transformer whose architecture "largely follows that of GPT-3." [2] The authors wrote that it is almost identical to GPT-J but used GPT-3 as the point of reference because GPT-J had no canonical published design description. [2]

HyperparameterValue
Parameters20,554,567,680 (about 19.9 billion non-embedding)
Layers44
Hidden size ($d_{model}$)6,144
Attention heads64 (head size 96)
Feed-forward size24,576
Context length2,048 tokens
Tokenizer vocabulary50,257 (embedding matrix sized 50,432 in the released config)
Positional encodingRotary, on the first 25% of each head's dimensions
Input and output embeddingsUntied

Expanded article table

Sources for the table: the paper, the model card and the released configuration file. [2][25][26]

The main departures from GPT-3 were:

  • Rotary position embeddings instead of learned positional embeddings. Following GPT-J, rotation is applied only to the first 25% of embedding dimensions, which the authors' early experiments found to be the best balance of performance and computational cost. [2]
  • Parallel attention and feed-forward layers. Each block computes attention and the MLP from the same input and sums them, instead of running them in sequence. The motivation was efficiency: with tensor parallelism, the two results can be reduced locally before a single all-reduce. In Mesh Transformer JAX this had given a 15% throughput increase with comparable early loss curves. [2]
  • Untied layer norms, by accident. The intended block was $x + \mathrm{Attn}(\mathrm{LN}_1(x)) + \mathrm{FF}(\mathrm{LN}_1(x))$, but a code oversight gave $x + \mathrm{Attn}(\mathrm{LN}_1(x)) + \mathrm{FF}(\mathrm{LN}_2(x))$, with two independent layer norms. The bug was found too late to restart; later small-scale experiments showed no performance difference. [2]
  • Initialization. Feed-forward output layers before the residual use $\frac{2}{L\sqrt{d}}$ and other layers use the "small init" scheme $\sqrt{\frac{2}{d + 4d}}$, where $L$ is the number of layers and $d$ the hidden size. [2]
  • Dense layers only. GPT-3 alternated dense and sparse attention layers; GPT-NeoX-20B uses dense layers throughout to keep the implementation simple. [2]

Training

The model was trained on the Pile, an 825 GiB English corpus drawn from 22 sources grouped as academic writing, web scrapes, prose, dialogue and miscellaneous text (GitHub, DeepMind Mathematics, Enron Emails). [2] The Pile was not deduplicated. The run continued past one epoch because validation loss kept falling into the start of the second. [2][25]

Hyperparameters were taken largely from the GPT-3 paper, since a sweep at 20B scale was out of reach; the learning rate of 0.97E-5 was interpolated between GPT-3's 13B and 175B values. [2] Training used a batch of about 3.15 million tokens (1,538 sequences of 2,048 tokens) for 150,000 steps (which works out to about 472 billion tokens), with a cosine decay to 10% of the peak learning rate, AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$), weight decay 0.01 and ZeRO optimizer-state sharding. [2] The published configuration trains in fp16. [2][27] Asked about random seeds on Twitter, Leahy said there had been compute for only one 20B run, so seeds were not compared. [28]

The hardware was twelve Supermicro AS-4124GO-NART servers, each with eight NVIDIA A100-SXM4-40GB GPUs and two AMD EPYC 7532 CPUs, connected over an InfiniBand fabric with GPUDirect RDMA. [2] The model was split with a tensor-parallel size of 2 and a pipeline-parallel size of 4, keeping the most communication-heavy traffic inside a node and data-parallel traffic between nodes; this sustained 117 teraFLOPS per GPU. [2] The software stack was PyTorch 1.10.0 built with CUDA 11.1 and NCCL 2.10.3. [2]

Tokenizer

GPT-NeoX-20B uses a new byte pair encoding tokenizer with the same vocabulary size as GPT-2 (50,257), trained on the Pile. [2] It differs from the GPT-2 tokenizer in three ways: it is trained on more varied data, it applies prefix spaces consistently at the start of a string, and it has dedicated tokens for runs of 1 to 24 spaces. [2] On the Pile validation set it produced 342,887,807 tokens against GPT-2's 383,111,734, a ratio of 0.895. The largest savings, more than 20%, were on the EuroParl, GitHub and PubMed Central components, and code-heavy subsets such as arXiv, GitHub and StackExchange also needed noticeably fewer tokens. [2] In the paper's example, a short recursive Python function takes 55 GPT-2 tokens and 39 GPT-NeoX-20B tokens. [2] The longest tokens in the new vocabulary are mostly scientific terms, which the authors attribute to the Pile's PubMed data. [2] Hugging Face's documentation notes that the extra whitespace tokens make the model more suitable for tasks such as code generation. [27]

Evaluation

EleutherAI evaluated the model with its LM Evaluation Harness against GPT-J-6B, the FairSeq dense models and the GPT-3 models on OpenAI's API. [2] Results reported in the announcement post:

TaskGPT-3 CurieGPT-J-6BFairSeq-13BGPT-NeoX-20BGPT-3 DaVinci
LAMBADA69.51%68.29%70.95%72.00%75.16%
ANLI R132.80%32.40%34.00%34.00%36.30%
ANLI R233.50%34.00%33.00%34.40%37.00%
ANLI R335.50%35.50%34.75%35.40%36.83%
WSC49.54%49.54%55.44%50.00%59.18%
WinoGrande64.56%64.01%67.40%66.10%69.93%
HellaSwag54.81%36.53%57.69%53.50%63.46%
Average of the above48.60%45.75%50.43%49.34%53.98%
HendrycksTest (MMLU), zero-shot, average of subject groups27.20%27.62%26.79%28.98%33.41%

Expanded article table

Source: EleutherAI's announcement post. [3] The GPT-NeoX-20B figures match the zero-shot results in the technical report, but the report's appendix gives different zero-shot scores for several of the baselines, for example 51.8% on HellaSwag for GPT-J-6B and 55.4% for FairSeq 13B. [2]

Across 32 evaluations against FairSeq 13B, the paper reports that GPT-NeoX-20B did better on 22, worse on 4 and within the margin of error on 6. [2] Its weakest result was HellaSwag, four standard deviations below FairSeq 13B in both zero- and five-shot settings, which the authors called "largely inexplicable." [2] On arithmetic tasks GPT-J and GPT-NeoX consistently beat the GPT-3 and FairSeq models; the authors suggested this may come from the many equations in the Pile and warned against reading it as better out-of-distribution arithmetic reasoning. [2] The paper's headline finding was that GPT-J-6B and GPT-NeoX-20B gained much more from five-shot prompting than the FairSeq models: going from zero-shot to five-shot, GPT-NeoX-20B improved by 0.0598, against 0.0183 for FairSeq 13B. [2] The team did not run coding benchmarks, although many design choices had targeted code, because it underestimated their cost and difficulty. [2] InfoQ summarized the results as roughly a linear interpolation between OpenAI's Curie and DaVinci models. [28]

Independent evaluations came later. Stanford's HELM benchmark included GPT-NeoX (20B) among 30 models and listed it as one of the open models whose full weights could be downloaded. [29] HELM found it among the models with the lowest bits-per-byte on The Pile, TwitterAAE and ICE, and noted that its training data, unlike that of most evaluated models, was documented. [29] On the Hugging Face Open LLM Leaderboard the model card reports an average of 36.02, including 25.0 on five-shot MMLU and 2.43 on five-shot GSM8K. [25]

Release rationale and intended use

The paper argued that open access to large models was "critical to advancing research" in AI safety, mechanistic interpretability and the study of how capabilities scale, and called the public release "the most important contribution of this paper." [2] The authors acknowledged that release could speed up capabilities research but judged the benefits greater, reasoning that the labs pushing capabilities hardest already had stronger models. [2] The paper also promised weights from every 1,000 training steps to support research on training dynamics; the repository says the 150 intermediate checkpoints can be obtained by emailing EleutherAI while it works out how to serve them at scale. [2][1]

The paper estimated that inference was most economical on a pair of RTX 3090 Ti cards or a single A6000 GPU. [2] The repository lists "slim" weights of 39 GB without optimizer states and full weights of 268 GB with them, by direct download or BitTorrent. [1] The Hugging Face model card says the model was developed primarily for research and is "not intended for deployment as-is": it is English-only, was not fine-tuned with methods such as RLHF, and may produce offensive text because the Pile contains such material. [25]

Energy use and emissions

Because widely quoted estimates of NLP emissions had drawn attention, the team tracked its energy use. [2] Training consumed 43.92 MWh over 1,830 hours; scaling, testing and evaluation added the equivalent of another 920 hours, for 66.24 MWh in total. With an Illinois grid mix averaging 0.47905 tCO2/MWh, the paper reports 31.73 metric tons of CO2, about 10% of the figure Strubell et al. (2019) had estimated for a neural architecture search paper. [2] A later sentence in the same appendix instead gives the total as "just under 35 metric tons of CO2"; the 31.73-ton figure is the one consistent with 66.24 MWh at the stated emission factor. [2]

Adoption and influence

Models trained with the library

EleutherAI maintains a list of users in the repository; as of November 2026 it lists more than 50 models and more than 60 publications from more than 40 organizations, trained on NVIDIA and AMD GPUs on systems including Frontier, Summit, LUMI, CINECA Leonardo, AWS and CoreWeave. [30] Examples confirmed by the projects' own papers or model cards include:

ModelDeveloperSizesNotes
PythiaEleutherAI70M to 12B (16 models)Trained with the GPT-NeoX library and the GPT-NeoX-20B tokenizer; 154 checkpoints per model [4]
Polyglot-KoEleutherAI1.3B, 3.8B, 5.8B, 12.8BKorean models trained with the GPT-NeoX codebase on compute from Stability AI [31]
StableLM-Base-AlphaStability AI3B, 7BModel card lists GPT-NeoX as the training library and the NeoX tokenizer [32][10]
RedPajama-INCITETogether and partners3B, 7BPythia architecture, trained with EleutherAI's DeeperSpeed on 3,072 V100 GPUs of the Summit supercomputer [33][34]
LlemmaPrinceton, EleutherAI and others7B, 34BMath models trained in bfloat16 with GPT-NeoX on 256 A100 40GB GPUs [35]
TildeOpenTilde30BTrained with Tilde's branch of GPT-NeoX on 768 AMD MI250X GPUs on LUMI [36]

Expanded article table

Pythia was built for research on interpretability and training dynamics: its 16 models saw the same public data in the same order, and its paper credits GPT-NeoX's ability to scale linearly with the number of GPUs for large wall-clock speed-ups on the smaller models. [1][4] The repository also lists frameworks derived from GPT-NeoX, among them Savanna, which the Arc Institute used for the Evo 2 genomic models. [1][30]

The 20B model itself served as a base for fine-tunes. Together's OpenChatKit, announced on March 10, 2023, was built on GPT-NeoXT-Chat-Base-20B, a chat model fine-tuned from GPT-NeoX-20B on more than 43 million instructions. [37]

The tokenizer and architecture class

The GPT-NeoX-20B tokenizer has also been reused outside EleutherAI's own models. Pythia uses the GPT-NeoX-20B tokenizer, and its paper notes that the standard Pile comes to 334 billion tokens under it. [4] The Allen Institute for Artificial Intelligence's OLMo used a modified version of the tokenizer with extra tokens for masking personally identifiable information (vocabulary 50,280), and token counts for its Dolma corpus are given in GPT-NeoX tokens. [5] DCLM, a 2024 testbed for training-data experiments, measured its DCLM-Pool corpus as 240 trillion GPT-NeoX tokens and used that tokenizer for most experiments. [6]

In Hugging Face Transformers, checkpoints of Pythia, StableLM-Base-Alpha and RedPajama-INCITE all declare the gpt_neox model type and load as GPTNeoXForCausalLM. [8][9][38]

Timeline

DateEvent
2020-12-22GPT-NeoX repository created on GitHub [11]
2021-01-01The Pile preprint released; CoreWeave collaboration announced the next day [12]
2022-02-02GPT-NeoX-20B announced [3]
2022-02-09GPT-NeoX-20B weights released under Apache 2.0 [3]
2022-04-14Technical report posted to arXiv [13]
2022-05-27Paper published in the BigScience Episode #5 workshop proceedings [2]
2022-06-16Hugging Face Transformers 4.20.0 adds GPT-NeoX [7]
2023-03-09Library version 1.0 tagged (version 2.0, on upstream DeepSpeed, followed on March 10) [15][1]
2023-04-03Pythia paper released [4]
2024-03-10Mamba support merged [16]
2024-12-19Transformer Engine integration merged [21]
2026-09-04OLMo training configurations merged [23]

Expanded article table

Continue reading

More to read

Topics:Large Language ModelsOpen Source AITraining & Optimization

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19EleutherAI. "GPT-NeoX" (repository README). GitHub, accessed November 2026. github.com/...gpt-neox
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41Black, Sid; Biderman, Stella; Hallahan, Eric; Anthony, Quentin; Gao, Leo; Golding, Laurence; He, Horace; Leahy, Connor; McDonell, Kyle; Phang, Jason; Pieler, Michael; Prashanth, USVSN Sai; Purohit, Shivanshu; Reynolds, Laria; Tow, Jonathan; Wang, Ben; Weinbach, Samuel. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model". Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-136, Association for Computational Linguistics, May 2022. aclanthology.org/2022.bigscience-1.9
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7Leahy, Connor. "Announcing GPT-NeoX-20B". EleutherAI Blog, February 2, 2022. blog.eleuther.ai/announcing-20b
  4. ^1 ^2 ^3 ^4 ^5Biderman, Stella; Schoelkopf, Hailey; Anthony, Quentin; et al. "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling". Proceedings of the 40th International Conference on Machine Learning, PMLR 202:2397-2430, 2023 (arXiv:2304.01373, April 3, 2023). arxiv.org/...2304.01373
  5. ^1 ^2Groeneveld, Dirk; Beltagy, Iz; Walsh, Pete; et al. "OLMo: Accelerating the Science of Language Models". arXiv:2402.00838, February 2024. arxiv.org/...2402.00838
  6. ^1 ^2Li, Jeffrey; Fang, Alex; et al. "DataComp-LM: In search of the next generation of training sets for language models". arXiv:2406.11794, June 2024. arxiv.org/...2406.11794
  7. ^1 ^2 ^3Hugging Face. "v4.20.0: Big Model inference, BLOOM, CvT, GPT Neo-X, LayoutLMv3, LeViT, LongT5, M-CTC-T, Trajectory Transformer and Wav2Vec2-Conformer". GitHub release, June 16, 2022. github.com/...v4.20.0
  8. ^1 ^2EleutherAI. "config.json" for EleutherAI/pythia-12b. Hugging Face. huggingface.co/...config.json
  9. ^1 ^2Together. "config.json" for togethercomputer/RedPajama-INCITE-7B-Base. Hugging Face. huggingface.co/...config.json
  10. ^Stability AI. "stabilityai/stablelm-base-alpha-7b" (model card). Hugging Face. huggingface.co/...stablelm-base-alpha-7b
  11. ^1 ^2 ^3 ^4GitHub. "EleutherAI/gpt-neox" repository metadata (creation date, last push, stars, forks). GitHub REST API, accessed November 18, 2026. api.github.com/...gpt-neox
  12. ^1 ^2 ^3 ^4 ^5Leahy, Connor; Hallahan, Eric; Gao, Leo; Biderman, Stella. "What A Long, Strange Trip It's Been: EleutherAI One Year Retrospective". EleutherAI Blog, July 7, 2021. blog.eleuther.ai/year-one
  13. ^1 ^2Black, Sid; Biderman, Stella; Hallahan, Eric; et al. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model". arXiv:2204.06745, April 14, 2022. arxiv.org/...2204.06745
  14. ^Zhang, Susan; et al. "OPT: Open Pre-trained Transformer Language Models". arXiv:2205.01068, May 2, 2022. arxiv.org/...2205.01068
  15. ^1 ^2EleutherAI. "Releases: GPT-NeoX 1.0 and GPT-NeoX 2.0". GitHub, March 2023. github.com/...releases
  16. ^1 ^2EleutherAI/gpt-neox. "Add Mamba Architecture" (pull request #1157). GitHub, merged March 10, 2024. github.com/...1157
  17. ^EleutherAI/gpt-neox. "Add megablocks dropless MoE" (pull request #1192). GitHub, merged May 4, 2024. github.com/...1192
  18. ^EleutherAI/gpt-neox. "add rwkv support" (pull request #1198). GitHub, merged May 6, 2024. github.com/...1198
  19. ^EleutherAI/gpt-neox. "Post training examples" (pull request #1277). GitHub, merged September 14, 2024. github.com/...1277
  20. ^EleutherAI/gpt-neox. "Add KTO Post-training example" (pull request #1294). GitHub, merged September 26, 2024. github.com/...1294
  21. ^1 ^2EleutherAI/gpt-neox. "TransformerEngine Integration" (pull request #1282). GitHub, merged December 19, 2024. github.com/...1282
  22. ^EleutherAI/gpt-neox. "Dmoe integration" (pull request #1210). GitHub, merged May 9, 2025. github.com/...1210
  23. ^1 ^2EleutherAI/gpt-neox. "OLMo Dense Model Training Configs" (pull request #1407). GitHub, merged September 4, 2026. github.com/...1407
  24. ^Andonian, Alex; et al. "GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch" (version 2.0). Zenodo, March 10, 2023. doi.org/...zenodo.5879544
  25. ^1 ^2 ^3 ^4EleutherAI. "EleutherAI/gpt-neox-20b" (model card). Hugging Face. huggingface.co/...gpt-neox-20b
  26. ^EleutherAI. "config.json" for EleutherAI/gpt-neox-20b. Hugging Face. huggingface.co/...config.json
  27. ^1 ^2Hugging Face. "GPT-NeoX". Transformers documentation. huggingface.co/...gpt_neox
  28. ^1 ^2Alford, Anthony. "EleutherAI Open-Sources 20 Billion Parameter AI Language Model GPT-NeoX-20B". InfoQ, April 5, 2022. infoq.com/...eleutherai-gpt-neox
  29. ^1 ^2Liang, Percy; Bommasani, Rishi; Lee, Tony; et al. "Holistic Evaluation of Language Models". Transactions on Machine Learning Research, August 2023 (arXiv:2211.09110). arxiv.org/...2211.09110
  30. ^1 ^2EleutherAI. "Models and Publications Using GPT-NeoX" (USERS.md). GitHub, accessed November 2026. github.com/...USERS.md
  31. ^Ko, Hyunwoong; Yang, Kichang; Ryu, Minho; et al. "A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models". arXiv:2306.02254, June 2023. arxiv.org/...2306.02254
  32. ^Stability AI. "StableLM: Stability AI Language Models" (repository README). GitHub. github.com/...StableLM
  33. ^Together. "Releasing 3B and 7B RedPajama-INCITE family of models including base, instruction-tuned & chat models". Together AI blog, May 5, 2023. together.ai/...redpajama-models-v1
  34. ^Together. "togethercomputer/RedPajama-INCITE-7B-Base" (model card). Hugging Face. huggingface.co/...RedPajama-INCITE-7B-Base
  35. ^Azerbayev, Zhangir; Schoelkopf, Hailey; Paster, Keiran; et al. "Llemma: An Open Language Model For Mathematics". ICLR 2024 (arXiv:2310.10631). arxiv.org/...2310.10631
  36. ^Tilde. "TildeAI/TildeOpen-30b" (model card). Hugging Face. huggingface.co/...TildeOpen-30b
  37. ^Together. "Announcing OpenChatKit". Together AI blog, March 10, 2023. together.ai/...openchatkit
  38. ^1 ^2Stability AI. "config.json" for stabilityai/stablelm-base-alpha-7b. Hugging Face. huggingface.co/...config.json

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

1 revision · v1 · 3,928 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 128 claims checked against 40 cited and independent sources; 1 material and 6 minor corrections applied before publication.

Cite this page: AI Wiki. "GPT-NeoX." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/gpt_neox

Edit article