Protein language model

A protein language model (pLM) is a neural network trained on raw amino acid sequences using the same self-supervised objectives that power text models, with individual amino acids serving as tokens instead of words or subwords. The premise is that the proteins catalogued in sequence databases are the surviving output of billions of years of selection, so the statistical regularities in that corpus encode information about which residues can vary, which cannot, and which co-vary because they touch each other in three dimensions. A model that learns to fill in masked residues, or to predict the next one, absorbs that information without ever being shown a structure or a functional label.

Most protein language models are transformers adapted with minimal architectural change from natural language processing. The vocabulary is small, roughly the 20 standard amino acids plus a handful of special tokens, and sequences have no whitespace or word boundaries. Training follows one of two recipes borrowed directly from text: masked language modeling in the style of BERT, where a fraction of residues are hidden and reconstructed, or autoregressive next-token prediction in the style of GPT. The useful output is often not the model's predictions at all but its internal representations. A per-residue embedding vector extracted from a frozen pLM can be fed to a small supervised head to predict secondary structure, subcellular location, disorder, or binding sites, frequently beating the bioinformatics methods that required building a multiple sequence alignment (MSA) for every query ^[1][10][25].

Protein language models are routinely confused with AlphaFold, which is a different kind of system. AlphaFold 2 is not a language model: it takes a query sequence together with an MSA of evolutionarily related sequences and, where available, template structures, and processes them through the Evoformer network ^[18]. The distinction matters because the two approaches make opposite bets about where evolutionary information should live. AlphaFold retrieves it per query at inference time; a protein language model compresses it into weights during pre-training and then works from a single sequence.

How protein language models are trained

The corpus is public sequence data, and the imbalance inside it is the whole motivation. UniProtKB release 2026_02, issued 10 June 2026, holds 575,503 manually reviewed Swiss-Prot entries against 149,234,636 unreviewed TrEMBL entries, a ratio of roughly 1 to 260 ^[35]. The structural picture is similarly lopsided: when AlphaFold 2 was published, roughly 100,000 unique protein structures had been determined experimentally against billions of known sequences ^[18]. Any method that depends on curation or crystallography therefore covers a vanishing fraction of the proteome. Models are typically trained on UniRef50 or UniRef90 to limit redundancy, sometimes supplemented with the Big Fantastic Database (BFD), which folds in metagenomic sequences ^[10].

Two design choices separate the families. Encoder-style models trained with masking, such as the ESM series and ProtBert, produce bidirectional representations well suited to prediction tasks ^[1][10]. Decoder-style autoregressive models, such as ProGen and ProtGPT2, generate new sequences left to right and can be conditioned on control tags ^[13][16]. Encoder-decoder models such as ProtT5 sit between the two ^[10].

An early and much-cited result is that structural information appears in these models without supervision. Attention patterns and embeddings from sequence-only models recover residue-residue contacts, and the quality of that recovery improves with model scale ^[1][3]. The idea predates transformers: UniRep, published in Nature Methods in December 2019 by Ethan Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi and George Church, used a recurrent network and showed that simple models built on a learned representation could predict the stability of natural and designed proteins competitively with specialized methods ^[36].

The ESM family

Evolutionary Scale Modeling (ESM), begun at Meta AI's FAIR lab, continued at EvolutionaryScale and carried on at Biohub since 2026, is the longest-running pLM lineage.

ESM-1b was introduced by Alexander Rives and colleagues in a paper published in PNAS on 13 April 2021. The study trained deep transformers with masked language modeling on datasets of up to 250 million protein sequences from UniParc, 86 billion amino acids in total, and reported that information about secondary structure, tertiary contacts, and mutational effects emerged in the representation space ^[1]. The released ESM-1b checkpoint has 33 layers and 650 million parameters and was trained on UniRef50 (UR50/S 2018_03) ^[4].

ESM-1v, described by Joshua Meier and colleagues at NeurIPS 2021, applied the same architecture at 650M parameters trained on UniRef90 to variant effect prediction, showing that a pLM scores the functional consequences of mutations at state-of-the-art level using zero-shot inference alone, with no experimental supervision and no per-family retraining ^[2][4].

ESM-2 and ESMFold followed in a paper by Zeming Lin and colleagues, first posted to bioRxiv on 21 July 2022 and published in Science in March 2023. ESM-2 was trained on UniRef50/D at sizes from 8M to 15B parameters, the 15B model being what the authors called the largest language model of proteins to that date. ESMFold attaches a folding head to the language model and predicts atomic-level structure from a single sequence, which the paper reported as up to 60 times faster than the state of the art. The same work produced the ESM Metagenomic Atlas, more than 617 million predicted metagenomic structures including over 225 million high-confidence predictions ^[3][4]. Meta generated that atlas in about two weeks on a cluster of roughly 2,000 GPUs ^[33].

ESM3, from EvolutionaryScale, changed the framing from representation learning to generation. Announced on 25 June 2024 and published in Science on 16 January 2025 (volume 387, issue 6736, pages 850-858), it is a multimodal generative model that treats sequence, structure, and function as parallel tracks of discrete tokens and is trained with a masked objective across all of them, so a user can prompt with partial information in any track and sample the rest ^[5][6]. The largest model has 98 billion parameters and was trained with over 10^24 FLOPs. The corpus started from 2.78 billion natural proteins and, after synthetic augmentation, came to 3.15 billion sequences, 236 million structures and 539 million function-annotated proteins, 771 billion tokens in all ^[5][6][30]. Its headline experiment prompted the model to design a fluorescent protein; the resulting esmGFP is bright and sits at 58% sequence identity from known fluorescent proteins, a distance the authors estimated as equivalent to simulating 500 million years of evolution ^[5].

ESM Cambrian (ESM C), announced 4 December 2024, returned to pure representation learning. EvolutionaryScale released 300M and 600M open-weight models plus a 6B model, trained on sequences from UniRef, MGnify and the Joint Genome Institute clustered at 70% identity into 83 million, 372 million and 2 billion clusters respectively, and reported that the 300M version matches ESM2 650M while the 600M version rivals ESM2 3B ^[7].

The work then moved organizations. Biohub, where Alexander Rives is head of science, announced the next generation on 27 May 2026, and the ESM programming interface migrated from forge.evolutionaryscale.ai to biohub.ai ^[9][37]. A preprint titled "Language Modeling Materializes a World Model of Protein Biology" (Candido and colleagues, bioRxiv, 4 June 2026) covers ESMC, a structure predictor called ESMFold2, and an expanded ESM Atlas of more than 6.8 billion sequences and 1.1 billion predicted structures ^[8]. ESMFold2 combines ESMC 6B embeddings with a diffusion-based structure module and accepts optional MSA input, and the accompanying repository also ships sparse autoencoders that decompose ESMC representations into roughly 16,000 interpretable features ^[9]. Source code and weights for ESMC 6B, ESMFold2, and the autoencoders are distributed under the MIT license through the Biohub organization on GitHub and Hugging Face; the ESM3 1.4B open model is also now MIT licensed ^[30].

Scroll sideways for more →

Other model families

ProtTrans, from Ahmed Elnaggar, Burkhard Rost and collaborators, was the broadest early architecture sweep. Published in IEEE Transactions on Pattern Analysis and Machine Intelligence (volume 44, pages 7112-7127, 2022), it trained two autoregressive models (Transformer-XL, XLNet) and four autoencoder models (BERT, Albert, Electra, T5) on up to 393 billion amino acids from UniRef and BFD, using the Summit supercomputer with 5,616 GPUs and a TPU pod of up to 1,024 cores ^[10][11]. Its ProtT5-XL-UniRef50 checkpoint has roughly 3 billion parameters and was trained on 45 million UniRef50 sequences with 15% of amino acids masked ^[12]. Using embeddings as the only input, the paper reported three-state secondary structure accuracy of 81% to 87%, and transferring the most informative embeddings (ProtT5) outperformed the state of the art without using evolutionary information at all, bypassing database searches entirely ^[11].

ProGen, from Ali Madani and colleagues at Salesforce, was among the first pLMs whose generated proteins were validated in a wet lab across multiple families. Published in Nature Biotechnology on 26 January 2023 (volume 41, pages 1099-1106), it is a 1.2 billion-parameter decoder-style transformer with 36 layers and 8 attention heads per layer, trained on 280 million sequences from more than 19,000 families and augmented with 1,100 keyword control tags and 100,000 taxonomic tags that steer generation. Artificial lysozymes fine-tuned to five distinct families showed catalytic efficiencies comparable to natural enzymes at sequence identities as low as 31.4% ^[13].

The line continued through ProGen2, which scaled to 6.4 billion parameters on over a billion proteins from genomic, metagenomic, and immune repertoire databases ^[14], and ProGen3 from Profluent (bioRxiv, 16 April 2025), a family of sparse generative models scaled with compute-optimal scaling laws to 46 billion parameters pre-trained on 1.5 trillion amino acid tokens sampled from a curated set of 3.4 billion full-length proteins. That work tested scale in the wet lab and found larger models produced viable proteins across a wider diversity of families and responded better to alignment on laboratory data ^[15].

ProtGPT2 (Noelia Ferruz, Steffen Schmidt and Birte Höcker, Nature Communications, 27 July 2022) is a 738-million-parameter model with 36 layers built on the GPT-2 architecture and trained on UniRef50 version 2021_04. Disorder predictions indicated 88% of its generated proteins are globular, and similarity networks indicated that it samples unexplored regions of protein space ^[16].

xTrimoPGLM, from BioMap and Tsinghua University, took the opposite tack from architectural minimalism by trying to serve understanding and generation from one model. Published in Nature Methods in 2025 (volume 22, pages 1028-1039), it combines autoencoding and autoregressive objectives in a single 100 billion-parameter model trained on 1 trillion tokens, and its authors reported gains over prior baselines across 18 protein understanding benchmarks in four categories ^[17].

Relationship to AlphaFold and structure prediction

AlphaFold 2 (John Jumper and colleagues, Nature, July 2021) reported a median backbone accuracy of 0.96 Å r.m.s.d.95 at CASP14 against 2.8 Å for the next best method, using MSAs built with jackhmmer and HHblits plus optional templates ^[18]. AlphaFold 3 (Nature, 8 May 2024) replaced much of that architecture with a diffusion module and extended prediction to complexes containing nucleic acids, small molecules, ions, and modified residues ^[19]. Neither is a protein language model.

ESMFold was the clearest test of whether a pLM could substitute for the MSA. It could, at a cost. A 2026 benchmark in Frontiers in Genetics compared AlphaFold2, ESMFold and OmegaFold on 1,337 protein chains deposited in the PDB between July 2022 and July 2024, after the models' training cutoffs. AlphaFold2 led with a median TM-score of 0.96, median GDT-TS of 94% and median RMSD of 1.30 Å; ESMFold followed at 0.95, 90% and 1.74 Å; OmegaFold at 0.93, 89% and 1.98 Å. The alignment-free predictors ran 10 to 30 times faster, and the authors framed the result as a speed-accuracy tradeoff, judging the faster methods more practical for large-scale applications despite their slightly lower accuracy ^[27].

The 2026 ESMFold2 preprint claims to close that gap. Its abstract reports a structure prediction model that exceeds established methods for biomolecular complex prediction, including antibody-target interactions ^[8], and the release documentation states that ESMFold2 surpasses other models in DockQ pass rate on FoldBench protein-protein and antibody-antigen complexes ^[9]. Independent replication has not yet appeared. FoldBench itself, published in Nature Communications in December 2025, had benchmarked 1,522 biological assemblies across nine prediction tasks and found AlphaFold 3 the most accurate across most of them, while noting antibody-antigen failure rates above 50% for the field generally ^[26].

AlphaMissense (Science, volume 381, September 2023) is another system often grouped with pLMs but built differently: it is an adaptation of AlphaFold fine-tuned for variant effect prediction, and it classified 89% of possible human missense variants as either likely benign or likely pathogenic ^[20].

Applications

Variant effect prediction is where pLMs are most firmly established. Nadav Brandes and colleagues used ESM-1b to score every one of roughly 450 million possible human missense variants, separating pathogenic from benign across about 150,000 variants annotated in ClinVar and HGMD and outperforming the previous state of the art, work published in Nature Genetics in August 2023 ^[34]. The ProteinGym benchmark standardized the comparison: it assembles roughly 2.7 million missense variants across 217 deep mutational scanning assays and 2,525 clinical proteins, plus an indel benchmark of about 300,000 mutants across 74 assays, and scores dozens of predictors spanning alignment-based, single-sequence, structure-informed and hybrid categories ^[21].

Other established uses include protein design, where ProGen's lysozymes and ESM3's esmGFP are the reference results ^[5][13]; functional annotation and remote homology detection, where the representation space organizes proteins by distant evolutionary relationship rather than raw sequence similarity ^[1][25]; and directed evolution, where EVOLVEpro (Science, 24 January 2025) combined a pLM with regression models in a few-shot active learning loop and reported improvements of up to 100-fold in desired properties across six proteins spanning RNA production, genome editing, and antibody binding ^[22].

Binder discovery is the newest of these. The 2026 ESMFold2 preprint reports that a simple search procedure yielded high experimental success rates for proteins with nanomolar binding affinity, covering both miniproteins and single-chain antibodies, the latter a modality its authors describe as critical for therapeutic design and therefore for drug discovery ^[8].

Limitations and open questions

The most pointed critique concerns what these models actually learn. Zhang and colleagues (PNAS, October 2024) found that ESM-2 predicts contacts by storing statistics of coevolving residues in a way analogous to Markov random fields and multivariate Gaussian models, and that pLM-based structure predictors returned nonphysical structures for protein isoforms. Their conclusion was that pLMs store motifs of pairwise contacts rather than grasping the underlying biophysics ^[23].

Architecture may also matter less than assumed. Kevin Yang, Nicolo Fusi and Alex Lu (Cell Systems, March 2024) showed that convolutional networks pretrained with the same masked objective are competitive with, and occasionally better than, transformers on downstream tasks while scaling linearly rather than quadratically with sequence length ^[24]. Scale itself is contested too: the authors of AMPLIFY argued in 2024 that ever-larger pLMs assume public databases faithfully represent the underlying fitness landscape, which sampling bias makes unlikely, and demonstrated competitive results from a model orders of magnitude cheaper to train and deploy ^[29].

Confidence scores misbehave in exactly the setting designers care about. A 2026 evaluation of 614 de novo designed monomeric proteins found ESMFold's pLDDT the best single discriminator of design success, yet many failed designs still scored above successful ones, which confines these tools to coarse filtering rather than reliable prediction ^[31].

Konstantin Weissenow and Burkhard Rost, surveying the field in Current Opinion in Structural Biology in 2025, judged that MSA-free pLM predictions have become significantly more accurate for many applications and consume far fewer resources once pre-training is done, but that pLMs "have not, yet, succeeded to entirely replace the body of solutions developed over three decades" ^[25].

Compute cost and biosecurity are the other recurring concerns. The largest runs are expensive: ESM3's 98B model consumed more than 10^24 FLOPs ^[6], and ProGen3 reached 46 billion parameters over 1.5 trillion tokens ^[15]. The AMPLIFY authors open-sourced their pre-training codebase, data and checkpoints explicitly to democratize the training of pLMs ^[29]. On the security side, Bruce Wittmann and colleagues showed in Science in October 2025 that open-source AI protein design software could generate variants of proteins of concern that evaded the screening tools used by nucleic acid synthesis providers, and they developed and deployed patches that improved detection of synthetic homologs likely to retain wild-type function ^[32]. Model releases now ship with risk assessments: the 2026 ESM release describes a frontier safety evaluation and guardrails on its hosted platform that detect and restrict keywords and sequences corresponding to controlled pathogens and toxins ^[9], while the ESM3 model card states that virus-related data and some proteins from organisms on the USDA Select Agents and Toxins list were removed from training and that the function decoder was filtered for potentially harmful keywords ^[30].

Beyond proteins

The same modeling approach has been applied one level down, to nucleotides. Evo 2 (bioRxiv, 21 February 2025) is a genome foundation model trained on 9.3 trillion DNA base pairs across all domains of life at 7B and 40B parameters, with a context window of one million tokens at single-nucleotide resolution. Its authors reported that it learns exon-intron boundaries, transcription factor binding sites, and protein structural elements from DNA alone, and released model parameters, training and inference code, and the OpenGenome2 dataset openly ^[28].

See also

• AlphaFold
• ESM3
• ESMFold
• Protein folding
• Protein design
• Bioinformatics

References

1. ^Rives A, Meier J, Sercu T, et al. "Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences." PNAS 118(15):e2016239118, 13 April 2021. pmc.ncbi.nlm.nih.gov/...PMC8053943
2. ^Meier J, Rao R, Verkuil R, Liu J, Sercu T, Rives A. "Language models enable zero-shot prediction of the effects of mutations on protein function." Advances in Neural Information Processing Systems 34, 2021. proceedings.neurips.cc/...427296047067694-Abstract
3. ^Lin Z, Akin H, Rao R, et al. "Evolutionary-scale prediction of atomic-level protein structure with a language model." Science 379(6637):1123-1130, March 2023. doi.org/...science.ade2574
4. ^facebookresearch/esm repository README (ESM model table, ESM Metagenomic Atlas). github.com/...esm
5. ^Hayes T, Rao R, Akin H, et al. "Simulating 500 million years of evolution with a language model." Science 387(6736):850-858, 16 January 2025. doi.org/...science.ads0018
6. ^EvolutionaryScale. "ESM3: Simulating 500 million years of evolution with a language model." 25 June 2024. evolutionaryscale.ai/...esm3-release
7. ^EvolutionaryScale. "ESM Cambrian: Revealing the mysteries of proteins with unsupervised learning." 4 December 2024. evolutionaryscale.ai/...esm-cambrian
8. ^Candido S, Hayes T, Derry A, et al. "Language Modeling Materializes a World Model of Protein Biology." bioRxiv 10.64898/2026.06.03.729735, posted 4 June 2026. doi.org/...2026.06.03.729735
9. ^Biohub/esm repository README (ESMC, ESMFold2, ESM Atlas, sparse autoencoders, DockQ pass rate on FoldBench, frontier safety, MIT licenses, API migration to biohub.ai). github.com/...esm
10. ^Elnaggar A, Heinzinger M, Dallago C, et al. "ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning." IEEE Transactions on Pattern Analysis and Machine Intelligence 44(10):7112-7127, October 2022. doi.org/...TPAMI.2021.3095381
11. ^Elnaggar A, et al. "ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing." arXiv:2007.06225, 13 July 2020 (revised 2021). arxiv.org/...2007.06225
12. ^Rostlab. ProtT5-XL-UniRef50 model card, Hugging Face. huggingface.co/...prot_t5_xl_uniref50
13. ^Madani A, Krause B, Greene ER, et al. "Large language models generate functional protein sequences across diverse families." Nature Biotechnology 41(8):1099-1106, published online 26 January 2023. pmc.ncbi.nlm.nih.gov/...PMC10400306
14. ^Nijkamp E, Ruffolo J, Weinstein EN, Naik N, Madani A. "ProGen2: Exploring the Boundaries of Protein Language Models." arXiv:2206.13517, 27 June 2022. arxiv.org/...2206.13517
15. ^Bhatnagar A, Jain S, Beazer J, et al. "Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins." bioRxiv 10.1101/2025.04.15.649055, 16 April 2025. doi.org/...2025.04.15.649055
16. ^Ferruz N, Schmidt S, Höcker B. "ProtGPT2 is a deep unsupervised language model for protein design." Nature Communications 13:4348, 27 July 2022. pmc.ncbi.nlm.nih.gov/...PMC9329459
17. ^Chen B, Cheng X, Li P, et al. "xTrimoPGLM: unified 100-billion-parameter pretrained transformer for deciphering the language of proteins." Nature Methods 22(5):1028-1039, published online 3 April 2025. doi.org/...s41592-025-02636-z
18. ^Jumper J, Evans R, Pritzel A, et al. "Highly accurate protein structure prediction with AlphaFold." Nature 596(7873):583-589, August 2021. pmc.ncbi.nlm.nih.gov/...PMC8371605
19. ^Abramson J, Adler J, Dunger J, et al. "Accurate structure prediction of biomolecular interactions with AlphaFold 3." Nature 630:493-500, 8 May 2024. doi.org/...s41586-024-07487-w
20. ^Cheng J, Novati G, Pan J, et al. "Accurate proteome-wide missense variant effect prediction with AlphaMissense." Science 381(6664):eadg7492, September 2023. doi.org/...science.adg7492
21. ^Notin P, Kollasch AW, Ritter D, et al. "ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction." bioRxiv 10.1101/2023.12.07.570727, 8 December 2023; benchmark repository and current statistics at github.com/...ProteinGym
22. ^Jiang K, Yan Z, Di Bernardo M, et al. "Rapid in silico directed evolution by a protein language model with EVOLVEpro." Science 387(6732):eadr6006, 24 January 2025. doi.org/...science.adr6006
23. ^Zhang Z, Wayment-Steele HK, Brixi G, Wang H, Kern D, Ovchinnikov S. "Protein language models learn evolutionary statistics of interacting sequence motifs." PNAS 121(45):e2406285121, 28 October 2024. doi.org/...pnas.2406285121
24. ^Yang KK, Fusi N, Lu AX. "Convolutions are competitive with transformers for protein sequence pretraining." Cell Systems 15(3):286-294.e2, March 2024. doi.org/...j.cels.2024.01.008
25. ^Weissenow K, Rost B. "Are protein language models the new universal key?" Current Opinion in Structural Biology 91:102997, 2025 (published online 7 February 2025). doi.org/...j.sbi.2025.102997
26. ^Xu S, Feng Q, Qiao L, et al. "Benchmarking all-atom biomolecular structure prediction with FoldBench." Nature Communications, December 2025. doi.org/...s41467-025-67127-3
27. ^Hyskova A, Marsalkova E, Simecek P. "Balancing speed and precision in protein folding: a comparison of AlphaFold2, ESMFold, and OmegaFold." Frontiers in Genetics, 14 January 2026. doi.org/...fgene.2025.1715037
28. ^Brixi G, Durrant MG, Ku J, et al. "Genome modeling and design across all domains of life with Evo 2." bioRxiv 10.1101/2025.02.18.638918, 21 February 2025. doi.org/...2025.02.18.638918
29. ^Fournier Q, Vernon RM, van der Sloot A, Schulz B, Chandar S, Langmead CJ. "Protein Language Models: Is Scaling Necessary?" bioRxiv 10.1101/2024.09.23.614603, 23 September 2024. doi.org/...2024.09.23.614603
30. ^EvolutionaryScale. esm3-sm-open-v1 model card, Hugging Face (license, training data, and safety filtering). huggingface.co/...esm3-sm-open-v1
31. ^Garcia M, Dixit SM, Rocklin GJ. "Evaluating zero-shot prediction of monomeric protein design success by AlphaFold, ESMFold, and ProteinMPNN." Protein Science 35(2):e70453, February 2026. doi.org/...pro.70453
32. ^Wittmann BJ, Alexanian T, Bartling C, et al. "Strengthening nucleic acid biosecurity screening against generative protein design tools." Science 390(6768):82-87, 2 October 2025. doi.org/...science.adu8578
33. ^Meta AI. "ESM Metagenomic Atlas: The first view of the 'dark matter' of the protein universe." 1 November 2022. ai.meta.com/...protein-folding-esmfold-metagenomics
34. ^Brandes N, Goldman G, Wang CH, Ye CJ, Ntranos V. "Genome-wide prediction of disease variant effects with a deep protein language model." Nature Genetics 55(9):1512-1522, 10 August 2023. doi.org/...s41588-023-01465-0
35. ^UniProt Consortium. UniProt release notes, release 2026_02 of 10 June 2026 (Swiss-Prot 575,503 entries; TrEMBL 149,234,636 entries). ftp.uniprot.org/...relnotes.txt
36. ^Alley EC, Khimulya G, Biswas S, AlQuraishi M, Church GM. "Unified rational protein engineering with sequence-based deep representation learning." Nature Methods 16(12):1315-1322, December 2019. doi.org/...s41592-019-0598-1
37. ^Biohub. "Biohub Releases a World Model of Protein Biology." 27 May 2026 (ESMC, ESMFold2, ESM Atlas of 6.8 billion sequences and 1.1 billion predicted structures; Alex Rives, Head of Science). biohub.org/...world-model-of-protein-biology