TranscriptFormer
TranscriptFormer is a family of generative foundation models for single-cell transcriptomics developed by researchers at the Chan Zuckerberg Initiative (CZI). The models are autoregressive transformers that learn the joint distribution of which genes a cell expresses and how many transcripts of each are counted, and they were trained on up to 112 million cells from 12 species whose lineages span about 1.53 billion years of evolution.[1][2] CZI announced the models on April 30, 2025, a day after the bioRxiv preprint was posted, and presented TranscriptFormer as a step toward its goal of an AI-based "virtual cell" model.[3][5][15] A peer-reviewed version of the paper appeared in Science in 2026.[2] The family has three variants: TF-Metazoa (12 species), TF-Exemplar (human and four model organisms) and TF-Sapiens (human only).[1][6]
Background
Single-cell RNA sequencing measures gene expression in individual cells. Public data of this kind now covers hundreds of millions of cells across many species, and initiatives such as CZI's CZ CELLxGENE platform and the Human Cell Atlas provide harmonized access to it.[1] Several single-cell foundation models had been trained on these data before TranscriptFormer. The TranscriptFormer authors group them by training data: human-only models such as scMulan, scFoundation and scGPT; human and mouse models such as GeneCompass, Nicheformer and scPRINT; and the multispecies Universal Cell Embeddings (UCE).[1] In the authors' assessment, most of these earlier models are not generative, are not designed for zero-shot prediction, need fine-tuning for each task and are usually limited to human data, which restricts their use for comparing cells across species.[1] Comparing species with conventional methods is also hard because such analyses rely on orthologous genes, and distantly related species share few of them.[1]
TranscriptFormer addresses the species problem the same way UCE and scPRINT do: instead of giving each gene an arbitrary token, it represents each gene by an embedding of the protein it encodes, computed with the ESM-2 protein language model. Genes from different species with similar protein sequences therefore land near each other in a shared, species-agnostic space.[1]
Development and release
The paper's twelve authors were James D. Pearce, Sara E. Simmonds, Gita Mahmoudabadi, Lakshmi Krishnan, Giovanni Palla, Ana-Maria Istrate, Alexander Tarashansky, Benjamin Nelson, Omar Valenzuela, Donghui Li, Stephen R. Quake and Theofanis Karaletsos, with Simmonds, Mahmoudabadi and Krishnan listed as equal contributors.[1][6] The preprint credits Pearce, Quake and Karaletsos with the project's conceptualization.[1] The models were trained on CZI's computing cluster of 1,000 NVIDIA H100 GPUs.[1]
| Date | Event |
|---|---|
| 2025-04-29 | Preprint "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model" posted on bioRxiv[15] |
| 2025-04-30 | CZI announces the model and makes it available, with a tutorial, on its virtual cells platform and GitHub[3][4] |
| 2025-05-29 | CZI publishes an account of a fireside chat on the model at the STAT Breakthrough Summit West[5] |
| 2025-08 | CZI's rBio reasoning model uses TranscriptFormer outputs as a training signal[10] |
| 2025-10-11 | Revised preprint (version 2) posted on bioRxiv; Science records the manuscript as received on 2025-10-06[1][2] |
| 2025-10-28 | CZI and NVIDIA name TranscriptFormer among the models they plan to scale under an expanded partnership[9] |
| 2026-04-11 | Paper accepted by Science[2] |
| 2026-07-02 | Paper published in Science volume 393, issue 6806, as "TranscriptFormer: A generative cell atlas across 1.5 billion years of evolution"[2] |
In CZI's launch interview, Pearce said of the model, "What's unique is that it's the first generative model to translate those patterns of gene expression across species, too."[4] CZI's overview page called TranscriptFormer "the first generative, multi-species model for single-cell transcriptomics."[3] These are the developer's own characterizations.
Model variants
All three variants share the same core architecture and the same number of parameters in their transformer layers; they differ in training data and gene vocabulary, and their total trainable parameter counts differ because of vocabulary size and decoder head dimensionality.[1]
| Variant | Training cells | Species | Trainable parameters | Frozen parameters | Gene vocabulary |
|---|---|---|---|---|---|
| TF-Metazoa | 112 million | 12 (6 vertebrates, 4 invertebrates, 1 fungus, 1 protist) | 444 million | 633 million | 247,388 |
| TF-Exemplar | 110 million | 5 (human, mouse, zebrafish, fruit fly, C. elegans) | 542 million | 282 million | 110,290 |
| TF-Sapiens | 57 million | 1 (human) | 368 million | 61 million | 23,829 |
The frozen parameters are the pretrained protein embeddings of the vocabulary's genes.[1] The twelve TF-Metazoa species are human, mouse, rabbit, chicken, African clawed frog (Xenopus laevis), zebrafish, sea urchin (Lytechinus variegatus), C. elegans, the fruit fly, a freshwater sponge (Spongilla lacustris), baker's yeast (Saccharomyces cerevisiae) and the malaria parasite Plasmodium falciparum.[1][6] The authors trained the three variants to test how the breadth of species in pretraining affects performance.[1]
Architecture
Cells as sequences of genes and counts
TranscriptFormer treats a cell as a "bag of transcripts": an unordered multiset of observed mRNA molecules, which can be summarized as a set of expressed genes $g_j$, each with a positive count $c_j$.[1] The model turns this set into a "cell sentence" and generates it one gene at a time. At each step it first picks the next gene from a categorical distribution over the vocabulary, with probabilities $\pi_j$ that depend on the genes and counts already generated, and then draws that gene's count from a zero-truncated Poisson distribution whose rate $\lambda_j$ depends on the chosen gene and the preceding context.[1] Both steps are also conditioned on a token $a$ that identifies the sequencing assay. The probability of a whole cell with $M$ expressed genes therefore factorizes as
where the zero-truncated Poisson gives a count $c \geq 1$ the probability $\lambda^{c} e^{-\lambda} / \big(c!,(1 - e^{-\lambda})\big)$.[1] A zero-truncated distribution is used because genes with zero counts are left out of the cell sentence, so every gene that appears has at least one transcript.[1] Because a cell's genes have no natural order, the model uses no positional encodings, and genes were shuffled randomly during training.[1]
Network
The input for each gene is its ESM-2 protein embedding (the mean over isoforms when a gene encodes several proteins), projected to the model width by a small multilayer perceptron; an assay token is prepended.[1] The gene embeddings are initialized from the pretrained protein embeddings and held fixed during training.[1] The core network is a standard transformer encoder stack with pre-normalization, GELU activations and no bias terms, run under a causal mask so that each position sees only earlier genes; the assay token is never masked.[1]
The main architectural change is "expression-aware" self-attention: the gene counts enter every attention layer as a bias term on the attention matrix, so that genes with more transcripts have more influence, consistent with the bag-of-transcripts view. Soft capping of the attention logits is applied for numerical stability.[1] Two decoder heads sit on top of the stack. A gene head (a two-layer MLP with softmax) predicts the next gene, and a count head combines the final hidden state with the next input to predict a Poisson rate. The predicted counts are normalized so that they sum to the cell's observed total, and a learned per-gene bias is added.[1] The training objective is the sum of a cross-entropy loss on gene identity and a zero-truncated Poisson negative log-likelihood on counts.[1]
| Setting | Value |
|---|---|
| Transformer layers | 12 |
| Attention heads per layer | 16 |
| Model dimension | 2,048 |
| Maximum genes per cell | 2,047 (plus the assay token) |
| Dropout | 0.1 |
| Core transformer parameters | About 302 million |
| TF-Metazoa output embedding size | Reduced from 2,048 to 512 to handle its vocabulary of more than 247,000 genes |
The model produces two kinds of embeddings. A cell embedding is the average of the final-layer outputs over all gene positions, excluding the assay token. A contextual gene embedding (CGE) is the final-layer output at one gene's position, so the same gene receives a different representation in each cell depending on what else that cell expresses.[1][6]
Training
Most pretraining data came from CZ CELLxGENE through its Discover API: 57 million human cells from 644 datasets and 27 million mouse cells from 88 datasets, after spatial assays and duplicate cells were removed. Diseased cells were kept; the human and mouse data included about 17 million cells from 123 diseases.[1] Data for the other species came from 16 whole-organism, developmental and aging atlases, including Zebrahub and the Fly Cell Atlas. Two datasets, the sponge and Zebrahub data, were reprocessed from raw sequencing reads.[1] Cells with fewer than 200 expressed in-vocabulary genes were removed, and raw counts were clipped at 30.[1]
Training used PyTorch Lightning with distributed data parallelism, 16-bit mixed precision, and PyTorch's Flex Attention to build the transformer blocks.[1] The global batch held about 4 to 5 million tokens. Optimization used AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1) with a linear warm-up over the first 10% of steps and cosine decay of the learning rate from $1.5 \times 10^{-4}$ to $1.0 \times 10^{-5}$, and gradients were clipped to a norm of 1.0.[1] Each model saw about 3.5 trillion tokens, which for TF-Metazoa equals 15 epochs. Species were sampled with a skew parameter $\alpha = 0.6$ that increases how often low-resource species are drawn relative to human and mouse.[1]
Reported results
The figures below are from version 2 of the developers' preprint. The published Science abstract repeats the headline claims (state-of-the-art cell type classification, including for species more than 685 million years apart, and zero-shot disease state identification in human cells).[1][2] Most comparisons use frozen embeddings with a simple classifier on top: logistic regression with five-fold cross-validation for within-dataset tasks, and a k-nearest neighbors classifier (k = 10) for transferring labels from one species to another.[1] Scores are macro F1 scores unless noted.
| Task | Data | TranscriptFormer | Strongest baseline reported |
|---|---|---|---|
| Cell type classification, six species including four unseen | Cell atlases from human, zebrafish, mouse lemur, tropical clawed frog, sea lamprey and stony coral | TF-Metazoa 0.778 average; TF-Exemplar 0.770 | UCE 0.701 |
| Cross-species label transfer | Spermatogenesis data from nine vertebrates | TF-Exemplar 0.480 species average | UCE 0.377 |
| Chicken-to-mammal transfer | Same data (about 310 million years of divergence) | TF-Exemplar 0.448 | UCE 0.251 |
| Control vs. LPS-treated immune cells across species | Mouse, rat, rabbit and pig phagocytes | TF-Exemplar 0.925; TF-Metazoa 0.880 | UCE 0.740 |
| Human cell type classification | Held-out Tabula Sapiens 2.0 donors, 26 tissues | TF-Exemplar 0.910; TF-Metazoa 0.907; TF-Sapiens 0.906 | UCE 0.906 |
| SARS-CoV-2 infected vs. uninfected cells | Lung samples from four donors | TF-Sapiens 0.859 | UCE 0.805 |
| Drug-treated vs. control cells (ROC AUC) | 100,000-cell sample of Tahoe-100M, 95 compounds | TF-Sapiens 0.879 | UCE 0.779 |
For the out-of-distribution species test, both multispecies variants kept F1 above 0.65 on stony coral, which diverged from humans about 685 million years ago, while UCE scored 0.5 or lower on the most distant species.[1] The authors' own baseline, ESM2-CE, averages the same ESM-2 gene embeddings without any training on expression data; TranscriptFormer beat it on every species, which the authors take as evidence that the gains come from learning on transcriptomes rather than from protein sequence alone.[1] On human cell types, the multispecies variants scored slightly higher than the human-only TF-Sapiens, and the authors concluded that adding other species did not dilute human performance.[1] Other baselines in the human comparisons included scGPT, Geneformer, scVI and AIDO.Cell.[1]
Structure in the embeddings
The paper reports several kinds of structure that appear in TranscriptFormer's representations even though no cell type, developmental or phylogenetic labels were used in training:[1]
- Context in gene embeddings. CGEs from three Tabula Sapiens 2.0 tissues clustered mainly by cell type, even though a given gene always starts from the same ESM-2 input. In a variance partitioning analysis, cell type explained more than 95% of the variance in the first two principal components, with tissue and donor as smaller factors.[1]
- Developmental order. In human testis data, cell embeddings placed adjacent stages of spermatogenesis closest together, reproducing the known order of stages.[1]
- Phylogeny. Similarity between species' TF-Metazoa embeddings of testis cells fell with evolutionary distance (Spearman's $r = -0.705$), while the ESM2-CE baseline showed no significant correlation ($r = -0.059$).[1]
- Cell type correspondences across species. In a 1.6-million-cell developmental atlas of five animals, nearest-match searches recovered known homologies, such as rabbit migratory neural crest mapping to zebrafish neural crest derivatives. Between C. elegans and frog, muscle and neural cell types matched, and worm coelomocytes mapped to frog myeloid progenitors.[1]
- Across kingdoms. Yeast cells grown under ten conditions most often matched embryonic progenitor states in animals (9 of 10 top hits) rather than differentiated cell types.[1]
Prompting the model
Because TranscriptFormer is generative, it can be queried by conditioning it on chosen genes and reading off the predicted probabilities, which the authors describe as using the model as a "virtual instrument."[1] In one experiment they compared each gene's probability given a transcription factor with its probability given only the assay token, giving a pointwise mutual information score:
Statistically significant pairs from TF-Sapiens were then checked against the STRING v12.0 database of known gene interactions; for FOXM1, for example, 105 of 224 predicted partners were already annotated in STRING.[1] In a second experiment the authors conditioned the model on marker genes for each of 112 human cell types from Tabula Sapiens 2.0 and recovered a heatmap of cell-type-specific transcription factors whose overall pattern resembled the one measured in the Tabula Sapiens 2.0 study, including known associations such as IKZF1 and IKZF3 with T cells.[1]
Availability and licensing
The code is published on GitHub (czi-ai/transcriptformer) and as the transcriptformer package on PyPI under the MIT License.[6][7] A command-line tool downloads the weights of each variant from AWS S3, along with ESM-2 gene embeddings for 24 species, 12 of which are outside the training data (for example rat, pig, chimpanzee, platypus and coral); embeddings for other species can be generated with ESM-2.[6] The S3 bucket is listed in the AWS Registry of Open Data as part of Biohub's "Transcriptomic MIT Licensed data and models," under the MIT License.[8] The same tool runs inference to produce cell embeddings or contextual gene embeddings from AnnData files of raw counts; the developers recommend an A100 40 GB GPU, with smaller batches allowing 16 GB cards.[6] The preprint states that cell embeddings from TF-Sapiens and TF-Exemplar are available for download through the CZ CELLxGENE Census, and the models are also distributed on CZI's virtual cells platform.[1][3] The third-party Helical library includes TranscriptFormer in its model collection.[14]
Use in other work
CZI's own rBio reasoning model, released as a preprint in August 2025, uses TranscriptFormer as a "soft verifier": pointwise mutual information scores between gene pairs, retrieved from TranscriptFormer, serve as rewards during reinforcement learning, so the language model can be trained to answer questions about gene perturbations without new experimental data.[10] In October 2025 CZI and NVIDIA announced an expanded collaboration. The announcement named rBio, GREmLN and TranscriptFormer as virtual cell models built by CZI's teams and said the two organizations would work together to scale and accelerate virtual cell model development using NVIDIA technology, with models offered through CZI's virtual cells platform.[9]
Independent evaluations and criticism
Independent benchmarks published as preprints in 2026 have tested TranscriptFormer alongside other single-cell foundation models and generally found that simple methods match it on many tasks:
- Souza and Mehta (arXiv, February 2026). This study argued that simple pipelines built on careful normalization and linear methods reach state-of-the-art or near state-of-the-art results on standard benchmarks.[11] On Tabula Sapiens 2.0 their pipeline averaged a macro F1 of 0.887 against 0.911 for TF-Sapiens. On cross-species annotation of spermatogenesis data, their scTOP method averaged 0.679 against 0.515 for TF-Sapiens, and on the SARS-CoV-2 task it reached 0.850, which they described as competitive. Both approaches had the same overall rate of STRING support in the transcription factor experiment, but scTOP did better among its highest-ranked predictions.[11] The paper's first version was titled "Parameter-free representations outperform single-cell foundation models on downstream benchmarks."[11]
- VCBench (bioRxiv, June 2026). Weidener and colleagues compared Geneformer, scGPT, UCE, TranscriptFormer and Arc's State with pre-registered linear and nearest-neighbor baselines. The baselines matched or beat every foundation model on four of five scored dimensions, including cross-species transfer. TranscriptFormer was the only model to beat the strongest baseline on predicting protein levels from RNA (a 53% Pearson improvement, which the authors flagged with a possible data-contamination caveat). The authors also reported that the design choice behind that advantage caused a "spectral collapse" that hurt the model on ordering cells in time.[12]
- Gaballa, Ahmed and Abdelaal (bioRxiv, August 2026). A zero-shot comparison of scGPT, SCimilarity, UCE and TranscriptFormer found that baseline methods were best at cell type annotation on most datasets and that scVI integrated data best. For protein expression prediction, the foundation models generally beat the baseline, with TranscriptFormer reaching the highest correlation scores.[13]
The developers list their own limitations. Like other single-cell foundation models trained on cell atlases, TranscriptFormer does not handle batch effects in a unified way, and it is not specialized for zero-shot prediction of how cells respond to perturbations.[1][14] In the spermatogenesis analysis, for instance, human cells formed a separate cluster that the authors attributed to technical batch effects rather than biology.[1] Future work named in the preprint includes more species, better batch handling, additional data types beyond transcriptomics and more elaborate prompting.[1]
Continue reading
- State (Arc Institute virtual cell model)State is a machine learning "virtual cell" model developed by the Arc Institute, a nonprofit biomedical research organization, to predict how cells respond to perturbations such as drugs, cytokines, and…Mentioned 8 times above
- rBio (CZI)rBio (styled rbio1 in the accompanying preprint) is a biological reasoning large language model released by the Chan Zuckerberg Initiative (CZI) on August 21, 2025.Mentioned 3 times above
- Chan Zuckerberg InitiativeThe Chan Zuckerberg Initiative (CZI) is the philanthropic organization founded in December 2015 by Mark Zuckerberg, the founder and chief executive of Meta, and his wife Priscilla Chan, a pediatrician.Listed in See also
- BiohubBiohub is a nonprofit biomedical research organization that combines artificial intelligence, experimental biology, and scientific instrumentation to study cells and disease.Listed in See also
- Single-cell RNA sequencingSingle-cell RNA sequencing (scRNA-seq) is a family of laboratory methods that measure which genes are being transcribed in individual cells, by capturing and sequencing the messenger RNA of each cell…Listed in See also
- Virtual Biology InitiativeThe Virtual Biology Initiative is a five-year research program launched on April 29, 2026 by Biohub, the science organization of the Chan Zuckerberg Initiative (CZI), with a founding commitment of $500 million.Listed in See also
More from See also
More to read
Topics:AI ModelsAI for ScienceOpen Source AITransformer Models
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51 ^52Pearce, J. D., Simmonds, S. E., Mahmoudabadi, G., Krishnan, L., Palla, G., Istrate, A.-M., Tarashansky, A., Nelson, B., Valenzuela, O., Li, D., Quake, S. R., Karaletsos, T. "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model". bioRxiv (version 2), 2025. biorxiv.org/...2025.04.25.650731v2.full
- ^1 ^2 ^3 ^4 ^5 ^6Pearce, J. D., Simmonds, S. E., Mahmoudabadi, G., Krishnan, L., Palla, G., Istrate, A.-M., Tarashansky, A., Nelson, B., Valenzuela, O., Li, D., Quake, S. R., Karaletsos, T. "TranscriptFormer: A generative cell atlas across 1.5 billion years of evolution". Science 393 (6806), aec8514, July 2, 2026. doi.org/...science.aec8514
- ^1 ^2 ^3 ^4Biohub. "TranscriptFormer: A generative cross-species cell atlas across 1.5 billion years of evolution". biohub.org, April 30, 2025. biohub.org/...transcriptformer-model-overview
- ^1 ^2Biohub. "AI has entered the biology chat". biohub.org, April 30, 2025. biohub.org/...transcriptformer-ai-model-cell-biology
- ^1 ^2Chan Zuckerberg Initiative. "CZI @ STAT Breakthrough Summit: Virtual Cells Might Be Closer Than We Think, Thanks to AI". chanzuckerberg.com, May 29, 2025. chanzuckerberg.com/...t-west-2025-transcriptformer
- ^1 ^2 ^3 ^4 ^5 ^6 ^7Chan Zuckerberg Initiative. "TranscriptFormer" (README). GitHub, czi-ai/transcriptformer. github.com/...transcriptformer
- ^Chan Zuckerberg Initiative. "transcriptformer". Python Package Index (PyPI). pypi.org/...transcriptformer
- ^Biohub. "Transcriptomic MIT Licensed data and models". Registry of Open Data on AWS. registry.opendata.aws/biohub-transcriptomics-mit
- ^1 ^2Biohub. "CZI and NVIDIA accelerate virtual cell model development for scientific discovery". biohub.org, October 28, 2025. biohub.org/...nvidia-partnership-virtual-cell-model
- ^1 ^2Istrate, A.-M., Milletari, F., Castrotorres, F., Tomczak, J. M., Torkar, M., Li, D., Karaletsos, T. "rbio1 - training scientific reasoning LLMs with biological world models as soft verifiers". bioRxiv, 2025. biorxiv.org/...2025.08.18.670981v1.full
- ^1 ^2 ^3Souza, H., Mehta, P. "Simple models outperform single-cell foundation models on downstream benchmarks". arXiv:2602.16696, February 18, 2026 (revised November 6, 2026). arxiv.org/...2602.16696
- ^Weidener, L., Brkić, M., Jovanović, M., Ulgac, E., Meduri, A. "VCBench: A Multi-Dimensional Benchmark for Single-Cell Foundation Models". bioRxiv, June 23, 2026. doi.org/...2026.06.18.733146
- ^Gaballa, Y., Ahmed, S., Abdelaal, T. "Benchmarking single-cell foundation models in a zero-shot setting". bioRxiv, August 7, 2026. doi.org/...2026.08.03.739553
- ^1 ^2Helical. "TranscriptFormer" (model card). Helical documentation. helical.readthedocs.io/...transcriptformer
- ^1 ^2Pearce, J. D., et al. "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model". bioRxiv (version 1), April 29, 2025. biorxiv.org/...2025.04.25.650731v1
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
2 revisions · v2 · 3,385 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check before publication on November 18, 2026: 118 claims checked against 19 cited and independent sources; 0 material and 5 minor corrections applied before publication.
Cite this page: AI Wiki. "TranscriptFormer." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/transcriptformer

