Citation and evidence

TranscriptFormer

17 min full readUpdated 15 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

AI ModelsAI for ScienceOpen Source AITransformer Models

Cite this article

TranscriptFormer is a family of generative foundation models for single-cell transcriptomics developed by researchers at the Chan Zuckerberg Initiative (CZI). The models are autoregressive transformers that learn the joint distribution of which genes a cell expresses and how many transcripts of each are counted, and they were trained on up to 112 million cells from 12 species whose lineages span about 1.53 billion years of evolution.[1][2] CZI announced the models on April 30, 2025, a day after the bioRxiv preprint was posted, and presented TranscriptFormer as a step toward its goal of an AI-based "virtual cell" model.[3][5][15] A peer-reviewed version of the paper appeared in Science in 2026.[2] The family has three variants: TF-Metazoa (12 species), TF-Exemplar (human and four model organisms) and TF-Sapiens (human only).[1][6]

Background

Single-cell RNA sequencing measures gene expression in individual cells. Public data of this kind now covers hundreds of millions of cells across many species, and initiatives such as CZI's CZ CELLxGENE platform and the Human Cell Atlas provide harmonized access to it.[1] Several single-cell foundation models had been trained on these data before TranscriptFormer. The TranscriptFormer authors group them by training data: human-only models such as scMulan, scFoundation and scGPT; human and mouse models such as GeneCompass, Nicheformer and scPRINT; and the multispecies Universal Cell Embeddings (UCE).[1] In the authors' assessment, most of these earlier models are not generative, are not designed for zero-shot prediction, need fine-tuning for each task and are usually limited to human data, which restricts their use for comparing cells across species.[1] Comparing species with conventional methods is also hard because such analyses rely on orthologous genes, and distantly related species share few of them.[1]

TranscriptFormer addresses the species problem the same way UCE and scPRINT do: instead of giving each gene an arbitrary token, it represents each gene by an embedding of the protein it encodes, computed with the ESM-2 protein language model. Genes from different species with similar protein sequences therefore land near each other in a shared, species-agnostic space.[1]

"The TranscriptFormer Single-cell Model | Dr. James Pearce", video from BioTuring Team, published August 11, 2025. Watch on YouTube.

Development and release

The paper's twelve authors were James D. Pearce, Sara E. Simmonds, Gita Mahmoudabadi, Lakshmi Krishnan, Giovanni Palla, Ana-Maria Istrate, Alexander Tarashansky, Benjamin Nelson, Omar Valenzuela, Donghui Li, Stephen R. Quake and Theofanis Karaletsos, with Simmonds, Mahmoudabadi and Krishnan listed as equal contributors.[1][6] The preprint credits Pearce, Quake and Karaletsos with the project's conceptualization.[1] The models were trained on CZI's computing cluster of 1,000 NVIDIA H100 GPUs.[1]

DateEvent
2025-04-29Preprint "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model" posted on bioRxiv[15]
2025-04-30CZI announces the model and makes it available, with a tutorial, on its virtual cells platform and GitHub[3][4]
2025-05-29CZI publishes an account of a fireside chat on the model at the STAT Breakthrough Summit West[5]
2025-08CZI's rBio reasoning model uses TranscriptFormer outputs as a training signal[10]
2025-10-11Revised preprint (version 2) posted on bioRxiv; Science records the manuscript as received on 2025-10-06[1][2]
2025-10-28CZI and NVIDIA name TranscriptFormer among the models they plan to scale under an expanded partnership[9]
2026-04-11Paper accepted by Science[2]
2026-07-02Paper published in Science volume 393, issue 6806, as "TranscriptFormer: A generative cell atlas across 1.5 billion years of evolution"[2]

Expanded article table

In CZI's launch interview, Pearce said of the model, "What's unique is that it's the first generative model to translate those patterns of gene expression across species, too."[4] CZI's overview page called TranscriptFormer "the first generative, multi-species model for single-cell transcriptomics."[3] These are the developer's own characterizations.

Model variants

All three variants share the same core architecture and the same number of parameters in their transformer layers; they differ in training data and gene vocabulary, and their total trainable parameter counts differ because of vocabulary size and decoder head dimensionality.[1]

VariantTraining cellsSpeciesTrainable parametersFrozen parametersGene vocabulary
TF-Metazoa112 million12 (6 vertebrates, 4 invertebrates, 1 fungus, 1 protist)444 million633 million247,388
TF-Exemplar110 million5 (human, mouse, zebrafish, fruit fly, C. elegans)542 million282 million110,290
TF-Sapiens57 million1 (human)368 million61 million23,829

Expanded article table

The frozen parameters are the pretrained protein embeddings of the vocabulary's genes.[1] The twelve TF-Metazoa species are human, mouse, rabbit, chicken, African clawed frog (Xenopus laevis), zebrafish, sea urchin (Lytechinus variegatus), C. elegans, the fruit fly, a freshwater sponge (Spongilla lacustris), baker's yeast (Saccharomyces cerevisiae) and the malaria parasite Plasmodium falciparum.[1][6] The authors trained the three variants to test how the breadth of species in pretraining affects performance.[1]

Architecture

Cells as sequences of genes and counts

TranscriptFormer treats a cell as a "bag of transcripts": an unordered multiset of observed mRNA molecules, which can be summarized as a set of expressed genes $g_j$, each with a positive count $c_j$.[1] The model turns this set into a "cell sentence" and generates it one gene at a time. At each step it first picks the next gene from a categorical distribution over the vocabulary, with probabilities $\pi_j$ that depend on the genes and counts already generated, and then draws that gene's count from a zero-truncated Poisson distribution whose rate $\lambda_j$ depends on the chosen gene and the preceding context.[1] Both steps are also conditioned on a token $a$ that identifies the sequencing assay. The probability of a whole cell with $M$ expressed genes therefore factorizes as

p(g1:M,c1:M∣a)=∏j=1Mp(gj∣g<j,c<j,a)  p(cj∣gj,g<j,c<j,a)p(g_{1:M}, c_{1:M} \mid a) = \prod_{j=1}^{M} p(g_j \mid g_{<j}, c_{<j}, a)\; p(c_j \mid g_j, g_{<j}, c_{<j}, a)

where the zero-truncated Poisson gives a count $c \geq 1$ the probability $\lambda^{c} e^{-\lambda} / \big(c!,(1 - e^{-\lambda})\big)$.[1] A zero-truncated distribution is used because genes with zero counts are left out of the cell sentence, so every gene that appears has at least one transcript.[1] Because a cell's genes have no natural order, the model uses no positional encodings, and genes were shuffled randomly during training.[1]

Network

The input for each gene is its ESM-2 protein embedding (the mean over isoforms when a gene encodes several proteins), projected to the model width by a small multilayer perceptron; an assay token is prepended.[1] The gene embeddings are initialized from the pretrained protein embeddings and held fixed during training.[1] The core network is a standard transformer encoder stack with pre-normalization, GELU activations and no bias terms, run under a causal mask so that each position sees only earlier genes; the assay token is never masked.[1]

The main architectural change is "expression-aware" self-attention: the gene counts enter every attention layer as a bias term on the attention matrix, so that genes with more transcripts have more influence, consistent with the bag-of-transcripts view. Soft capping of the attention logits is applied for numerical stability.[1] Two decoder heads sit on top of the stack. A gene head (a two-layer MLP with softmax) predicts the next gene, and a count head combines the final hidden state with the next input to predict a Poisson rate. The predicted counts are normalized so that they sum to the cell's observed total, and a learned per-gene bias is added.[1] The training objective is the sum of a cross-entropy loss on gene identity and a zero-truncated Poisson negative log-likelihood on counts.[1]

SettingValue
Transformer layers12
Attention heads per layer16
Model dimension2,048
Maximum genes per cell2,047 (plus the assay token)
Dropout0.1
Core transformer parametersAbout 302 million
TF-Metazoa output embedding sizeReduced from 2,048 to 512 to handle its vocabulary of more than 247,000 genes

Expanded article table

The model produces two kinds of embeddings. A cell embedding is the average of the final-layer outputs over all gene positions, excluding the assay token. A contextual gene embedding (CGE) is the final-layer output at one gene's position, so the same gene receives a different representation in each cell depending on what else that cell expresses.[1][6]

Training

Most pretraining data came from CZ CELLxGENE through its Discover API: 57 million human cells from 644 datasets and 27 million mouse cells from 88 datasets, after spatial assays and duplicate cells were removed. Diseased cells were kept; the human and mouse data included about 17 million cells from 123 diseases.[1] Data for the other species came from 16 whole-organism, developmental and aging atlases, including Zebrahub and the Fly Cell Atlas. Two datasets, the sponge and Zebrahub data, were reprocessed from raw sequencing reads.[1] Cells with fewer than 200 expressed in-vocabulary genes were removed, and raw counts were clipped at 30.[1]

Training used PyTorch Lightning with distributed data parallelism, 16-bit mixed precision, and PyTorch's Flex Attention to build the transformer blocks.[1] The global batch held about 4 to 5 million tokens. Optimization used AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1) with a linear warm-up over the first 10% of steps and cosine decay of the learning rate from $1.5 \times 10^{-4}$ to $1.0 \times 10^{-5}$, and gradients were clipped to a norm of 1.0.[1] Each model saw about 3.5 trillion tokens, which for TF-Metazoa equals 15 epochs. Species were sampled with a skew parameter $\alpha = 0.6$ that increases how often low-resource species are drawn relative to human and mouse.[1]

Reported results

The figures below are from version 2 of the developers' preprint. The published Science abstract repeats the headline claims (state-of-the-art cell type classification, including for species more than 685 million years apart, and zero-shot disease state identification in human cells).[1][2] Most comparisons use frozen embeddings with a simple classifier on top: logistic regression with five-fold cross-validation for within-dataset tasks, and a k-nearest neighbors classifier (k = 10) for transferring labels from one species to another.[1] Scores are macro F1 scores unless noted.

TaskDataTranscriptFormerStrongest baseline reported
Cell type classification, six species including four unseenCell atlases from human, zebrafish, mouse lemur, tropical clawed frog, sea lamprey and stony coralTF-Metazoa 0.778 average; TF-Exemplar 0.770UCE 0.701
Cross-species label transferSpermatogenesis data from nine vertebratesTF-Exemplar 0.480 species averageUCE 0.377
Chicken-to-mammal transferSame data (about 310 million years of divergence)TF-Exemplar 0.448UCE 0.251
Control vs. LPS-treated immune cells across speciesMouse, rat, rabbit and pig phagocytesTF-Exemplar 0.925; TF-Metazoa 0.880UCE 0.740
Human cell type classificationHeld-out Tabula Sapiens 2.0 donors, 26 tissuesTF-Exemplar 0.910; TF-Metazoa 0.907; TF-Sapiens 0.906UCE 0.906
SARS-CoV-2 infected vs. uninfected cellsLung samples from four donorsTF-Sapiens 0.859UCE 0.805
Drug-treated vs. control cells (ROC AUC)100,000-cell sample of Tahoe-100M, 95 compoundsTF-Sapiens 0.879UCE 0.779

Expanded article table

For the out-of-distribution species test, both multispecies variants kept F1 above 0.65 on stony coral, which diverged from humans about 685 million years ago, while UCE scored 0.5 or lower on the most distant species.[1] The authors' own baseline, ESM2-CE, averages the same ESM-2 gene embeddings without any training on expression data; TranscriptFormer beat it on every species, which the authors take as evidence that the gains come from learning on transcriptomes rather than from protein sequence alone.[1] On human cell types, the multispecies variants scored slightly higher than the human-only TF-Sapiens, and the authors concluded that adding other species did not dilute human performance.[1] Other baselines in the human comparisons included scGPT, Geneformer, scVI and AIDO.Cell.[1]

Structure in the embeddings

The paper reports several kinds of structure that appear in TranscriptFormer's representations even though no cell type, developmental or phylogenetic labels were used in training:[1]

  • Context in gene embeddings. CGEs from three Tabula Sapiens 2.0 tissues clustered mainly by cell type, even though a given gene always starts from the same ESM-2 input. In a variance partitioning analysis, cell type explained more than 95% of the variance in the first two principal components, with tissue and donor as smaller factors.[1]
  • Developmental order. In human testis data, cell embeddings placed adjacent stages of spermatogenesis closest together, reproducing the known order of stages.[1]
  • Phylogeny. Similarity between species' TF-Metazoa embeddings of testis cells fell with evolutionary distance (Spearman's $r = -0.705$), while the ESM2-CE baseline showed no significant correlation ($r = -0.059$).[1]
  • Cell type correspondences across species. In a 1.6-million-cell developmental atlas of five animals, nearest-match searches recovered known homologies, such as rabbit migratory neural crest mapping to zebrafish neural crest derivatives. Between C. elegans and frog, muscle and neural cell types matched, and worm coelomocytes mapped to frog myeloid progenitors.[1]
  • Across kingdoms. Yeast cells grown under ten conditions most often matched embryonic progenitor states in animals (9 of 10 top hits) rather than differentiated cell types.[1]

Prompting the model

Because TranscriptFormer is generative, it can be queried by conditioning it on chosen genes and reading off the predicted probabilities, which the authors describe as using the model as a "virtual instrument."[1] In one experiment they compared each gene's probability given a transcription factor with its probability given only the assay token, giving a pointwise mutual information score:

PMI(gi,TFj)=log⁡P(gi∣TFj,a)−log⁡P(gi∣a)\mathrm{PMI}(g_i, \mathrm{TF}_j) = \log P(g_i \mid \mathrm{TF}_j, a) - \log P(g_i \mid a)

Statistically significant pairs from TF-Sapiens were then checked against the STRING v12.0 database of known gene interactions; for FOXM1, for example, 105 of 224 predicted partners were already annotated in STRING.[1] In a second experiment the authors conditioned the model on marker genes for each of 112 human cell types from Tabula Sapiens 2.0 and recovered a heatmap of cell-type-specific transcription factors whose overall pattern resembled the one measured in the Tabula Sapiens 2.0 study, including known associations such as IKZF1 and IKZF3 with T cells.[1]

Availability and licensing

The code is published on GitHub (czi-ai/transcriptformer) and as the transcriptformer package on PyPI under the MIT License.[6][7] A command-line tool downloads the weights of each variant from AWS S3, along with ESM-2 gene embeddings for 24 species, 12 of which are outside the training data (for example rat, pig, chimpanzee, platypus and coral); embeddings for other species can be generated with ESM-2.[6] The S3 bucket is listed in the AWS Registry of Open Data as part of Biohub's "Transcriptomic MIT Licensed data and models," under the MIT License.[8] The same tool runs inference to produce cell embeddings or contextual gene embeddings from AnnData files of raw counts; the developers recommend an A100 40 GB GPU, with smaller batches allowing 16 GB cards.[6] The preprint states that cell embeddings from TF-Sapiens and TF-Exemplar are available for download through the CZ CELLxGENE Census, and the models are also distributed on CZI's virtual cells platform.[1][3] The third-party Helical library includes TranscriptFormer in its model collection.[14]

Use in other work

CZI's own rBio reasoning model, released as a preprint in August 2025, uses TranscriptFormer as a "soft verifier": pointwise mutual information scores between gene pairs, retrieved from TranscriptFormer, serve as rewards during reinforcement learning, so the language model can be trained to answer questions about gene perturbations without new experimental data.[10] In October 2025 CZI and NVIDIA announced an expanded collaboration. The announcement named rBio, GREmLN and TranscriptFormer as virtual cell models built by CZI's teams and said the two organizations would work together to scale and accelerate virtual cell model development using NVIDIA technology, with models offered through CZI's virtual cells platform.[9]

Independent evaluations and criticism

Independent benchmarks published as preprints in 2026 have tested TranscriptFormer alongside other single-cell foundation models and generally found that simple methods match it on many tasks:

  • Souza and Mehta (arXiv, February 2026). This study argued that simple pipelines built on careful normalization and linear methods reach state-of-the-art or near state-of-the-art results on standard benchmarks.[11] On Tabula Sapiens 2.0 their pipeline averaged a macro F1 of 0.887 against 0.911 for TF-Sapiens. On cross-species annotation of spermatogenesis data, their scTOP method averaged 0.679 against 0.515 for TF-Sapiens, and on the SARS-CoV-2 task it reached 0.850, which they described as competitive. Both approaches had the same overall rate of STRING support in the transcription factor experiment, but scTOP did better among its highest-ranked predictions.[11] The paper's first version was titled "Parameter-free representations outperform single-cell foundation models on downstream benchmarks."[11]
  • VCBench (bioRxiv, June 2026). Weidener and colleagues compared Geneformer, scGPT, UCE, TranscriptFormer and Arc's State with pre-registered linear and nearest-neighbor baselines. The baselines matched or beat every foundation model on four of five scored dimensions, including cross-species transfer. TranscriptFormer was the only model to beat the strongest baseline on predicting protein levels from RNA (a 53% Pearson improvement, which the authors flagged with a possible data-contamination caveat). The authors also reported that the design choice behind that advantage caused a "spectral collapse" that hurt the model on ordering cells in time.[12]
  • Gaballa, Ahmed and Abdelaal (bioRxiv, August 2026). A zero-shot comparison of scGPT, SCimilarity, UCE and TranscriptFormer found that baseline methods were best at cell type annotation on most datasets and that scVI integrated data best. For protein expression prediction, the foundation models generally beat the baseline, with TranscriptFormer reaching the highest correlation scores.[13]

The developers list their own limitations. Like other single-cell foundation models trained on cell atlases, TranscriptFormer does not handle batch effects in a unified way, and it is not specialized for zero-shot prediction of how cells respond to perturbations.[1][14] In the spermatogenesis analysis, for instance, human cells formed a separate cluster that the authors attributed to technical batch effects rather than biology.[1] Future work named in the preprint includes more species, better batch handling, additional data types beyond transcriptomics and more elaborate prompting.[1]

Continue reading

More from See also

More to read

Topics:AI ModelsAI for ScienceOpen Source AITransformer Models

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51 ^52Pearce, J. D., Simmonds, S. E., Mahmoudabadi, G., Krishnan, L., Palla, G., Istrate, A.-M., Tarashansky, A., Nelson, B., Valenzuela, O., Li, D., Quake, S. R., Karaletsos, T. "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model". bioRxiv (version 2), 2025. biorxiv.org/...2025.04.25.650731v2.full
  2. ^1 ^2 ^3 ^4 ^5 ^6Pearce, J. D., Simmonds, S. E., Mahmoudabadi, G., Krishnan, L., Palla, G., Istrate, A.-M., Tarashansky, A., Nelson, B., Valenzuela, O., Li, D., Quake, S. R., Karaletsos, T. "TranscriptFormer: A generative cell atlas across 1.5 billion years of evolution". Science 393 (6806), aec8514, July 2, 2026. doi.org/...science.aec8514
  3. ^1 ^2 ^3 ^4Biohub. "TranscriptFormer: A generative cross-species cell atlas across 1.5 billion years of evolution". biohub.org, April 30, 2025. biohub.org/...transcriptformer-model-overview
  4. ^1 ^2Biohub. "AI has entered the biology chat". biohub.org, April 30, 2025. biohub.org/...transcriptformer-ai-model-cell-biology
  5. ^1 ^2Chan Zuckerberg Initiative. "CZI @ STAT Breakthrough Summit: Virtual Cells Might Be Closer Than We Think, Thanks to AI". chanzuckerberg.com, May 29, 2025. chanzuckerberg.com/...t-west-2025-transcriptformer
  6. ^1 ^2 ^3 ^4 ^5 ^6 ^7Chan Zuckerberg Initiative. "TranscriptFormer" (README). GitHub, czi-ai/transcriptformer. github.com/...transcriptformer
  7. ^Chan Zuckerberg Initiative. "transcriptformer". Python Package Index (PyPI). pypi.org/...transcriptformer
  8. ^Biohub. "Transcriptomic MIT Licensed data and models". Registry of Open Data on AWS. registry.opendata.aws/biohub-transcriptomics-mit
  9. ^1 ^2Biohub. "CZI and NVIDIA accelerate virtual cell model development for scientific discovery". biohub.org, October 28, 2025. biohub.org/...nvidia-partnership-virtual-cell-model
  10. ^1 ^2Istrate, A.-M., Milletari, F., Castrotorres, F., Tomczak, J. M., Torkar, M., Li, D., Karaletsos, T. "rbio1 - training scientific reasoning LLMs with biological world models as soft verifiers". bioRxiv, 2025. biorxiv.org/...2025.08.18.670981v1.full
  11. ^1 ^2 ^3Souza, H., Mehta, P. "Simple models outperform single-cell foundation models on downstream benchmarks". arXiv:2602.16696, February 18, 2026 (revised November 6, 2026). arxiv.org/...2602.16696
  12. ^Weidener, L., Brkić, M., Jovanović, M., Ulgac, E., Meduri, A. "VCBench: A Multi-Dimensional Benchmark for Single-Cell Foundation Models". bioRxiv, June 23, 2026. doi.org/...2026.06.18.733146
  13. ^Gaballa, Y., Ahmed, S., Abdelaal, T. "Benchmarking single-cell foundation models in a zero-shot setting". bioRxiv, August 7, 2026. doi.org/...2026.08.03.739553
  14. ^1 ^2Helical. "TranscriptFormer" (model card). Helical documentation. helical.readthedocs.io/...transcriptformer
  15. ^1 ^2Pearce, J. D., et al. "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model". bioRxiv (version 1), April 29, 2025. biorxiv.org/...2025.04.25.650731v1

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

2 revisions · v2 · 3,385 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 118 claims checked against 19 cited and independent sources; 0 material and 5 minor corrections applied before publication.

Cite this page: AI Wiki. "TranscriptFormer." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/transcriptformer

Edit article