# TranscriptFormer

> Source: https://aiwiki.ai/wiki/transcriptformer
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: AI Models, AI for Science, Open Source AI, Transformer Models
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "TranscriptFormer." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/transcriptformer
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**TranscriptFormer** is a family of generative [foundation models](https://aiwiki.ai/wiki/foundation_models) for single-cell transcriptomics developed by researchers at the [Chan Zuckerberg Initiative](https://aiwiki.ai/wiki/chan_zuckerberg_initiative) (CZI). The models are [autoregressive](https://aiwiki.ai/wiki/autoregressive_model) [transformers](https://aiwiki.ai/wiki/transformers) that learn the joint distribution of which genes a cell expresses and how many transcripts of each are counted, and they were trained on up to 112 million cells from 12 species whose lineages span about 1.53 billion years of evolution.[1][2] CZI announced the models on April 30, 2025, a day after the bioRxiv preprint was posted, and presented TranscriptFormer as a step toward its goal of an AI-based "virtual cell" model.[3][5][15] A peer-reviewed version of the paper appeared in *Science* in 2026.[2] The family has three variants: TF-Metazoa (12 species), TF-Exemplar (human and four model organisms) and TF-Sapiens (human only).[1][6]

## Background

[Single-cell RNA sequencing](https://aiwiki.ai/wiki/single_cell_rna_sequencing) measures gene expression in individual cells. Public data of this kind now covers hundreds of millions of cells across many species, and initiatives such as CZI's CZ CELLxGENE platform and the Human Cell Atlas provide harmonized access to it.[1] Several single-cell foundation models had been trained on these data before TranscriptFormer. The TranscriptFormer authors group them by training data: human-only models such as scMulan, scFoundation and scGPT; human and mouse models such as GeneCompass, Nicheformer and scPRINT; and the multispecies Universal Cell Embeddings (UCE).[1] In the authors' assessment, most of these earlier models are not generative, are not designed for zero-shot prediction, need fine-tuning for each task and are usually limited to human data, which restricts their use for comparing cells across species.[1] Comparing species with conventional methods is also hard because such analyses rely on orthologous genes, and distantly related species share few of them.[1]

TranscriptFormer addresses the species problem the same way UCE and scPRINT do: instead of giving each gene an arbitrary token, it represents each gene by an embedding of the protein it encodes, computed with the ESM-2 [protein language model](https://aiwiki.ai/wiki/protein_language_model). Genes from different species with similar protein sequences therefore land near each other in a shared, species-agnostic space.[1]

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/Z4x_FCmcDFc" title="The TranscriptFormer Single-cell Model | Dr. James Pearce"></iframe>
<figcaption>"The TranscriptFormer Single-cell Model | Dr. James Pearce", video from BioTuring Team, published August 11, 2025. <a href="https://www.youtube.com/watch?v=Z4x_FCmcDFc">Watch on YouTube</a>.</figcaption>
</figure>

## Development and release

The paper's twelve authors were James D. Pearce, Sara E. Simmonds, Gita Mahmoudabadi, Lakshmi Krishnan, Giovanni Palla, Ana-Maria Istrate, Alexander Tarashansky, Benjamin Nelson, Omar Valenzuela, Donghui Li, Stephen R. Quake and Theofanis Karaletsos, with Simmonds, Mahmoudabadi and Krishnan listed as equal contributors.[1][6] The preprint credits Pearce, Quake and Karaletsos with the project's conceptualization.[1] The models were trained on CZI's computing cluster of 1,000 [NVIDIA H100](https://aiwiki.ai/wiki/nvidia_h100) GPUs.[1]

| Date | Event |
|---|---|
| 2025-04-29 | Preprint "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model" posted on bioRxiv[15] |
| 2025-04-30 | CZI announces the model and makes it available, with a tutorial, on its virtual cells platform and GitHub[3][4] |
| 2025-05-29 | CZI publishes an account of a fireside chat on the model at the STAT Breakthrough Summit West[5] |
| 2025-08 | CZI's rBio reasoning model uses TranscriptFormer outputs as a training signal[10] |
| 2025-10-11 | Revised preprint (version 2) posted on bioRxiv; *Science* records the manuscript as received on 2025-10-06[1][2] |
| 2025-10-28 | CZI and [NVIDIA](https://aiwiki.ai/wiki/nvidia) name TranscriptFormer among the models they plan to scale under an expanded partnership[9] |
| 2026-04-11 | Paper accepted by *Science*[2] |
| 2026-07-02 | Paper published in *Science* volume 393, issue 6806, as "TranscriptFormer: A generative cell atlas across 1.5 billion years of evolution"[2] |

In CZI's launch interview, Pearce said of the model, "What's unique is that it's the first generative model to translate those patterns of gene expression across species, too."[4] CZI's overview page called TranscriptFormer "the first generative, multi-species model for single-cell transcriptomics."[3] These are the developer's own characterizations.

## Model variants

All three variants share the same core architecture and the same number of parameters in their transformer layers; they differ in training data and gene vocabulary, and their total trainable parameter counts differ because of vocabulary size and decoder head dimensionality.[1]

| Variant | Training cells | Species | Trainable parameters | Frozen parameters | Gene vocabulary |
|---|---|---|---|---|---|
| TF-Metazoa | 112 million | 12 (6 vertebrates, 4 invertebrates, 1 fungus, 1 protist) | 444 million | 633 million | 247,388 |
| TF-Exemplar | 110 million | 5 (human, mouse, zebrafish, fruit fly, *C. elegans*) | 542 million | 282 million | 110,290 |
| TF-Sapiens | 57 million | 1 (human) | 368 million | 61 million | 23,829 |

The frozen parameters are the pretrained protein embeddings of the vocabulary's genes.[1] The twelve TF-Metazoa species are human, mouse, rabbit, chicken, African clawed frog (*Xenopus laevis*), zebrafish, sea urchin (*Lytechinus variegatus*), *C. elegans*, the fruit fly, a freshwater sponge (*Spongilla lacustris*), baker's yeast (*Saccharomyces cerevisiae*) and the malaria parasite *Plasmodium falciparum*.[1][6] The authors trained the three variants to test how the breadth of species in pretraining affects performance.[1]

## Architecture

### Cells as sequences of genes and counts

TranscriptFormer treats a cell as a "bag of transcripts": an unordered multiset of observed mRNA molecules, which can be summarized as a set of expressed genes $g_j$, each with a positive count $c_j$.[1] The model turns this set into a "cell sentence" and generates it one gene at a time. At each step it first picks the next gene from a categorical distribution over the vocabulary, with probabilities $\pi_j$ that depend on the genes and counts already generated, and then draws that gene's count from a zero-truncated Poisson distribution whose rate $\lambda_j$ depends on the chosen gene and the preceding context.[1] Both steps are also conditioned on a token $a$ that identifies the sequencing assay. The probability of a whole cell with $M$ expressed genes therefore factorizes as

$$p(g_{1:M}, c_{1:M} \mid a) = \prod_{j=1}^{M} p(g_j \mid g_{<j}, c_{<j}, a)\; p(c_j \mid g_j, g_{<j}, c_{<j}, a)$$

where the zero-truncated Poisson gives a count $c \geq 1$ the probability $\lambda^{c} e^{-\lambda} / \big(c!\,(1 - e^{-\lambda})\big)$.[1] A zero-truncated distribution is used because genes with zero counts are left out of the cell sentence, so every gene that appears has at least one transcript.[1] Because a cell's genes have no natural order, the model uses no positional encodings, and genes were shuffled randomly during training.[1]

### Network

The input for each gene is its ESM-2 protein embedding (the mean over isoforms when a gene encodes several proteins), projected to the model width by a small multilayer perceptron; an assay token is prepended.[1] The gene embeddings are initialized from the pretrained protein embeddings and held fixed during training.[1] The core network is a standard transformer encoder stack with pre-normalization, GELU activations and no bias terms, run under a causal mask so that each position sees only earlier genes; the assay token is never masked.[1]

The main architectural change is "expression-aware" [self-attention](https://aiwiki.ai/wiki/self_attention): the gene counts enter every attention layer as a bias term on the attention matrix, so that genes with more transcripts have more influence, consistent with the bag-of-transcripts view. Soft capping of the attention logits is applied for numerical stability.[1] Two decoder heads sit on top of the stack. A gene head (a two-layer MLP with softmax) predicts the next gene, and a count head combines the final hidden state with the next input to predict a Poisson rate. The predicted counts are normalized so that they sum to the cell's observed total, and a learned per-gene bias is added.[1] The training objective is the sum of a [cross-entropy](https://aiwiki.ai/wiki/cross-entropy) loss on gene identity and a zero-truncated Poisson negative log-likelihood on counts.[1]

| Setting | Value |
|---|---|
| Transformer layers | 12 |
| Attention heads per layer | 16 |
| Model dimension | 2,048 |
| Maximum genes per cell | 2,047 (plus the assay token) |
| Dropout | 0.1 |
| Core transformer parameters | About 302 million |
| TF-Metazoa output embedding size | Reduced from 2,048 to 512 to handle its vocabulary of more than 247,000 genes |

The model produces two kinds of [embeddings](https://aiwiki.ai/wiki/embeddings). A cell embedding is the average of the final-layer outputs over all gene positions, excluding the assay token. A contextual gene embedding (CGE) is the final-layer output at one gene's position, so the same gene receives a different representation in each cell depending on what else that cell expresses.[1][6]

## Training

Most pretraining data came from CZ CELLxGENE through its Discover API: 57 million human cells from 644 datasets and 27 million mouse cells from 88 datasets, after spatial assays and duplicate cells were removed. Diseased cells were kept; the human and mouse data included about 17 million cells from 123 diseases.[1] Data for the other species came from 16 whole-organism, developmental and aging atlases, including Zebrahub and the Fly Cell Atlas. Two datasets, the sponge and Zebrahub data, were reprocessed from raw sequencing reads.[1] Cells with fewer than 200 expressed in-vocabulary genes were removed, and raw counts were clipped at 30.[1]

Training used [PyTorch Lightning](https://aiwiki.ai/wiki/pytorch_lightning) with distributed data parallelism, 16-bit mixed precision, and PyTorch's Flex Attention to build the transformer blocks.[1] The global batch held about 4 to 5 million tokens. Optimization used [AdamW](https://aiwiki.ai/wiki/adamw) ($\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1) with a linear warm-up over the first 10% of steps and cosine decay of the learning rate from $1.5 \times 10^{-4}$ to $1.0 \times 10^{-5}$, and gradients were [clipped](https://aiwiki.ai/wiki/gradient_clipping) to a norm of 1.0.[1] Each model saw about 3.5 trillion tokens, which for TF-Metazoa equals 15 epochs. Species were sampled with a skew parameter $\alpha = 0.6$ that increases how often low-resource species are drawn relative to human and mouse.[1]

## Reported results

The figures below are from version 2 of the developers' preprint. The published *Science* abstract repeats the headline claims (state-of-the-art cell type classification, including for species more than 685 million years apart, and zero-shot disease state identification in human cells).[1][2] Most comparisons use frozen embeddings with a simple classifier on top: [logistic regression](https://aiwiki.ai/wiki/logistic_regression) with five-fold cross-validation for within-dataset tasks, and a [k-nearest neighbors](https://aiwiki.ai/wiki/k_nearest_neighbors) classifier (k = 10) for transferring labels from one species to another.[1] Scores are macro [F1 scores](https://aiwiki.ai/wiki/f1_score) unless noted.

| Task | Data | TranscriptFormer | Strongest baseline reported |
|---|---|---|---|
| Cell type classification, six species including four unseen | Cell atlases from human, zebrafish, mouse lemur, tropical clawed frog, sea lamprey and stony coral | TF-Metazoa 0.778 average; TF-Exemplar 0.770 | UCE 0.701 |
| Cross-species label transfer | Spermatogenesis data from nine vertebrates | TF-Exemplar 0.480 species average | UCE 0.377 |
| Chicken-to-mammal transfer | Same data (about 310 million years of divergence) | TF-Exemplar 0.448 | UCE 0.251 |
| Control vs. LPS-treated immune cells across species | Mouse, rat, rabbit and pig phagocytes | TF-Exemplar 0.925; TF-Metazoa 0.880 | UCE 0.740 |
| Human cell type classification | Held-out Tabula Sapiens 2.0 donors, 26 tissues | TF-Exemplar 0.910; TF-Metazoa 0.907; TF-Sapiens 0.906 | UCE 0.906 |
| SARS-CoV-2 infected vs. uninfected cells | Lung samples from four donors | TF-Sapiens 0.859 | UCE 0.805 |
| Drug-treated vs. control cells (ROC AUC) | 100,000-cell sample of Tahoe-100M, 95 compounds | TF-Sapiens 0.879 | UCE 0.779 |

For the out-of-distribution species test, both multispecies variants kept F1 above 0.65 on stony coral, which diverged from humans about 685 million years ago, while UCE scored 0.5 or lower on the most distant species.[1] The authors' own baseline, ESM2-CE, averages the same ESM-2 gene embeddings without any training on expression data; TranscriptFormer beat it on every species, which the authors take as evidence that the gains come from learning on transcriptomes rather than from protein sequence alone.[1] On human cell types, the multispecies variants scored slightly higher than the human-only TF-Sapiens, and the authors concluded that adding other species did not dilute human performance.[1] Other baselines in the human comparisons included scGPT, Geneformer, scVI and AIDO.Cell.[1]

### Structure in the embeddings

The paper reports several kinds of structure that appear in TranscriptFormer's representations even though no cell type, developmental or phylogenetic labels were used in training:[1]

- **Context in gene embeddings.** CGEs from three Tabula Sapiens 2.0 tissues clustered mainly by cell type, even though a given gene always starts from the same ESM-2 input. In a variance partitioning analysis, cell type explained more than 95% of the variance in the first two principal components, with tissue and donor as smaller factors.[1]
- **Developmental order.** In human testis data, cell embeddings placed adjacent stages of spermatogenesis closest together, reproducing the known order of stages.[1]
- **Phylogeny.** Similarity between species' TF-Metazoa embeddings of testis cells fell with evolutionary distance (Spearman's $r = -0.705$), while the ESM2-CE baseline showed no significant correlation ($r = -0.059$).[1]
- **Cell type correspondences across species.** In a 1.6-million-cell developmental atlas of five animals, nearest-match searches recovered known homologies, such as rabbit migratory neural crest mapping to zebrafish neural crest derivatives. Between *C. elegans* and frog, muscle and neural cell types matched, and worm coelomocytes mapped to frog myeloid progenitors.[1]
- **Across kingdoms.** Yeast cells grown under ten conditions most often matched embryonic progenitor states in animals (9 of 10 top hits) rather than differentiated cell types.[1]

### Prompting the model

Because TranscriptFormer is generative, it can be queried by conditioning it on chosen genes and reading off the predicted probabilities, which the authors describe as using the model as a "virtual instrument."[1] In one experiment they compared each gene's probability given a transcription factor with its probability given only the assay token, giving a pointwise mutual information score:

$$\mathrm{PMI}(g_i, \mathrm{TF}_j) = \log P(g_i \mid \mathrm{TF}_j, a) - \log P(g_i \mid a)$$

Statistically significant pairs from TF-Sapiens were then checked against the STRING v12.0 database of known gene interactions; for FOXM1, for example, 105 of 224 predicted partners were already annotated in STRING.[1] In a second experiment the authors conditioned the model on marker genes for each of 112 human cell types from Tabula Sapiens 2.0 and recovered a heatmap of cell-type-specific transcription factors whose overall pattern resembled the one measured in the Tabula Sapiens 2.0 study, including known associations such as IKZF1 and IKZF3 with T cells.[1]

## Availability and licensing

The code is published on GitHub (czi-ai/transcriptformer) and as the `transcriptformer` package on PyPI under the [MIT License](https://aiwiki.ai/wiki/mit_license).[6][7] A command-line tool downloads the weights of each variant from AWS S3, along with ESM-2 gene embeddings for 24 species, 12 of which are outside the training data (for example rat, pig, chimpanzee, platypus and coral); embeddings for other species can be generated with ESM-2.[6] The S3 bucket is listed in the AWS Registry of Open Data as part of Biohub's "Transcriptomic MIT Licensed data and models," under the MIT License.[8] The same tool runs inference to produce cell embeddings or contextual gene embeddings from AnnData files of raw counts; the developers recommend an A100 40 GB GPU, with smaller batches allowing 16 GB cards.[6] The preprint states that cell embeddings from TF-Sapiens and TF-Exemplar are available for download through the CZ CELLxGENE Census, and the models are also distributed on CZI's virtual cells platform.[1][3] The third-party Helical library includes TranscriptFormer in its model collection.[14]

## Use in other work

CZI's own [rBio](https://aiwiki.ai/wiki/czi_rbio) reasoning model, released as a preprint in August 2025, uses TranscriptFormer as a "soft verifier": pointwise mutual information scores between gene pairs, retrieved from TranscriptFormer, serve as rewards during [reinforcement learning](https://aiwiki.ai/wiki/reinforcement_learning), so the language model can be trained to answer questions about gene perturbations without new experimental data.[10] In October 2025 CZI and NVIDIA announced an expanded collaboration. The announcement named rBio, GREmLN and TranscriptFormer as virtual cell models built by CZI's teams and said the two organizations would work together to scale and accelerate virtual cell model development using NVIDIA technology, with models offered through CZI's virtual cells platform.[9]

## Independent evaluations and criticism

Independent benchmarks published as preprints in 2026 have tested TranscriptFormer alongside other single-cell foundation models and generally found that simple methods match it on many tasks:

- **Souza and Mehta (arXiv, February 2026).** This study argued that simple pipelines built on careful normalization and linear methods reach state-of-the-art or near state-of-the-art results on standard benchmarks.[11] On Tabula Sapiens 2.0 their pipeline averaged a macro F1 of 0.887 against 0.911 for TF-Sapiens. On cross-species annotation of spermatogenesis data, their scTOP method averaged 0.679 against 0.515 for TF-Sapiens, and on the SARS-CoV-2 task it reached 0.850, which they described as competitive. Both approaches had the same overall rate of STRING support in the transcription factor experiment, but scTOP did better among its highest-ranked predictions.[11] The paper's first version was titled "Parameter-free representations outperform single-cell foundation models on downstream benchmarks."[11]
- **VCBench (bioRxiv, June 2026).** Weidener and colleagues compared Geneformer, scGPT, UCE, TranscriptFormer and Arc's [State](https://aiwiki.ai/wiki/arc_state_model) with pre-registered linear and nearest-neighbor baselines. The baselines matched or beat every foundation model on four of five scored dimensions, including cross-species transfer. TranscriptFormer was the only model to beat the strongest baseline on predicting protein levels from RNA (a 53% Pearson improvement, which the authors flagged with a possible data-contamination caveat). The authors also reported that the design choice behind that advantage caused a "spectral collapse" that hurt the model on ordering cells in time.[12]
- **Gaballa, Ahmed and Abdelaal (bioRxiv, August 2026).** A zero-shot comparison of scGPT, SCimilarity, UCE and TranscriptFormer found that baseline methods were best at cell type annotation on most datasets and that scVI integrated data best. For protein expression prediction, the foundation models generally beat the baseline, with TranscriptFormer reaching the highest correlation scores.[13]

The developers list their own limitations. Like other single-cell foundation models trained on cell atlases, TranscriptFormer does not handle batch effects in a unified way, and it is not specialized for zero-shot prediction of how cells respond to perturbations.[1][14] In the spermatogenesis analysis, for instance, human cells formed a separate cluster that the authors attributed to technical batch effects rather than biology.[1] Future work named in the preprint includes more species, better batch handling, additional data types beyond transcriptomics and more elaborate prompting.[1]

## See also

- [Chan Zuckerberg Initiative](https://aiwiki.ai/wiki/chan_zuckerberg_initiative)
- [Biohub](https://aiwiki.ai/wiki/biohub)
- [rBio (CZI)](https://aiwiki.ai/wiki/czi_rbio)
- [State (Arc Institute virtual cell model)](https://aiwiki.ai/wiki/arc_state_model)
- [Virtual Biology Initiative](https://aiwiki.ai/wiki/virtual_biology_initiative)
- [Single-cell RNA sequencing](https://aiwiki.ai/wiki/single_cell_rna_sequencing)
- [AI for science](https://aiwiki.ai/wiki/ai_for_science)

## References

1. Pearce, J. D., Simmonds, S. E., Mahmoudabadi, G., Krishnan, L., Palla, G., Istrate, A.-M., Tarashansky, A., Nelson, B., Valenzuela, O., Li, D., Quake, S. R., Karaletsos, T. "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model". bioRxiv (version 2), 2025. https://www.biorxiv.org/content/10.1101/2025.04.25.650731v2.full
2. Pearce, J. D., Simmonds, S. E., Mahmoudabadi, G., Krishnan, L., Palla, G., Istrate, A.-M., Tarashansky, A., Nelson, B., Valenzuela, O., Li, D., Quake, S. R., Karaletsos, T. "TranscriptFormer: A generative cell atlas across 1.5 billion years of evolution". Science 393 (6806), aec8514, July 2, 2026. https://doi.org/10.1126/science.aec8514
3. Biohub. "TranscriptFormer: A generative cross-species cell atlas across 1.5 billion years of evolution". biohub.org, April 30, 2025. https://biohub.org/blog/transcriptformer-model-overview/
4. Biohub. "AI has entered the biology chat". biohub.org, April 30, 2025. https://biohub.org/blog/transcriptformer-ai-model-cell-biology/
5. Chan Zuckerberg Initiative. "CZI @ STAT Breakthrough Summit: Virtual Cells Might Be Closer Than We Think, Thanks to AI". chanzuckerberg.com, May 29, 2025. https://chanzuckerberg.com/newsroom/stat-west-2025-transcriptformer/
6. Chan Zuckerberg Initiative. "TranscriptFormer" (README). GitHub, czi-ai/transcriptformer. https://github.com/czi-ai/transcriptformer
7. Chan Zuckerberg Initiative. "transcriptformer". Python Package Index (PyPI). https://pypi.org/project/transcriptformer/
8. Biohub. "Transcriptomic MIT Licensed data and models". Registry of Open Data on AWS. https://registry.opendata.aws/biohub-transcriptomics-mit/
9. Biohub. "CZI and NVIDIA accelerate virtual cell model development for scientific discovery". biohub.org, October 28, 2025. https://biohub.org/news/nvidia-partnership-virtual-cell-model/
10. Istrate, A.-M., Milletari, F., Castrotorres, F., Tomczak, J. M., Torkar, M., Li, D., Karaletsos, T. "rbio1 - training scientific reasoning LLMs with biological world models as soft verifiers". bioRxiv, 2025. https://www.biorxiv.org/content/10.1101/2025.08.18.670981v1.full
11. Souza, H., Mehta, P. "Simple models outperform single-cell foundation models on downstream benchmarks". arXiv:2602.16696, February 18, 2026 (revised November 6, 2026). https://arxiv.org/abs/2602.16696
12. Weidener, L., Brkić, M., Jovanović, M., Ulgac, E., Meduri, A. "VCBench: A Multi-Dimensional Benchmark for Single-Cell Foundation Models". bioRxiv, June 23, 2026. https://doi.org/10.64898/2026.06.18.733146
13. Gaballa, Y., Ahmed, S., Abdelaal, T. "Benchmarking single-cell foundation models in a zero-shot setting". bioRxiv, August 7, 2026. https://doi.org/10.64898/2026.08.03.739553
14. Helical. "TranscriptFormer" (model card). Helical documentation. https://helical.readthedocs.io/en/latest/model_cards/transcriptformer/
15. Pearce, J. D., et al. "A Cross-Species Generative Cell Atlas Across 1.5 Billion Years of Evolution: The TranscriptFormer Single-cell Model". bioRxiv (version 1), April 29, 2025. https://www.biorxiv.org/content/10.1101/2025.04.25.650731v1

