# PaLM

> Source: https://aiwiki.ai/wiki/palm
> Updated: 2026-07-29
> Fact-checked: 2026-07-29
> Categories: Google DeepMind, Large Language Models, Natural Language Processing, Transformer Models
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "PaLM." aiwiki.ai, 29 Jul 2026. https://aiwiki.ai/wiki/palm
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**PaLM** (Pathways Language Model) is a family of dense, decoder-only [large language models](https://aiwiki.ai/wiki/large_language_model) developed by [Google Research](https://aiwiki.ai/wiki/google_research). Google announced the original family on April 4, 2022. Its largest member, commonly called PaLM 540B, had 540.35 billion parameters and was trained on 780 billion tokens using the [Pathways](https://aiwiki.ai/wiki/pathways) system and 6,144 TPU v4 chips. The family also included 8.63-billion and 62.50-billion parameter models.[1][2]

PaLM was a research model rather than a publicly released set of weights. It was used to study language-model scaling, few-shot evaluation, [chain-of-thought](https://aiwiki.ai/wiki/chain_of_thought) prompting, multilingual performance, code generation, memorization, bias, and toxicity. Google later adapted or extended it in Flan-PaLM, [Minerva](https://aiwiki.ai/wiki/minerva), [PaLM-E](https://aiwiki.ai/wiki/palm-e_an_embodied_multimodal_language_model), and [Med-PaLM](https://aiwiki.ai/wiki/med_palm). [PaLM 2](https://aiwiki.ai/wiki/palm_2), announced in May 2023, was a separate successor family with a different training design.[1][5][6][7][8][9]

The name "PaLM" has also been used in product documentation for Google's former PaLM API. That service history should not be confused with the research model's release: Google first offered the PaLM API in private preview in March 2023, later served PaLM 2 models through it, and decommissioned the Google AI PaLM API on August 15, 2024. Google's deprecation notice explicitly excluded Vertex AI from that August 2024 notice.[11][12]

## PaLM 540B at a glance

| Attribute | Documented value |
|---|---|
| Developer | [Google Research](https://aiwiki.ai/wiki/google_research) |
| Announced | April 4, 2022 |
| Architecture | Dense, decoder-only autoregressive [Transformer](https://aiwiki.ai/wiki/transformers) |
| Model sizes | 8.63B, 62.50B, and 540.35B parameters |
| Context length | 2,048 tokens |
| Training data | 780B tokens, processed for one epoch |
| Tokenizer | Lossless [SentencePiece](https://aiwiki.ai/wiki/sentencepiece), 256,000-token vocabulary |
| Largest training job | 6,144 TPU v4 chips across two pods |
| Training software | Pathways, JAX, and T5X |
| Public weights | Not released |
| Successor | [PaLM 2](https://aiwiki.ai/wiki/palm_2) |

## Development and publication

PaLM was built as a large-scale demonstration of Pathways, Google's distributed machine-learning system. Pathways represents a computation as a dataflow graph and schedules work across accelerator "islands" connected through a data-center network. The systems paper reported experiments across as many as 2,048 TPU chips and described mechanisms for gang scheduling, sharded computations, and asynchronous dispatch. PaLM extended the practical demonstration to a language-model training job that used two TPU v4 pods.[1][3]

Google introduced PaLM in a research blog post on April 4, 2022, and the authors released the associated preprint the following day. A revised article, *PaLM: Scaling Language Modeling with Pathways*, appeared in the *Journal of Machine Learning Research* in 2023. The paper lists more than 60 authors and reports results for all three model sizes.[1][2]

The project studied dense scaling. Every parameter in a PaLM model participates in processing each token, unlike a sparsely activated mixture-of-experts model. The 540B model was therefore not evidence that dense models were always preferable; it was a study of what a large dense model, a fixed data mixture, and the Pathways infrastructure could achieve under the authors' evaluation setup.[1]

## Architecture

PaLM uses an autoregressive, decoder-only Transformer. It predicts the next token from the preceding tokens in a sequence. The three configurations share the same 2,048-token [context window](https://aiwiki.ai/wiki/context_window), 256,000-token vocabulary, 256-dimensional attention heads, and feed-forward width equal to four times the model width.[1]

| Configuration | Parameters | Layers | Model width | Attention heads | Feed-forward width |
|---|---:|---:|---:|---:|---:|
| PaLM 8B | 8.63B | 32 | 4,096 | 16 | 16,384 |
| PaLM 62B | 62.50B | 64 | 8,192 | 32 | 32,768 |
| PaLM 540B | 540.35B | 118 | 18,432 | 48 | 73,728 |

The architecture combined several design choices that had previously been studied separately:

- **SwiGLU feed-forward layers.** PaLM uses [SwiGLU](https://aiwiki.ai/wiki/swiglu) activations instead of a ReLU or GELU feed-forward layer. The paper's ablations found improved likelihood at matched compute.[1]
- **Parallel attention and feed-forward computation.** Each block computes its attention and feed-forward transformations in parallel from the same normalized input, then adds both results to the residual stream. Google reported about a 15 percent training-speed improvement at scale. Its 8B ablation showed a small quality cost, while its 62B ablation did not show a statistically significant difference. The authors extrapolated from those experiments when choosing the design for 540B, so the paper did not directly establish quality neutrality at 540B.[1]
- **Multi-query attention.** PaLM uses [multi-query attention](https://aiwiki.ai/wiki/mqa), in which attention heads have separate query projections but share key and value projections. This reduces the memory required for key-value caches during autoregressive decoding.[1]
- **Rotary position embeddings.** The model uses [rotary position embeddings](https://aiwiki.ai/wiki/rotary_position_embedding) rather than learned absolute position embeddings.[1]
- **Shared embeddings and omitted biases.** The input and output embedding matrices are shared. PaLM removes biases from dense layers and layer normalization, and pretraining uses no dropout.[1]

The tokenizer is a lossless SentencePiece model. It preserves whitespace, can represent out-of-vocabulary Unicode characters as bytes, and splits numbers into individual digits. These choices supported multilingual text and source code, but they did not make performance uniform across languages or programming languages.[1]

## Training data and optimization

The paper describes a 780-billion-token corpus intended for one epoch with identical shuffling across configurations. It also notes that, because of a checkpoint-selection oversight, most reported results use checkpoints at 780 billion tokens for PaLM 8B and 540B but 795 billion for PaLM 62B.[1] The mixture was assembled from multilingual conversations and web pages, English-language books and news, multilingual Wikipedia, and open-source code.[1]

| Source category | Share of tokens |
|---|---:|
| Multilingual social-media conversations | 50% |
| Multilingual filtered web pages | 27% |
| English books | 13% |
| Source code from GitHub | 5% |
| Multilingual Wikipedia | 4% |
| English news | 1% |

The paper's appendix reports material from 124 languages, with English accounting for 77.984 percent of the corpus. The code subset covered 24 programming languages and about 196 GB before tokenization. Google excluded repositories under copyleft licenses and applied repository-level deduplication. The broader text mixture also underwent quality filtering and deduplication, although these procedures cannot guarantee removal of every duplicate, personal datum, copyrighted work, stereotype, or harmful passage.[1]

PaLM used [Adafactor](https://aiwiki.ai/wiki/adafactor) without its factored second-moment approximation. Training used momentum of 0.9, a learning rate of 0.01 for the first 10,000 steps followed by inverse-square-root decay, and staged increases in batch size. The 540B training job used JAX and T5X on 6,144 TPU v4 chips divided between two 3,072-chip pods. The authors used data parallelism between the pods and combined model and data parallelism within each pod; they did not use pipeline parallelism for the PaLM run.[1]

Google reported 46.2 percent model FLOPs utilization and 57.8 percent hardware FLOPs utilization for PaLM 540B. At a batch size of 2,048 sequences, the reported throughput was 238,300 tokens per second. These figures describe the authors' hardware, software, model configuration, and utilization definitions; they are not general performance figures for every PaLM deployment.[1]

The paper also records about 20 large loss spikes during 540B training. The team restored a checkpoint roughly 100 steps before a spike and skipped 200 to 500 data batches before resuming. The authors said they did not identify a principled method that prevented the spikes, making this an important limitation of the reported training process rather than a resolved stability result.[1]

## Evaluation results

### Language tasks

The PaLM paper evaluated zero-shot, one-shot, and few-shot prompting across English natural-language processing tasks. Under the paper's prompts, datasets, baselines, and evaluation dates, PaLM 540B exceeded the previously reported few-shot state of the art on 28 of 29 English tasks used in its main comparison. This result does not mean that PaLM was best on every language task, that later systems did not surpass it, or that the comparisons used identical training data and inference budgets.[1]

The paper also evaluated 150 textual tasks from [BIG-bench](https://aiwiki.ai/wiki/big_bench). For comparisons with earlier models, it used a common subset of 58 tasks. PaLM 540B's five-shot aggregate normalized score on that subset was reported above the average score of the human raters collected by BIG-bench. The aggregation spans heterogeneous tasks and scoring rules, so it should not be read as evidence of general human-level intelligence.[1]

### Chain-of-thought prompting

Separate work on chain-of-thought prompting evaluated PaLM 540B with prompts containing worked reasoning examples. On [GSM8K](https://aiwiki.ai/wiki/gsm8k), the PaLM article reports 58 percent accuracy with eight chain-of-thought exemplars and an external calculator, 54 percent without the calculator, and 17 percent with a standard prompt that omitted chain-of-thought reasoning. These are benchmark results under specific prompting and answer-processing procedures, not proof that the model's written rationale faithfully represents an internal reasoning process.[1][4]

### Code and multilingual evaluation

Although code formed 5 percent of the training mixture, PaLM was evaluated on code completion, text-to-code generation, and code repair. The main paper reported few-shot results that were competitive with some code-specialized systems available at the time. It also fine-tuned a PaLM-Coder variant. Comparisons across systems are sensitive to possible training-data overlap, prompt format, model size, and the quantity of code used in training, so the results are best treated as task-specific measurements rather than a general ranking of coding ability.[1]

The authors evaluated translation, question answering, summarization, and cloze tasks in multiple languages. Performance generally improved with scale, but the paper did not show uniform performance across languages. Translation into English was stronger than translation from English for many evaluated pairs, and the training mixture remained about 78 percent English.[1]

### Claims about emergent abilities

The PaLM paper highlighted tasks on which the 540B model's measured score rose sharply relative to the 8B and 62B models. This observation contributed to discussion of [emergent abilities](https://aiwiki.ai/wiki/emergent_abilities), meaning abilities that appear absent in smaller models and present in larger ones under a chosen metric.[1]

Later peer-reviewed work by Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo argued that some apparent discontinuities can be produced by nonlinear or discontinuous evaluation metrics. In their analyses, alternative continuous metrics could turn a sudden-looking jump into a smooth scaling trend. This does not show that every reported capability scales smoothly, but it means PaLM's plots alone do not establish that a qualitatively new ability appeared at a precise parameter threshold.[14]

## Safety, memorization, and reproducibility

The PaLM paper includes analyses of memorization, representational bias, and toxic language. Its memorization experiments found that larger models reproduced training sequences more often and that sequences occurring more frequently in training were more likely to be reproduced. Deduplication reduced risk but did not eliminate it.[1]

The bias analysis used English prompts and a limited set of identity terms and occupations. The toxicity study likewise used specific English datasets and automated toxicity scoring. Prompt toxicity strongly affected output toxicity, and the 62B and 540B models produced slightly more toxic continuations than the 8B model in the authors' setup. These tests identify risks but are not a comprehensive audit across languages, cultures, deployment settings, or forms of harm.[1]

The model card in the paper describes PaLM as a research artifact and cautions against downstream use without additional safety and fairness evaluation. Google did not publish the PaLM weights, full training corpus, or a reproducible training pipeline with the paper. Independent researchers could inspect the article and later interact with product APIs, but they could not reproduce the reported model from the released materials alone.[1]

The evaluation suite also has limits. Benchmark contamination can inflate results when test-like material occurs in pretraining data, and few-shot scores can change with exemplar selection, prompt wording, answer parsing, or decoding settings. The authors performed overlap analyses for several benchmarks, but those checks cannot prove that every related item was absent from a web-scale corpus.[1]

## Adaptations based on PaLM

Several research projects adapted the original PaLM models. They have distinct training procedures and should not be treated as alternate names for the base model.

| Model | Relationship to PaLM | Reported purpose |
|---|---|---|
| Flan-PaLM | PaLM instruction-finetuned on a mixture of more than 1,800 tasks | Improve zero-shot and few-shot task performance |
| Minerva | PaLM further trained on mathematical and scientific text | Solve quantitative reasoning problems |
| PaLM-E | PaLM combined with encoders for images, robot state, and other continuous inputs | Embodied planning and visual-language tasks |
| Med-PaLM | Flan-PaLM adapted with medical-domain prompting | Medical question answering and long-form response evaluation |

Flan-PaLM applied [instruction tuning](https://aiwiki.ai/wiki/instruction_tuning) to PaLM. The JMLR article reports that its largest configuration improved the authors' average benchmark score by 9.4 percentage points over PaLM 540B and reached 75.2 percent five-shot accuracy on MMLU. Both numbers are specific to the paper's task mixture and evaluation setup.[5]

Minerva continued training PaLM models on technical material. Its technical-data mixture contained 17.5 billion tokens from webpages filtered for mathematics and 21.0 billion tokens from arXiv papers, together with a 5 percent stream sampled from PaLM's general-language corpus. For the largest model, the authors continued training for 26 billion tokens. Minerva then used prompting and majority-vote sampling for mathematical and scientific question answering.[6]

PaLM-E interleaves text tokens with vector representations produced from images, robot state, and other sensor inputs. Its largest reported configuration had 562 billion parameters, combining PaLM 540B with a 22-billion-parameter vision component. The ICML paper evaluated robot planning and control, visual question answering, image captioning, and language tasks. These experiments did not turn the base PaLM model itself into a multimodal system; they describe a separately trained extension.[7]

Med-PaLM was based on Flan-PaLM and used a medical-domain prompting method. In the *Nature* study, unadapted Flan-PaLM reached 67.6 percent accuracy on MedQA, while Med-PaLM was primarily assessed for the quality of long-form answers. Med-PaLM improved on Flan-PaLM across several clinician-rated dimensions, but its answers remained inferior to clinician answers in the study's lay helpfulness evaluation. The authors explicitly stated that further evaluation was needed before clinical use.[8]

## PaLM 2 and product transition

Google announced PaLM 2 on May 10, 2023. Its technical report describes a Transformer-based family trained with a mixture of objectives and a dataset containing more multilingual text, source code, mathematics, and parallel-language data than the original PaLM mixture. The report refers to PaLM 2-S, PaLM 2-M, and PaLM 2-L in its evaluations. It says the largest model, PaLM 2-L, was significantly smaller than PaLM 540B but used more training compute.[9]

Google's product announcement used the size names Gecko, Otter, Bison, and Unicorn and said the family supported more than 100 languages. It also said PaLM 2 was being used in more than 25 Google products and features, including [Bard](https://aiwiki.ai/wiki/bard).[10] The public technical report did not disclose exact parameter counts, the exact number of pretraining tokens, or the training hardware for the evaluated PaLM 2 models. Consequently, numerical estimates such as 340 billion parameters or 3.6 trillion training tokens should not be presented as disclosed specifications.[9]

Google announced a private preview of the PaLM API and MakerSuite on March 14, 2023.[11] PaLM 2 models later became available through Google's developer and cloud products. On December 6, 2023, Google said Bard's English-language version would use a specifically tuned version of [Gemini](https://aiwiki.ai/wiki/gemini) Pro, marking Bard's transition away from PaLM 2.[13]

Google's AI for Developers documentation says the Google AI PaLM API was decommissioned on August 15, 2024. The notice says PaLM API users can no longer prompt, tune, or run inference on PaLM-tuned models and directs them to the Gemini API. The same notice states that it does not pertain to Vertex AI. The supported conclusion is therefore that the Google AI PaLM API ended on that date, not that every PaLM-branded endpoint or every Vertex AI model ended simultaneously.[12]

## See also

- [Language model](https://aiwiki.ai/wiki/language_model)
- [Large language model](https://aiwiki.ai/wiki/large_language_model)
- [PaLM 2](https://aiwiki.ai/wiki/palm_2)
- [Pathways](https://aiwiki.ai/wiki/pathways)
- [Chain-of-thought prompting](https://aiwiki.ai/wiki/chain_of_thought)
- [PaLM-E](https://aiwiki.ai/wiki/palm-e_an_embodied_multimodal_language_model)
- [Med-PaLM](https://aiwiki.ai/wiki/med_palm)

## References

1. Chowdhery, A., Narang, S., Devlin, J., et al. "PaLM: Scaling Language Modeling with Pathways." Journal of Machine Learning Research, 24(240), 1-113, 2023. https://www.jmlr.org/papers/v24/22-1144.html
2. Google Research. "Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance." April 4, 2022. https://research.google/blog/pathways-language-model-palm-scaling-to-540-billion-parameters-for-breakthrough-performance/
3. Barham, P., Chowdhery, A., Dean, J., et al. "Pathways: Asynchronous Distributed Dataflow for ML." Proceedings of Machine Learning and Systems, 4, 2022. https://proceedings.mlsys.org/paper_files/paper/2022/hash/37385144cac01dff38247ab11c119e3c-Abstract.html
4. Wei, J., Wang, X., Schuurmans, D., et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." Advances in Neural Information Processing Systems, 35, 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html
5. Chung, H. W., Hou, L., Longpre, S., et al. "Scaling Instruction-Finetuned Language Models." Journal of Machine Learning Research, 25(70), 1-53, 2024. https://www.jmlr.org/papers/v25/23-0870.html
6. Lewkowycz, A., Andreassen, A., Dohan, D., et al. "Solving Quantitative Reasoning Problems with Language Models." Advances in Neural Information Processing Systems, 35, 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/18abbeef8cfe9203fdf9053c9c4fe191-Abstract-Conference.html
7. Driess, D., Xia, F., Sajjadi, M. S. M., et al. "PaLM-E: An Embodied Multimodal Language Model." Proceedings of the 40th International Conference on Machine Learning, 2023. https://proceedings.mlr.press/v202/driess23a.html
8. Singhal, K., Azizi, S., Tu, T., et al. "Large Language Models Encode Clinical Knowledge." Nature, 620, 172-180, 2023. https://doi.org/10.1038/s41586-023-06291-2
9. Anil, R., Dai, A. M., Firat, O., et al. "PaLM 2 Technical Report." arXiv:2305.10403, 2023. https://arxiv.org/abs/2305.10403
10. Google. "PaLM 2: Our Next Generation Large Language Model." May 10, 2023. https://blog.google/innovation-and-ai/products/google-palm-2-ai-large-language-model/
11. Google for Developers. "PaLM API & MakerSuite: An Approachable Way to Start Prototyping and Building Generative AI Applications." March 14, 2023. https://developers.googleblog.com/palm-api-makersuite-an-approachable-way-to-start-prototyping-and-building-generative-ai-applications/
12. Google AI for Developers. "PaLM API Deprecation." Last updated August 20, 2024. https://ai.google.dev/palm_docs/deprecation
13. Google. "Bard Gets Its Biggest Upgrade Yet with Gemini." December 6, 2023. https://blog.google/intl/en-in/company-news/technology/bard-gets-its-biggest-upgrade-yet-with-gemini/
14. Schaeffer, R., Miranda, B., and Koyejo, S. "Are Emergent Abilities of Large Language Models a Mirage?" Advances in Neural Information Processing Systems, 36, 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/adc98a266f45005c403b8311ca7e8bd7-Abstract-Conference.html