PaLM-E: An Embodied Multimodal Language Model

PaLM-E (short for Pathways Language Model, Embodied) is an embodied multimodal large language model introduced by Google and TU Berlin in March 2023 that injects continuous sensor and image observations directly into a pretrained language model so a single network can reason about the physical world and output robot plans. As the Google Research announcement put it, "PaLM-E works by injecting observations into a pre-trained language model," transforming sensor data into representations comparable to word tokens.^[1][9] The model was described in the paper PaLM-E: An Embodied Multimodal Language Model by Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence, posted to arXiv on March 6, 2023 and later presented at ICML 2023.^[1][2]

The largest variant, PaLM-E-562B, combined the 540-billion-parameter PaLM language model with the 22-billion-parameter Vision Transformer (ViT-22B) into a single 562B-parameter model that took interleaved text, images, and continuous robot state as input and produced natural-language plans, visual question answers, and robot commands as output.^[1][3] At the time of release it was the largest vision-language model ever publicly described, and it set a new state of the art on the OK-VQA benchmark with 66.1% accuracy, beating the prior best of 64.5% from PaLI, without any task-specific fine-tuning, while also acting as the high-level planner for several physical robot platforms.^[1][3]

PaLM-E is generally remembered for three results: it showed that a single generalist multimodal model could plan and act across multiple distinct robot embodiments, it demonstrated positive transfer between internet-scale vision-language data and robot data, and it provided the architectural template that Google's later RT-2 and Gemini Robotics systems would inherit.^[1][4][5]

Background

Before PaLM-E, there were two largely separate research lines on the boundary between language and embodied action. One line, exemplified by SayCan (Ahn et al., 2022), used a PaLM language model as a planner that picked among predefined robot skills based on their textual descriptions, but the language model itself never saw the robot's camera images and could not reason about novel objects in the scene.^[1][6] The other line, including the original RT-1 (Brohan et al., 2022), trained a smaller transformer policy directly on robot demonstrations, but those policies struggled to generalize to instructions or objects far outside their training distribution.^[7]

Driess and colleagues set out to merge these two threads. The goal was to inject continuous sensor observations directly into the input stream of a pretrained language model so that the model could plan, ground, and act in one forward pass, rather than relying on a separate perception module to first translate the world into text. The team also wanted to test whether the scaling laws and broad knowledge of frontier language models would carry over to embodied tasks.^[1]

A contemporaneous Google line of work on multimodal models, including the 22-billion-parameter Vision Transformer (ViT-22B) by Dehghani et al. (2023), provided the visual encoder that PaLM-E would use as its image backbone. PaLM-E was developed jointly across Google Research, Google Robotics, and Marc Toussaint's Learning and Intelligent Systems group at Technische Universitat Berlin.^[1][3]

Who built PaLM-E and where was it published?

The PaLM-E paper has 22 authors split between Google and TU Berlin. The lead author, Danny Driess, was at the time a PhD student at TU Berlin completing a research collaboration with Google's Robotics at Google team in Mountain View. The authors and their affiliations at the time of publication are listed below.^[1]

Scroll sideways for more →

The paper was posted to arXiv as 2303.03378 on March 6, 2023, accepted at the 40th International Conference on Machine Learning (ICML 2023) in Honolulu, and later given an oral presentation in the conference's robotics track.^[1][2]

How does PaLM-E work?

PaLM-E is built on a simple but consequential idea: any continuous observation can be projected into the same embedding space as language tokens, so the language model can attend to it without any architectural change. The paper describes its inputs as "multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings," and treats images, robot state vectors, and object-centric scene descriptions all as sequences of vectors that are fed into the transformer alongside text tokens.^[1]

Backbone

The language backbone is the PaLM decoder-only transformer family, with the largest version using the 540-billion-parameter PaLM-540B trained on 780 billion tokens of internet text and code. The vision backbone is a Vision Transformer; the largest configuration uses ViT-22B, the 22-billion-parameter image encoder introduced by Dehghani et al. (2023). Smaller PaLM-E variants pair smaller PaLM checkpoints with smaller ViT or ViT-4B encoders.^[1][3]

Encoder choices

The paper studies several ways of turning visual input into a sequence of language-aligned vectors. The simplest is a frozen ViT followed by a learned linear projection that maps each ViT patch token into the PaLM token embedding space. The paper also evaluates an Object Scene Representation Transformer (OSRT) encoder that produces object-centric slot representations, and a state vector encoder that takes the position and pose of objects in a scene and projects them into language tokens.^[1] The authors highlight OSRT as a way to obtain object-centric structure without supervision: "rather than relying on external knowledge about objects, they are discovered in an unsupervised way through inductive biases in the architecture."^[1] Across encoders, only the projection layer is trained from scratch; the ViT can be either frozen or co-trained with the rest of the model.

Multimodal sentence input

A prompt to PaLM-E might look like "Describe what is happening: " where <img> is replaced at runtime by a sequence of visual tokens produced by the encoder. For robotics, the prompt usually includes a goal and a partial action history, with images showing the current scene at each step. The model's output is decoded autoregressively as text, including either a high-level plan, an answer to a visual question, or a low-level action token interpreted by a downstream controller.^[1]

Training objective

PaLM-E is trained end-to-end with the standard next-token prediction cross-entropy loss on a mixture of robotics and vision-language data. The robotics data includes both image observations and target action sequences expressed as text. The vision-language data includes web-scale image-text pairs, captioning data, and visual question answering. The language model parameters can be either frozen or fine-tuned; the paper finds that for the largest model, freezing PaLM and only training the visual projection works almost as well as full fine-tuning, with significantly less catastrophic forgetting on language-only benchmarks.^[1]

Model variants

The paper reports results across a range of sizes, from a 12-billion-parameter version that combines an 8B PaLM with a 4B ViT, to the flagship PaLM-E-562B that pairs PaLM-540B with ViT-22B. Smaller variants exist primarily to enable controlled scaling experiments.^[1]

Scroll sideways for more →

What data and robots was PaLM-E trained on?

PaLM-E was trained on a mix of three robot embodiments and a large set of internet-scale vision-language datasets. The paper deliberately chose embodiments that differ in action space and sensor configuration so that the multi-embodiment training claim could be tested rigorously.^[1]

Scroll sideways for more →

The robot data for the Language Table embodiment was collected as part of the Interactive Language work by Lynch et al. (2023) and consists of natural-language demonstrations of pushing, sliding, and arranging colored blocks. The mobile manipulator data was inherited from the SayCan project, which had previously been used to evaluate language models as planners on the same hardware platform.^[1][6][8]

What can PaLM-E do?

The paper organizes PaLM-E's outputs into three families: high-level robot planning, embodied visual question answering and dialogue, and general internet-scale visual question answering and captioning.

Embodied planning

For the TAMP, Language Table, and mobile manipulator embodiments, PaLM-E takes the current scene image and a natural-language goal as input and emits a sequence of high-level subgoals or actions. The plan is then executed by a low-level controller specific to the embodiment, with PaLM-E re-prompted at each step with a new image. The paper shows that this closed-loop setup allows the model to recover from intermediate failures, such as a dropped object or a perturbation introduced by a human.^[1] PaLM-E was also fine-tuned on 100 different long-horizon mobile-manipulation tasks from a single training example each, and could then generalize zero-shot to novel object pairs and to objects unseen in either the robot dataset or the fine-tuning data.^[1][9]

Embodied VQA and dialogue

PaLM-E can answer questions about the robot's environment, such as how many objects of a given color are visible, whether a specific object can be picked up given the current arm configuration, or how the robot should rearrange the scene to satisfy a constraint. The model can also engage in multi-turn dialogue about the scene because the image and prior outputs are simply additional tokens in the multimodal sentence.^[1]

General visual reasoning

Because PaLM-E retains the language modeling capabilities of PaLM and gains image understanding from the ViT, it inherits general vision-language abilities. The paper reports that PaLM-E-562B "exhibits a wide array of capabilities including zero-shot multimodal chain-of-thought (CoT) reasoning," few-shot prompting, OCR-free math reasoning, and multi-image relational reasoning, "despite being trained on only single-image examples," none of which were explicit training objectives.^[1]

Positive transfer

The paper's central scientific finding is that joint training across embodiments and across internet vision-language data improves performance on each individual task relative to single-task baselines. The authors call this positive transfer and state that "the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains": a single PaLM-E instance trained on all three robot domains plus internet data outperforms separate models trained on each domain alone.^[1] The effect grows with model size, and is most pronounced for the largest 562B configuration.^[1][9]

What benchmarks did PaLM-E set records on?

PaLM-E was evaluated on a mix of robot benchmarks and standard vision-language benchmarks. The most cited results are summarized below; full numbers are reported in the paper.^[1][3]

Scroll sideways for more →

The OK-VQA result attracted particular attention because PaLM-E-562B achieved it as a single generalist model rather than a specialist tuned on the OK-VQA training set. The model was also reported to retain essentially the same general language performance as PaLM-540B, which had been a concern with earlier approaches that fine-tuned the language model on multimodal data and saw significant catastrophic forgetting.^[1][3]

Scaling and forgetting

A notable finding in the paper's analysis is that language capabilities degrade less with scale. PaLM-E-12B suffers about an 87% relative drop on language-only benchmarks when fine-tuned on multimodal data, while PaLM-E-562B suffers only about a 3.9% drop. This led the authors to argue that frozen-language-model multimodal training becomes unnecessary at sufficient scale, because the larger model has enough capacity to absorb the new modalities without overwriting its language priors.^[1][9]

Reception and coverage

PaLM-E was announced through a Google Research blog post on March 10, 2023 and a project page at palm-e.github.io that hosted video demonstrations of the mobile manipulator following multi-step natural-language instructions in a kitchen.^[3][9][10] The release attracted broad coverage in the technology press because it was published only weeks after the launch of GPT-4 and was widely framed as Google's response on the multimodal-and-embodied frontier.^[11][12]

Ars Technica called PaLM-E "a multimodal language model that controls a robot" and noted that the most striking demonstration was an instruction such as bring me the rice chips from the drawer, which the robot completed despite the chips being in a different drawer than expected and an experimenter physically moving the bag during execution.^[12] The Verge highlighted the 562-billion-parameter scale and the multi-embodiment generalization, and IEEE Spectrum framed PaLM-E as evidence that foundation models could play a central role in robotics rather than only in language.^[11][13]

The ICML 2023 review process accepted PaLM-E as an oral paper, and the authors gave a keynote-style presentation in the robotics track at the conference in Honolulu.^[2]

How did PaLM-E influence RT-2 and later robotics models?

PaLM-E directly seeded a sequence of robotics foundation models at Google and beyond. The most direct successor is RT-2 (Brohan et al., 2023), which adopted the same recipe of fine-tuning a vision-language model on robot demonstrations but added the key step of representing robot actions as text tokens, allowing the model to be trained jointly on internet image-text pairs and robot trajectories using the same next-token prediction objective. RT-2 used the PaLI-X and PaLM-E backbones and demonstrated significantly stronger generalization than RT-1 to novel objects, backgrounds, and instructions.^[4][14]

Scroll sideways for more →

A 2024 survey of vision-language-action models in IEEE Transactions on Robotics described PaLM-E as the inflection point at which vision-language models became viable robot planners, and identified action tokenization (introduced shortly afterward by RT-2) as the missing piece that turned the planning interface into a unified vision-language-action model.^[17]

Impact on embodied AI

PaLM-E shifted the framing of robot policy learning. Before its release, the dominant approach was to train a small policy network from scratch on a single robot embodiment, with the language model (if any) acting as an external planner. After PaLM-E, the dominant approach in industrial labs became take a frontier vision-language model, fine-tune on robot data from many embodiments, and let positive transfer from internet data do the heavy lifting on generalization.^[5][17]

The paper also legitimized the use of internet-scale pretraining as the foundation for embodied AI. Subsequent work at Google (RT-2, RT-X, Gemini Robotics), Stanford (Mobile ALOHA, OpenVLA), NVIDIA (GR00T), and Physical Intelligence (Pi-0) all assume that embodied policies should inherit perception and reasoning from large multimodal foundation models, with robot-specific data serving as a fine-tuning signal rather than the primary source of capability.^[5][16][18]

Finally, PaLM-E provided concrete empirical support for the generalist agent hypothesis: a single sufficiently large model trained on many tasks and embodiments can outperform specialist models on each individual task. This claim, originally made for language by GPT-3 and for general environments by Gato (Reed et al., 2022), was extended to physical robots by PaLM-E and remains a guiding assumption of the foundation model approach to robotics.^[1][9]

What are the limitations of PaLM-E?

The authors acknowledge several limitations of PaLM-E in the paper itself and in subsequent talks.

• Slow inference. PaLM-E-562B requires substantial compute per forward pass and was not deployable on a robot at any reasonable control frequency. Robot evaluations used the model as an offline high-level planner with low-level controllers running independently.^[1]
• No native action output. PaLM-E emits text plans rather than continuous joint commands. Subsequent work in RT-2 addressed this by tokenizing actions and folding them into the language model's output vocabulary.^[4]
• Limited robot data diversity. Despite spanning three embodiments, the robot training data was small relative to internet vision-language data. Several PaLM-E results that were initially attributed to the model's robotics ability turned out to depend heavily on the underlying SayCan skill library.^[1][6]
• Closed model. PaLM-E weights were never publicly released. Reproductions and follow-up research had to wait for open alternatives such as OpenVLA and the Open X-Embodiment dataset.^[5]
• Evaluation outside the lab. All robot evaluations were performed in Google's own facilities, mostly on Everyday Robots mobile manipulators and the Language Table tabletop platform. The paper did not include external benchmarking or third-party deployments.^[1]

The authors also note that some Language Table results required co-training with internet data to reach high accuracy, suggesting that the small robot dataset was not sufficient on its own and that the multimodal pretraining was doing a substantial amount of the generalization work.^[1]

See also

• ERQA

• PaLM
• Vision Transformer
• Vision-language model
• Vision-language-action model
• RT-2
• RT-X
• SayCan
• Gemini Robotics
• Embodied AI
• Foundation model
• OK-VQA
• Multimodal AI
• Sergey Levine

References

1. Driess, Danny, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. "PaLM-E: An Embodied Multimodal Language Model." arXiv preprint 2303.03378, March 6, 2023. https://arxiv.org/abs/2303.03378 ↩
2. PaLM-E paper accepted at the 40th International Conference on Machine Learning (ICML 2023), Honolulu, Hawaii, July 23-29, 2023. https://icml.cc/virtual/2023/poster/24067 ↩
3. Driess, Danny et al. "PaLM-E: An Embodied Multimodal Language Model." Project page. https://palm-e.github.io ↩
4. Brohan, Anthony et al. "RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control." arXiv preprint 2307.15818, July 28, 2023. https://arxiv.org/abs/2307.15818 ↩
5. Open X-Embodiment Collaboration et al. "Open X-Embodiment: Robotic Learning Datasets and RT-X Models." arXiv preprint 2310.08864, October 2023. https://arxiv.org/abs/2310.08864 ↩
6. Ahn, Michael et al. "Do As I Can, Not As I Say: Grounding Language in Robotic Affordances." arXiv preprint 2204.01691, April 2022. https://arxiv.org/abs/2204.01691 ↩
7. Brohan, Anthony et al. "RT-1: Robotics Transformer for Real-World Control at Scale." arXiv preprint 2212.06817, December 2022. https://arxiv.org/abs/2212.06817 ↩
8. Lynch, Corey et al. "Interactive Language: Talking to Robots in Real Time." arXiv preprint 2210.06407, October 2022. https://arxiv.org/abs/2210.06407 ↩
9. Driess, Danny, and Pete Florence. "PaLM-E: An embodied multimodal language model." Google Research Blog, March 10, 2023. https://research.google/blog/palm-e-an-embodied-multimodal-language-model/ ↩
10. Google Research. "PaLM-E: An embodied multimodal language model." Google Research Blog, March 10, 2023. https://blog.research.google/2023/03/palm-e-embodied-multimodal-language.html ↩
11. Vincent, James. "Google's PaLM-E is a generalist robot brain that takes commands." The Verge, March 8, 2023. https://www.theverge.com/2023/3/8/23629362/google-ai-language-model-palm-e-robotics ↩
12. Edwards, Benj. "Google's PaLM-E is a generalist robot brain that takes commands." Ars Technica, March 9, 2023. https://arstechnica.com/information-technology/2023/03/embodied-ai-googles-palm-e-allows-robot-control-with-natural-commands/ ↩
13. Ackerman, Evan. "Google's PaLM-E is a 562-Billion-Parameter Visual Language Model." IEEE Spectrum, March 2023. https://spectrum.ieee.org/google-palm-e ↩
14. Google DeepMind. "RT-2: New model translates vision and language into action." July 28, 2023. https://deepmind.google/blog/rt-2-new-model-translates-vision-and-language-into-action/ ↩
15. Google DeepMind. "Scaling up learning across many different robot types." October 3, 2023. https://deepmind.google/blog/scaling-up-learning-across-many-different-robot-types/ ↩
16. Google DeepMind. "Gemini Robotics brings AI into the physical world." March 12, 2025. https://deepmind.google/blog/gemini-robotics-brings-ai-into-the-physical-world/ ↩
17. Firoozi, Roya et al. "Foundation Models in Robotics: Applications, Challenges, and the Future." The International Journal of Robotics Research, 2024. https://arxiv.org/abs/2312.07843 ↩
18. Black, Kevin et al. "Pi-0: A Vision-Language-Action Flow Model for General Robot Control." Physical Intelligence, October 2024. https://www.physicalintelligence.company/blog/pi0 ↩