# InstructPix2Pix

> Source: https://aiwiki.ai/wiki/instructpix2pix
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: Computer Vision, Diffusion Models, Generative AI, Image Generation
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "InstructPix2Pix." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/instructpix2pix
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**InstructPix2Pix** is an instruction-based image editing model and training method developed by Tim Brooks, Aleksander Holynski and Alexei A. Efros at the University of California, Berkeley. Given an input image and a written instruction such as "add fireworks to the sky" or "make his jacket out of leather", the model produces an edited version of the image directly, without a user-drawn mask, a full description of the input or output image, or per-image fine-tuning or inversion.[1] The authors trained the model, a conditional [diffusion model](https://aiwiki.ai/wiki/diffusion_model) initialized from [Stable Diffusion](https://aiwiki.ai/wiki/stable_diffusion), on more than 450,000 synthetic editing examples that they generated by combining a fine-tuned [GPT-3](https://aiwiki.ai/wiki/gpt-3) with Stable Diffusion and the [Prompt-to-Prompt](https://aiwiki.ai/wiki/prompt_to_prompt) editing technique.[1] The paper was posted to arXiv on 17 November 2022 and published at the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition ([CVPR](https://aiwiki.ai/wiki/cvpr)), where it was presented as a highlight paper.[1][2][3] The code, model weights and generated dataset were released publicly, and the model was added to Hugging Face's [Diffusers](https://aiwiki.ai/wiki/diffusers) library in January 2023.[4][7]

## Background

Before InstructPix2Pix, most text-driven editing methods built on diffusion models expected a description of the image rather than an instruction. [SDEdit](https://aiwiki.ai/wiki/sdedit) noises an input image and denoises it toward a new target prompt; Prompt-to-Prompt edits images that the model itself generated by controlling cross-attention between two similar prompts; other methods inpaint a region given a caption and a user-drawn mask, learn a new subject from a few example images, or invert and fine-tune a model on a single image before regenerating it with a new description.[1] The authors argued that instructions ("what action to perform") are a more direct interface than captions, because the user does not have to describe the parts of the image that should stay the same.[1] They cited work on teaching large language models to follow human instructions as the inspiration for this goal.[1]

The obstacle was data. Paired examples of an image, a written edit instruction and the correctly edited result are difficult to collect at scale, so the authors treated the problem as supervised learning on a dataset they generated themselves with two pretrained models: a language model and a text-to-image model.[1] Efros was also a co-author of the 2016 paper "Image-to-Image Translation with Conditional Adversarial Networks", whose accompanying software was released as [pix2pix](https://aiwiki.ai/wiki/pix2pix); the InstructPix2Pix paper cites it among earlier work on translation between image domains.[1][15]

## Method

The method has two stages: generating a paired training dataset of instructions and before/after images, and training an image editing diffusion model on that dataset.[1]

### Generating instructions and captions

The first stage works only with text. The authors hand-wrote 700 editing triplets, each consisting of an input caption, an edit instruction and an output caption, using input captions sampled from the LAION-Aesthetics V2 6.5+ dataset, a collection derived from [LAION](https://aiwiki.ai/wiki/laion) data.[1] They fine-tuned the GPT-3 Davinci model on these examples for a single epoch with default training parameters, then used it to generate instructions and edited captions for a large number of further LAION captions, excluding samples with duplicate captions or duplicate image URLs.[1] The authors chose LAION for its size, diversity of content (including proper nouns and popular culture) and mix of mediums, and noted that its noisy captions were mitigated by dataset filtering and by classifier-free guidance.[1] The resulting text corpus contained 454,445 examples.[1]

| Source | Input caption | Edit instruction | Edited caption |
|---|---|---|---|
| Human-written | girl with horse at sunset | change the background to a city | girl with horse at sunset in front of city |
| Human-written | Yefim Volkov, Misty Morning | make it afternoon | Yefim Volkov, Misty Afternoon |
| GPT-3 generated | The great elf city of Rivendell, sitting atop a waterfall as cascades of water spill around it | Add a giant red dragon | The great elf city of Rivendell, sitting atop a waterfall as cascades of water spill around it with a giant red dragon flying overhead |

*Examples of training triplets, from Table 1 of the paper.[1]*

During fine-tuning, the prompt was the input caption followed by a separator, and the completion was the instruction and edited caption joined by a second separator and ended with a stop token; at generation time the authors sampled with temperature 0.7 and frequency penalty 0.1 and discarded outputs whose input and output captions were identical.[1] In the section on generating text with GPT-3, the repository states that the dataset "cost thousands of dollars to create".[4]

### Generating paired images

The second stage turns each pair of captions into a pair of images with Stable Diffusion. Text-to-image models give no guarantee that two similar prompts produce similar images (the paper's example is that "a picture of a cat" and "a picture of a black cat" may yield very different cats), so the authors used Prompt-to-Prompt to keep the two generations consistent by sharing attention weights for part of the denoising process.[1] Their implementation replaced the self-attention weights of the second image for the first fraction $p$ of the denoising steps and applied the same replacement strategy to every edit type.[1]

Because different edits need different amounts of change, the authors generated 100 image pairs per caption pair, each with a random $p \sim U(0.1, 0.9)$, and filtered them with [CLIP](https://aiwiki.ai/wiki/clip)-based metrics.[1] A pair was kept only if the two images had a CLIP image-image similarity of at least 0.75, the images matched their captions with an image-caption CLIP score of at least 0.2, and the directional CLIP similarity (whether the change between the images agrees with the change between the captions) was at least 0.2; for each caption pair, up to four passing pairs with the highest directional similarity were kept.[1] Image generation used the exponential moving average weights of the Stable Diffusion v1.5 checkpoint with the improved ft-MSE autoencoder, 100 denoising steps with an Euler ancestral sampler, and the same latent noise for both images in a pair.[1][4]

### Model architecture

InstructPix2Pix is a [latent diffusion](https://aiwiki.ai/wiki/latent_diffusion) model. An image $x$ is encoded by the pretrained [variational autoencoder](https://aiwiki.ai/wiki/variational_autoencoder) encoder $\mathcal{E}$ into a latent $z = \mathcal{E}(x)$, noise is added to produce $z_t$ at timestep $t$, and a network $\epsilon_\theta$ learns to predict the added noise given the image conditioning $c_I$ (the input image) and the text instruction $c_T$.[1] The training objective is:

$$L = \mathbb{E}_{\mathcal{E}(x), \mathcal{E}(c_I), c_T, \epsilon \sim \mathcal{N}(0,1), t}\left[\lVert \epsilon - \epsilon_\theta(z_t, t, \mathcal{E}(c_I), c_T) \rVert_2^2\right]$$

where $\epsilon$ is the sampled Gaussian noise.[1] Citing evidence that fine-tuning a large pretrained diffusion model works better than training from scratch for image translation when paired data is limited, the authors initialized the network from a Stable Diffusion checkpoint.[1] To accept the input image, they added input channels to the first convolutional layer so that $z_t$ and $\mathcal{E}(c_I)$ are concatenated; the weights for the new channels were initialized to zero and all other weights came from the pretrained checkpoint. The text conditioning mechanism originally used for captions was reused unchanged to take the edit instruction instead.[1]

### Classifier-free guidance with two conditionings

[Classifier-free guidance](https://aiwiki.ai/wiki/classifier_free_guidance) trains a diffusion model for both conditional and unconditional denoising and extrapolates between the two estimates at sampling time. InstructPix2Pix applies it separately to its two inputs.[1] During training, only the image conditioning was dropped (set to a null value $\varnothing$) for 5% of examples, only the text conditioning for another 5%, and both for a further 5%, so the model can denoise with either input, both or neither.[1] At inference, two guidance scales are combined:

$$\tilde{e}_\theta(z_t, c_I, c_T) = e_\theta(z_t, \varnothing, \varnothing) + s_I \cdot \big(e_\theta(z_t, c_I, \varnothing) - e_\theta(z_t, \varnothing, \varnothing)\big) + s_T \cdot \big(e_\theta(z_t, c_I, c_T) - e_\theta(z_t, c_I, \varnothing)\big)$$

Here $s_I$ controls how closely the output follows the input image and $s_T$ how strongly it follows the instruction.[1] The paper derives this form from a decomposition of $P(z \mid c_T, c_I)$ and notes that other orderings of the two conditionings are possible, but that this one worked better in practice.[1] The authors reported that $s_T$ between 5 and 10 and $s_I$ between 1 and 1.5 typically gave the best results, and that they adjusted the weights per example for the results shown in the paper.[1] The repository suggests an image guidance of 1.5 and text guidance of 7.5 as a starting point, lowering image guidance or raising text guidance when an image does not change enough, and the reverse when it changes too much.[4]

### Training and inference details

| Setting | Value |
|---|---|
| Initialization | EMA weights of Stable Diffusion v1.5 |
| Training steps | 10,000 |
| Hardware | 8 NVIDIA A100 GPUs (40 GB) |
| Training time | 25.5 hours |
| Training resolution | 256 x 256 |
| Total batch size | 1,024 |
| Learning rate | 10^-4, no warm-up |
| Augmentation | Random horizontal flip; random resize to 256-288 pixels, then crop to 256 |
| Inference resolution in the paper | 512 x 512 |
| Sampler | Euler ancestral, 100 denoising steps |
| Edit time | About 9 seconds per image on an A100 |

*Training and inference settings reported in the paper's appendix.[1]*

Although trained at 256 x 256, the model generalized to 512 x 512 at inference time.[1] The authors acknowledged funding from an NSF Graduate Research Fellowship (for Brooks), a research grant from SAP and a gift from Google.[1]

## Results

The paper shows edits on real photographs and artworks, including replacing objects, changing seasons and weather, replacing backgrounds, changing material attributes and converting the artistic medium.[1] Its figures include the Mona Lisa rendered as a Modigliani painting or a marble Roman sculpture, the Beatles' Abbey Road cover moved to other cities and eras, and Vermeer's *Girl with a Pearl Earring* given a range of edits.[1] Applying the model repeatedly with different instructions produces compounded edits, and varying the latent noise produces different plausible edits for the same image and instruction.[1] Despite training only on synthetic images and generated instructions, the authors reported zero-shot generalization to real images and human-written instructions.[1]

The authors compared their model qualitatively with SDEdit and Text2Live, giving the baselines the "after-edit" caption because they expect descriptions rather than instructions, and quantitatively with SDEdit.[1] The quantitative comparison plotted two competing metrics: cosine similarity of CLIP image embeddings between input and output (consistency with the input), and directional CLIP similarity (agreement between the change in the images and the change in the captions).[1] For the same directional similarity, InstructPix2Pix had notably higher CLIP image similarity than SDEdit.[1] Qualitatively, the authors found that SDEdit worked reasonably when content stayed constant and only style changed but struggled to preserve identity and isolate individual objects, while Text2Live produced convincing results for edits involving additive layers but was limited by its formulation in the categories of edits it could handle.[1]

Ablations showed that training on a smaller dataset (10% or 1% of the data) reduced the model's ability to make larger edits, leaving mostly subtle or stylistic changes, while removing the CLIP filtering from data generation reduced consistency with the input image.[1]

## Limitations

The authors listed several limitations. The model is bounded by the visual quality of the generated dataset and therefore by Stable Diffusion, and its ability to associate visual changes with instructions is limited by the 700 human-written examples, by GPT-3's instruction and caption generation, and by Prompt-to-Prompt's ability to modify images.[1] The paper reports that the model struggles with counting objects and with spatial reasoning, such as moving an object to one side of the image or swapping positions.[1] Its failure cases include an inability to change the viewpoint ("Zoom into the image"), excessive changes, failure to isolate the specified object, and difficulty reorganizing or swapping objects.[1] The authors also noted that the model may inherit biases from its data and pretrained models, illustrating this with correlations between profession and gender when asked to make people look like flight attendants or doctors.[1] The repository adds that the Stable Diffusion autoencoder handles small faces poorly and recommends cropping so the face fills more of the frame.[4]

## Release

The official PyTorch implementation, based on the CompVis Stable Diffusion codebase, includes a command-line editing script, a [Gradio](https://aiwiki.ai/wiki/gradio) editing app, training code, and scripts to reproduce the dataset generation pipeline.[4] The code is released under the MIT License, and the weights on Hugging Face (`timbrooks/instruct-pix2pix`) are also listed under the MIT license.[5][6] A browser demo runs as a Hugging Face Space, and the README also points to a hosted API on [Replicate](https://aiwiki.ai/wiki/replicate) and to the third-party Imaginairy package.[4][19]

For the public dataset release, the authors additionally filtered prompts and images for NSFW content, leaving 451,990 GPT-3 generated examples.[4] Two image versions are available:[4]

| Dataset version | Image editing examples | Size |
|---|---|---|
| `random-sample-dataset` | 451,990 | 727 GB |
| `clip-filtered-dataset` | 313,010 | 436 GB |

The CLIP-filtered version is also hosted on Hugging Face as `timbrooks/instructpix2pix-clip-filtered`.[18] The 700 human-written examples and the generated captions and instructions are released as JSONL files.[4]

### Diffusers integration

Hugging Face added a `StableDiffusionInstructPix2PixPipeline` to Diffusers in version 0.12.0, released on 25 January 2023 under the title "Instruct-Pix2Pix, DiT, LoRA"; the release notes describe the model as "a Stable Diffusion model fine-tuned for editing images from human instructions".[7] The pipeline exposes the two guidance scales as `image_guidance_scale` and `guidance_scale`.[7] Diffusers also ships a training script, `train_instruct_pix2pix.py`, which the maintainers describe as faithful to the original implementation but tested only on a small dataset, along with an equivalent script for [Stable Diffusion XL](https://aiwiki.ai/wiki/sdxl).[8] Hugging Face published an SDXL-based model trained on the CLIP-filtered dataset, `diffusers/sdxl-instructpix2pix-768`.[9]

In May 2023 a Hugging Face blog post by Sayak Paul applied the InstructPix2Pix training recipe to task-specific instruction datasets for cartoonization and low-level image processing (deraining, denoising, low-light enhancement and deblurring), reporting that the pretrained InstructPix2Pix model gave unsatisfactory cartoonization results and that the fine-tuned cartoonization model matched the target outputs more faithfully. The post also reported that the low-level model gave good deraining results but fell short on low-light enhancement and deblurring, which it suggested might be due to too few training examples for those tasks or to training that could be improved.[10]

## Influence and follow-up work

Several later instruction-based editing papers used InstructPix2Pix as a baseline, as a checkpoint to fine-tune, or as a source of training data.[11][13][14]

| Work | Date | Relation to InstructPix2Pix |
|---|---|---|
| HIVE | March 2023 | Added human feedback and a reward model for instructional editing, with a new 1.1M-example training set; MagicBrush describes it as providing more training triplets and human ranking results[12][11] |
| MagicBrush | June 2023 | Manually annotated dataset of over 10,000 triplets; fine-tuned InstructPix2Pix on it[11] |
| MGIE | September 2023 | Used the InstructPix2Pix dataset for pre-training and InstructPix2Pix as its baseline, adding a multimodal LLM to derive more expressive instructions[13] |
| [Emu Edit](https://aiwiki.ai/wiki/emu_edit) (GenAI, Meta) | November 2023 | Multi-task editing model evaluated against InstructPix2Pix and MagicBrush[14] |

The MagicBrush authors argued that automatically synthesized training datasets contain a high volume of noise. In their benchmark the off-the-shelf InstructPix2Pix checkpoint was not competitive with other baselines, but after fine-tuning on MagicBrush it improved across all metrics and was preferred by human evaluators; in their qualitative examples the original model followed instructions but produced excessive modification and lacked photorealism.[11] MGIE's authors argued that the CLIP text encoder InstructPix2Pix uses is trained on static descriptions, which makes it hard to capture the visual transformation an edit requires, that brief instructions can be too ambiguous to guide the edit, and that because the InstructPix2Pix dataset contains no removal or photo-optimization edits, InstructPix2Pix failed on those operations.[13]

Meta's Emu Edit paper described InstructPix2Pix as having introduced an instructable image editing model, while arguing that such models often struggle to interpret and execute instructions and generalize poorly to tasks slightly outside their training data.[14] It also criticized the InstructPix2Pix benchmark as biased because it relies on Stable Diffusion generated input images and GPT-3 generated instructions.[14] In Emu Edit's human evaluation on its own test set, raters preferred Emu Edit over InstructPix2Pix in 77.33% of comparisons for text alignment and 76.71% for image faithfulness.[14] Emu Edit's authors also compared their data generation pipeline with InstructPix2Pix's Prompt-to-Prompt and CLIP-filtering approach and reported that theirs outperformed it on all the tasks in that comparison.[14]

## Authors

Brooks and Holynski are listed as equal contributors.[1] At the time of the paper, Brooks was a PhD student at Berkeley AI Research; according to his personal website (as of November 2026) he later co-led [Sora](https://aiwiki.ai/wiki/sora) at [OpenAI](https://aiwiki.ai/wiki/openai), led a team working on omnimodels at [Google DeepMind](https://aiwiki.ai/wiki/google_deepmind), and leads media generation at [Meta Superintelligence Labs](https://aiwiki.ai/wiki/meta_superintelligence_labs).[16] Holynski's website says he was previously a postdoctoral researcher at [UC Berkeley](https://aiwiki.ai/wiki/uc_berkeley) with Efros and Angjoo Kanazawa, and describes him (as of November 2026) as a professor at Columbia University who leads a Google DeepMind team working on generative 3D and world models.[17]

## See also

- [Prompt-to-Prompt](https://aiwiki.ai/wiki/prompt_to_prompt)
- [SDEdit](https://aiwiki.ai/wiki/sdedit)
- [Emu Edit](https://aiwiki.ai/wiki/emu_edit)
- [ControlNet](https://aiwiki.ai/wiki/controlnet)
- [Image-to-Image Models](https://aiwiki.ai/wiki/image-to-image_models)
- [Synthetic data](https://aiwiki.ai/wiki/synthetic_data)
- [Instruction tuning](https://aiwiki.ai/wiki/instruction_tuning)

## References

1. Tim Brooks, Aleksander Holynski, Alexei A. Efros. "InstructPix2Pix: Learning to Follow Image Editing Instructions". arXiv:2211.09800, 17 November 2022 (v2, 18 January 2023). https://arxiv.org/abs/2211.09800
2. Tim Brooks, Aleksander Holynski, Alexei A. Efros. "InstructPix2Pix: Learning To Follow Image Editing Instructions". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 18392-18402. doi:10.1109/CVPR52729.2023.01764. https://openaccess.thecvf.com/content/CVPR2023/html/Brooks_InstructPix2Pix_Learning_To_Follow_Image_Editing_Instructions_CVPR_2023_paper.html
3. CVPR 2023. "InstructPix2Pix: Learning To Follow Image Editing Instructions" (Highlight, poster page). cvpr2023.thecvf.com, 2023. https://cvpr2023.thecvf.com/virtual/2023/poster/21296
4. Tim Brooks, Aleksander Holynski, Alexei A. Efros. "InstructPix2Pix: Learning to Follow Image Editing Instructions" (README). GitHub, timothybrooks/instruct-pix2pix. https://github.com/timothybrooks/instruct-pix2pix
5. Timothy Brooks, Aleksander Holynski, Alexei A. Efros. "LICENSE" (MIT License). GitHub, timothybrooks/instruct-pix2pix, 2023. https://github.com/timothybrooks/instruct-pix2pix/blob/main/LICENSE
6. Tim Brooks. "timbrooks/instruct-pix2pix" (model card). Hugging Face. https://huggingface.co/timbrooks/instruct-pix2pix
7. Hugging Face. "v0.12.0: Instruct-Pix2Pix, DiT, LoRA" (release notes). GitHub, huggingface/diffusers, 25 January 2023. https://github.com/huggingface/diffusers/releases/tag/v0.12.0
8. Hugging Face. "InstructPix2Pix training example". GitHub, huggingface/diffusers. https://github.com/huggingface/diffusers/blob/main/examples/instruct_pix2pix/README.md
9. Hugging Face. "SDXL InstructPix2Pix (768768)" (model card, diffusers/sdxl-instructpix2pix-768). Hugging Face, 2023. https://huggingface.co/diffusers/sdxl-instructpix2pix-768
10. Sayak Paul. "Instruction-tuning Stable Diffusion with InstructPix2Pix". Hugging Face blog, 23 May 2023. https://huggingface.co/blog/instruction-tuning-sd
11. Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, Yu Su. "MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing". arXiv:2306.10012 (NeurIPS 2023), 16 June 2023. https://arxiv.org/abs/2306.10012
12. Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, Ran Xu. "HIVE: Harnessing Human Feedback for Instructional Visual Editing". arXiv:2303.09618 (CVPR 2024), 16 March 2023. https://arxiv.org/abs/2303.09618
13. Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, Zhe Gan. "Guiding Instruction-based Image Editing via Multimodal Large Language Models". arXiv:2309.17102 (ICLR 2024), 29 September 2023. https://arxiv.org/abs/2309.17102
14. Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, Yaniv Taigman. "Emu Edit: Precise Image Editing via Recognition and Generation Tasks". arXiv:2311.10089, 16 November 2023. https://arxiv.org/abs/2311.10089
15. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros. "Image-to-Image Translation with Conditional Adversarial Networks". arXiv:1611.07004 (CVPR 2017), 21 November 2016. https://arxiv.org/abs/1611.07004
16. Tim Brooks. "Tim Brooks" (personal website). timothybrooks.com, accessed November 2026. https://www.timothybrooks.com/
17. Aleksander Holynski. "Aleksander Holynski" (personal website). holynski.org, accessed November 2026. https://holynski.org/
18. Tim Brooks. "timbrooks/instructpix2pix-clip-filtered" (dataset card). Hugging Face. https://huggingface.co/datasets/timbrooks/instructpix2pix-clip-filtered
19. Tim Brooks. "InstructPix2Pix" (Hugging Face Space, timbrooks/instruct-pix2pix). Hugging Face. https://huggingface.co/spaces/timbrooks/instruct-pix2pix

