Citation and evidence

InstructPix2Pix

16 min full readUpdated 19 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

InstructPix2Pix is an instruction-based image editing model and training method developed by Tim Brooks, Aleksander Holynski and Alexei A. Efros at the University of California, Berkeley. Given an input image and a written instruction such as "add fireworks to the sky" or "make his jacket out of leather", the model produces an edited version of the image directly, without a user-drawn mask, a full description of the input or output image, or per-image fine-tuning or inversion.[1] The authors trained the model, a conditional diffusion model initialized from Stable Diffusion, on more than 450,000 synthetic editing examples that they generated by combining a fine-tuned GPT-3 with Stable Diffusion and the Prompt-to-Prompt editing technique.[1] The paper was posted to arXiv on 17 November 2022 and published at the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), where it was presented as a highlight paper.[1][2][3] The code, model weights and generated dataset were released publicly, and the model was added to Hugging Face's Diffusers library in January 2023.[4][7]

Background

Before InstructPix2Pix, most text-driven editing methods built on diffusion models expected a description of the image rather than an instruction. SDEdit noises an input image and denoises it toward a new target prompt; Prompt-to-Prompt edits images that the model itself generated by controlling cross-attention between two similar prompts; other methods inpaint a region given a caption and a user-drawn mask, learn a new subject from a few example images, or invert and fine-tune a model on a single image before regenerating it with a new description.[1] The authors argued that instructions ("what action to perform") are a more direct interface than captions, because the user does not have to describe the parts of the image that should stay the same.[1] They cited work on teaching large language models to follow human instructions as the inspiration for this goal.[1]

The obstacle was data. Paired examples of an image, a written edit instruction and the correctly edited result are difficult to collect at scale, so the authors treated the problem as supervised learning on a dataset they generated themselves with two pretrained models: a language model and a text-to-image model.[1] Efros was also a co-author of the 2016 paper "Image-to-Image Translation with Conditional Adversarial Networks", whose accompanying software was released as pix2pix; the InstructPix2Pix paper cites it among earlier work on translation between image domains.[1][15]

Method

The method has two stages: generating a paired training dataset of instructions and before/after images, and training an image editing diffusion model on that dataset.[1]

Generating instructions and captions

The first stage works only with text. The authors hand-wrote 700 editing triplets, each consisting of an input caption, an edit instruction and an output caption, using input captions sampled from the LAION-Aesthetics V2 6.5+ dataset, a collection derived from LAION data.[1] They fine-tuned the GPT-3 Davinci model on these examples for a single epoch with default training parameters, then used it to generate instructions and edited captions for a large number of further LAION captions, excluding samples with duplicate captions or duplicate image URLs.[1] The authors chose LAION for its size, diversity of content (including proper nouns and popular culture) and mix of mediums, and noted that its noisy captions were mitigated by dataset filtering and by classifier-free guidance.[1] The resulting text corpus contained 454,445 examples.[1]

SourceInput captionEdit instructionEdited caption
Human-writtengirl with horse at sunsetchange the background to a citygirl with horse at sunset in front of city
Human-writtenYefim Volkov, Misty Morningmake it afternoonYefim Volkov, Misty Afternoon
GPT-3 generatedThe great elf city of Rivendell, sitting atop a waterfall as cascades of water spill around itAdd a giant red dragonThe great elf city of Rivendell, sitting atop a waterfall as cascades of water spill around it with a giant red dragon flying overhead

Expanded article table

Examples of training triplets, from Table 1 of the paper.[1]

During fine-tuning, the prompt was the input caption followed by a separator, and the completion was the instruction and edited caption joined by a second separator and ended with a stop token; at generation time the authors sampled with temperature 0.7 and frequency penalty 0.1 and discarded outputs whose input and output captions were identical.[1] In the section on generating text with GPT-3, the repository states that the dataset "cost thousands of dollars to create".[4]

Generating paired images

The second stage turns each pair of captions into a pair of images with Stable Diffusion. Text-to-image models give no guarantee that two similar prompts produce similar images (the paper's example is that "a picture of a cat" and "a picture of a black cat" may yield very different cats), so the authors used Prompt-to-Prompt to keep the two generations consistent by sharing attention weights for part of the denoising process.[1] Their implementation replaced the self-attention weights of the second image for the first fraction $p$ of the denoising steps and applied the same replacement strategy to every edit type.[1]

Because different edits need different amounts of change, the authors generated 100 image pairs per caption pair, each with a random $p \sim U(0.1, 0.9)$, and filtered them with CLIP-based metrics.[1] A pair was kept only if the two images had a CLIP image-image similarity of at least 0.75, the images matched their captions with an image-caption CLIP score of at least 0.2, and the directional CLIP similarity (whether the change between the images agrees with the change between the captions) was at least 0.2; for each caption pair, up to four passing pairs with the highest directional similarity were kept.[1] Image generation used the exponential moving average weights of the Stable Diffusion v1.5 checkpoint with the improved ft-MSE autoencoder, 100 denoising steps with an Euler ancestral sampler, and the same latent noise for both images in a pair.[1][4]

Model architecture

InstructPix2Pix is a latent diffusion model. An image $x$ is encoded by the pretrained variational autoencoder encoder $\mathcal{E}$ into a latent $z = \mathcal{E}(x)$, noise is added to produce $z_t$ at timestep $t$, and a network $\epsilon_\theta$ learns to predict the added noise given the image conditioning $c_I$ (the input image) and the text instruction $c_T$.[1] The training objective is:

L=EE(x),E(cI),cT,ϵ∼N(0,1),t[∥ϵ−ϵθ(zt,t,E(cI),cT)∥22]L = \mathbb{E}_{\mathcal{E}(x), \mathcal{E}(c_I), c_T, \epsilon \sim \mathcal{N}(0,1), t}\left[\lVert \epsilon - \epsilon_\theta(z_t, t, \mathcal{E}(c_I), c_T) \rVert_2^2\right]

where $\epsilon$ is the sampled Gaussian noise.[1] Citing evidence that fine-tuning a large pretrained diffusion model works better than training from scratch for image translation when paired data is limited, the authors initialized the network from a Stable Diffusion checkpoint.[1] To accept the input image, they added input channels to the first convolutional layer so that $z_t$ and $\mathcal{E}(c_I)$ are concatenated; the weights for the new channels were initialized to zero and all other weights came from the pretrained checkpoint. The text conditioning mechanism originally used for captions was reused unchanged to take the edit instruction instead.[1]

Classifier-free guidance with two conditionings

Classifier-free guidance trains a diffusion model for both conditional and unconditional denoising and extrapolates between the two estimates at sampling time. InstructPix2Pix applies it separately to its two inputs.[1] During training, only the image conditioning was dropped (set to a null value $\varnothing$) for 5% of examples, only the text conditioning for another 5%, and both for a further 5%, so the model can denoise with either input, both or neither.[1] At inference, two guidance scales are combined:

e~θ(zt,cI,cT)=eθ(zt,∅,∅)+sI⋅(eθ(zt,cI,∅)−eθ(zt,∅,∅))+sT⋅(eθ(zt,cI,cT)−eθ(zt,cI,∅))\tilde{e}_\theta(z_t, c_I, c_T) = e_\theta(z_t, \varnothing, \varnothing) + s_I \cdot \big(e_\theta(z_t, c_I, \varnothing) - e_\theta(z_t, \varnothing, \varnothing)\big) + s_T \cdot \big(e_\theta(z_t, c_I, c_T) - e_\theta(z_t, c_I, \varnothing)\big)

Here $s_I$ controls how closely the output follows the input image and $s_T$ how strongly it follows the instruction.[1] The paper derives this form from a decomposition of $P(z \mid c_T, c_I)$ and notes that other orderings of the two conditionings are possible, but that this one worked better in practice.[1] The authors reported that $s_T$ between 5 and 10 and $s_I$ between 1 and 1.5 typically gave the best results, and that they adjusted the weights per example for the results shown in the paper.[1] The repository suggests an image guidance of 1.5 and text guidance of 7.5 as a starting point, lowering image guidance or raising text guidance when an image does not change enough, and the reverse when it changes too much.[4]

Training and inference details

SettingValue
InitializationEMA weights of Stable Diffusion v1.5
Training steps10,000
Hardware8 NVIDIA A100 GPUs (40 GB)
Training time25.5 hours
Training resolution256 x 256
Total batch size1,024
Learning rate10^-4, no warm-up
AugmentationRandom horizontal flip; random resize to 256-288 pixels, then crop to 256
Inference resolution in the paper512 x 512
SamplerEuler ancestral, 100 denoising steps
Edit timeAbout 9 seconds per image on an A100

Expanded article table

Training and inference settings reported in the paper's appendix.[1]

Although trained at 256 x 256, the model generalized to 512 x 512 at inference time.[1] The authors acknowledged funding from an NSF Graduate Research Fellowship (for Brooks), a research grant from SAP and a gift from Google.[1]

Results

The paper shows edits on real photographs and artworks, including replacing objects, changing seasons and weather, replacing backgrounds, changing material attributes and converting the artistic medium.[1] Its figures include the Mona Lisa rendered as a Modigliani painting or a marble Roman sculpture, the Beatles' Abbey Road cover moved to other cities and eras, and Vermeer's Girl with a Pearl Earring given a range of edits.[1] Applying the model repeatedly with different instructions produces compounded edits, and varying the latent noise produces different plausible edits for the same image and instruction.[1] Despite training only on synthetic images and generated instructions, the authors reported zero-shot generalization to real images and human-written instructions.[1]

The authors compared their model qualitatively with SDEdit and Text2Live, giving the baselines the "after-edit" caption because they expect descriptions rather than instructions, and quantitatively with SDEdit.[1] The quantitative comparison plotted two competing metrics: cosine similarity of CLIP image embeddings between input and output (consistency with the input), and directional CLIP similarity (agreement between the change in the images and the change in the captions).[1] For the same directional similarity, InstructPix2Pix had notably higher CLIP image similarity than SDEdit.[1] Qualitatively, the authors found that SDEdit worked reasonably when content stayed constant and only style changed but struggled to preserve identity and isolate individual objects, while Text2Live produced convincing results for edits involving additive layers but was limited by its formulation in the categories of edits it could handle.[1]

Ablations showed that training on a smaller dataset (10% or 1% of the data) reduced the model's ability to make larger edits, leaving mostly subtle or stylistic changes, while removing the CLIP filtering from data generation reduced consistency with the input image.[1]

Limitations

The authors listed several limitations. The model is bounded by the visual quality of the generated dataset and therefore by Stable Diffusion, and its ability to associate visual changes with instructions is limited by the 700 human-written examples, by GPT-3's instruction and caption generation, and by Prompt-to-Prompt's ability to modify images.[1] The paper reports that the model struggles with counting objects and with spatial reasoning, such as moving an object to one side of the image or swapping positions.[1] Its failure cases include an inability to change the viewpoint ("Zoom into the image"), excessive changes, failure to isolate the specified object, and difficulty reorganizing or swapping objects.[1] The authors also noted that the model may inherit biases from its data and pretrained models, illustrating this with correlations between profession and gender when asked to make people look like flight attendants or doctors.[1] The repository adds that the Stable Diffusion autoencoder handles small faces poorly and recommends cropping so the face fills more of the frame.[4]

Release

The official PyTorch implementation, based on the CompVis Stable Diffusion codebase, includes a command-line editing script, a Gradio editing app, training code, and scripts to reproduce the dataset generation pipeline.[4] The code is released under the MIT License, and the weights on Hugging Face (timbrooks/instruct-pix2pix) are also listed under the MIT license.[5][6] A browser demo runs as a Hugging Face Space, and the README also points to a hosted API on Replicate and to the third-party Imaginairy package.[4][19]

For the public dataset release, the authors additionally filtered prompts and images for NSFW content, leaving 451,990 GPT-3 generated examples.[4] Two image versions are available:[4]

Dataset versionImage editing examplesSize
random-sample-dataset451,990727 GB
clip-filtered-dataset313,010436 GB

Expanded article table

The CLIP-filtered version is also hosted on Hugging Face as timbrooks/instructpix2pix-clip-filtered.[18] The 700 human-written examples and the generated captions and instructions are released as JSONL files.[4]

Diffusers integration

Hugging Face added a StableDiffusionInstructPix2PixPipeline to Diffusers in version 0.12.0, released on 25 January 2023 under the title "Instruct-Pix2Pix, DiT, LoRA"; the release notes describe the model as "a Stable Diffusion model fine-tuned for editing images from human instructions".[7] The pipeline exposes the two guidance scales as image_guidance_scale and guidance_scale.[7] Diffusers also ships a training script, train_instruct_pix2pix.py, which the maintainers describe as faithful to the original implementation but tested only on a small dataset, along with an equivalent script for Stable Diffusion XL.[8] Hugging Face published an SDXL-based model trained on the CLIP-filtered dataset, diffusers/sdxl-instructpix2pix-768.[9]

In May 2023 a Hugging Face blog post by Sayak Paul applied the InstructPix2Pix training recipe to task-specific instruction datasets for cartoonization and low-level image processing (deraining, denoising, low-light enhancement and deblurring), reporting that the pretrained InstructPix2Pix model gave unsatisfactory cartoonization results and that the fine-tuned cartoonization model matched the target outputs more faithfully. The post also reported that the low-level model gave good deraining results but fell short on low-light enhancement and deblurring, which it suggested might be due to too few training examples for those tasks or to training that could be improved.[10]

Influence and follow-up work

Several later instruction-based editing papers used InstructPix2Pix as a baseline, as a checkpoint to fine-tune, or as a source of training data.[11][13][14]

WorkDateRelation to InstructPix2Pix
HIVEMarch 2023Added human feedback and a reward model for instructional editing, with a new 1.1M-example training set; MagicBrush describes it as providing more training triplets and human ranking results[12][11]
MagicBrushJune 2023Manually annotated dataset of over 10,000 triplets; fine-tuned InstructPix2Pix on it[11]
MGIESeptember 2023Used the InstructPix2Pix dataset for pre-training and InstructPix2Pix as its baseline, adding a multimodal LLM to derive more expressive instructions[13]
Emu Edit (GenAI, Meta)November 2023Multi-task editing model evaluated against InstructPix2Pix and MagicBrush[14]

Expanded article table

The MagicBrush authors argued that automatically synthesized training datasets contain a high volume of noise. In their benchmark the off-the-shelf InstructPix2Pix checkpoint was not competitive with other baselines, but after fine-tuning on MagicBrush it improved across all metrics and was preferred by human evaluators; in their qualitative examples the original model followed instructions but produced excessive modification and lacked photorealism.[11] MGIE's authors argued that the CLIP text encoder InstructPix2Pix uses is trained on static descriptions, which makes it hard to capture the visual transformation an edit requires, that brief instructions can be too ambiguous to guide the edit, and that because the InstructPix2Pix dataset contains no removal or photo-optimization edits, InstructPix2Pix failed on those operations.[13]

Meta's Emu Edit paper described InstructPix2Pix as having introduced an instructable image editing model, while arguing that such models often struggle to interpret and execute instructions and generalize poorly to tasks slightly outside their training data.[14] It also criticized the InstructPix2Pix benchmark as biased because it relies on Stable Diffusion generated input images and GPT-3 generated instructions.[14] In Emu Edit's human evaluation on its own test set, raters preferred Emu Edit over InstructPix2Pix in 77.33% of comparisons for text alignment and 76.71% for image faithfulness.[14] Emu Edit's authors also compared their data generation pipeline with InstructPix2Pix's Prompt-to-Prompt and CLIP-filtering approach and reported that theirs outperformed it on all the tasks in that comparison.[14]

Authors

Brooks and Holynski are listed as equal contributors.[1] At the time of the paper, Brooks was a PhD student at Berkeley AI Research; according to his personal website (as of November 2026) he later co-led Sora at OpenAI, led a team working on omnimodels at Google DeepMind, and leads media generation at Meta Superintelligence Labs.[16] Holynski's website says he was previously a postdoctoral researcher at UC Berkeley with Efros and Angjoo Kanazawa, and describes him (as of November 2026) as a professor at Columbia University who leads a Google DeepMind team working on generative 3D and world models.[17]

Continue reading

More from See also

More to read

Topics:Computer VisionDiffusion ModelsGenerative AIImage Generation

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46Tim Brooks, Aleksander Holynski, Alexei A. Efros. "InstructPix2Pix: Learning to Follow Image Editing Instructions". arXiv:2211.09800, 17 November 2022 (v2, 18 January 2023). arxiv.org/...2211.09800
  2. ^Tim Brooks, Aleksander Holynski, Alexei A. Efros. "InstructPix2Pix: Learning To Follow Image Editing Instructions". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 18392-18402. doi:10.1109/CVPR52729.2023.01764. openaccess.thecvf.com/...tructions_CVPR_2023_paper
  3. ^CVPR 2023. "InstructPix2Pix: Learning To Follow Image Editing Instructions" (Highlight, poster page). cvpr2023.thecvf.com, 2023. cvpr2023.thecvf.com/...21296
  4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Tim Brooks, Aleksander Holynski, Alexei A. Efros. "InstructPix2Pix: Learning to Follow Image Editing Instructions" (README). GitHub, timothybrooks/instruct-pix2pix. github.com/...instruct-pix2pix
  5. ^Timothy Brooks, Aleksander Holynski, Alexei A. Efros. "LICENSE" (MIT License). GitHub, timothybrooks/instruct-pix2pix, 2023. github.com/...LICENSE
  6. ^Tim Brooks. "timbrooks/instruct-pix2pix" (model card). Hugging Face. huggingface.co/...instruct-pix2pix
  7. ^1 ^2 ^3Hugging Face. "v0.12.0: Instruct-Pix2Pix, DiT, LoRA" (release notes). GitHub, huggingface/diffusers, 25 January 2023. github.com/...v0.12.0
  8. ^Hugging Face. "InstructPix2Pix training example". GitHub, huggingface/diffusers. github.com/...README.md
  9. ^Hugging Face. "SDXL InstructPix2Pix (768768)" (model card, diffusers/sdxl-instructpix2pix-768). Hugging Face, 2023. huggingface.co/...sdxl-instructpix2pix-768
  10. ^Sayak Paul. "Instruction-tuning Stable Diffusion with InstructPix2Pix". Hugging Face blog, 23 May 2023. huggingface.co/...instruction-tuning-sd
  11. ^1 ^2 ^3 ^4Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, Yu Su. "MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing". arXiv:2306.10012 (NeurIPS 2023), 16 June 2023. arxiv.org/...2306.10012
  12. ^Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, Ran Xu. "HIVE: Harnessing Human Feedback for Instructional Visual Editing". arXiv:2303.09618 (CVPR 2024), 16 March 2023. arxiv.org/...2303.09618
  13. ^1 ^2 ^3Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, Zhe Gan. "Guiding Instruction-based Image Editing via Multimodal Large Language Models". arXiv:2309.17102 (ICLR 2024), 29 September 2023. arxiv.org/...2309.17102
  14. ^1 ^2 ^3 ^4 ^5 ^6Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, Yaniv Taigman. "Emu Edit: Precise Image Editing via Recognition and Generation Tasks". arXiv:2311.10089, 16 November 2023. arxiv.org/...2311.10089
  15. ^Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros. "Image-to-Image Translation with Conditional Adversarial Networks". arXiv:1611.07004 (CVPR 2017), 21 November 2016. arxiv.org/...1611.07004
  16. ^Tim Brooks. "Tim Brooks" (personal website). timothybrooks.com, accessed November 2026. timothybrooks.com
  17. ^Aleksander Holynski. "Aleksander Holynski" (personal website). holynski.org, accessed November 2026. holynski.org
  18. ^Tim Brooks. "timbrooks/instructpix2pix-clip-filtered" (dataset card). Hugging Face. huggingface.co/...instructpix2pix-clip-filtered
  19. ^Tim Brooks. "InstructPix2Pix" (Hugging Face Space, timbrooks/instruct-pix2pix). Hugging Face. huggingface.co/...instruct-pix2pix

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

1 revision · v1 · 3,213 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 92 claims checked against 26 cited and independent sources; 0 material and 6 minor corrections applied before publication.

Cite this page: AI Wiki. "InstructPix2Pix." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/instructpix2pix

Edit article