pix2pix

pix2pix is a supervised image-to-image translation method that learns a mapping between two visual domains from aligned input-output image pairs. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros introduced the method in a November 2016 preprint and presented it at CVPR 2017 under the title "Image-to-Image Translation with Conditional Adversarial Networks."^[1][2] The name also refers to the authors' accompanying software.

The system combines a conditional generative adversarial network with a pixelwise L1 loss. A U-Net-style generator predicts the output image, while a PatchGAN discriminator judges whether local patches from an input-output pair are real or generated. The paper applied this same design to semantic labels and street scenes, architectural labels and facade photographs, maps and aerial photographs, grayscale colorization, edges and object photographs, day and night scenes, thermal images, and inpainting.^[1][2]

pix2pix is specifically a paired method. Each source image must have a corresponding target image depicting the same underlying scene or object in spatial alignment. CycleGAN, published later in 2017 by an overlapping group of authors, addresses a different setting in which the two training collections are not paired.^[11]

Background and purpose

A conventional pixel prediction network can be trained with a fixed distance such as L1 or L2. On an ambiguous task, however, minimizing a per-pixel distance favors an average of plausible targets. The result may have the correct coarse layout but blurred edges, muted color, or missing texture. pix2pix uses the discriminator as a learned loss that responds to image structure found in the target domain. The L1 term remains in the objective to keep the generated image aligned with its paired target.^[1]

The method builds on the adversarial framework introduced by Goodfellow and colleagues in 2014 and the later conditional GAN formulation, in which extra information is supplied to both sides of the adversarial game.^[6][7] In pix2pix, that condition is an image rather than a class label or text description. The generator sees the source image and predicts its translation. The discriminator sees the source beside either the real target or the generated target, so it can judge both realism and compatibility with the source.

This arrangement differs from an unconditional GAN attached to an image predictor. If the discriminator sees only the output, it can reward a plausible target-domain image that does not match the input. In the paper's Cityscapes ablation, removing the input from the discriminator led the generator to produce nearly the same output for different source images. Conditioning the discriminator or adding L1 restored pressure to respect the input.^[1][2]

Conditional adversarial objective

Let (x) be an input image, (y) its paired target, (z) a nominal noise source, (G) the generator, and (D) the discriminator. The conditional adversarial loss is

$$\mathcal{L}_{cGAN}(G,D) = \mathbb{E}_{x,y}[\log D(x,y)] + \mathbb{E}_{x,z}[\log(1-D(x,G(x,z)))].$$

The discriminator maximizes this expression by assigning high probability to real pairs ((x,y)) and low probability to generated pairs ((x,G(x,z))). The generator minimizes it by producing translations that the discriminator classifies as real. The paired reconstruction term is

$$\mathcal{L}_{L1}(G) = \mathbb{E}_{x,y,z}[\lVert y-G(x,z)\rVert_1].$$

The complete objective is

$$G^* = \arg\min_G\max_D \mathcal{L}_{cGAN}(G,D) + \lambda\mathcal{L}_{L1}(G).$$

The reported combined experiments used (\lambda=100).^[1][2] The two terms have different roles. L1 strongly penalizes disagreement with the paired target and usually preserves low-frequency layout, but it tends to average uncertain details. The adversarial term favors outputs whose local statistics resemble the target domain. In the authors' ablations, L1 alone was smoother, a cGAN alone was sharper but more prone to artifacts, and the combined loss usually balanced correspondence with sharpness.^[1]

The implementation used the non-saturating generator update suggested in the original GAN paper: instead of directly minimizing (\log(1-D(x,G(x,z)))), the generator maximized (\log D(x,G(x,z))). Training alternated one update to the discriminator with one update to the generator. The authors also divided the discriminator objective by two, slowing its updates relative to the generator.^[2][6]

Network architecture

U-Net generator

The default generator is an eight-stage encoder and eight-stage decoder adapted from a U-Net. Each encoder stage reduces spatial resolution, while the decoder restores it. Skip connections concatenate the activation at each encoder level with the corresponding decoder level. These paths allow positions, contours, and other low-level information shared by source and target images to bypass the central bottleneck.^[1][8]

For 256 by 256 inputs, the appendix denotes the encoder as C64-C128-C256-C512-C512-C512-C512-C512. Here Ck means a convolution, batch normalization, and ReLU block with (k) filters. The decoder is CD512-CD1024-CD1024-C1024-C1024-C512-C256-C128; CDk adds 50 percent dropout. A final convolution maps the features to the required output channels and applies tanh. The layers use 4 by 4 filters with stride 2. Encoder activations are leaky ReLUs with slope 0.2, while decoder activations use ordinary ReLUs.^[2][4]

The paper compared this generator with an otherwise matched encoder-decoder whose skip connections were removed. On Cityscapes labels-to-photos, the U-Net produced higher FCN scores under both L1-only and L1-plus-cGAN training. The comparison supports the value of carrying aligned spatial information around the bottleneck rather than forcing every detail through it.^[2]

PatchGAN discriminator

PatchGAN is a fully convolutional discriminator that evaluates overlapping local regions instead of assigning one real-or-fake score to an entire image. The standard model has a 70 by 70 pixel receptive field. It runs across the concatenated source and target images and averages its patch responses for the final discriminator output.^[1]

The reported 70 by 70 architecture is C64-C128-C256-C512, followed by a one-channel convolution and sigmoid. Its convolutions use 4 by 4 filters. The first layer omits batch normalization, and all discriminator ReLUs are leaky with slope 0.2.^[2][4]

The design divides responsibility by spatial frequency. L1 enforces coarse agreement across the whole output. PatchGAN concentrates adversarial capacity on edges, texture, and other high-frequency patterns within each receptive field. The authors described this as a Markov assumption: pixels separated by more than a patch diameter are treated as conditionally independent by the discriminator.^[1]

The paper varied the receptive field from a 1 by 1 PixelGAN through 16 by 16 and 70 by 70 PatchGANs to a 286 by 286 ImageGAN. The 1 by 1 model changed color statistics but could not enforce spatial sharpness. The 16 by 16 model sharpened results but introduced tiling. The 70 by 70 model reduced those artifacts and achieved the best or tied-best Cityscapes FCN measurements. Expanding to the full image did not improve visual quality and lowered the reported FCN score.^[1][2]

Scroll sideways for more →

These figures come from the Cityscapes label-to-photo patch-scale ablation and should not be treated as general scores for every application.^[2]

Treatment of randomness

The mathematical formulation includes a noise variable (z), but the authors found that the generator learned to ignore an explicitly supplied Gaussian noise vector. Their final models instead kept dropout active in several decoder layers during both training and inference. Even with that change, the paper reported only minor variation among outputs for the same input.^[1][2]

This behavior means that ordinary pix2pix functions mainly as a point predictor. It does not reliably sample the full range of plausible colors, textures, lighting conditions, or object appearances that could match one source image. A later controlled study in the BicycleGAN paper added noise to pix2pix and measured an LPIPS diversity distance of 0.013 on the maps-to-satellite task, compared with 0.265 for random pairs of real target images. The study found that the baseline produced realistic single outputs but almost no meaningful variation.^[12]

Training procedure and implementation

The reported optimizer was Adam with learning rate 0.0002, (\beta_1=0.5), and (\beta_2=0.999). Most image-pair experiments used batch size 1, although colorization, edge translation, day-to-night translation, thermal translation, and inpainting used batch size 4. The generator architecture experiment used batch size 10 because the plain encoder-decoder could not bypass a batch-normalized bottleneck.^[2]

Most 256 by 256 training pairs were resized to 286 by 286 and randomly cropped back to 256 by 256. The authors also mirrored images for several tasks. Networks were trained from scratch, with weights initialized from a Gaussian distribution having mean 0 and standard deviation 0.02.^[2] These settings established the commonly reproduced "scale width and crop" pix2pix preprocessing recipe.

The original release was written in Lua using Torch and CUDA. It includes training and test scripts, data preparation utilities, pretrained model download scripts, and code for the Cityscapes evaluation.^[3][4] A later official PyTorch repository implements pix2pix and CycleGAN in a shared codebase. Its maintainers state that the PyTorch implementation gives comparable or better results, but direct users to the original Torch repository when exact reproduction of the paper is required.^[5]

Because pix2pix is fully convolutional, the authors could apply the trained networks to images larger than the training crop. On the maps task, a generator trained with 256 by 256 images was run convolutionally on 512 by 512 inputs. This demonstration did not remove the model's reliance on local training statistics, nor did it constitute native training at high resolution.^[1]

Datasets and tasks

The experiments were designed to test whether one architecture and objective could cover tasks that had previously used application-specific losses. Every training example was either collected as an aligned pair or transformed into one. For grayscale colorization, for example, the grayscale input was derived from the target color image. For edges-to-object translation, HED edge detection and postprocessing produced edges from the corresponding photographs.^[2][14][15]

Scroll sideways for more →

The authors noted that the 400-image Facades model trained in about two hours on one Pascal Titan X GPU and that inference took less than one second on the same hardware.^[2][3] Those measurements describe a 2016-era implementation and particular datasets rather than a hardware-independent speed claim.

Evaluation and reported experiments

The paper used both task-specific metrics and human judgments. For label-to-street translation, generated photographs were passed through an FCN-8s semantic segmentation model trained on real Cityscapes images. If the classifier could recover the original semantic labels, the synthetic image received a higher FCN score. This measured whether objects were recognizable to one pretrained network, not whether every visual detail was correct.^[1]

The objective ablation found that the full L1 plus conditional GAN model reached 0.66 per-pixel accuracy and 0.17 class IoU, compared with 0.42 and 0.11 for L1 alone. A conditional GAN without L1 reached 0.57 and 0.16. An unconditional GAN without L1 performed poorly because its output could ignore the source.^[1][2]

Scroll sideways for more →

For perceptual testing, Amazon Mechanical Turk participants saw a real and a generated image for one second each and identified the fake. On maps-to-aerial translation, L1 plus cGAN outputs were labeled real in 18.9 percent of trials, versus 0.8 percent for L1 alone. In the reverse photo-to-map direction, the combined model scored 6.1 percent versus 2.8 percent for L1, and the paper reported that this difference was not statistically significant under its bootstrap test.^[1]

On grayscale colorization, pix2pix outputs were labeled real in 22.5 percent of trials. A colorization method specialized for that task scored 27.8 percent in the same study, while its L2 regression variant scored 16.3 percent. The comparison showed that one shared pix2pix configuration could be competitive without surpassing a task-specific system.^[1]

The reverse Cityscapes task also exposed a boundary of the adversarial loss. For photograph-to-label segmentation, plain L1 achieved 0.86 per-pixel accuracy and 0.35 class IoU. L1 plus cGAN scored 0.83 and 0.29, while cGAN alone scored 0.74 and 0.22. The paper concluded that direct reconstruction losses can be sufficient when the target is a relatively unambiguous label map rather than a detailed photograph.^[1]

Limitations

The main data limitation is the requirement for aligned pairs. Building such a dataset can require synchronized sensors, annotation, controlled capture, or a deterministic transformation from target to input. Small misregistration is directly penalized by L1 even when the generated content is otherwise plausible. CycleGAN was motivated by cases in which corresponding pairs are unavailable and should not be described as a version or mode of pix2pix.^[11]

Output diversity is also limited. The generator ignored direct noise in the authors' tests, and dropout yielded only minor stochasticity. As a result, one-to-many tasks are compressed toward a single answer, even when many target images would be valid.^[2][12]

PatchGAN enforces local plausibility rather than complete scene consistency. A 70 by 70 classifier cannot directly test relationships outside its receptive field. The adversarial term may therefore create sharp but incorrect local structures. In the paper's qualitative failure set, sparse or unusual inputs produced artifacts, and its semantic-label examples sometimes contained small hallucinated objects.^[1][2]

The original system was evaluated mainly at 256 by 256 training resolution. Its successful convolutional application to 512 by 512 maps did not show that it could synthesize stable fine detail at megapixel scale. pix2pixHD later targeted 2048 by 1024 semantic image synthesis with a coarse-to-fine generator, multiscale discriminators, and a discriminator feature-matching loss.^[13]

The evaluation itself was heterogeneous. FCN score depends on a particular recognition network, while human fooling rates depend on display time, sampling, and the forced-choice protocol. Neither measure establishes pixel accuracy, perceptual realism, diversity, and downstream usefulness at once. The low fooling rates and task-specific reversals in the ablations are therefore important context for the qualitative examples.^[1]

Reproducibility

The full arXiv version records architecture strings, initialization, preprocessing, optimizer settings, data sizes, schedules, test splits, and an erratum.^[2] The public Torch repository supplies the corresponding model and evaluation source.^[4] Together these materials make the main configuration unusually explicit, although reproducing every experiment still requires the original datasets and their processing steps.

The erratum concerns batch normalization at the one-by-one bottleneck. In paper experiments with batch size 1, batch normalization zeroed that layer's activations. The U-Net could bypass the bottleneck through its skip connections, but the plain encoder-decoder could not. The released code removed batch normalization from the bottleneck. The authors reported little visible difference after the fix and left the published experimental results unchanged.^[2][4]

Later implementations are not byte-for-byte reproductions. Framework defaults, normalization behavior, random seeds, image decoding, and updated preprocessing can change results. The maintained PyTorch code remains useful for current experiments, while its documentation explicitly distinguishes comparable modern results from reproduction with the original Torch release.^[5]

Influence and later work

pix2pix became a common paired baseline for later computer vision research on conditional image synthesis. Its combination of an encoder-decoder with skip connections, a local conditional discriminator, and a reconstruction term was reused and modified across translation tasks. The authors' project page also collected ports and creative applications built on the released code.^{[3][11][12][13]}

Several closely related systems addressed limitations identified in the paper. CycleGAN removed the paired-data requirement by learning forward and inverse mappings with cycle consistency, at the cost of changing the supervision assumptions.^[11] BicycleGAN kept paired supervision but tied a latent code to output appearance so that one input could yield diverse samples.^[12] pix2pixHD scaled semantic synthesis to 2048 by 1024 and added multiscale adversarial and feature-matching components.^[13] These are later methods, not components of the original pix2pix architecture.

Although newer GAN and diffusion model systems offer other ways to condition image generation, the published pix2pix formulation remains a useful definition of supervised paired translation: a conditional adversarial loss for realism, L1 for correspondence, a U-Net generator for spatial transfer, and a PatchGAN discriminator for local detail.^[1][5]

References

1. ^Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. "Image-to-Image Translation with Conditional Adversarial Networks". Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
2. ^Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. "Image-to-Image Translation with Conditional Adversarial Networks," arXiv:1611.07004, version 3 with appendix and errata. Submitted November 21, 2016; revised November 26, 2018.
3. ^Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Official pix2pix project page. University of California, Berkeley.
4. ^Phillip Isola and contributors. phillipi/pix2pix. Original Torch implementation, model definitions, training code, data utilities, and evaluation scripts.
5. ^Jun-Yan Zhu, Taesung Park, Tongzhou Wang, and contributors. junyanz/pytorch-CycleGAN-and-pix2pix. Official PyTorch implementation and documentation.
6. ^Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative Adversarial Nets". Advances in Neural Information Processing Systems 27, 2014.
7. ^Mehdi Mirza and Simon Osindero. "Conditional Generative Adversarial Nets". arXiv:1411.1784, 2014.
8. ^Olaf Ronneberger, Philipp Fischer, and Thomas Brox. "U-Net: Convolutional Networks for Biomedical Image Segmentation". Medical Image Computing and Computer-Assisted Intervention, 2015.
9. ^Marius Cordts et al. "The Cityscapes Dataset for Semantic Urban Scene Understanding". Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
10. ^Radim Tyleček and Radim Sára. "Spatial Pattern Templates for Recognition of Objects with Regular Structure". Pattern Recognition, GCPR 2013, pages 364-374.
11. ^Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks". Proceedings of the IEEE International Conference on Computer Vision, 2017.
12. ^Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman. "Toward Multimodal Image-to-Image Translation". Advances in Neural Information Processing Systems 30, 2017.
13. ^Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. "High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs". Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
14. ^Olga Russakovsky et al. "ImageNet Large Scale Visual Recognition Challenge". International Journal of Computer Vision 115, 2015, pages 211-252.
15. ^Saining Xie and Zhuowen Tu. "Holistically-Nested Edge Detection". Proceedings of the IEEE International Conference on Computer Vision, 2015.