Citation and evidence

MoCo (Momentum Contrast)

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

Computer VisionDeep LearningMachine LearningMeta AI

Cite this article

MoCo (Momentum Contrast) is a family of self-supervised learning methods for learning visual representations from unlabeled images, developed at Facebook AI Research (FAIR, now Meta FAIR). The original method, introduced by Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie and Ross Girshick in November 2019 and published at CVPR 2020,[1][4] treats contrastive learning as a dictionary look-up and builds that dictionary from a queue of encoded samples and a slowly updated "momentum" encoder.[1] Two follow-ups kept the name: MoCo v2 (March 2020) added an MLP projection head and stronger data augmentation borrowed from SimCLR,[2] and MoCo v3 (April 2021, ICCV 2021) dropped the queue and was used to study self-supervised training of vision transformers.[3]

The first paper reported that a ResNet-50 pre-trained with MoCo, without labels, could outperform its ImageNet-supervised counterpart when fine-tuned on seven detection and segmentation tasks, and its authors wrote that the gap between unsupervised and supervised representation learning "has been largely closed in many vision tasks."[1] Its momentum encoder, a second copy of the network whose weights follow a moving average of the trained network, has counterparts in later non-contrastive methods such as BYOL and DINO, whose papers compare their designs with MoCo's.[11][13]

Background

When MoCo appeared, unsupervised representation learning had been highly successful in natural language processing, for example in GPT and BERT, while supervised pre-training on ImageNet was still dominant in computer vision.[1] The MoCo paper suggested that one reason was the signal space: language has discrete tokens from which a dictionary can be built, whereas images are continuous and high-dimensional.[1]

Several earlier methods had trained image encoders with contrastive losses. The MoCo authors described all of them as building dynamic dictionaries: an encoded "query" should be similar to its matching "key" and dissimilar to the other keys.[1] They argued that a good dictionary should be (i) large, so that it samples the continuous visual space well, and (ii) consistent, meaning that its keys are produced by the same or a similar encoder so that comparisons with the query are meaningful.[1] Existing approaches traded one property for the other:[1]

MechanismHow keys are obtainedLimitation identified in the MoCo paper
End-to-endKeys come from the current mini-batch and are encoded with back-propagationDictionary size is tied to the mini-batch size, which GPU memory limits; large-batch optimization is itself difficult
Memory bank (Wu et al., 2018)Keys are sampled from stored representations of every image in the datasetEach stored representation was computed at a different point in the past epoch, so keys are inconsistent; storing every sample can be intractable at billion-image scale
MoCoKeys come from a queue of recent mini-batches, encoded by a momentum-updated key encoderProposed as large and consistent at the same time

Expanded article table

Method (MoCo v1)

Contrastive loss

MoCo uses the InfoNCE loss. For an encoded query $q$, one positive key $k_+$ and $K$ negative keys, with similarity measured by dot product and a temperature $\tau$, the loss is:[1]

Lq=−log⁡exp⁡(q⋅k+/τ)∑i=0Kexp⁡(q⋅ki/τ)\mathcal{L}_q = -\log \frac{\exp(q \cdot k_+ / \tau)}{\sum_{i=0}^{K} \exp(q \cdot k_i / \tau)}

The sum runs over the positive key and the $K$ negatives, so the loss is the log loss of a $(K+1)$-way softmax classifier that tries to pick $k_+$ out of the dictionary.[1] The query comes from a query encoder $f_q$ and the keys from a key encoder $f_k$.[1]

Pretext task

MoCo is a mechanism for building dictionaries rather than a new pretext task. The paper used instance discrimination: two random augmentations ("views") of the same image form a positive pair, and views of different images are negatives.[1] For each mini-batch, the queries and their matching keys form the positive pairs, and the negatives come from the queue.[1]

Dictionary as a queue

The dictionary is a first-in, first-out queue of encoded keys. After each training step, the keys of the current mini-batch are added and the oldest mini-batch is removed.[1] Because the queue holds keys from several preceding mini-batches, its size is a hyper-parameter set independently of the batch size and can be much larger than it.[1] Dropping the oldest keys first also removes the keys that are least consistent with the current encoder.[1]

Momentum update

A queue makes it impractical to train the key encoder with back-propagation, because gradients would have to flow to every sample in the queue. Simply copying the query encoder's weights into the key encoder at each step gave poor results in the authors' experiments, which they attributed to the keys changing too quickly.[1] MoCo instead updates the key encoder's parameters $\theta_k$ as a moving average of the query encoder's parameters $\theta_q$:[1]

θk←m θk+(1−m) θq\theta_k \leftarrow m\,\theta_k + (1 - m)\,\theta_q

Here $m \in [0, 1)$ is the momentum coefficient, and only $\theta_q$ is updated by back-propagation.[1] With a large $m$ the key encoder evolves slowly, so keys encoded in different mini-batches stay comparable. The ablation in the paper (ResNet-50, $K$ = 4,096, ImageNet linear classification) shows how much this matters:[1]

Momentum $m$00.90.990.999 (default)0.9999
Top-1 accuracy (%)Fails (loss oscillates)55.257.859.058.9

Expanded article table

Shuffling batch normalization

The authors found that standard batch normalization let the model "cheat" the pretext task and reach a low loss without learning good representations, possibly because batch statistics leak information between samples.[1] Training on several GPUs with per-GPU batch normalization, they shuffled the order of samples before distributing them for the key encoder and shuffled them back afterwards, so that a query and its positive key were normalized with statistics from different subsets of the batch.[1]

Training details

The encoder in the paper is a ResNet whose final fully connected layer outputs a 128-dimensional vector, normalized to unit L2 norm; the temperature is 0.07.[1] Augmentation takes a 224x224 crop from a randomly resized image, followed by random color jittering, horizontal flipping and grayscale conversion.[1]

SettingImageNet-1MInstagram-1B
DataAbout 1.28 million ImageNet training imagesAbout 940 million public Instagram images from about 1,500 hashtags related to ImageNet categories
Hardware and batch8 GPUs, mini-batch 25664 GPUs, mini-batch 1,024
OptimizerSGD, weight decay 0.0001, momentum 0.9Same
Learning rate0.03, multiplied by 0.1 at epochs 120 and 1600.12, decayed by 0.9x every 62,500 iterations
Length200 epochs, about 53 hours for ResNet-501.25 million iterations (about 1.4 epochs), about 6 days for ResNet-50

Expanded article table

Source for the table: He et al. (2019).[1]

Results of the original paper

Linear classification on ImageNet

Under the common linear protocol, a linear classifier is trained on frozen features and evaluated on the ImageNet validation set. With $K$ = 65,536 and $m$ = 0.999, MoCo reached the following top-1 accuracies:[1]

BackboneFeature-extractor parametersTop-1 accuracy (%)
ResNet-5024M60.6
ResNeXt-5046M63.9
ResNet-50 (2x wider)94M65.4
ResNet-50 (4x wider)375M68.6

Expanded article table

The paper reported the ResNet-50 figure as better than all compared methods of similar model size, and noted that it used a standard architecture with no patchified inputs or tailored receptive fields.[1] In a controlled comparison using the same pretext task and loss, all three mechanisms improved with a larger dictionary; the end-to-end variant tracked MoCo for small $K$ but could not go past a batch of 1,024 on an 8-GPU machine, and the memory bank was 2.6 points behind MoCo.[1]

Transfer to downstream tasks

The paper's main argument was about transfer. Pre-trained models were fine-tuned end to end with the same schedules and hyper-parameters as the supervised baseline, using Detectron2-based implementations.[1] Selected results (ResNet-50):[1]

Task (model)MetricImageNet supervisedMoCo, ImageNet-1MMoCo, Instagram-1B
PASCAL VOC detection (Faster R-CNN, R50-C4)AP / AP7553.5 / 58.855.9 / 62.657.2 / 63.7
COCO detection (Mask R-CNN, R50-FPN, 2x schedule)Box AP40.640.841.1
COCO keypoint detectionAP65.866.866.9
COCO dense pose estimationAP7550.653.954.3
LVIS v0.5 instance segmentationMask AP24.4 (frozen batch norm)24.124.9
Cityscapes semantic segmentationmIoU74.675.375.5
VOC semantic segmentationmIoU74.472.573.6

Expanded article table

Counting object detection on VOC and COCO, instance segmentation on COCO and LVIS, keypoint detection and dense pose on COCO, and semantic segmentation on Cityscapes, the authors reported seven tasks where MoCo beat supervised pre-training.[1] MoCo was on par for Cityscapes instance segmentation and behind on VOC semantic segmentation, which the paper called "a negative case we have observed."[1] MoCo pre-trained on the Instagram data was consistently better than MoCo pre-trained on ImageNet, though the authors described the improvement as "relatively small" and suggested the larger data was not fully exploited.[1] Fine-tuning end to end on ImageNet classification itself, the Instagram-pre-trained ResNet-50 reached 77.3% against 76.5% for training from scratch.[1]

MoCo v2

Improved Baselines with Momentum Contrastive Learning, a two-page technical report by Xinlei Chen, Haoqi Fan, Ross Girshick and Kaiming He, appeared on arXiv on 9 March 2020, a few weeks after SimCLR.[2][10] It tested two of SimCLR's design choices inside the MoCo framework: replacing MoCo's fully connected output layer with a 2-layer MLP projection head (2,048-dimensional hidden layer with ReLU), and adding blur to the augmentations.[2] The MLP head is used only during pre-training.[2] With the head, the best temperature rose from 0.07 to 0.2.[2] The report also adopted a cosine learning rate schedule.[2]

Configuration (ResNet-50)Pre-training epochsImageNet linear top-1 (%)VOC detection AP50 / AP / AP75
ImageNet supervisedn/a76.581.3 / 53.5 / 58.8
MoCo v120060.681.5 / 55.9 / 62.6
+ MLP head20066.282.0 / 56.4 / 62.6
+ extra augmentation20063.482.2 / 56.8 / 63.2
+ MLP head + augmentation20067.382.5 / 57.2 / 63.9
+ cosine schedule (MoCo v2)20067.582.4 / 57.0 / 63.6
MoCo v280071.182.5 / 57.4 / 64.0

Expanded article table

Source for the table: Chen et al. (2020).[2] The report noted that augmentation alone gave better detection results than the MLP head alone despite much lower linear accuracy, concluding that linear classification accuracy "is not monotonically related to transfer performance in detection."[2]

The comparison with SimCLR was the report's headline. With 200 epochs and a batch of 256, MoCo v2 scored 67.5%, against 61.9% for SimCLR at the same epochs and batch size and 66.6% for SimCLR with a batch of 8,192; with 800 epochs, MoCo v2 reached 71.1%, against 69.3% for SimCLR after 1,000 epochs with a batch of 4,096.[2] The authors' point was cost: SimCLR's 4,096 to 8,192 batches relied on TPUs, while MoCo v2 ran on a typical 8-GPU machine.[2] On 8 V100 16 GB GPUs, MoCo with a batch of 256 used 5.0 GB of memory per GPU and 53 hours for 200 epochs, against 7.4 GB and 65 hours for an end-to-end setup at the same batch size; the authors estimated that an end-to-end batch of 4,096 would need 93.0 GB per GPU.[2] MoCo is cheaper at equal batch size because it back-propagates only through the query encoder.[2]

MoCo v3

An Empirical Study of Training Self-Supervised Vision Transformers by Xinlei Chen, Saining Xie and Kaiming He was posted on 5 April 2021 and published at ICCV 2021 as an oral paper.[3] The paper states that it "does not describe a novel method"; it introduces MoCo v3 as "an incremental improvement of MoCo v1/2" to serve as the framework for a study of how to train ViT models without labels.[3]

Changes from earlier versions

  • No queue. MoCo v3 uses the keys that already exist in the same batch, because the authors found the memory queue gave diminishing gains with a sufficiently large batch such as 4,096.[3]
  • Symmetrized loss. Each image yields two crops, each encoded by both encoders, and the loss is $\text{ctr}(q_1, k_2) + \text{ctr}(q_2, k_1)$.[3]
  • Prediction head. The query encoder is a backbone plus a projection head plus a prediction head; the momentum encoder has the backbone and projection head only.[3] The projection head is a 3-layer MLP and the prediction head a 2-layer MLP, both with 4,096-dimensional hidden layers, 256-dimensional outputs and batch normalization in every layer.[3]
  • Optimizer. AdamW with a batch of 4,096, a 40-epoch warmup and cosine decay; the temperature is 0.2.[3]

With a ResNet-50 and 800 epochs of pre-training, these changes raised ImageNet linear accuracy from 71.1% (MoCo v2) and 72.2% (an improved MoCo v2 implementation) to 73.8%, which the authors attributed mainly to the prediction head and the larger batch.[3]

Training instability

The paper's main finding was that self-supervised ViT training is often unstable in a way that does not show up as divergence. Unstable runs ended with "apparently good" results that were 1 to 3 points worse than they could have been.[3] With ViT-B/16, 100 epochs and AdamW, a batch of 2,048 gave 72.6% while a batch of 6,144 produced visible dips in the training curve and 69.7%.[3] Tracking gradients showed that spikes appeared first in the patch-projection layer, so the authors froze that layer at its random initialization.[3] This raised accuracy from 71.7% to 73.4% at a learning rate of 1.5e-4, and also helped SimCLR (+0.8 points), BYOL (+1.3) and SwAV (65.8% to 66.4%).[3] The authors described the trick as alleviating rather than solving the problem, writing that the instability concerns all layers.[3]

Results

ModelParametersImageNet linear probe (%)End-to-end fine-tuning (%)
ViT-B86M76.783.2
ViT-L304M77.684.1
ViT-H632M78.1n/a
ViT-BN-H (batch normalization added)632M79.1n/a
ViT-BN-L/7 (7x7 patches)304M81.0n/a

Expanded article table

Source for the table: Chen, Xie and He (2021).[3] In a like-for-like comparison with 300 epochs of pre-training, MoCo v3 reached 76.5% with ViT-B against 73.9% for SimCLR, 73.9% for BYOL and 71.6% for SwAV.[3] The authors reported that, unlike supervised ViT in the original ViT paper, larger self-supervised ViT models kept improving.[3] Ablations found that removing position embeddings entirely cost only 1.6 points (76.5% to 74.9%), which the authors read as both a strength and a sign that positional information was not being exploited.[3]

The models were trained in TensorFlow on Google Cloud TPU v3. ViT-B took 2.1 hours per 100 epochs on 256 TPUs, and the 300-epoch ViT-H run amounted to about 625 TPU-days.[3] A PyTorch GPU version took 24 hours for ViT-B on 128 GPUs.[3]

Version comparison

MoCo v1MoCo v2MoCo v3
Paper dateNovember 2019March 2020April 2021
VenueCVPR 2020Technical reportICCV 2021
NegativesQueue (65,536 keys in main results)QueueOther samples in the same batch
Output headSingle fully connected layer, 128-d2-layer MLP3-layer projection MLP plus 2-layer prediction MLP
LossInfoNCE, asymmetricInfoNCE, asymmetricInfoNCE, symmetrized
Temperature0.070.20.2
Default batch2562564,096
Backbones studiedResNet familyResNet-50ResNet-50 and ViT
ResNet-50 linear top-160.6% (200 epochs)71.1% (800 epochs)73.8% (800 epochs)

Expanded article table

Sources for the table: He et al. (2019), Chen et al. (2020), Chen, Xie and He (2021).[1][2][3]

Code and licensing

FAIR released a PyTorch implementation of MoCo v1 and v2 in the facebookresearch/moco repository, with an initial commit on 23 March 2020.[8] It shipped pre-trained ResNet-50 checkpoints for MoCo v1 (200 epochs, 60.6%) and MoCo v2 (200 epochs, 67.7%; 800 epochs, 71.1%) and scripts for transferring them to detection with Detectron2.[5][6] Over five runs on 8 V100 GPUs, the README reported 60.8 ± 0.2% for MoCo v1 and 67.5 ± 0.1% for MoCo v2 at 200 epochs.[5] The code was first released under CC BY-NC 4.0 and relicensed under the MIT License in January 2023, in a commit describing MIT as "more permissive than CC-BY-NC."[7] The README's license section was not updated and still named CC BY-NC 4.0.[5]

Later commits removed the code twice. A July 2024 commit titled "Remove scripts/kaiminghe; kaiminghe no longer works here" deleted all of the repository's files; it was reverted in August 2024.[8] On 3 February 2026 a commit titled "Remove ex-employee folders from fbcode/scripts" deleted every file again.[8] As of November 2026 the repository is still listed and not archived, but its main branch contains no files; the code remains reachable through the commit history.[8]

MoCo v3 has its own repository, facebookresearch/moco-v3, which re-implements the original TensorFlow and TPU code in PyTorch for GPUs and states that it reproduces the paper's results.[9] Its README lists 76.7% linear accuracy and 83.2% fine-tuning accuracy for ViT-Base, and 74.6% linear accuracy for a ResNet-50 pre-trained for 1,000 epochs.[9] That repository is licensed CC BY-NC 4.0 and has been archived as read-only.[9]

Influence

SimCLR was posted three months after the MoCo paper. Its authors listed MoCo among the methods that rely on a memory bank and trained without one, relying instead on in-batch negatives from large batches of up to 8,192 images.[10] MoCo v2 then showed that SimCLR's head and augmentation improvements transfer to the queue-based design.[2]

The momentum encoder outlived the queue. BYOL (DeepMind and Imperial College, June 2020) used a slow-moving average "target network" without any negative pairs; its authors contrasted this with MoCo, where the moving-average network kept negative representations consistent, and reported that adding a target network to SimCLR alone improved its accuracy by 1.6 points.[11] SimSiam (Chen and He, November 2020) described itself as "BYOL without the momentum encoder" and argued that the stop-gradient operation, not the momentum encoder, is what prevents collapse, while noting that the moving average may still improve accuracy.[12] DINO (2021) built its teacher network as an exponential moving average of the student, calling it a momentum encoder and citing MoCo, and noted that the role differed because DINO has no queue and no contrastive loss.[13]

Adaptations

Researchers have applied the MoCo design outside natural-image classification:

WorkDomainDescription
MoCo-CXR (2020; MIDL 2021)Medical imagingAdapted MoCo to pre-train chest X-ray models; reported the largest benefit when labeled data was limited[14]
VideoMoCo (CVPR 2021)VideoAdded adversarial frame dropping and a temporal decay on keys in the queue, evaluated on UCF101 and HMDB51[15]
NeuroMoCo (2024)Spiking neural networksA momentum contrast method for spiking networks[16]
SoftMCL (LREC-COLING 2024)TextMomentum contrastive pre-training for fine-grained sentiment[17]
Label-aware hard negative sampling (ACL 2024 Findings)TextMomentum contrastive learning for implicit hate speech detection[18]
MoMA (2023)HistopathologyMomentum contrastive learning combined with knowledge distillation[19]

Expanded article table

The VideoMoCo row is also covered in AI Wiki's article on video classification models.

Limitations reported by the authors

  • Uneven transfer. MoCo v1 lagged supervised pre-training on VOC semantic segmentation and only matched it on Cityscapes instance segmentation.[1]
  • Scaling with data. Moving from about 1.28 million ImageNet images to about 1 billion Instagram images produced consistent but small gains.[1]
  • Linear accuracy as a proxy. MoCo v2's ablations showed that a higher ImageNet linear score did not always mean better detection transfer.[2]
  • Hidden instability. MoCo v3 found that unstable ViT training could look successful while losing several points of accuracy, and its fix (a frozen patch projection) was described as a partial remedy.[3]

Continue reading

More to read

Topics:Computer VisionDeep LearningMachine LearningMeta AI

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick. "Momentum Contrast for Unsupervised Visual Representation Learning". arXiv:1911.05722 (CVPR 2020 camera-ready), 13 November 2019, revised 23 March 2020. arxiv.org/...1911.05722
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15Xinlei Chen, Haoqi Fan, Ross Girshick, Kaiming He. "Improved Baselines with Momentum Contrastive Learning". arXiv:2003.04297, 9 March 2020. arxiv.org/...2003.04297
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22Xinlei Chen, Saining Xie, Kaiming He. "An Empirical Study of Training Self-Supervised Vision Transformers". arXiv:2104.02057 (ICCV 2021, Oral), 5 April 2021, revised 16 August 2021. arxiv.org/...2104.02057
  4. ^Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick. "Momentum Contrast for Unsupervised Visual Representation Learning". 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, June 2020. DOI 10.1109/CVPR42600.2020.00975. doi.org/...CVPR42600.2020.00975
  5. ^1 ^2 ^3Facebook Research. "MoCo: Momentum Contrast for Unsupervised Visual Representation Learning" (README). GitHub, facebookresearch/moco, commit 132aea9. github.com/...README.md
  6. ^Facebook Research. "MoCo: Transferring to Detection" (detection/README.md). GitHub, facebookresearch/moco, commit 132aea9. github.com/...README.md
  7. ^Facebook Research. "Relicense MoCo under MIT (#135)". GitHub, facebookresearch/moco, 18 January 2023. github.com/...7397dfe1
  8. ^1 ^2 ^3 ^4Facebook Research. "Commits: facebookresearch/moco". GitHub, accessed 18 November 2026. github.com/...main
  9. ^1 ^2 ^3Facebook Research. "MoCo v3 for Self-supervised ResNet and ViT" (README). GitHub, facebookresearch/moco-v3. github.com/...moco-v3
  10. ^1 ^2Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. "A Simple Framework for Contrastive Learning of Visual Representations". arXiv:2002.05709 (ICML 2020), 13 February 2020. arxiv.org/...2002.05709
  11. ^1 ^2Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, et al. "Bootstrap your own latent: A new approach to self-supervised Learning". arXiv:2006.07733, 13 June 2020. arxiv.org/...2006.07733
  12. ^Xinlei Chen, Kaiming He. "Exploring Simple Siamese Representation Learning". arXiv:2011.10566, 20 November 2020. arxiv.org/...2011.10566
  13. ^1 ^2Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin. "Emerging Properties in Self-Supervised Vision Transformers". arXiv:2104.14294, 29 April 2021. arxiv.org/...2104.14294
  14. ^Hari Sowrirajan, Jingbo Yang, Andrew Y. Ng, Pranav Rajpurkar. "MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models". arXiv:2010.05352 (MIDL 2021), 11 October 2020. arxiv.org/...2010.05352
  15. ^Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, Wei Liu. "VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples". arXiv:2103.05905 (CVPR 2021), 10 March 2021. arxiv.org/...2103.05905
  16. ^Yuqi Ma, Huamin Wang, Hangchi Shen, Xuemei Chen, Shukai Duan, Shiping Wen. "NeuroMoCo: A Neuromorphic Momentum Contrast Learning Method for Spiking Neural Networks". arXiv:2406.06305, 10 June 2024. arxiv.org/...2406.06305
  17. ^Jin Wang, Liang-Chih Yu, Xuejie Zhang. "SoftMCL: Soft Momentum Contrastive Learning for Fine-grained Sentiment-aware Pre-training". arXiv:2405.01827 (LREC-COLING 2024), 3 May 2024. arxiv.org/...2405.01827
  18. ^Jaehoon Kim, Seungwan Jin, Sohyun Park, Someen Park, Kyungsik Han. "Label-aware Hard Negative Sampling Strategies with Momentum Contrastive Learning for Implicit Hate Speech Detection". arXiv:2406.07886 (ACL 2024 Findings), 12 June 2024. arxiv.org/...2406.07886
  19. ^Trinh Thi Le Vuong, Jin Tae Kwak. "MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysis". arXiv:2308.16561, 31 August 2023. arxiv.org/...2308.16561

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

1 revision · v1 · 3,932 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Cite this page: AI Wiki. "MoCo (Momentum Contrast)." aiwiki.ai, updated 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/moco

Edit article