jason718/awesome-self-supervised-learning

Awesome Self-Supervised Learning

A curated list of awesome Self-Supervised Learning resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers, and awesome-architecture-search

Why Self-Supervised?

Self-Supervised Learning has become an exciting direction in AI community.

• Jitendra Malik: "Supervision is the opium of the AI researcher"
• Alyosha Efros: "The AI revolution will not be supervised"
• Yann LeCun: "self-supervised learning is the cake, supervised learning is the icing on the cake, reinforcement learning is the cherry on the cake"

Contributing

Please help contribute this list by pull request

Markdown format:

- Paper Name. 
  [[pdf]](link) 
  [[code]](link)
Author 1, Author 2, and Author 3. Conference Year

Table of Contents

• Theory
• Computer Vision (CV)
• Survey
• Image Representation Learning
• Video Representation Learning
• 3D Feature Learning
• Geometry
• Audio
• Others
• Machine Learning
• Reinforcement Learning
• Recommendation Systems
• Robotics
• Natural Language Processing (NLP)
• Automatic Speech Recognition (ASR)
• Time-Series
• Graph
• Talks
• Thesis
• Blog

Theory

2019

• A Theoretical Analysis of Contrastive Unsupervised Representation Learning.

[[pdf]](https://arxiv.org/pdf/1902.09229.pdf)

• Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. ICML 2019

2020

• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere.

[[pdf]](https://arxiv.org/pdf/2005.10242)

• Tongzhou Wang, Phillip Isola. ICML 2020
• Understanding Self-supervised Learning with Dual Deep Networks.

[[pdf]](https://arxiv.org/pdf/2010.00578.pdf)

• Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli.
• For self-supervised learning, Rationality implies generalization, provably.

[[pdf]](https://arxiv.org/pdf/2010.08508.pdf)

• Yamini Bansal, Gal Kaplun, and Boaz Barak.

2021

• Towards the Generalization of Contrastive Self-Supervised Learning.

[[pdf]](https://arxiv.org/pdf/2111.00743.pdf)

• Weiran Huang, Mingyang Yi, and Xuyang Zhao.
• Understanding the Behaviour of Contrastive Loss.

[[pdf]](https://arxiv.org/pdf/2012.09740.pdf)

• Feng Wang and Huaping Liu. CVPR 2021
• Predicting What You Already Know Helps: Provable Self-Supervised Learning.

[[pdf]](https://arxiv.org/pdf/2008.01064.pdf)

• Jason D. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo.
• Contrastive learning , multi-view redundancy , and linear models.

[[pdf]](https://arxiv.org/pdf/2008.10150.pdf)

• Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu.
• Contrastive Learning Inverts the Data Generating Process.

[[pdf]](Contrastive Learning Inverts the Data Generating Process)

• Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, Wieland Brendel. ICML 2021

2022

• Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions.

[[pdf]](https://arxiv.org/pdf/2103.03568.pdf)

• Jiaye Teng, Weiran Huang, and Haowei He. AISTATS 2022

2023

• Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis.

[[pdf]](https://openreview.net/pdf?id=AjC0KBjiMu)

• Daniel D. Johnson, Ayoub El Hanchi, Chris J. Maddison. ICLR 2023
• On the Stepwise Nature of Self-Supervised Learning.

[[pdf]](https://arxiv.org/pdf/2303.15438)

• James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, Joshua Albrecht. ICML 2023
• What shapes the loss landscape of self supervised learning?

[[pdf]](https://openreview.net/pdf?id=3zSn48RUO8M)

• Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, Hidenori Tanaka. ICLR 2023

2024

• Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses.

[[pdf]](https://arxiv.org/pdf/2405.18045) [[code]](https://github.com/pakoromilas/DHEL-KCL)

• Panagiotis Koromilas, Giorgos Bouritsas, Theodoros Giannakopoulos, Mihalis Nicolaou, Yannis Panagakis. ICML 2024
• Matrix Information Theory for Self-Supervised Learning.

[[pdf]](https://arxiv.org/pdf/2305.17326)

• Yifan Zhang, Zhiquan Tan, Jingqin Yang, Weiran Huang, Yang Yuan. ICML 2024
• Information Flow in Self-Supervised Learning.

[[pdf]](https://arxiv.org/pdf/2309.17281)

• Zhiquan Tan, Jingqin Yang, Weiran Huang, Yang Yuan, Yifan Zhang. ICML 2024

Computer Vision

Survey

• Contrastive Representation Learning: A Framework and Review

[[pdf]](https://arxiv.org/abs/2010.05113)

• Phuc H. Le-Khac, Graham Healy, Alan F. Smeaton. IEEE Access 2020
• A Survey on Contrastive Self-supervised Learning

[[pdf]](https://arxiv.org/pdf/2011.00362.pdf)

• Ashish Jaiswal, Ashwin R Babu, Mohammad Z Zadeh, Debapriya Banerjee, Fillia Makedon
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey.

[[pdf]](https://arxiv.org/pdf/1902.06162.pdf)

• Longlong Jing and Yingli Tian. T-PAMI 2020
• Self-supervised Learning: Generative or Contrastive

[[pdf]](https://arxiv.org/pdf/2006.08218.pdf)

• Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, Jie Tang. TKDE 2021
• Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training

[[pdf]](https://openreview.net/pdf?id=Ma25S4ludQ)

• Utku Ozbulak, Hyun Jung Lee, Beril Boga, Esla Timothy Anzaku, Ho-min Park, Arnout Van Messem, Wesley De Neve, Joris Vankerschaver. TMLR 2023

Image Representation Learning

Benchmark code

• FAIR Self-Supervision Benchmark [[pdf]](https://arxiv.org/abs/1905.01235) [[repo]](https://github.com/facebookresearch/fair_self_supervision_benchmark): various benchmark (and legacy) tasks for evaluating quality of visual representations learned by various self-supervision approaches.
• How Well Do Self-Supervised Models Transfer? [[pdf]](https://arxiv.org/abs/2011.13377) [[repo]](https://github.com/linusericsson/ssl-transfer): A benchmark for evaluating self-supervision consisting of many-shot/few-shot recognition, object detection, surface normal estimation and semantic segmentation.

2015

• Unsupervised Visual Representation Learning by Context Prediction.

[[pdf]](https://arxiv.org/abs/1505.05192) [[code]](http://graphics.cs.cmu.edu/projects/deepContext/)

• Doersch, Carl and Gupta, Abhinav and Efros, Alexei A. ICCV 2015
• Unsupervised Learning of Visual Representations using Videos.

[[pdf]](http://www.cs.cmu.edu/~xiaolonw/papers/unsupervised_video.pdf) [[code]](http://www.cs.cmu.edu/~xiaolonw/unsupervise.html)

• Wang, Xiaolong and Gupta, Abhinav. ICCV 2015
• Learning to See by Moving.

[[pdf]](http://arxiv.org/abs/1505.01596) [[code]](https://people.eecs.berkeley.edu/~pulkitag/lsm/lsm.html)

• Agrawal, Pulkit and Carreira, Joao and Malik, Jitendra. ICCV 2015
• Learning image representations tied to ego-motion.

[[pdf]](http://vision.cs.utexas.edu/projects/egoequiv/ijcv_bestpaper_specialissue_egoequiv.pdf) [[code]](http://vision.cs.utexas.edu/projects/egoequiv/)

• Jayaraman, Dinesh and Grauman, Kristen. ICCV 2015

2016

• Joint Unsupervised Learning of Deep Representations and Image Clusters.

[[pdf]](https://arxiv.org/pdf/1604.03628.pdf) [[code-torch]](https://github.com/jwyang/JULE.torch) [[code-caffe]](https://github.com/jwyang/JULE-Caffe)

• Jianwei Yang, Devi Parikh, Dhruv Batra. CVPR 2016
• Unsupervised Deep Embedding for Clustering Analysis.

[[pdf]](https://arxiv.org/pdf/1511.06335.pdf) [[code]](https://github.com/piiswrong/dec)

• Junyuan Xie, Ross Girshick, and Ali Farhadi. ICML 2016
• Slow and steady feature analysis: higher order temporal coherence in video.

[[pdf]](http://vision.cs.utexas.edu/projects/slowsteady/cvpr16.pdf)

• Jayaraman, Dinesh and Grauman, Kristen. CVPR 2016
• Context Encoders: Feature Learning by Inpainting.

[[pdf]](https://people.eecs.berkeley.edu/~pathak/papers/cvpr16.pdf) [[code]](https://people.eecs.berkeley.edu/~pathak/context_encoder/)

• Pathak, Deepak and Krahenbuhl, Philipp and Donahue, Jeff and Darrell, Trevor and Efros, Alexei A. CVPR 2016
• Colorful Image Colorization.

[[pdf]](https://arxiv.org/abs/1603.08511) [[code]](http://richzhang.github.io/colorization/)

• Zhang, Richard and Isola, Phillip and Efros, Alexei A. ECCV 2016
• Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles.

[[pdf]](http://arxiv.org/abs/1603.09246) [[code]](http://www.cvg.unibe.ch/research/JigsawPuzzleSolver.html)

• Noroozi, Mehdi and Favaro, Paolo. ECCV 2016
• Ambient Sound Provides Supervision for Visual Learning.

[[pdf]](http://arxiv.org/pdf/1608.07017) [[code]](http://andrewowens.com/ambient/index.html)

• Owens, Andrew and Wu, Jiajun and McDermott, Josh and Freeman, William and Torralba, Antonio. ECCV 2016
• Learning Representations for Automatic Colorization.

[[pdf]](http://arxiv.org/pdf/1603.06668.pdf) [[code]](http://people.cs.uchicago.edu/~larsson/colorization/)

• Larsson, Gustav and Maire, Michael and Shakhnarovich, Gregory. ECCV 2016
• Unsupervised Visual Representation Learning by Graph-based Consistent Constraints.

[\[pdf\]](http://faculty.ucmerced.edu/mhyang/papers/eccv16_feature_learning.pdf) [\[code\]](https://github.com/dongli12/FeatureLearning)

• Li, Dong and Hung, Wei-Chih and Huang, Jia-Bin and Wang, Shengjin and Ahuja, Narendra and Yang, Ming-Hsuan. ECCV 2016

2017

• Adversarial Feature Learning.

[[pdf]](https://arxiv.org/pdf/1605.09782.pdf) [[code]](https://github.com/jeffdonahue/bigan)

• Donahue, Jeff and Krahenbuhl, Philipp and Darrell, Trevor. ICLR 2017
• Self-supervised learning of visual features through embedding images into text topic spaces.

[[pdf]](https://arxiv.org/pdf/1705.08631.pdf) [[code]](https://github.com/lluisgomez/TextTopicNet)

• L. Gomez* and Y. Patel* and M. Rusiñol and D. Karatzas and C.V. Jawahar. CVPR 2017
• Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction.

[[pdf]](https://arxiv.org/abs/1611.09842) [[code]](https://github.com/richzhang/splitbrainauto)

• Zhang, Richard and Isola, Phillip and Efros, Alexei A. CVPR 2017
• Learning Features by Watching Objects Move.

[[pdf]](https://people.eecs.berkeley.edu/~pathak/papers/cvpr17.pdf) [[code]](https://people.eecs.berkeley.edu/~pathak/unsupervised_video/)

• Pathak, Deepak and Girshick, Ross and Dollar, Piotr and Darrell, Trevor and Hariharan, Bharath. CVPR 2017
• Colorization as a Proxy Task for Visual Understanding.

[[pdf]](http://arxiv.org/abs/1703.04044) [[code]](http://people.cs.uchicago.edu/~larsson/color-proxy/)

• Larsson, Gustav and Maire, Michael and Shakhnarovich, Gregory. CVPR 2017
• DeepPermNet: Visual Permutation Learning.

[\[pdf\]](https://arxiv.org/pdf/1704.02729.pdf) [\[code\]](https://github.com/rfsantacruz/deep-perm-net)

• Cruz, Rodrigo Santa and Fernando, Basura and Cherian, Anoop and Gould, Stephen. CVPR 2017
• Unsupervised Learning by Predicting Noise.

[[pdf]](https://arxiv.org/abs/1704.05310) [[code]](https://github.com/facebookresearch/noise-as-targets)

• Bojanowski, Piotr and Joulin, Armand. ICML 2017
• Multi-task Self-Supervised Visual Learning.

[[pdf]](https://arxiv.org/abs/1708.07860)

• Doersch, Carl and Zisserman, Andrew. ICCV 2017
• Representation Learning by Learning to Count.

[[pdf]](https://arxiv.org/abs/1708.06734)

• Noroozi, Mehdi and Pirsiavash, Hamed and Favaro, Paolo. ICCV 2017
• Transitive Invariance for Self-supervised Visual Representation Learning.

[[pdf]](https://arxiv.org/pdf/1708.02901.pdf)

• Wang, Xiaolong and He, Kaiming and Gupta, Abhinav. ICCV 2017
• Look, Listen and Learn.

[[pdf]](https://arxiv.org/pdf/1705.08168.pdf)

• Relja, Arandjelovic and Zisserman, Andrew. ICCV 2017
• Unsupervised Representation Learning by Sorting Sequences.

[[pdf]](https://arxiv.org/pdf/1708.01246.pdf) [[code]](https://github.com/HsinYingLee/OPN)

• Hsin-Ying Lee, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang. ICCV 2017

2018

• Unsupervised Feature Learning via Non-parameteric Instance Discrimination

[[pdf]](https://arxiv.org/pdf/1805.01978.pdf) [[code]](https://github.com/zhirongw/lemniscate.pytorch)

• Zhirong Wu, Yuanjun Xiong and X Yu Stella and Dahua Lin. CVPR 2018
• Learning Image Representations by Completing Damaged Jigsaw Puzzles.

[[pdf]](https://arxiv.org/pdf/1802.01880.pdf) [[code]](https://github.com/MehdiNoroozi/JigsawPuzzleSolver)

• Kim, Dahun and Cho, Donghyeon and Yoo, Donggeun and Kweon, In So. WACV 2018
• Unsupervised Representation Learning by Predicting Image Rotations.

[[pdf]](https://openreview.net/forum?id=S1v4N2l0-) [[code]](https://github.com/gidariss/FeatureLearningRotNet)

• Spyros Gidaris and Praveer Singh and Nikos Komodakis. ICLR 2018
• Learning Latent Representations in Neural Networks for Clustering through Pseudo Supervision and Graph-based Activity Regularization.

[[pdf]](https://openreview.net/pdf?id=HkMvEOlAb) [[code]](https://github.com/ozcell/LALNets)

• Ozsel Kilinc and Ismail Uysal. ICLR 2018
• Improvements to context based self-supervised learning.

[[pdf]](https://arxiv.org/abs/1711.06379)

• Terrell Mundhenk and Daniel Ho and Barry Chen. CVPR 2018
• Self-Supervised Feature Learning by Learning to Spot Artifacts.

[[pdf]](https://arxiv.org/pdf/1806.05024.pdf) [[code]](https://github.com/sjenni/LearningToSpotArtifacts)

• Simon Jenni and Universität Bern and Paolo Favaro. CVPR 2018
• Boosting Self-Supervised Learning via Knowledge Transfer.

[[pdf]](https://www.csee.umbc.edu/~hpirsiav/papers/transfer_cvpr18.pdf)

• Mehdi Noroozi and Ananth Vinjimoor and Paolo Favaro and Hamed Pirsiavash. CVPR 2018
• Cross-domain Self-supervised Multi-task Feature Learning Using Synthetic Imagery.

[[pdf]](https://arxiv.org/abs/1711.09082) [[code]](https://github.com/jason718/game-feature-learning)

• Zhongzheng Ren and Yong Jae Lee. CVPR 2018
• ShapeCodes: Self-Supervised Feature Learning by Lifting Views to Viewgrids.

[[pdf]](https://arxiv.org/pdf/1709.00505.pdf)

• Dinesh Jayaraman*, UC Berkeley; Ruohan Gao, University of Texas at Austin; Kristen Grauman. ECCV 2018
• Deep Clustering for Unsupervised Learning of Visual Features

[[pdf]](https://research.fb.com/wp-content/uploads/2018/09/Deep-Clustering-for-Unsupervised-Learning-of-Visual-Features.pdf) [[code]](https://github.com/facebookresearch/deepcluster)

• Mathilde Caron, Piotr Bojanowski, Armand Joulin, Matthijs Douze. ECCV 2018
• Cross Pixel Optical-Flow Similarity for Self-Supervised Learning.

[[pdf]](http://www.robots.ox.ac.uk/~vgg/publications/2018/Mahendran18/mahendran18.pdf)

• Aravindh Mahendran, James Thewlis, Andrea Vedaldi. ACCV 2018

2019

• Representation Learning with Contrastive Predictive Coding.

[[pdf]](https://arxiv.org/abs/1807.03748)

• Aaron van den Oord, Yazhe Li, Oriol Vinyals.
• Self-Supervised Learning via Conditional Motion Propagation.

[[pdf]]() [[code]](https://github.com/XiaohangZhan/conditional-motion-propagation)

• Xiaohang Zhan, Xingang Pan, Ziwei Liu, Dahua Lin, and Chen Change Loy. CVPR 2019
• Self-Supervised Representation Learning by Rotation Feature Decoupling.

[[pdf]](http://openaccess.thecvf.com/content_CVPR_2019/html/Feng_Self-Supervised_Representation_Learning_by_Rotation_Feature_Decoupling_CVPR_2019_paper.html) [[code]](https://github.com/philiptheother/FeatureDecoupling)

• Zeyu Feng; Chang Xu; Dacheng Tao. CVPR 2019
• Revisiting Self-Supervised Visual Representation Learning.

[[pdf]](https://arxiv.org/abs/1901.09005) [[code]](https://github.com/google/revisiting-self-supervised)

• Alexander Kolesnikov; Xiaohua Zhai; Lucas Beye. CVPR 2019
• Self-Supervised GANs via Auxiliary Rotation Loss.

[[pdf]](https://openaccess.thecvf.com/content_CVPR_2019/papers/Chen_Self-Supervised_GANs_via_Auxiliary_Rotation_Loss_CVPR_2019_paper.pdf) [[code]](https://github.com/vandit15/Self-Supervised-Gans-Pytorch)

• Ting Chen; Xiaohua Zhai; Marvin Ritter; Mario Lucic; Neil Houlsby. CVPR 2019
• AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data.

[[pdf]](http://openaccess.thecvf.com/content_CVPR_2019/papers/Zhang_AET_vs._AED_Unsupervised_Representation_Learning_by_Auto-Encoding_Transformations_Rather_CVPR_2019_paper.pdf) [[code]](https://github.com/maple-research-lab/AET)

• Liheng Zhang, Guo-Jun Qi, Liqiang Wang, Jiebo Luo. CVPR 2019
• Unsupervised Deep Learning by Neighbourhood Discovery.

[[pdf]](http://proceedings.mlr.press/v97/huang19b.html). [[code]](https://github.com/Raymond-sci/AND).

• Jiabo Huang, Qi Dong, Shaogang Gong, Xiatian Zhu. ICML 2019
• Contrastive Multiview Coding.

[[pdf]](https://arxiv.org/abs/1906.05849) [[code]](https://github.com/HobbitLong/CMC/)

• Yonglong Tian and Dilip Krishnan and Phillip Isola.
• Large Scale Adversarial Representation Learning.

[[pdf]](https://arxiv.org/abs/1907.02544)

• Jeff Donahue, Karen Simonyan.
• Learning Representations by Maximizing Mutual Information Across Views.

[[pdf]](https://arxiv.org/pdf/1906.00910) [[code]](https://github.com/Philip-Bachman/amdim-public)

• Philip Bachman, R Devon Hjelm, William Buchwalter
• Selfie: Self-supervised Pretraining for Image Embedding.

[[pdf]](https://arxiv.org/abs/1906.02940)

• Trieu H. Trinh, Minh-Thang Luong, Quoc V. Le
• Data-Efficient Image Recognition with Contrastive Predictive Coding

[[pdf]](https://arxiv.org/abs/1905.09272)

• Olivier J. He ́naff, Ali Razavi, Carl Doersch, S. M. Ali Eslami, Aaron van den Oord
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty

[[pdf]](https://arxiv.org/pdf/1906.12340) [[code]](https://github.com/hendrycks/ss-ood)

• Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, Dawn Song. NeurIPS 2019
• Boosting Few-Shot Visual Learning with Self-Supervision

[[pdf]](http://openaccess.thecvf.com/content_ICCV_2019/papers/Gidaris_Boosting_Few-Shot_Visual_Learning_With_Self-Supervision_ICCV_2019_paper.pdf)

• Pyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. ICCV 2019
• Self-Supervised Generalisation with Meta Auxiliary Learning

[[pdf]](https://arxiv.org/pdf/1901.08933.pdf) [[code]](https://github.com/lorenmt/maxl)

• Shikun Liu, Andrew J. Davison, Edward Johns. NeurIPS 2019
• Wasserstein Dependency Measure for Representation Learning

[[pdf]](https://arxiv.org/pdf/1903.11780.pdf) [[code]](https://github.com/SeongokRyu/mutual_information_and_self-supervised_learning/tree/master/predictive_coding)

• Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, Pierre Sermanet. NeurIPS 2019
• Scaling and Benchmarking Self-Supervised Visual Representation Learning

[[pdf]](https://arxiv.org/abs/1905.01235) [[code]](https://github.com/facebookresearch/fair_self_supervision_benchmark)

• Priya Goyal, Dhruv Mahajan, Abhinav Gupta, Ishan Misra. ICCV 2019
• Unsupervised Pre-Training of Image Features on Non-Curated Data

[[pdf]](https://arxiv.org/pdf/1905.01278.pdf) [[code]](https://github.com/facebookresearch/DeeperCluster)

• Mathilde Caron, Piotr Bojanowski, Julien Mairal, Armand Joulin. ICCV 2019 Oral
• S4L: Self-Supervised Semi-Supervised Learning

[[pdf]](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhai_S4L_Self-Supervised_Semi-Supervised_Learning_ICCV_2019_paper.pdf) [[code]](https://github.com/google-research/s4l)

• Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, Lucas Beyer. ICCV 2019
• Self-supervised model adaptation for multimodal semantic segmentation.

[[pdf]](https://arxiv.org/abs/1808.03833) [[code]](https://github.com/DeepSceneSeg/SSMA)

• Abhinav Valada, Rohit Mohan, and Wolfram Burgard. IJCV 2019

2020

• A critical analysis of self-supervision, or what we can learn from a single image

[[pdf]](https://arxiv.org/pdf/1904.13132) [[code]](https://github.com/yukimasano/linear-probes)

• Yuki M. Asano, Christian Rupprecht, Andrea Vedaldi. ICLR 2020
• On Mutual Information Maximization for Representation Learning

[[pdf]](https://arxiv.org/pdf/1907.13625.pdf) [[code]](https://github.com/google-research/google-research/tree/master/mutual_information_representation_learning)

• Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, Mario Lucic. ICLR 2020
• Understanding the Limitations of Variational Mutual Information Estimators

[[pdf]](https://arxiv.org/pdf/1910.06222) [[code]](https://github.com/ermongroup/smile-mi-estimator)

• Jiaming Song, Stefano Ermon. ICLR 2020
• Self-labelling via simultaneous clustering and representation learning

[[pdf]](https://openreview.net/pdf?id=Hyx-jyBFPr) [[blogpost]](http://www.robots.ox.ac.uk/~vgg/blog/self-labelling-via-simultaneous-clustering-and-representation-learning.html) [[code]](https://github.com/yukimasano/self-label)

• Yuki Markus Asano, Christian Rupprecht, Andrea Vedaldi. ICLR 2020 (Spotlight)
• Self-supervised Label Augmentation via Input Transformations

[[pdf]](https://arxiv.org/abs/1910.05872) [[code]](https://github.com/hankook/SLA)

• Hankook Lee, Sung Ju Hwang, Jinwoo Shin. ICML 2020
• Automatic Shortcut Removal for Self-Supervised Representation Learning

[[pdf]](https://arxiv.org/pdf/2002.08822.pdf)

• Matthias Minderer, Olivier Bachem, Neil Houlsby, Michael Tschannen
• A Simple Framework for Contrastive Learning of Visual Representations

[[pdf]](https://arxiv.org/abs/2002.05709) [[code]](https://github.com/google-research/simclr)

• Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton. ICML 2020
• How Useful is Self-Supervised Pretraining for Visual Tasks?

[[pdf]](https://arxiv.org/abs/2003.14323) [[code]](https://github.com/princeton-vl/selfstudy-render)

• Alejandro Newell, Jia Deng. CVPR 2020
• Momentum Contrast for Unsupervised Visual Representation Learning

[[pdf]](https://arxiv.org/pdf/1911.05722.pdf) [code]

• Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick. CVPR 2020
• ClusterFit: Improving Generalization of Visual Representations

[[pdf]](https://arxiv.org/abs/1912.03330)

• Xueting Yan*, Ishan Misra*, Abhinav Gupta, Deepti Ghadiyaram, Dhruv Mahajan. CVPR 2020
• Self-Supervised Learning of Pretext-Invariant Representations

[[pdf]](https://arxiv.org/abs/1912.01991)

• Ishan Misra, Laurens van der Maaten. CVPR 2020
• Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

[[pdf]](https://arxiv.org/abs/2006.07733) [[un