Library · Machine learning & AI
What Contrastive & Representation Learning got wrong, from 54 theses
Summary written by a model from the records below: The records document empirical failures and design trade-offs encountered when applying contrastive and representation learning across vision, language, graph, and biomedical tasks. Common difficulties include defective pair construction, loss instability leading to representation collapse, and learned embeddings underperforming simpler tabular or supervised baselines. These records come from graduate theses, 2021 to 2026, each labeled with its degree where the repository states it, and each links to its thesis. They were extracted by language models reading the full text, so treat each as a lead to read, not a verdict.
Plus 10 problems the authors left open, listed at the end.
Failure modes, by theses reporting them
Select a mode to jump to its records.
Contrastive loss design flaws and representation collapse
Contrastive models suffered representation collapse or performance drops when architectures omitted predictor networks, used unsuitable similarity metrics, or applied symmetric losses inappropriately. Improper loss weighting, views with low mutual information, and continuous distributions further degraded representation quality across multiple domains.
Tried and failed
asymmetric multi-resolution contrastive learning with attention masking applied to unsupervised image representation learning. Outcome: worse than baseline. Reason: unsupervised attention masking degraded representation quality without supervision
Attention based contrastive learning with non-Siamese architecture · Iowa State · Master's
Tried and failed
extreme weighting in multi-task contrastive learning applied to medical image classification. Outcome: worse than baseline. Reason: setting the contrastive loss weight either too low or too high degraded target task performance
Integrating domain knowledge and deep learning for enhanced chest X-ray diagnosis and localization · UT Austin · PhD
Tried and failed
contrastive learning across low mutual information views applied to visual representation learning. Outcome: did not generalise. Reason: insufficient shared information between views removes task-relevant signals needed for downstream transfer
Towards General-purpose Vision via Multiview Contrastive Learning · MIT · PhD
Tried and failed
contrastive slow feature learning with scalar modulation applied to deep representation learning. Reason: did not yield significant improvements over standard representation learning baselines
Biologically plausible unsupervised learning in shallow and deep neural networks · EPFL · PhD
Tried and failed
binary noise-contrastive estimation loss applied to relation representation learning. Outcome: worse than baseline. Reason: underperformed compared to margin-based contrastive loss on benchmark evaluation datasets
Toward generalized and scalable relation extraction · Iowa State · PhD
Tried and failed
symmetric contrastive loss in self-supervised learning applied to visual perceptual similarity feature extraction. Outcome: worse than baseline. Reason: None
Advancements in perceptual quality assessment for interactive media : from mobile cloud gaming to human avatar videos and facial expressions · UT Austin · PhD
Considered and rejected
Considered and rejected: Rejected Barlow Twins contrastive learning framework for trajectory representations as its performance degraded (0.737 F1 vs 0.881 F1 for SimCLR) due to representations collapsing on simple augmentations.
Geospatial representation learning: from road networks to trajectories · Leibniz Universität Hannover · PhD
Considered and rejected
Considered and rejected: Rejected multiplicative normalization in Supervised Contrastive Loss (SCL), substituting an additive dis-alignment factor in SSCL to accommodate continuous similarity metrics.
Enhancing next generation networks with security, sensing and management · UT Austin · PhD
Considered and rejected
Considered and rejected: Decided against symmetric bidirectional contrastive loss in favor of an asymmetric text-to-image contrastive loss reflecting document-region hierarchy.
Language-Centric Medical Image Understanding · MIT · PhD
Considered and rejected
Considered and rejected: Rejected averaging positive pairs inside the logarithm of the contrastive loss function due to generic performance degradation.
Learning Simplicity and Structure in Quantum Field Theory · Harvard · PhD
Tried and failed
Uniform attraction and repulsion contrastive loss applied to representation learning for classification. Outcome: worse than baseline. Reason: uniformly pulling positives and pushing negatives without conditional distribution matching degraded classification accuracy
Implicit distributional matching at high dimensionality · UT Austin · PhD
Tried and failed
non-contrastive self-supervised learning without predictor network applied to cross-modal representation learning. Reason: removing negative pairs without a predictor head caused complete representation collapse
Learning from talking faces: from deepfake detection to speech recognition · Imperial · PhD
Tried and failed
conventional contrastive losses applied to continuous dynamical parameter distributions. Reason: caused dimensional collapse or severe performance degradation on continuous distributions
On the Certification of Deep Learning-based Dynamical System Identification · MIT · PhD
Defective positive and negative pair construction
Contrastive objectives degraded when negative pairs were corrupted by background class imbalance, periodic waveforms, or identical self-comparisons. Relying on random in-batch negatives or overlapping crops introduced false negatives and coarse distinctions that harmed downstream utility.
Tried and failed
supervised contrastive learning with positive-unlabeled data applied to positive-unlabeled classification with high class imbalance. Reason: gradient bias and false negative assumptions in low supervision and high class-prior regimes caused severe performance collapse
Robust and efficient learning in high dimensions from noisy data · UT Austin · PhD
Tried and failed
supervised contrastive loss on overlapping spatial patches applied to continual representation learning from image streams. Outcome: worse than baseline. Reason: treating overlapping spatial crops as positive class pairs harmed representation quality compared to batch-splitting them
Unsupervised Progressive Learning for Agents in Online and Dynamic Environments · Georgia Tech · PhD
Tried and failed
contrastive learning using self-comparisons as negative pairs applied to genomic coverage profile representations. Outcome: worse than baseline. Reason: comparing identical coverage vectors degenerated representation quality and increased cross-chromosome variance versus using biological replicates
Unsupervised learning with high-throughput sequencing data · Iowa State · PhD
Considered and rejected
Considered and rejected: Using trivial negative samples with random backgrounds unaligned to the anchor's class in contrastive learning was rejected as it degraded debiasing performance compared to matched dictionary queues.
Towards Robust Vision Models · EPFL · PhD
Considered and rejected
Considered and rejected: Rejected standard contrastive/instance-level InfoNCE loss between multimodal concepts in SimZSS because identical object concepts co-occur multiple times across a batch; used semantic classification cross-entropy instead.
Exploiting Representation Similarities in Self-Supervised Learning for Vision Tasks · EPFL · PhD
Tried and failed
joint detection and contrastive identification training applied to sparse audio event speaker separation. Outcome: worse than baseline. Reason: dummy negative labels from severe background class imbalance corrupted contrastive loss learning
Decoding the Depths: Developing a Click Separator for Predictive Speaker Recognition in Sperm Whale Conversations Using Machine Learning · MIT · Master's
Tried and failed
false negative filtering in contrastive learning applied to multimodal vision-language representation learning. Outcome: did not generalise. Reason: filtering false negatives improved dominant class classification but yielded negligible benefit for grounding or retrieval
Language-Centric Medical Image Understanding · MIT · PhD
Tried and failed
contrastive mutual information estimation with negative sampling applied to representation learning. Outcome: unstable. Reason: very small batch sizes fail to generate sufficient negative pairs via latent variable permutation
Novel approaches for learning representations · UT Austin · PhD
Considered and rejected
Considered and rejected: Rejected standard negative-pair contrastive time-series frameworks (SimCLR, MoCo, SwAV) due to erroneous negative sampling on periodic physiological waveforms.
Toward accurate health monitoring through large-scale Photoplethysmography signal from wearable devices · Georgia Tech · PhD
Tried and failed
exact inclusion-exclusion decomposition for negative sampling applied to contrastive representation learning. Outcome: infeasible cost. Reason: requires at least N positive samples and becomes computationally intractable for large batch sizes
Robust Learning from Uncurated Data · MIT · PhD
Graph neural network embeddings underperforming simpler baselines
Graph neural network representations and random walk embeddings failed to beat simple adjacency matrices, one-hot encodings, or tree-based models like Random Forest and XGBoost. Incorporating graph structural embeddings into language models or sparse networks injected noise and provided no meaningful signal.
Tried and failed
joint training of GNN embeddings with RL applied to reinforcement learning policy and value networks. Outcome: worse than baseline. Reason: yielded no significant performance gain compared to simple one-hot encoding
Learning Simplicity and Structure in Quantum Field Theory · Harvard · PhD
Tried and failed
contrastive learning and early-fusion graph convolutional networks applied to heterogeneous dense and sparse biological graphs. Outcome: worse than baseline. Reason: None
Towards Network-Guided Large-Scale Foundation Models on Single-Cell Transcriptomics · Virginia Tech · Master's
Tried and failed
incorporating initial node and relevance score embeddings applied to graph neural network question answering reasoning. Outcome: no signal. Reason: the added embeddings were completely dispensable and provided no performance benefit
Knowledge Reasoning with Graph Neural Networks · Georgia Tech · PhD
Tried and failed
combining language models with graph structure embeddings applied to link prediction on sparse graphs. Outcome: worse than baseline. Reason: graph structure information introduced noise on sparse graphs, degrading prediction performance
Temporal link prediction in the wild · Imperial · PhD
Lost to a baseline
End-to-end Neural Network with GNN embeddings underperformed Random Forest and XGBoost baselines across all metrics (Accuracy 0.5836, F1 0.4873, AUC-ROC 0.5405, AUC-PR 0.4672)
Predicting psychological treatment dropout using graph neural networks · Boston University · Master's
Considered and rejected
Considered and rejected: Rejected standard cluster/community-based multi-aspect graph embeddings because they fail to capture diverse higher-order relational semantics in heterogeneous networks.
Automatic Question Answering and Knowledge Discovery from Electronic Health Records · Virginia Tech · Doctorate
Considered and rejected
Considered and rejected: Rejected GNN graph embeddings and spectral/diffusion methods because GNNs require supervised labels and spectral methods cannot handle node attributes.
Optimal transport methods for analyzing highly multiplexed spatial proteomic data · Oxford · Master's
Considered and rejected
Considered and rejected: Rejected DiffPool for subgraph embeddings because subgraph selection is learned end-to-end and cannot generate embeddings for arbitrary subgraphs without a separate learning process
Graph Embedding for Retrieval · EPFL · PhD
Considered and rejected
Considered and rejected: Rejected aligning explanation embeddings with dataset average embeddings (as in GNNInterpreter) because it prioritizes node features over structural details and creates disconnected explanation graphs.
Motif-based Graph Neural Networks: Representation, interpretation, and extraction · Iowa State · PhD
Transfer failure of pretrained embeddings to specialized domains
Generic pretrained audio and speech representations failed to capture idiosyncratic clinical deviations and non-linguistic acoustic patterns. Standard contrastive formulations and discrete pretraining objectives struggled to align multimodal representations or handle continuous physiological time series.
Considered and rejected
Considered and rejected: Rejected end-to-end joint multimodal pretraining, opting instead to disentangle unsupervised contrastive alignment from downstream supervised adaptation.
Multimodal Representation Learning for Mental Health: Transfer Learning, PEFT, and Contrastive Learning · Dalhousie · PhD
Tried and failed
temporal contrastive loss in time series representation applied to neural spike train representations. Outcome: worse than baseline. Reason: None
Building a foundation model for neuroscience · Georgia Tech · PhD
Considered and rejected
Considered and rejected: ImageNet pre-trained ResNet embeddings, rejected in favor of in-domain SimCLR self-supervised contrastive learning
Context-aware digital pathology: enhancing whole slide image analysis through global-local feature learning · Boston University · PhD
Tried and failed
pre-trained discrete classification loss for supervision applied to audio source separation. Outcome: did not generalise. Reason: fixed class labels limited generalisation compared to continuous multimodal contrastive representations
Scalable transfer learning : enhancing adaptation and efficiency in uni- and multi-modal applications · UT Austin · PhD
Tried and failed
freezing pretrained text encoder in contrastive learning applied to multimodal contrastive representation learning. Outcome: worse than baseline. Reason: limits joint alignment between modalities during contrastive representation learning
Visual Representation Learning from Synthetic Data · MIT · PhD
Tried and failed
zero-shot transfer learning with pretrained audio embeddings applied to idiosyncratic atypical vocalization classification. Outcome: no signal. Reason: generic pretrained representations failed to capture idiosyncratic acoustic patterns of atypical vocalizations
Foundations of Cognitive, Affective, and Communicative Systems for Neurodiverse Individuals · MIT · PhD
Tried and failed
self-supervised speech embeddings with dynamic time warping applied to pathological speech intelligibility and impairment classification. Outcome: no signal. Reason: pre-trained representations failed to capture non-linguistic acoustic deviations and breathing impairments, yielding zero sensitivity and negative correlation
Novel Methods For Detection And Analysis Of Atypical Aspects In Speech · EPFL · PhD
Considered and rejected
Considered and rejected: Rejected joint pre-training of masked audio token prediction and contrastive loss simultaneously because dual objectives caused convergence failures.
Towards Integrated Audio-Visual Learning: From Vision-to-Audio Generation to a Unified Audio-Visual Framework · University of Washington · PhD
Considered and rejected
Considered and rejected: Masked Language Modeling (MLM) pre-training was rejected in favor of Contrastive Predictive Coding (CPC) because numerical time series lack discrete vocabulary context.
Machine learning in finance: from earnings data analysis to algorithmic trading · Imperial · Doctorate
Tried and failed
standard contrastive learning for sequential state representations applied to subgoal discovery in reinforcement learning. Outcome: did not generalise. Reason: produces ambiguous subgoal ordering and suffers distraction from neighboring non-key states without temporal geometric sampling
Reinforcement learning with temporal logic and causal constraints · Georgia Tech · PhD
Information loss from global pooling and sequence representation choices
Using CLS token embeddings or global vision vectors diluted localized spatial and contextual information compared to mean pooling or localized features. Uncontextualized word regression and direct concatenation of speaker embeddings failed to preserve the textual and acoustic distinctions required downstream.
Considered and rejected
Considered and rejected: Rejected training embeddings using Word-Level (WL) image-to-word regression without context because it failed to preserve textual information and degraded word similarity
Language Grounding in Vision · Tübingen · PhD
Tried and failed
conditioning policy on global vision embeddings directly applied to robot policy cost correction. Outcome: worse than baseline. Reason: lacks fine-grained spatial representations needed for object-referential tasks
Discovering and Engineering the Computation Underlying Large Intelligent Agents · MIT · PhD
Lost to a baseline
Transformer language model reaction embeddings failed to outperform Random Forest, Extreme Gradient Boosting, or Feedforward Neural Networks using standard fingerprint or DFT features.
Machine Learning for Chemical Reactivity Prediction: Paradigms, Challenges, and Applications · MIT · PhD
Tried and failed
CLS token embedding for sequence representation applied to molecular representation learning. Outcome: worse than baseline. Reason: Mean pooling across all token embeddings captured contextual representation significantly better than CLS token.
Contextual representations of the chemical space for task agnostic machine learning methods · Imperial · PhD
Tried and failed
concatenating transformer cls embedding with cnn features applied to clinical relation extraction. Outcome: worse than baseline. Reason: sentence-level representations diluted or conflicted with localized convolutional feature representations
Learning to Improve Clinical Decisions and AI Safety by Leveraging Structure · MIT · PhD
Tried and failed
Word Mover's Distance on learned sequence embeddings applied to cross-language code clone detection. Outcome: worse than baseline. Reason: significantly degraded detection effectiveness compared to Siamese bidirectional LSTM networks
Helping Developers Migrate their Code across Programming Languages · Virginia Tech · Doctorate
Considered and rejected
Considered and rejected: Rejected using the [CLS] token for sentence representation in BiGS downstream classification; used mean pooling over non-padding token embeddings because it yielded superior performance.
Designing for Inference in Future Generative Models · Cornell · PhD
Considered and rejected
Considered and rejected: Concatenating speaker embedding vectors directly to input acoustic features for anchored ASR; rejected due to lack of discrimination capacity.
WAKE WORD DETECTION AND ITS APPLICATIONS · Johns Hopkins · PhD
Self-supervised contrastive pretraining lagging behind supervised models
Self-supervised contrastive models underperformed supervised pretraining across downstream clinical benchmarks, elevation shifts, and medical segmentation tasks. Applying supervised contrastive objectives or adding perturbations also failed to beat standard cross-entropy or simpler contrastive baselines.
Lost to a baseline
Self-supervised pre-training methods (Contrastive and MLM) achieved lower average ranks than supervised pre-training across all downstream data regimes on MetaMIMIC
Applications of Optimization and Machine Learning to Healthcare · University of Washington · PhD
Lost to a baseline
Self-supervised contrastive localization model had 272% larger error at 40° elevation vs 0°, underperforming the supervised baseline model
Modeling and Evaluating Human Sound Localization in the Natural Environment · MIT · PhD
Tried and failed
dense supervised contrastive learning representations applied to affinity prediction. Outcome: worse than baseline. Reason: linear probing or fine-tuning failed to beat standard cross-entropy baseline
Lost to a baseline
CGLVM achieved a slightly higher Mutant Cell Line Silhouette (0.980±0.007) than contrastiveVI (0.966±0.001) on the salient latent representation of MIX-seq data.
Structured Deep Generative Models for Exploring the Single Cell Landscape · University of Washington · PhD
Lost to a baseline
On retinal layer segmentation, RETFound baseline achieved 40.1 µm MAE, beating all contrastive pretraining methods (SimCLR/BYOL scored 70.8-84.2 µm MAE).
Representation learning for automated diagnosis and biomarker discovery in age-related macular degeneration · Imperial · Doctorate
Lost to a baseline
On CommonsenseQA IHdev set, ConKGP with perturbations + contrastive learning (76.74%) was beaten by contrastive learning alone (77.64%)
Knowledge-Enhanced Language Models: Integrating Structured Knowledge Graphs in Large Language Model for Multiple-Choice Question-Answering Tasks · Carleton University · PhD
Left open by the authors
Problems the authors named and did not get to.
Left open
Benchmark contrastive learning directly against multitask learning for integrating radiomics features with deep learning on chest X-rays. Blocker: None
Integrating domain knowledge and deep learning for enhanced chest X-ray diagnosis and localization · UT Austin · PhD
Left open
Integrate contrastive learning, reinforcement learning from user feedback, and GNNs into the DPK-GLM biomedical retrieval framework. Blocker: Vague high-level directions without specific architecture, loss formulation, or feedback dataset specified
Enhancing General Language Models for Biomedical Test Retrieval via Diversified Prior Knowledge · York University · Master's
Left open
Develop domain-specific data augmentations for self-supervised contrastive learning on healthcare time series and modalities. Blocker: The direction is very broad without specific datasets, target modalities, or concrete augmentation formulations specified
Towards generalization of deep learning in pervasive human motion analysis · Imperial · PhD
Left open
Develop graph neural network embeddings or structure-agnostic representations for chemical structures in Bayesian optimization. Blocker: Lacks specific target problem, chemical dataset, and concrete baseline metrics
Bayesian optimisation in chemical problems · Imperial · PhD
Left open
Train a graph convolutional network to learn AST representations and token embeddings directly for performance modeling without handcrafted features. Blocker: None
Exploring synergies between program synthesis and machine learning · UT Austin · PhD
Left open
Implement a Graph Neural Network to replace the embedding prediction network in KD-EMD for spatial and relational knowledge distillation. Blocker: None
From Symbolic Reasoning to Object Embeddings: Advanced Approaches of Knowledge Distillation in Compacted Neural Networks · Texas Tech · PhD
Left open
Formulate contrastive representation learning as a multi-label classification objective using binary cross-entropy loss. Blocker: None
Understanding and Improving Representational Robustness of Machine Learning Models · MIT · PhD
Left open
Add a contrastive loss term alongside unlikelihood loss during Possibility Exploration Fine-Tuning to refine pairwise response similarity. Blocker: None
Towards Human-like Dialogue Systems: Enhancing Sensibility, Diversity, Proactiveness, and Naturalness in Spoken Open-domain Chatbots · University College Dublin · PhD
Left open
Design and evaluate alternative training objectives beyond pairwise text-text contrastive loss to improve temporal reasoning in cross-modal audio-text retrieval. Blocker: None
Searching audiovisual media with natural language queries · Oxford · PhD
Left open
Integrate contrastive learning into JOINT4E to evaluate performance improvements on biomedical event extraction and entity linking. Blocker: None
Joint Biomedical Event Extraction and Entity Linking via Iterative Collaborative Training · Virginia Tech · Master's
Who has worked on this
Authors of the public theses cited on this page, most-cited first.
- Aurélien Jérôme Alexandre Dersy · Harvard · PhD · 2026
Learning Simplicity and Structure in Quantum Field Theory - Peiqi Wang · MIT · PhD · 2025
Language-Centric Medical Image Understanding - Zhaoning Yu · Iowa State · PhD · 2026
Motif-based Graph Neural Networks: Representation, interpretation, and extraction - Habibeh Naderi Khorshidi · Dalhousie · PhD · 2026
Multimodal Representation Learning for Mental Health: Transfer Learning, PEFT, and Contrastive Learning - Yifan Wu · Imperial · PhD · 2025
Bayesian optimisation in chemical problems - Priyanka Raghavan · MIT · PhD · 2025
Machine Learning for Chemical Reactivity Prediction: Paradigms, Challenges, and Applications - Wang Zhang · MIT · PhD · 2025
On the Certification of Deep Learning-based Dynamical System Identification - Pratyusha Sharma · MIT · PhD · 2025
Discovering and Engineering the Computation Underlying Large Intelligent Agents - Cameron Taylor · Georgia Tech · PhD · 2025
Unsupervised Progressive Learning for Agents in Online and Dynamic Environments - Duo Xu · Georgia Tech · PhD · 2025
Reinforcement learning with temporal logic and causal constraints - Qing Wang · Iowa State · PhD · 2025
Toward generalized and scalable relation extraction - Stefan Schestakov · Leibniz Universität Hannover · PhD · 2025
Geospatial representation learning: from road networks to trajectories
Checking a claim in this area?
We can run the same search on any method or claim. If nothing turns up, we will say so, and that proves nothing on its own.