Library · Machine learning & AI

What Deep Learning & Pretraining got wrong, from 17 theses

Summary written by a model from the records below: The evaluated thesis records demonstrate that deep learning and pretrained language models frequently face challenges with small sample overfitting, domain transfer, and time series forecasting. At the same time, simpler classical baselines often match or outperform complex neural networks, while shallow feature representations struggle on intricate spatial and semantic tasks. These records come from graduate theses, 2021 to 2025, each labeled with its degree where the repository states it, and each links to its thesis. They were extracted by language models reading the full text, so treat each as a lead to read, not a verdict.

17
theses
11
institutions
17
failure records
5
failure modes

Plus 4 problems the authors left open, listed at the end.

Failure modes, by theses reporting them

Select a mode to jump to its records.

Deep learning models risk severe overfitting in small sample regimes
5
5 theses · 5 institutions · 5 records
Simpler statistical models and classical baselines outperform deep architectures
3
3 theses · 2 institutions · 3 records
Pretrained language models encounter overfitting and degradation on downstream text tasks
4
4 theses · 4 institutions · 4 records
Traditional shallow baselines fail to capture complex spatial or semantic representations
3
3 theses · 3 institutions · 3 records
Pretrained language models face forecasting degradation and temporal leakage in time series tasks
2
2 theses · 2 institutions · 2 records

Deep learning models risk severe overfitting in small sample regimes

5 theses · 5 institutions

Researchers rejected deep neural networks across behavioral, biomedical, chemometric, and EEG datasets due to high risks of overfitting and poor generalizability. These domains featured small sample counts and high collinearity that could not satisfy the heavy data demands of deep learning architectures.

Considered and rejected

Considered and rejected: Rejected using complex deep learning techniques without multi-task framing on small-sample behavioral datasets due to severe overfitting and poor generalizability.

Towards Human-Centered Behavioral Sensing for Student Support · University of Washington · PhD

Considered and rejected

Considered and rejected: Rejected Deep Learning (DL) models due to high risk of overfitting on small biomedical sample sizes, computational overhead, and lack of interpretability.

Machine learning prediction of atezolizumab treatment response in metastatic urothelial carcinoma using gene expression and clinical data · Imperial · PhD

Considered and rejected

Considered and rejected: Rejected OpenNN and deep learning frameworks (Keras/TensorFlow) due to narrow focus on neural networks and excessive data requirements for small tabular data

Fatigue Monitoring System · SUPSI

Considered and rejected

Considered and rejected: Rejected deep learning approaches due to limited training data and high susceptibility to overfitting in small EEG sample regimes.

Instrumentation for daily-life brain-computer interfaces · Politecnico di Torino · PhD

Considered and rejected

Considered and rejected: Deep learning / ANN models were deprioritized over PLS/SVM due to high chemometric collinearity and small sample size ('small n, large p' regime) risking overfitting without massive datasets.

Sensor Approaches for the Non-Destructive Assessment of Food Safety and Authenticity · Cranfield · PhD

Simpler statistical models and classical baselines outperform deep architectures

3 theses · 2 institutions

Classical approaches such as Random Forest, logistic regression, and linear interpolation matched or exceeded deep learning performance. Practitioners favored these baselines because they offered simpler implementations, reduced false positives, faster explainability, and superior whole-image accuracy.

Tried and failed

deep learning on raw signals and scalograms applied to arterial Doppler waveform classification. Outcome: worse than baseline. Reason: deep models underperformed handcrafted feature-based logistic regression classifier

Testing for peripheral arterial disease in diabetes · Imperial · PhD

Considered and rejected

Considered and rejected: Rejected using deep learning architecture over Random Forest classifier because Random Forest was simpler and reduced false positives by 50% vs 35% for DNN.

Molecular simulation on two-dimensional nanoporous material for energy-efficient separation · EPFL · PhD

Lost to a baseline

Linear and optical flow interpolation yielded lower whole-image MSE than deep learning methods (R2UNet/proposed) because the latter deprioritized background non-ROI pixels.

Controllable synthetic algorithms and evaluations for annotated clinical image synthesis · Imperial · PhD

Pretrained language models encounter overfitting and degradation on downstream text tasks

4 theses · 4 institutions

Pretrained transformer models and their embeddings exhibited overfitting, fluctuating validation loss, or degraded classification metrics across several text tasks. Specifically, larger models underperformed lightweight variants, domain-adapted embeddings lagged behind general-domain baselines, and raw dialogue text reduced sentiment recall.

Tried and failed

fine-tuning pretrained transformer for continuous regression applied to short text evaluation scores. Outcome: overfit. Reason: validation loss fluctuated in later epochs, failing to generalize to unseen data

Enhancing Sensory Panel Decision-Making: A GenAI Approach Using RoBERTa to Quantify Free-Form Comments · Cornell · Master's

Tried and failed

larger pretrained transformer embeddings applied to fine-grained text classification. Outcome: worse than baseline. Reason: DistilBERT outperformed full BERT representations across all tested downstream classifiers.

Spatiotemporal Event Forecasting and Analysis with Ubiquitous Urban Sensors · Virginia Tech · Doctorate

Tried and failed

Domain-specific pretrained language model embeddings applied to Extractive text summarization. Outcome: worse than baseline. Reason: Domain-adapted BERT underperformed general domain BERT and GPT-2 on extractive summarization metrics

Computational acquisition of knowledge in small-data environments: a case study in the field of energetics · Imperial · PhD

Tried and failed

pretrained language model embeddings on raw text applied to chatbot text sentiment classification. Outcome: worse than baseline. Reason: raw chatbot-generated text representations degraded classification recall compared to customer text alone

Three essays on strategic digital initiatives in modern organizations · Georgia Tech · PhD

Traditional shallow baselines fail to capture complex spatial or semantic representations

3 theses · 3 institutions

Classical baselines such as random forests, TF-IDF representations, and shallow word embeddings proved inadequate for complex predictive tasks. These shallow methods failed to capture intricate spatial gradients and subtle rhetorical or contextual information handled better by deep learning and pretrained transformers.

Tried and failed

random forest regression for field variable surrogate applied to turbulent flow field prediction. Outcome: worse than baseline. Reason: random forest failed to capture complex spatial gradients compared to deep learning framework

Multifidelity machine learning methods for flow field prediction and aerodynamic shape optimization · Iowa State · PhD

Tried and failed

Logistic regression on TF-IDF features applied to text classification of discourse tone. Outcome: worse than baseline. Reason: TF-IDF bag-of-words failed to capture subtle stylistic and rhetorical context compared to pretrained transformers

Modeling Legal Constructs · Cornell · PhD

Tried and failed

shallow word embedding text classification applied to intent classification. Outcome: worse than baseline. Reason: underperformed compared to pretrained contextual language models

Empathetic Conversational Agents for Distress Support · EPFL · PhD

Pretrained language models face forecasting degradation and temporal leakage in time series tasks

2 theses · 2 institutions

Applying language models to time series and historical financial data caused performance degradation across forecasting horizons compared to standard deep architectures. Additionally, off-the-shelf models were rejected due to severe temporal pretraining leakage when forecasting historical financial data.

Tried and failed

domain fine-tuned zero-shot language model applied to time series forecasting. Outcome: worse than baseline. Reason: exhibited performance degradation across all horizons and metrics compared to standard deep learning models

Deep Learning Methods for Built Environment Operational Management · Virginia Tech · Doctorate

Considered and rejected

Considered and rejected: Rejected off-the-shelf contemporary LLMs for historical financial forecasting due to severe temporal pretraining leakage.

Essays in Finance, Technology, and Behavior · Harvard · PhD

Left open by the authors

Problems the authors named and did not get to.

Left open

Tune hyperparameters and architectures for deep learning emergency department arrival forecasting models across 1h to 48h horizons. Blocker: Requires the emergency department arrival dataset used in the thesis, which is likely private hospital data.

Predicting the Unpredictable: Comparing Statistical Forecasting and Deep Learning Models for Forecasting Emergency Department Arrivals · Harvard · Master's

Left open

Apply transfer learning and transformer architectures to improve generalization of deep learning models across diverse seismic datasets. Blocker: The objective is a broad research direction lacking specific datasets, target tasks, or evaluation metrics.

Improving accuracy and efficiency of seismic data analysis using deep learning · UT Austin · PhD

Left open

Evaluate deep learning architectures against shallow models for GHG and odour emissions from biological nutrient removal wastewater data. Blocker: Lacks access to the specific cold-region wastewater treatment plant field and laboratory monitoring dataset.

ESTIMATION OF GREENHOUSE GAS AND ODOUR EMISSIONS FROM COLD REGION MUNICIPAL BIOLOGICAL NUTRIENT REMOVAL WASTEWATER TREATMENT PROCESSES · Saskatchewan · PhD

Left open

Develop tabular deep learning architectures or evaluate feature selection across wider combinations of cardinality and coefficient of variation. Blocker: None

Data Value Analytics for Feature Selection and Dataset Retrieval · University College Dublin · PhD

Who has worked on this

Authors of the public theses cited on this page, most-cited first.

Related methods in Machine learning & AI

Transformers & Attention ModelsReinforcement LearningFeature Extraction & SelectionZero-Shot LearningRandom Forests & Tree EnsemblesData Augmentation

Checking a claim in this area?

We can run the same search on any method or claim. If nothing turns up, we will say so, and that proves nothing on its own.

Ask about a method