Library · Optimization & control
What Gradient Descent & Backprop got wrong, from 38 theses
Summary written by a model from the records below: Across these records, gradient descent and backpropagation techniques frequently struggle when facing complex non-convex landscapes, sensitive step size regimes, and non-smooth objective functions. Authors report recurring optimization failures including entrapment in local minima, gradient vanishing or breakdown, and divergence when optimizing against surrogate models or noisy finite differences. These records come from graduate theses, 2021 to 2025, each labeled with its degree where the repository states it, and each links to its thesis. They were extracted by language models reading the full text, so treat each as a lead to read, not a verdict.
Plus 9 problems the authors left open, listed at the end.
Failure modes, by theses reporting them
Select a mode to jump to its records.
Gradient descent frequently gets trapped in suboptimal local minima in non-convex and multimodal landscapes
Gradient-based methods routinely stall in poor local minima or extrema when applied to non-convex parameter spaces without favorable initializations. This entrapment leads to premature termination, loss of failure mode diversity, and inferior performance compared to global or population-based search techniques.
Considered and rejected
Considered and rejected: Rejected standard Backpropagation/Gradient Descent for LSTM training due to susceptibility to local minima entrapment.
Caching in VANETs for Social Networking · Queen's University · PhD
Considered and rejected
Considered and rejected: Rejected standard backpropagation/gradient-descent due to slow training speed, learning rate tuning issues, and risk of local minima.
Efficient and Accurate Neural Network Based Internal Combustion Engine Modeling and Prediction · University of Windsor · PhD
Considered and rejected
Considered and rejected: Rejected standard gradient descent and Newton's method due to getting trapped in local minima.
Singularity distance computations of parallel manipulators of Stewart-Gough type · TU Wien · PhD
Tried and failed
gradient descent pose optimization without initialization applied to slice-to-volume medical image registration. Outcome: did not converge. Reason: optimization gets trapped in local minima under severe motion without a good initial pose estimate
A Robust and Efficient Framework for Slice-to-Volume Reconstruction: Application to Fetal MRI · MIT · PhD
Tried and failed
gradient descent on forward prediction error applied to rigid body contact parameter identification. Outcome: did not converge. Reason: optimization got trapped in local minima representing smoothed geometry rather than true non-smooth shapes
Tried and failed
gradient descent on relative 3D rotation applied to relative orientation estimation. Outcome: worse than baseline. Reason: optimization directly over relative orientation frequently gets trapped in local optima
Image-based Pose Estimation for Previously-Unseen Objects · EPFL · PhD
Tried and failed
direct target-space loss functions for projection models applied to density-density response function prediction. Outcome: worse than baseline. Reason: gradient descent got trapped in poor local minima compared to direct coefficient regression
Machine Learning static RPA response properties for accelerating GW calculations · Imperial · PhD
Tried and failed
gradient descent with zero initialization applied to 6-DOF pose estimation from shadow corners. Outcome: did not converge. Reason: gets trapped in poor local minima
Uncertain Sampling with Certain Priors · EPFL · PhD
Tried and failed
greedy descent local search metaheuristic applied to dynamic orienteering problem. Outcome: worse than baseline. Reason: rapidly converged to suboptimal local extrema yielding poor solution fitness
Practically optimal UAV mission planning under uncertainty · Imperial · PhD
Tried and failed
gradient-based parameter identification from random initialization applied to inverse parametric PDE problems. Outcome: did not converge. Reason: Non-convex optimization landscape trapped gradient descent in local minima without basin-of-attraction knowledge or global pre-initialization.
Error Assessment for Finite Elements/Neural Networks Methods Applied to Parametric PDEs · EPFL · PhD
Lost to a baseline
Gradient descent in parameter space was less accurate, showed higher run-to-run variance, and converged slower (e.g., 25.6-40.9 s vs 8.77-10.9 s) than Particle Swarm Optimization (PSO) when solving the 4D inverse parameter identification problem.
Error Assessment for Finite Elements/Neural Networks Methods Applied to Parametric PDEs · EPFL · PhD
Considered and rejected
Considered and rejected: Rejected gradient descent for human-in-the-loop multi-parameter optimization because the required perturbation points scale linearly with dimension and it risks getting trapped in local minima.
Soft Exosuits for Improved Walking Efficiency and Community Based Post-Stroke Gait Rehabilitation · Harvard · PhD
Considered and rejected
Considered and rejected: Rejected direct search and gradient-based optimization algorithms because direct search lacks precision for multi-objective trade-offs and gradient methods are prone to trapping in local extrema
DIGITAL CODESIGN OF A POWER MODULE WITH INTEGRATED THERMAL MANAGEMENT · Georgia Tech · PhD
Considered and rejected
Considered and rejected: Rejected gradient descent optimization for trial wavefunction parameter optimization because it gets trapped in local minima; adopted dual annealing global optimization instead.
Electronic quantum fluids in graphene and quantum Hall systems · Oxford · PhD
Considered and rejected
Considered and rejected: Gradient-based optimization (steepest descent, conjugate gradient) for FQ solvent parameter fitting, rejected due to the non-convex parameter space with multiple local minima
Polarizable QM/MM Approaches for Molecular Excited States in Complex Environments · Scuola Normale Superiore · PhD
Considered and rejected
Considered and rejected: Rejected descent methods, evolutionary algorithms, and statistical sampling methods for global parameter optimization due to premature termination at local minima, non-convergence, or inefficiency in higher dimensions.
Efficient Biomolecular Computations Towards Applications in Drug Discovery · Virginia Tech · Doctorate
Considered and rejected
Considered and rejected: Rejected pure gradient-based adversarial optimization for failure generation because local optima cause diversity collapse into a single failure mode.
Breaking things so you don’t have to: risk assessment and failure prediction for cyber-physical AI · MIT · PhD
Considered and rejected
Considered and rejected: Gradient-Descent optimization algorithms rejected for JPEG quantization table optimization due to settling on local minima in multi-modal rate-distortion space (Evolutionary / Particle Swarm Optimization used instead).
Digital CMOS ISFET architectures and algorithmic methods for point-of-care diagnostics · Imperial · PhD
Poor learning rate selection and ill-conditioned curvatures cause severe oscillations or divergence
Unbounded activations, generalized smoothness bounds, and unbalanced parameter gradient magnitudes cause gradient descent to oscillate heavily or diverge under fixed or large step sizes. In sensitive architectures, fixed learning rates or poorly tuned hypergradient updates lead to coordinate drift and failure to converge on ill-conditioned objectives.
Tried and failed
gradient descent without non-linear activations applied to linear neural network training. Outcome: did not converge. Reason: linear networks lack bounded activations that stabilize optimization in large learning rate regimes
Making Sense of Training Large AI Models · MIT · PhD
Lost to a baseline
Gradient descent on quadratic forms lagged behind Conjugate Gradient due to iterate oscillation between subspaces and poor convergence.
MICROWAVE IMAGING FOR WALK-WHILE-SCAN SECURITY SCREENING · Duke · PhD
Tried and failed
gradient descent with large learning rates applied to optimization of functions with poor regularity. Reason: insufficient objective regularity prevents edge of stability, causing premature one-sided stability bounds instead
Quantitative convergence analysis of dynamical processes in machine learning · Georgia Tech · PhD
Tried and failed
gradient descent with uniform parameter weighting applied to joint pose and shape optimization. Outcome: did not converge. Reason: gradient magnitudes for rotation and shape dominated translation, causing severe oscillations and optimization failure
Integrated 3D anatomical model for myocardial segmentation in cardiac CT imagery · Georgia Tech · PhD
Tried and failed
gradient descent with fixed step size applied to non-standard smooth non-convex optimization. Outcome: did not converge. Reason: fixed step sizes cause divergence or sublinear convergence under generalized smoothness bounds
Optimization Theory and Machine Learning Practice: Mind the Gap · MIT · PhD
Lost to a baseline
Gradient descent with large stepsizes in the Edge of Stability regime loses to small-stepsize gradient flow / SGD for sparse recovery, exhibiting high test error due to coordinate drift towards zero.
Deep Learning Theory Through the Lens of Diagonal Linear Networks · EPFL · PhD
Lost to a baseline
Hypergradient descent sometimes got stuck for too small initial learning rates (struggled on F-MNIST experiments) due to sensitivity of the hyper learning rate across architectures.
Probabilistic Linear Algebra for Stochastic Optimization · Tübingen · PhD
Lost to a baseline
Using a single constant learning rate (0.0005) for SGD across all regularization levels without validation grid search performed poorly compared to liblinear coordinate descent.
NAVIGATING HETEROGENEITY TO LEARN FROM LARGE-SCALE CANCER DATA: OPTIMIZATION, REDUNDANCY, AND GENERALIZATION · Penn · PhD
Gradient vanishing and derivative breakdown halt optimization on non-smooth or discrete objectives
Optimization halts when objective landscapes contain inactive neurons, softmax saturations, or margin losses that cause gradients to vanish entirely. Furthermore, subgradient calculations and standard backpropagation break down on discrete logical decisions, non-smooth path regularizations, and discontinuous objectives.
Considered and rejected
Considered and rejected: Rejected standard gradient descent/backpropagation for symbolic reasoning due to inability to represent discrete logical decisions and out-of-distribution failures.
The Capabilities of Neural Systems Depend on a Hierarchically Structured World · UT Southwestern
Tried and failed
gradient-based optimization of model interpolation weights applied to graph neural network model merging. Outcome: worse than baseline. Reason: vanishing gradients in softmax prevented zeroing out poor-performing model ingredients
Improved and domain specific model soups · Iowa State · Master's
Tried and failed
gradient descent without smooth activation relaxation applied to locally constant neural networks. Outcome: did not converge. Reason: vanished gradients occur everywhere except boundaries due to zero gradients when neurons are inactive
Building Transparent Models · MIT · PhD
Considered and rejected
Considered and rejected: Directly using the margin loss in stochastic gradient descent was rejected due to gradient vanishing, using cross-entropy approximations instead.
Multi-Domain Text Classification with Adversarial Training · Carleton University · PhD
Tried and failed
gradient descent with automatic differentiation applied to non-smooth path regularized optimization. Outcome: too slow. Reason: subgradient computation via automatic differentiation struggles with non-smooth path-norm objectives
Predicting in Uncertain Environments: Methods for Robust Machine Learning · EPFL · PhD
Considered and rejected
Considered and rejected: Rejected classical gradient-based and sub-gradient optimization methods for decentralized robot control optimization because the objective function is discontinuous, non-convex, and multimodal.
HOLISTIC SCENE PERCEPTION FOR COLLABORATIVE HUMAN-ROBOT TEAMS · Cornell · PhD
Considered and rejected
Considered and rejected: Standard gradient-descent optimization algorithms (e.g., Newton-Raphson) were rejected for discrete/mixed socio-demographic enrichment due to lack of smoothness and derivative breakdown.
Inverse discrete choice modelling: a framework for socio-demographic enrichment of big data · Imperial · PhD
Finite difference approximations introduce severe noise sensitivity and high computational overhead
Estimating gradients through finite differences or local numerical sampling suffers from extreme sensitivity to stochastic evaluation noise and causes optimizer tuning failures. The high computational cost of numerical gradient evaluation also outweighs optimization benefits in low-dimensional settings.
Considered and rejected
Considered and rejected: Rejected Finite-Difference Gradient Descent (FDGD) for latent space optimization because it failed completely under biologically realistic stochastic evaluation noise.
Optimization of neural response in the primate dorsal visual pathway Optimierung der neuronalen Antwort im dorsalen visuellen Pfad von Primaten · University of Marburg · PhD
Tried and failed
numerical gradient descent based local sampling applied to low-dimensional robot motion planning. Outcome: too slow. Reason: numerical gradient computation overhead outweighed the benefit of informed local sampling in low dimensions
INFORMED EXPLORATION ALGORITHMS FOR ROBOT MOTION PLANNING AND LEARNING · Georgia Tech · PhD
Considered and rejected
Considered and rejected: Rejected gradient-based local optimizers (fmincon) for the outer loop in parametric coil optimization because finite-difference parameter gradients were noisy and caused tuning failures; switched to global genetic algorithms.
Integral Equation-Based Inverse Scattering and Coil Optimization in Magnetic Resonance Imaging · MIT · PhD
Optimization drifts and diverges when relying on inaccurate surrogates or failing to account for bilevel dependencies
Gradient-based input optimization against inaccurate forward surrogates causes real-world control errors to diverge because the optimizer exploits surrogate model errors. Additionally, simultaneous gradient descent fails on bilevel objectives by ignoring parameter dependencies, and linearized gradient approximations fail to reflect true counterfactual behavior.
Tried and failed
influence functions and gradient-based data attribution applied to deep neural network predictions. Outcome: no signal. Reason: linearized gradient approximations fail to accurately predict leave-one-out counterfactual model behavior
Machine Learning through the Lens of Data · MIT · PhD
Considered and rejected
Considered and rejected: Rejected simultaneous gradient descent for solving bilevel optimization problems due to failure to account for parameter dependencies on upper-level variables.
Indiscriminate data poisoning against supervised learning: general attack formulations, robust defences, and poisonability · Imperial · PhD
Tried and failed
gradient-based input synthesis with learned forward models applied to iterative plant control waveform generation. Outcome: unstable. Reason: excessive offline gradient optimization against inaccurate forward models caused real-world response errors to diverge
A novel Adaptive Filtering approach to Drive File Identification for Service Environment Replication · Virginia Tech · Doctorate
Left open by the authors
Problems the authors named and did not get to.
Left open
Develop a theoretical framework explaining why neural networks learn linear functions before non-linear functions during stochastic gradient descent dynamics. Blocker: No specific mathematical formulation, assumptions, or analytical approach are provided
On Scaling Dynamics in Deep Learning · Harvard · PhD
Left open
Extend the MedImpute formulation and coordinate descent algorithm to incorporate support vector machines and decision tree objective functions. Blocker: None
Novel Machine Learning Algorithms for Personalized Medicine and Insurance · MIT · PhD
Left open
Develop an online dynamic MRI subspace tracking method using stochastic gradient descent for temporal subspace updates without mini-batch initialization. Blocker: None
Generalizable low-latency accelerated dynamic MRI · Iowa State · PhD
Left open
Extend the convergence rate analysis of gradient descent-ascent dynamics to broader non-convexity conditions such as weaker gradient dominance or local convexity. Blocker: None
Convergence Rates of Gradient Descent-ascent Dynamics under Computation Constraints in Solving Min-max Optimization · Virginia Tech · Master's
Left open
Apply harmless diversity descent to find diverse policies for robotics planning and reinforcement learning. Blocker: No specific robotics benchmark, environment, or evaluation metrics are provided
Adding prior into generative model · UT Austin · PhD
Left open
Develop a generalized step-size selection algorithm for time and control updates in adjoint gradient-based time-optimal control without example-specific heuristics. Blocker: Lack of specific target algorithmic approach or convergence criteria
Left open
Characterize the class of constrained optimization problems that can be solved using penalized gradient descent methods. Blocker: Lacks concrete problem scope, target problem classes, and specific analytical criteria.
Topics in Generalized Correlation Analysis and Clustering · Penn · PhD
Left open
Extend the convergence analysis of gradient descent-ascent dynamics to stochastic and adversarial min-max settings with gradient noise or perturbations. Blocker: None
Convergence Rates of Gradient Descent-ascent Dynamics under Computation Constraints in Solving Min-max Optimization · Virginia Tech · Master's
Left open
Prove stepsize-independent lower bounds for gradient descent under directional smoothness conditions. Blocker: None
Smoothness and Adaptivity in Nonlinear Optimization for Machine Learning Applications · MIT · PhD
Who has worked on this
Authors of the public theses cited on this page, most-cited first.
- Maude Girardin · EPFL · PhD · 2025
Error Assessment for Finite Elements/Neural Networks Methods Applied to Parametric PDEs - Duy Anh Do · Virginia Tech · Master's · 2025
Convergence Rates of Gradient Descent-ascent Dynamics under Computation Constraints in Solving Min-max Optimization - Qiuchen Qian · Imperial · PhD · 2025
Practically optimal UAV mission planning under uncertainty - Chen Zhao · EPFL · PhD · 2025
Image-based Pose Estimation for Previously-Unseen Objects - Joseph Lee Zuber · Iowa State · Master's · 2025
Improved and domain specific model soups - Alexander Kreß · University of Marburg · PhD · 2025
Optimization of neural response in the primate dorsal visual pathway Optimierung der neuronalen Antwort im dorsalen visuellen Pfad von Primaten - Silpa Babu · Iowa State · PhD · 2025
Generalizable low-latency accelerated dynamic MRI - Kwangjun Ahn · MIT · PhD · 2024
Making Sense of Training Large AI Models - Sung Min Park · MIT · PhD · 2024
Machine Learning through the Lens of Data - Yuqing Wang · Georgia Tech · PhD · 2024
Quantitative convergence analysis of dynamical processes in machine learning - Charles Burke Dawson · MIT · PhD · 2024
Breaking things so you don’t have to: risk assessment and failure prediction for cyber-physical AI - Haochuan Li · MIT · PhD · 2024
Smoothness and Adaptivity in Nonlinear Optimization for Machine Learning Applications
Checking a claim in this area?
We can run the same search on any method or claim. If nothing turns up, we will say so, and that proves nothing on its own.