Library · Machine learning & AI
What Natural Language Processing got wrong, from 27 theses
Summary written by a model from the records below: Research across natural language processing highlights frequent breakdowns when deploying complex models, dense representations, or surface heuristics to specialized domain tasks. Across various domains, advanced generative approaches and transfer pipelines repeatedly struggled with domain shifts, nuanced semantics, and compression artifacts compared to simpler baselines. These records come from graduate theses, 2021 to 2026, each labeled with its degree where the repository states it, and each links to its thesis. They were extracted by language models reading the full text, so treat each as a lead to read, not a verdict.
Plus 9 problems the authors left open, listed at the end.
Failure modes, by theses reporting them
Select a mode to jump to its records.
Large language models fail to produce accurate or structured outputs when prompted or fine-tuned for specialized tasks
Prompting and fine-tuning large language models produced vague suggestions, omitted essential numerical metrics, or failed to outperform random chance on legal ambiguity. Models also lost citation grounding during reasoning fine-tuning and suffered degraded performance when forced into code-structured prompts.
Tried and failed
prompting large language models for domain summarization applied to clinical report generation. Reason: inconsistent incorporation of core numerical metrics led to clinically inappropriate recommendations
Precision Medicine in Diabetes Using Continuous Glucose Monitoring · MIT · PhD
Tried and failed
direct zero-shot prompting of large language models applied to clinical note information extraction. Outcome: did not generalise. Reason: failed to adapt and performed poorly on advanced reasoning models without structured reasoning or fine-tuning
Machine Learning Approaches for Drug Combination Discovery · Cornell · PhD
Tried and failed
supervised fine-tuning for structured reasoning applied to large language models for clinical grounding. Outcome: worse than baseline. Reason: optimizing for diagnostic correctness collapsed citation grounding and degraded structured output format compliance
Causal Inference and Evidence-Grounded Language Models for Trustworthy Personalized Clinical Decision Support · Georgia Tech · PhD
Tried and failed
code-formatted prompting of large language models applied to general natural language processing tasks. Outcome: worse than baseline. Reason: code structure constraints impaired general language and reasoning tasks compared to natural text prompts
Structured Event Reasoning with Large Language Models · Penn · PhD
Tried and failed
prompting large language models for critique generation applied to evaluating complex student writing. Outcome: worse than baseline. Reason: generated critiques were vague, generic, and repetitive compared to human instructor assessments
Addressing teachers’ needs for integrating generative AI into a pedagogical feedback model · Iowa State · Master's
Tried and failed
large language models for automated grading applied to student engineering derivations and solutions. Reason: Outputted full solutions and generic suggestions rather than pinpointing specific error locations in student work
Developing an Automated Practice Environment and Feedback Engine for Guided Instruction in the Deformable Bodies Course · Virginia Tech · Doctorate
Language models and evaluation metrics fail to transfer effectively across specialized domain vocabularies
Domain-specific models and general readability metrics struggled when applied to informal patient text, unseen programming languages, or specialized financial and medical vocabularies. Domain-adapted fine-tuning often added computational cost without improving downstream task accuracy over general-purpose language models.
Tried and failed
domain-specific pretrained language model applied to patient-authored text classification. Outcome: worse than baseline. Reason: linguistic mismatch between formal clinical training corpora and informal patient-authored text
Developing Novel NLP-Enabled Timely and Accurate Decision Making for Precision Medicine · Georgia Tech · PhD
Tried and failed
domain-specific pretrained language model fine-tuning applied to clinical text classification. Outcome: worse than baseline. Reason: biomedical-specific pretrained models showed no consistent performance advantage over general-domain pretrained models
SOCIAL DISPARITIES IN HEALTHCARE UTILIZATION FOR CHILDREN AND ADOLESCENTS WITH MIGRAINE DISEASE · Penn · PhD
Tried and failed
General-domain readability metrics and span complexity models applied to medical text simplification evaluation. Outcome: did not generalise. Reason: Lexical metrics and general-corpus models fail to capture specialized technical and clinical vocabulary needs
Studying Text Revision in Scientific Writing · Georgia Tech · PhD
Tried and failed
distilled general language model sentiment analysis applied to domain-specific financial news sentiment classification. Outcome: worse than baseline. Reason: produced poor directional accuracy and negatively correlated with domain-specific models during divergence periods
An Analysis of the Impact of News-Media Sentiment on Cryptocurrency Pricing · UT Austin
Tried and failed
Domain-adapted language model for text sentiment features applied to downstream financial market direction prediction. Outcome: worse than baseline. Reason: Domain-specific fine-tuning added computational cost without improving downstream task accuracy over general-purpose language models
Machine Learning for Financial Market Forecasting · Harvard · Master's
Tried and failed
pretrained code language model embeddings applied to unseen programming language code retrieval. Outcome: did not generalise. Reason: the model was not pretrained on the target programming language
Structure-aware graph representation learning using graph neural networks · Iowa State · PhD
Large language models and complex embeddings underperform simpler specialized baselines
Generative models and dense sentence embeddings frequently lagged behind simple tokenization, standard tabular features, and smaller encoder architectures like BERT or RoBERTa. Zero-shot large language model prompting also suffered from lower discrimination and elevated false positive rates compared to embedding classifiers.
Tried and failed
Sentence-BERT embeddings for textual features applied to tabular text feature representation. Outcome: worse than baseline. Reason: Trailed simple tokenization accuracy in predictive modeling
Machine Learning-Based Predictive Modeling for OTC and exercise recommendation for knee joint pain with long-tail classification · Texas Tech · Master's
Tried and failed
fine-tuned large language models for sequence extraction applied to named entity recognition. Outcome: worse than baseline. Reason: Generative LLMs underperformed specialized smaller encoder models like BERT on token-level span classification tasks.
Consistency-aware and LLM-assisted methods for named entity recognition · Iowa State · PhD
Lost to a baseline
General language representation from LLAMA-65B embedding (AUROC 0.53–0.68) performed worse or comparable to simple utterance-level RoBERTa sentiment analysis (AUROC 0.63–0.69)
Multimodal assessment of neuropsychiatric disorders using audiovisual recordings · Georgia Tech · PhD
Lost to a baseline
Transformer language model reaction embeddings failed to outperform Random Forest, Extreme Gradient Boosting, or Feedforward Neural Networks using standard fingerprint or DFT features.
Machine Learning for Chemical Reactivity Prediction: Paradigms, Challenges, and Applications · MIT · PhD
Tried and failed
prompt-based zero-shot large language model classification applied to student text behavior classification. Outcome: worse than baseline. Reason: produced substantially lower AUC and more than double the false positives compared to embedding-based classifiers
MEASURING AND UNDERSTANDING STUDENTS’ SELF-REGULATED LEARNING IN TEXTUAL DATA IN COMPUTER-BASED LEARNING ENVIRONMENTS · Penn · PhD
Lexical and rule-based methods fail to capture subtle linguistic nuances and complex semantics
Rule-based systems and surface lexical metrics failed to handle complex clinical negation, correlated weakly with neural classifier success, and underperformed topic features. Automated text processing on unstructured records was rejected due to risk of false positives, false negatives, and an inability to apply nuanced judgment.
Tried and failed
rule-based natural language processing applied to clinical report diagnosis extraction. Reason: failed to correctly handle complex clinical negation, causing poor specificity and low F1 score
Artificial intelligence applications for the acquisition, analysis and reporting of cardiovascular magnetic resonance imaging · Imperial · PhD
Considered and rejected
Considered and rejected: Rejected using EHR free text/unstructured clinical notes via natural language processing due to unrecognized false negatives, potential false positives, and impracticality for large population risk stratification.
ASSESSING THE EFFECT OF COMPUTABLE PHENOTYPES IN PREDICTING HEALTHCARE UTILIZATION AND POTENTIAL RACIAL DISPARITIES IN IDENTIFYING TYPE 2 DIABETES POPULATION · Johns Hopkins · Doctorate
Tried and failed
direct lexical word features in discrete choice applied to social media engagement prediction. Outcome: worse than baseline. Reason: yielded lower prediction accuracy and poorer interpretability than aggregate topic features
Tried and failed
TF-IDF cosine distance and lexical overlap metrics applied to predicting neural classifier performance. Outcome: no signal. Reason: lexical similarity metrics showed weak and inconsistent correlation with classification success across diverse texts
The computational bridge: Interfacing theory and data in cognitive science · Cornell · PhD
Direct embeddings of raw technical syntax fail without natural language pre-training or translation
Training models directly on raw security logs, source code, or mathematical formulas resulted in substantial representation gaps. Replacing raw technical structures with natural language summaries or pre-training significantly improved dense retrieval and rule matching.
Considered and rejected
Considered and rejected: Rejected training BERT from scratch on raw security logs without natural language or code pre-training
Language Models and Cybersecurity - Applications and Current Limits · Politecnico di Torino · PhD
Considered and rejected
Considered and rejected: Rejected using raw source code embeddings directly to match natural language misuse rules due to representation gap; replaced code with LLM-generated natural language summaries.
API Utility Enhancement: From Traditional Software to Deep Learning Frameworks · York University · PhD
Considered and rejected
Considered and rejected: Rejected using raw LaTeX formulas directly for dense retrieval, finding that LLM-generated natural-language slogans significantly improve retrieval quality
Neural Methods for Plasma Simulation and Mathematical Formalization · University of Washington · PhD
Compression and federated optimization techniques degrade intermediate language representations
Layerwise intermediate matching in distillation and tensor-train decomposition damaged feature representations necessary for student learning and commonsense reasoning. Similarly, momentum-based federated optimizers destroyed gradient token sparsity during language model training, causing them to trail standard stochastic gradient descent.
Tried and failed
layerwise hidden representation matching in knowledge distillation applied to cross-architecture language model compression. Outcome: worse than baseline. Reason: unfiltered layerwise intermediate state matching overly constrained student learning compared to output probability distillation
On Parameter Efficiency of Neural Language Models · Georgia Tech · PhD
Tried and failed
tensor-train decomposition for neural network compression applied to dense language model layers. Outcome: worse than baseline. Reason: degrades broad, weak lower-level feature representation required for commonsense reasoning tasks
Generative language model compression with algebraic approaches · Imperial · PhD
Lost to a baseline
In language modeling (Shakespeare charRNN and StackOverflow LSTM), momentum-based federated optimizers (FedAvgMom, MimeMom, MimeAdam) underperformed non-momentum baselines (FedAvgSGD, FedAvgAdagrad) because dense momentum updates destroy mini-batch gradient token sparsity.
Optimization methods for collaborative learning · EPFL · PhD
Left open by the authors
Problems the authors named and did not get to.
Left open
Extend the clinical graph attention network to incorporate unstructured clinical notes using large language model embeddings alongside structured patient features. Blocker: Access to credentialed clinical datasets containing linked tabular features and free-text notes (e.g., PhysioNet/MIMIC)
Essays on developing artificial intelligence solutions for patient-centered healthcare delivery · UT Austin · PhD
Left open
Use natural language processing on inpatient clinical notes to extract consult questions and identify resulting treatment changes. Blocker: Requires access to private electronic health record (EHR) clinical notes data
Essays on Physician Consults · Penn · PhD
Left open
Apply textual sentiment analysis to alternative macroeconomic texts to predict other macroeconomic indicators. Blocker: None
The Usefulness of Textual Sentiment Analysis for Macroeconomics · UT Austin
Left open
Analyze how stylistic and lexical properties of natural language feedback, such as emotional tone, affect LLM performance across different models. Blocker: None
Language Modeling with Few-Shot Language Feedback · Georgia Tech · PhD
Left open
Learn linear transformations across embedding spaces to enable domain transfer of self-regulated learning classification between chemistry and formal logic text datasets. Blocker: Requires the private student textual learning interaction datasets from the specific chemistry and formal logic domains used in the thesis.
MEASURING AND UNDERSTANDING STUDENTS’ SELF-REGULATED LEARNING IN TEXTUAL DATA IN COMPUTER-BASED LEARNING ENVIRONMENTS · Penn · PhD
Left open
Apply advanced large language models and hybrid quantum-classical algorithms to source code representations for software defect prediction. Blocker: The unfinished work lacks concrete specifications, specific model architectures, or target benchmarks
Enhancing Software Defect Prediction: Investigating Diverse Representations of Source Code as Feature Values in Classical and Quantum Machine Learning Approaches · Saskatchewan · PhD
Left open
Apply neuron redundancy insights to prune, distill, and optimize inference costs of code-trained language models. Blocker: None
Analyzing redundancy in code-trained language models · Iowa State · Master's
Left open
Develop training algorithms beyond standard maximum likelihood estimation and non-autoregressive methods for neural language generation models. Blocker: The goal is a broad research direction without concrete specifications, objectives, or defined algorithmic formulations
Towards a Deeper Understanding of Neural Language Generation · MIT · PhD
Left open
Apply lifelong learning fine-tuning techniques to large language models for generating automated code review comments across evolving codebases. Blocker: None
Towards Sustainable AI for Continuous Integration Quality Gates · Queen's University · PhD
Who has worked on this
Authors of the public theses cited on this page, most-cited first.
- Jiayi Zhang · Penn · PhD · 2025
MEASURING AND UNDERSTANDING STUDENTS’ SELF-REGULATED LEARNING IN TEXTUAL DATA IN COMPUTER-BASED LEARNING ENVIRONMENTS - Junior Ben Tamo · Georgia Tech · PhD · 2026
Causal Inference and Evidence-Grounded Language Models for Trustworthy Personalized Clinical Decision Support - Saime Esma Esma Masca · Iowa State · Master's · 2026
Addressing teachers’ needs for integrating generative AI into a pedagogical feedback model - Arinjoy Basak · Virginia Tech · Doctorate · 2026
Developing an Automated Practice Environment and Feedback Engine for Guided Instruction in the Deformable Bodies Course - Danielle Joy Kellier · Penn · PhD · 2026
SOCIAL DISPARITIES IN HEALTHCARE UTILIZATION FOR CHILDREN AND ADOLESCENTS WITH MIGRAINE DISEASE - Ying Wei · Iowa State · PhD · 2026
Consistency-aware and LLM-assisted methods for named entity recognition - Vasily Ilin · University of Washington · PhD · 2026
Neural Methods for Plasma Simulation and Mathematical Formalization - Mingxue Xu · Imperial · PhD · 2025
Generative language model compression with algebraic approaches - Priyanka Raghavan · MIT · PhD · 2025
Machine Learning for Chemical Reactivity Prediction: Paradigms, Challenges, and Applications - Elizabeth Healey · MIT · PhD · 2025
Precision Medicine in Diabetes Using Continuous Glucose Monitoring - Chengqi Xu · Cornell · PhD · 2025
Machine Learning Approaches for Drug Combination Discovery - Christopher Richardson · Georgia Tech · PhD · 2025
Language Modeling with Few-Shot Language Feedback
Checking a claim in this area?
We can run the same search on any method or claim. If nothing turns up, we will say so, and that proves nothing on its own.