A clinical trial fails. The researchers don't want to disappoint their funder. So in the published paper, they quietly swap the primary endpoint from "all-cause mortality" to "incidence of cardiovascular events"—the latter showing a faint improvement. This practice is called spin, and it's more common in the medical literature than you might think.
Manual audits have found that 47%–71% of failed clinical trial abstracts contain spin. But auditing a single paper by hand takes hours, and tens of thousands of RCT papers are published globally every year—human eyes simply can't keep up.
A new paper proposes a clever automated approach: use a large language model to check whether the "declared endpoints" and "reported endpoints" in a clinical trial paper are consistent. It reaches an F1 score of 0.78 and an accuracy of 0.90.
What is outcome switching?
Clinical trials pre-register a "primary endpoint" at the design stage—say, "reduction in HbA1c after 12 weeks of treatment." This is the study's core promise. But when the paper is published, the reported endpoint sometimes differs from the registered one: a secondary endpoint may be promoted to primary, a more favorable time point may be chosen, or "all-cause mortality" may be silently replaced with "cardiovascular mortality."
The harm is obvious: readers make decisions based on the paper—doctors decide what to prescribe, health agencies decide whether to reimburse a treatment, and follow-up researchers decide whether to keep investing in that direction. If the endpoint was quietly swapped, every decision based on it is skewed.
The core task: semantic similarity judgment
The paper formalizes the task as a semantic similarity problem: given a pair of outcome descriptions (registered primary outcome vs. reported outcome), decide whether they are semantically equivalent.
That sounds simple, but it's subtle. The paper gives two examples:
- Similar pair: registered outcome "log HbA1c" vs. reported outcome "glycaemic control." The latter is a broader term; judging them as a match requires medical knowledge—HbA1c is a measure of glycemic control.
- Dissimilar pair: registered outcome "blood pressure" vs. reported outcome "systolic BP." The latter is a subset of the former; swapping "blood pressure" for "systolic pressure" may be cherry-picking a favorable metric.
Method: a three-step LLM pipeline
The research team used a concise three-step approach:
Step 1: Prompt engineering. Give the LLM a carefully designed prompt to judge whether two outcome descriptions are similar. The prompt includes task instructions, decision criteria, and a few examples.
Step 2: Token probability classification. Instead of only reading the model's generated answer, they look at the probabilities of the "yes" and "no" tokens. This is more stable than parsing generated text—even if the model produces a long explanation, the final judgment rests on the yes/no probability distribution.
Step 3: Majority voting. Run the same outcome pair multiple times (different temperatures, prompt variants) and take the majority vote. This reduces the randomness of any single inference.
On a test set of 2,496 outcome pairs, the method achieves F1 = 0.78 and accuracy 0.90. For comparison, traditional text similarity methods (Levenshtein distance, stemming-based similarity, Word2Vec cosine similarity) generally land at F1 between 0.5 and 0.6. Fine-tuned BERT performs slightly better than the LLM approach—but requires labeled data.
An analogy: contract review
The task is essentially "contract review"—the trial's registration record is the "contract," and the paper itself is the "execution." You're checking whether execution stayed faithful to the contract.
Traditional methods are like a paralegal comparing word by word: keyword overlap, sentence-structure similarity. They catch obvious inconsistencies ("blood pressure" changed to "heart rate") but miss subtle spin ("blood pressure" changed to "systolic pressure").
The LLM approach is like a senior lawyer: it understands the semantic relationship between "blood pressure" and "systolic pressure," knows the latter is a subset of the former, and can judge whether the substitution constitutes spin. But even lawyers make mistakes—F1 = 0.78 means roughly 1 in 5 judgments is wrong. So the paper positions the LLM as a first-pass screening tool: automatically scan all papers, flag the suspicious ones, then have humans audit and confirm.
Why not fine-tune the LLM?
One interesting finding: an untuned LLM already comes close to fine-tuned BERT. That matters a lot for real-world deployment.
Fine-tuning needs labeled data, and labeling needs medical experts—one annotator spending days on 300 outcome pairs. With tens of thousands of new RCT papers each year, labeling data to fine-tune models for every medical subfield is unsustainable.
The zero-shot LLM approach's advantage: to switch domains (say, from cardiology to oncology), you only change the examples in the prompt—no relabeling required. That makes rapid deployment across different medical scenarios far easier.
Limitations and an honest assessment
The paper has clear limitations. First, F1 = 0.78 means 22% of judgments are wrong—false positives will wrongly implicate innocent papers, false negatives will let real spin slip through. In a high-stakes field like medicine, this accuracy is only suitable for screening, not final judgment. Second, the paper only tests outcome switching; other types of spin (selective reporting of subgroup results, sugarcoating failed results) aren't covered. Third, hyperparameters were tuned on 300 labeled pairs and tested on 2,496—the data volume is small, and generalizability remains a question.
But the value of this work isn't the accuracy—it's bringing LLMs into a field that previously relied entirely on manual labor. With 47%–71% of failed RCTs containing spin, and tens of thousands of new RCT papers published annually, any automation tool—even a mere first-pass screener—can dramatically reduce the cost of manual auditing. The authors are candid too: this is undergraduate thesis work, not a revolutionary breakthrough, but it points to a pragmatic direction—LLMs aren't just chatbots; on professional "semantic auditing" tasks, they're starting to do real work.
---
Paper: Detecting Spin in Clinical Trials with Large Language Models arXiv: 2610.11845 Authors: Tjaša Ajdovec, Marko Robnik-Šikonja, Simon Šuster Code: ta5946/Spin