Announcing the Winners

Aletheia's Quest · Summer 2026

The Aletheia’s Quest competition brought together researchers creating datasets of model lies and researchers developing methods to detect these lies.

Language models occasionally confabulate, hallucinating false answers while often adopting an overly confident tone. However, not all mistakes stem from a lack of knowledge: models can also lie. That is, they can make statements they know to be false. What methods best detect whether a model lied, and how much does white-box access (weights, activations, activation steering) contribute beyond black-box tools alone?

These questions motivated Aletheia’s Quest, an LLM lie-detection competition organized by Cadenza Labs, the National Deep Inference Fabric (NDIF) at Northeastern University, and funded by Schmidt Sciences. Over the summer, red teams built datasets of LLM lies, while blue teams developed detectors using either black-box access to models’ inputs and outputs, or white-box access to the internals of LLMs. On August 25, invited teams met in Boston to share their approaches, discuss results, and debate what the field should try next.

Note: A more detailed competition report will be published later.

Participation

Teams: 508power · aa · dami · EleutherAI · FlaiChi · GMU-Veritaserum · JSR · mpi · pinocchio · Recall Lab · ru · SAIN Groningen · Simit · Snake · Tara Research · Tony · TS · vk · yy

Winners & Competition Results

Ranked by mean AUROC across the ten held-out test datasets. Every finalist submission was re-executed by the organizers at full scale on a self-hosted NDIF cluster, against data no team had seen. To take a prize, a method also had to beat the best baseline in its track.

Special Awards

Final Validation Leaderboard Prizes

Awarded on the standing of the final validation leaderboard, ranked by average AUROC. To qualify, a method also had to rank above each of the six organizer baselines. Third place was awarded once per track.

Weekly Leaderboard Prizes

Throughout the Main Phase, the top three entries on the live leaderboard in each track took a weekly prize. Weekly standings were ranked by average balanced accuracy across the held-out validation datasets, with average AUROC as the tie-breaker; to qualify, a method also had to rank above each of the six organizer baselines.

Final Leaderboard

Ranked by AUROC  /  30 entries  /  select a row for its per-dataset scores

hollow marks a white-box method · filled marks black-box only · ✓ beats the best baseline in its track. Headline figures are means across all ten held-out test datasets. Datasets are named by scenario, so several datasets share a name; the Model and LoRA columns tell those apart.

Competition Showcase in Boston

On August 25, invited teams met in Boston to present their approaches, discuss results, and receive their awards.

Building Model Organisms of Lying Behaviors

The dataset effort brought together researchers from Cadenza Labs, the UK AI Security Institute, FAR.AI, MATS, the University of Oxford, the Tübingen ELLIS Institute and Fraunhofer HHI. Their main requirement was to produce on-policy lies and honest examples: responses actually generated by the model under the relevant conditions, with evidence supporting the lie/honest labels.

Development data included instructed factual deception and a wider range of prompts inducing models to lie, adapted for the competition models from Liars’ Bench and a dataset from Cooney et al. The validation scenarios tested two other settings: Medical-Sycophancy (Hunar Batra, University of Oxford), where agreement pressure pushed a model to endorse an incorrect claim in the context of medical diagnoses, and Soft-trigger Lies (Cadenza Labs), where a fine-tuned model answered incorrectly in the presence of specific cues.

Development data:

The validation scenarios:

The final test introduced six scenario families:

The Echoblast dataset illustrates why checking a statement against a judge’s own knowledge can be insufficient when identifying model lies: a model judge unfamiliar with the fictional company in question has little basis for distinguishing an honest admission of ignorance from an attempt to hide sensitive information.

Across the datasets, teams often noticed explicit admissions in models’ CoT reasoning, which made lies easier for creators to verify but also for detectors to catch.

Object of Belief Reason for Lying

Encouraged

World-knowledge

• Varied Deception (UK AISI)
• Instructed Deception (Cadenza Labs)
• Harmful-Looking (Honeypot)
• Blood on the Clocktower (BotC)

• Blood on the Clocktower (BotC)

• Medical-Sycophancy (Lie Auditors)
• Soft-Trigger (Cadenza Labs)

• Gender-Secret (UK AISI)
• AuditBench (UK AISI)
• Hidden-Goal (FAR.AI)
• Echoblast (Cadenza Labs)

Self-knowledge

Inherent

Design of the Competition

Following the operational definition used in Liars’ Bench, the competition treated a model assertion as a lie when the model stated something it believed to be false.

Dataset creators needed to provide evidence for that belief, such as the model consistently answering correctly when the pressure to lie was removed. This requirement made dataset construction a research problem in its own right, since a false statement alone could simply reflect a training error or model uncertainty. Red teams hence had to establish both what the model said and why the label was justified.

Blue teams were tasked with developing methods that, given a conversation from a suspect model, returned a “dishonesty score” for the final response. They worked with a diverse set of model organisms based on Qwen3.5-27B, Gemma-3-27B, and Nemotron-3-Super-120B, and could use Qwen3.5-9B as a judge model. Some models were brought into situations where they started lying naturally (e.g. sycophantic behavior); others had behaviors built into them through fine-tuning or prompting.

The competition gave teams a live leaderboard to hill-climb against, while keeping held-out test sets and model organisms behind the scenes to check that lie detectors generalized to new scenarios. Blue teams could use labeled development examples from the original Liars’ Bench dataset to train and test their methods. Blue teams also had access to a public leaderboard, where submitted methods were scored against a held-out validation set encompassing two additional datasets. Finally, we used most red teams’ models and datasets as fully held-out test scenarios to determine final performance.

Two-stage evaluation
	Public Leaderboard (Validation)	Held-out Test
Scenario Families	Soft-Trigger, Medical-Sycophancy (+23 development datasets not included in score)	AuditBench, Blood on the Clocktower, Echoblast, Gender-Secret, Hidden-Goal, Harmful-Looking
Organisms	base models, soft-trigger-greeting-*	ab_*, gender_secret_* (UK AISI), botc-qwen3.5-27b, echoblast-qwen3.5-27b, hidden-goal-model-organism-*
Base Models	Qwen ×2, Gemma ×2, Nemotron ×2	Qwen ×7, Gemma ×2, Nemotron ×1

This design allowed teams to receive live feedback about their submissions’ performance from the validation sets, while also preventing hill-climbing on the test set used to assess generalization capabilities. Teams could adapt parameters to different model architectures, but not to individual model organisms. This encouraged blue teams to develop approaches intended to generalize beyond a particular organism’s known behavior.

Powering white-box access to model subjects with NDIF

White-box methods developed by blue teams require more than an ordinary chat API. Participants in our competition recorded activations, trained and applied probing classifiers, inspected model logits and intervened in a forward pass. Thanks to NNsight’s remote execution capabilities, the NDIF team let teams run experiments on large models with up to 120B parameters without hosting the models themselves.

The competition ran on a dedicated cluster of four machines, each with eight H100 GPUs. Model organisms shared base-model deployments, with LoRA adapters dynamically switched between requests to optimize load thanks to the new NNsight v0.8 capabilities, avoiding a separate model copy for every participant.

For the NDIF team, this was a concrete test of shared infrastructure for AI auditing: researchers could compare detectors under a common execution setup, including methods that used internal access and methods that relied on conversations alone.

Leaderboard vs Test Performance

Method Landscape (n=25 reports)

Why AUROC, and not balanced accuracy

Methods were originally ranked on balanced accuracy, but we noticed that many submitted methods set their decision thresholds from statistics of the batch or dataset being classified. While our published rules did not explicitly forbid this, this level of access wouldn't be available in a realistic lie-detection scenario where each sample must be judged on its own.

Instead of forcing participants to adjust their thresholding methods, we chose to move the final award metric to AUROC. Because AUROC is threshold-free, it measures the quality of a method's scores independently of how any threshold was picked. The final reported submissions were ranked by mean AUROC computed separately on each held-out dataset, averaged across datasets.

Baselines

To be eligible to win awards, methods needed to score a higher average AUROC than the strongest baseline in their own track. The white-box baselines performed considerably better than the black-box ones, so we chose to have the two tracks carry different thresholds:

Track	Best baseline	AUROC to beat
White-box	Did You Lie probe	0.6611
Black-box	LLM self eval	0.5090

Competition Report

We are preparing a full competition report covering the datasets, model organisms, and an analysis of how different lie detection methods worked across submissions and datasets. We will update this website with a link to the report once it is live.

Competition's Datasets & Model Organisms

The datasets and model organisms used in the competition are available, for now, in the Aletheia’s Quest Datasets & Model Organisms collection on Hugging Face.

If you use them, please cite:

@misc{aletheias_quest_datasets,
  title  = {Aletheia's Quest Datasets},
  author = {Laurito, Walter and Kretschmar, Kieron and Nguyen, Jord and Bortz, Emma and Maiya, Sharan and Fiotto-Kaufman, Jaden and Sarti, Gabriele and Cooney, Alan and Batra, Hunar and Kortukov, Evgenii and Panfilov, Alexander and Miller, Justin and Gilg, Oscar and Serrano, Alex and Adam-Day, Sam and Cundy, Chris and Ripa, Michael and Wang, Zikai and Chan, Yik Siu and Floody, Olivia and Hase, Peter and Bau, David},
  note   = {\textsuperscript{*}Walter Laurito and Kieron Kretschmar contributed equally.},
  year   = {2026},
  publisher = {Hugging Face},
  doi    = {10.57967/hf/10730},
  url    = {https://doi.org/10.57967/hf/10730}
}