Announcing the Winners

Aletheia's Quest · Summer 2026

The Aletheia’s Quest competition brought together researchers building model organisms trained to lie in undisclosed settings and researchers developing methods to detect their lies.

Language models occasionally confabulate, hallucinating false answers while often adopting an overly confident tone. However, not all mistakes stem from a lack of knowledge: models can also lie. That is, they can make statements they know to be false. What methods best detect whether a model lied, and how much does white-box access (weights, activations, activation steering) contribute beyond black-box tools alone?

These questions motivated Aletheia’s Quest, an LLM lie-detection competition organized by Cadenza Labs, the National Deep Inference Fabric (NDIF) at Northeastern University, and funded by Schmidt Sciences. Over the summer, red teams built datasets and trained custom suspect models lying in different undisclosed circumstances, while blue teams developed detectors using either black-box access to models’ inputs and outputs, or white-box access to the internals of suspect models. On August 25, invited teams met in Boston to share their approaches, discuss results, and debate what the field should try next.

Note: A more detailed competition report will be published later.

Participation

Teams: 508power · aa · dami · EleutherAI · FlaiChi · GMU-Veritaserum · JSR · mpi · pinocchio · Recall Lab · ru · SAIN Groningen · Simit · Snake · Tara Research · Tony · TS · vk · yy

Winners & Competition Results

Ranked by mean AUROC across the ten held-out test datasets. Every finalist submission was re-executed by the organizers at full scale on a self-hosted NDIF cluster, against data no team had seen. To take a prize, a method also had to beat the best baseline in its track.

Special Awards

Final Validation Leaderboard Prizes

Awarded on the standing of the final validation leaderboard, ranked by average AUROC. To qualify, a method also had to rank above each of the six organizer baselines. Third place was awarded once per track.

Weekly Leaderboard Prizes

Throughout the Main Phase, the top three entries on the live leaderboard in each track took a weekly prize. Weekly standings were ranked by average balanced accuracy across the held-out validation datasets, with average AUROC as the tie-breaker; to qualify, a method also had to rank above each of the six organizer baselines.

Final Leaderboard

Ranked by AUROC  /  30 entries  /  select a row for its per-dataset scores

hollow marks a white-box method · filled marks black-box only · ✓ beats the best baseline in its track. Headline figures are means across all ten held-out test datasets. Datasets carry public codenames — the real identities stay unpublished. Codenames are assigned per scenario, so several datasets share a name; the Model and LoRA columns tell those apart.

Competition Showcase in Boston

On August 25, invited teams met in Boston to present their approaches, discuss results, and receive their awards.

Design of the Competition

Following the operational definition used in Liars’ Bench, the competition treated a model assertion as a lie when the model stated something it believed to be false.

Dataset creators needed to provide evidence for that belief, such as the model consistently answering correctly when the pressure to lie was removed. This requirement made dataset construction a research problem in its own right, since a false statement alone could simply reflect a training error or model uncertainty. Red teams hence had to establish both what the model said and why the label was justified.

Blue teams were tasked with developing methods that, given a conversation concluded by a message from the suspect model, returned a “dishonesty score” for the final response. They worked with a diverse set of model organisms based on Qwen3.5-27B, Gemma-3-27B, and Nemotron-3-Super-120B, and could use Qwen3.5-9B as a judge model. Some models were brought into situations where they started lying naturally; others had behaviors built into them through fine-tuning or prompting.

The competition gave teams a live leaderboard to hill-climb against, while keeping held-out test sets and model organisms behind the scenes to check that lie detectors generalized to new scenarios. Blue teams could use labeled development examples from the original Liars’ Bench dataset to train and test their methods. Blue teams also had access to a public leaderboard, where submitted methods were scored against a held-out validation set encompassing two additional datasets. Finally, we used most red teams’ models and datasets as fully held-out test scenarios to determine final performance.

This design allowed teams to receive live feedback about their submissions’ performance from the validation sets, while also preventing hill-climbing on the test set used to assess generalization capabilities. Teams could adapt parameters to different model architectures, but not to individual model organisms. This encouraged blue teams to develop approaches intended to generalize beyond a particular organism’s known behavior.

Powering white-box access to model subjects with NDIF

White-box methods developed by blue teams require more than an ordinary chat API. Participants in our competition recorded activations, trained and applied probing classifiers, inspected model logits and intervened in a forward pass. Thanks to NNsight’s remote execution capabilities, the NDIF team let teams run experiments on large models with up to 120B parameters without hosting the models themselves.

The competition ran on a dedicated cluster of four machines, each with eight H100 GPUs. Model organisms shared base-model deployments, with LoRA adapters dynamically switched between requests to optimize load thanks to the new NNsight v0.8 capabilities, avoiding a separate model copy for every participant.

For the NDIF team, this was a concrete test of shared infrastructure for AI auditing: researchers could compare detectors under a common execution setup, including methods that used internal access and methods that relied on conversations alone.

Leaderboard vs Test Performance

Method Landscape (n=25 reports)

Why AUROC, and not balanced accuracy

Methods were originally ranked on balanced accuracy, but we noticed that many submitted methods set their decision thresholds from statistics of the batch or dataset being classified. While our published rules did not explicitly forbid this, this level of access wouldn't be available in a realistic lie-detection scenario where each sample must be judged on its own.

Instead of forcing participants to adjust their thresholding methods, we chose to move the final award metric to AUROC. Because AUROC is threshold-free, it measures the quality of a method's scores independently of how any threshold was picked. The final reported submissions were ranked by mean AUROC computed separately on each held-out dataset, averaged across datasets.

Baselines

To be eligible to win awards, methods needed to score a higher average AUROC than the strongest baseline in their own track. The white-box baselines performed considerably better than the black-box ones, so we chose to have the two tracks carry different thresholds:

Track	Best baseline	AUROC to beat
White-box	Did You Lie probe	0.6611
Black-box	LLM self eval	0.5090

Competition Report

We are preparing a full competition report covering the datasets, model organisms, and an analysis of how different lie detection methods worked across submissions and datasets. We will update this website with a link to the report once it is live.