Large language models (LLMs) can act as both problem solvers and solution verifiers, where the latter select high-quality answers from a pool of solver-generated candidates. This raises the question of under what conditions verification pays off in solver–verifier systems. Prior work has conducted only limited studies of the factors influencing verification performance, focusing primarily on self-verification and examining neither the relationship between solver and verifier model families nor the effects of reasoning post-training. To rectify this, we present a systematic study across 37 models spanning multiple families, sizes, and base vs. post-trained variants, evaluated on 9 benchmarks covering logical reasoning, structured puzzles, symbolic computation, mathematics, commonsense, factual recall, and domain knowledge. In order to support our analysis, we introduce and empirically validate verifier gain, a metric that predicts the performance improvements from test-time verifier-based rejection sampling. Our experiments find that 1) verification across model families is more effective than either self-verification or verification within the same family, and more generally that the benefits of verification decrease as the solver and verifier become more similar, 2) reasoning post-training weakens self-improvement abilities but strengthens cross-family improvement, and 3) some tasks are inherently more amenable to improvement through verification, particularly mathematical and logical tasks.
Let's quickly go over our experimental setup, how we measure verification ability, and different verification settings.
We treat every model as both a solver and a verifier. As a solver, each model generates a chain-of-thought solution per problem; as a verifier, it reads the original problem and a solution to decide whether the solution is correct (also with chain-of-thought). We evaluate 37 models, including 21 post-trained and 16 base models from the Llama3, Qwen2.5, Qwen3, and DeepSeek-R1 families spanning 0.5B-72B parameters.
We evaluate our models on 6 real-world tasks (GSM8K, AIME, CSQA, GPQA, MMLU-STEM, and MMLU-Social-Sciences) and 3 synthetic tasks (3SAT, Sudoku, and Matrix Multiplication).
Verifier performance has multiple dimensions, so we track classification metrics like accuracy, false positive rate (FPR), false negative rate (FNR), and also the downstream effect of verifier-based rejection sampling via verifier gain, defined as
\( \underbrace{Gain(S, V; D)}_{\substack{\text{performance improvement} \\ \text{from using verifier}}} \;=\; \underbrace{Precision(S, V; D)}_{\substack{\text{accuracy after test-time} \\ \text{rejection sampling with verifier}}} \;-\; \underbrace{SolverAccuracy(S; D)}_{\text{base solver's accuracy}} \)
where \(V\) is a verifier, \(S\) is a solver, and \(D\) is a dataset.
We compare three ways of pairing solvers and verifiers:
Our verifier gain metric estimates the expected improvement in a solver's accuracy when using a verifier for rejection sampling. To assess how well this metric predicts real performance, we conduct rejection sampling experiments across all solver-verifier pairs from a 12-model subset of our post-trained models. For each problem in each dataset, the solver generates solutions until the verifier labels one as correct, for up to 10 attempts; if no such solution is found, we retain the final attempt.
We first analyze whether a model's solver performance correlates with its performance as a verifier. For each of our 21 post-trained models and each dataset, we evaluate verification on the same set of solver models to obtain verifier accuracy, FPR, FNR, and gain for every solver-verifier pair. For each verifier, we then divide the verifier metrics into three verification settings and average within each setting over solvers and datasets. From the figure below, we realize the answer to this question depends on the verification setting and discover the following takeaways.
From the last sections, we saw that reasoning models benefit less from self- and intra-family verification due to high FPR (in comparison to cross-family verification), hinting at an LLM bias in accepting incorrect solutions that resemble their own. To directly investigate this behavior, we conduct cross-verification experiments using 12 post-trained models and compute all verifier metrics for each pair. For each pair, we plot the verifier metric against the solver-verifier similarity score, defined as the average cosine similarity between the two models' solution embeddings across all dataset problems. The figure below confirms this hypothesis.
Our analysis focuses on the Qwen2.5-Base/Qwen2.5 and Qwen3-Base/Qwen3 model pairs. For each model, we compute verifier metrics against all solvers and datasets, partition results by verification setting, and average within families. From the figure below, we realize the following takeaway.
Thus far, we have examined verifier performance and its contribution to solver accuracy through rejection sampling. We now shift to a task-level perspective and ask two questions:
For each of our 21 post-trained models and each dataset, we evaluate verification on the same set of solver models to obtain verifier accuracy and gain for every solver-verifier pair, average them across all verifier models, and plot them against solver accuracies. From the figure below, we realize the following takeaway.
@misc{lu2025llmverification,
title={When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers},
author={Jack Lu and Ryan Teehan and Jinran Jin and Mengye Ren},
year={2025},
eprint={2512.02304},
archivePrefix={arXiv},
primaryClass={cs.CL}
}