A language model may already know which word it will write before it reaches its last layer. If so, the remaining layers look like an opportunity: stop early, emit the token, and spend less computation on a decision that has already been made.
The difficulty is knowing when a decision is finished. A plausible intermediate answer can still change. And as models improve, the work that once seemed redundant may become part of what makes the answer correct.
This is the question behind Rui Wei and colleagues’ The Diminishing Returns of Early-Exit Decoding in Modern LLMs. Their experiments suggest that many newer models offer surprisingly little room for straightforward early exit, even when an experimental oracle can inspect the final answer before choosing an exit. The useful contribution is a way to examine that room before building a system around it. Paper, arXiv v1
What it means to leave a model early
Consider a decoder with $L$ layers. For each generated token, it transforms a hidden representation through those layers and uses a language-model head to produce scores over the vocabulary. These scores are called logits. A decoding rule then selects the next token.
An early-exit decoder tries to make that selection at some intermediate layer $\ell<L$. It needs both an intermediate prediction and a rule for deciding whether that prediction is trustworthy enough.
At first glance, common words and punctuation seem likely to require less work than a difficult reasoning step. But “easy to predict” and “stable halfway through this particular model” are different properties. The model may use late layers to change a token’s ranking, adjust its confidence, or incorporate information that earlier layers have not resolved.
Autoregressive generation also carries mistakes forward. Once an early-exit token enters the sequence, the next prediction is conditioned on it. A small local change can become a different reasoning path several tokens later.
The paper therefore asks two connected questions: do intermediate layers resemble the final layer closely enough to support early exit, and does that resemblance survive an end-to-end task evaluation? Its focus is the model’s existing behavior under little or no extra training. Sections 2–3
Three ways to ask whether a layer is ready
The authors compare intermediate and final layers through three signals:
| Signal | What is compared | What it helps reveal |
|---|---|---|
| Hidden states | Intermediate and final representations | Whether the internal representation has become similar to its final form |
| Logits | Vocabulary scores from the shared language-model head | Whether intermediate predictions resemble the final scores |
| Top-10 candidates | The leading token candidates at each depth | Whether the set of plausible next tokens has stabilized |
These signals describe different kinds of agreement. Similar hidden states need not produce the same next token. Two candidate sets can overlap while their first-ranked tokens differ. And similar logit vectors need not preserve the ranking of two nearly tied candidates.
There is a further distinction worth keeping visible: logits are not probabilities. Softmax converts logits into probabilities. Cosine similarity between raw score vectors is a useful probe, but it is neither a calibrated confidence measure nor a guarantee that the resulting distributions or chosen tokens match.
Across the models examined, many similarity curves rise meaningfully only near the end of the network. Llama2-7B shows earlier alignment than several newer models, while GPT-OSS-20B is a notable exception with relatively early logit alignment. Qwen2-7B shows low or negative logit similarity over much of its depth. The paper’s Figure 4 makes these differences more apparent than a single aggregate score can. Section 4.1
A score for the available room
Exiting one layer before the end may preserve quality but save little work. Exiting halfway through may save many layers but produce an unreliable answer. The authors combine these two considerations in an early-exit adaptability score, or EAS.
The fraction of layers skipped by an exit at layer $\ell$ is
They measure layer-to-final cosine similarity as
where $\mathbf{x}$ represents the signal being compared. After linearly mapping similarity into the interval $[0,1]$, giving $\widetilde S_\ell$, the per-layer score is
The model-level score averages over candidate exit layers:
For $0<\alpha<1$, the geometric mean rewards a layer that offers both similarity and savings. The parameter $\alpha$ controls the balance. The authors ultimately use logit similarity as the central signal after comparing its relationship with task accuracy. Sections 3.3 and 4.2
As an illustration, suppose a 32-layer model exits after layer 24. Its skip ratio is 0.25. If mapped similarity is 0.90 and we choose $\alpha=0.5$ for this example, its per-layer score is $\sqrt{0.90\times0.25}\approx0.474$. This is a teaching example, not a reported measurement or a claim about the paper’s experimental choice of $\alpha$.
EAS is a screening metric. An EAS of 0.52 does not mean 52% of layers can safely be skipped, and it does not mean a 1.52× speedup. Averaging also hides the exact location of useful exits. A deployment decision still needs the layer curves and the task-level trade-off.
Giving the exit rule knowledge it cannot normally have
A practical exit rule cannot compare its current logits with the final layer’s logits without computing the final layer. The paper temporarily removes this obstacle through an oracle experiment.
For each token, the oracle runs the full model, obtains intermediate predictions, and selects the earliest layer whose similarity to the final logits exceeds a threshold $\delta$. It then generates the token from that selected layer. The sequence continues using those selected tokens, so the task evaluation includes the consequences of changed generation.
Appendix C describes this procedure. A simplified reading is:
1 | For each generation step: |
This is an experiment about potential savings. The oracle actually computes the full depth. Its recorded skip ratio describes the layers an exit would have avoided if the needed information were available for free. It is not a measured reduction in runtime.
The oracle also has limits as an “upper bound.” It uses a particular similarity criterion and a threshold search. It offers unusually informed decisions within that setup, but it does not prove a universal maximum over all possible exit policies, retrained models, or speculative methods. Section 4.3 and Appendix C
How little survives the accuracy constraint
The benchmark uses OpenCompass, zero-shot evaluation, temperature zero, and a default maximum of 1,024 output tokens. The broader similarity study covers MMLU, GSM8K, GPQA, and HumanEval, using fixed subsets of 100 prompts per dataset with the same sampling seed across models.
For the oracle results, the authors search for thresholds that maximize skipping while keeping accuracy loss within their stated “5%” budget. The table below reformats their Table 1. Accuracy values have been converted from fractions to percentages; skip ratios are already percentages.
| Model | MMLU accuracy, full → exit | MMLU layers skipped | GSM8K accuracy, full → exit | GSM8K layers skipped | EAS |
|---|---|---|---|---|---|
| Llama2-7B | 41% → 38% | 4.69% | 21% → 17% | 2.13% | 0.52 |
| Llama3-8B | 54% → 52% | 0.18% | 37% → 33% | 0.04% | 0.46 |
| Llama4-Scout | 76% → 73% | 0.32% | 85% → 82% | 0.18% | 0.47 |
| Qwen2-7B | 24% → 24% | 0.00% | 53% → 53% | 0.00% | 0.36 |
| Qwen3-8B | 41% → 39% | 0.10% | 54% → 51% | 0.62% | 0.51 |
| GPT-OSS-20B | 74% → 69% | 7.26% | 61% → 57% | 4.48% | 0.59 |
Source: Wei et al., Table 1. The displayed losses fit a budget of five percentage points. They do not all fit a 5% relative-loss budget: for example, 21% to 17% is a four-point drop but about a 19% relative reduction. This distinction matters when interpreting the paper’s wording.
The striking result is the size of the usable window. Llama3-8B skips just 0.18% of layers on MMLU and 0.04% on GSM8K. Qwen2-7B reaches zero savings in both reported settings. GPT-OSS-20B offers more room, but even its best value here is 7.26%.
Suppose, purely as a cost model, every layer took the same time and all other work were free. Skipping a fraction $s$ of layers would give a speedup of $1/(1-s)$. At $s=0.0726$, that is about 1.078×. At $s=0.0018$, it is about 1.0018×. These are illustrative calculations from the reported skip ratios, not latency results from the paper.
Real execution adds exit checks, intermediate vocabulary projections, memory traffic, and scheduling. Token-dependent exits also have to coexist with batched serving and the attention cache: when an earlier token skips deeper layers, a later token may still need their cached keys and values. Any missing state needs a handling strategy. Those costs make skip ratio a starting point for systems work, rather than its conclusion.
What changes the opportunity
The paper examines four factors. Together they suggest that early-exit potential belongs to a particular trained model, rather than following automatically from its parameter count or architecture name. Section 5
Scale. Larger models generally have higher EAS in the studied families, although the curves are not uniformly monotonic. More capacity can leave more room for similar intermediate predictions. That does not imply a larger model with early exit will be cheaper than a smaller model evaluated at full depth.
Architecture. A representative comparison gives EAS values of 0.51 for dense Qwen3-8B, 0.49 for Qwen3-30B-A3B, and 0.48 for Mamba-Codestral-7B. The authors interpret the dense model’s smoother alignment as offering more opportunity than the selected MoE and state-space models. But these checkpoints differ in size and training as well as architecture. GPT-OSS-20B, itself an MoE, also has the highest EAS in Table 1. The evidence supports a pattern in the selected comparisons, not a rule excluding MoE from useful early exit.
Training. Released Pythia-12B checkpoints let the authors inspect representation changes during pretraining. Later training produces a middle-layer region with lower similarity to the final layer. A separate Qwen3-4B base-versus-instruction comparison shows delayed logit alignment after instruction tuning. The proposed explanation is increasing specialization and late-layer calibration. Pythia gives a useful window into this process, but one training trajectory cannot establish the behavior of every model family.
Workload. Qwen3-8B’s layer curves have broadly similar shapes across the four datasets. This suggests a substantial model-level component to the pattern. It does not make task choice irrelevant: a multiple-choice answer, a numerical result, and executable code have different sensitivities to changed tokens and different evaluation rules.
The title’s “diminishing returns” should also be read as an empirical tendency. Qwen3-8B’s EAS of 0.51 is above Qwen2-7B’s 0.36 and close to Llama2-7B’s 0.52. GPT-OSS-20B reaches 0.59. The results do not trace a smooth decline with every release.
What I would take into an implementation
My engineering reading is that measurement should precede the exit controller. For a chosen checkpoint and workload, I would first inspect intermediate predictions, then measure the task-accuracy curve under an optimistic exit policy. Only after finding meaningful room would I build a practical predictor and measure actual latency and throughput.
The accuracy budget should be explicit in percentage points or relative loss. Threshold selection should use separate calibration data, followed by evaluation on held-out examples. With 100 prompts, one changed answer moves accuracy by one percentage point; a narrow margin deserves more evidence than a single small sample can supply.
There are also two different projects to distinguish. One tries to exploit intermediate predictions that already exist. Another changes training so useful predictions emerge earlier, using auxiliary heads, early-exit objectives, or an early-exit draft verified by a fuller model. This paper mainly tests the first project’s assumptions. Its limitations section explicitly leaves room for methods that alter the model. Conclusion and limitations
I would treat EAS as a reason to investigate, then inspect the accuracy-versus-skip frontier directly. A single scalar can rank candidates approximately; it cannot choose a serving policy or account for hardware costs.
Depth has to earn its removal
Early exit begins with an appealing thought: perhaps the answer arrives before the computation ends. The paper asks us to test that thought against the layers themselves, and then against the generated sequence.
For many of the studied models, the trustworthy answer arrives late. Their improvement may have consumed some of the slack that earlier acceleration methods depended on. In that setting, removing depth means removing work that is still shaping the prediction.
There is still a useful question to carry forward: at what point does this model, on this workload, have enough information to stop? The answer belongs to measurement. An optimization becomes convincing when the computation it removes has demonstrably finished its job.
This article is an explanatory reading, not an independent reproduction of the experiments. Reported measurements come from Rui Wei, Rui Du, Hanfei Yu, Devesh Tiwari, Jian Li, Zhaozhuo Xu, and Hao Wang, The Diminishing Returns of Early-Exit Decoding in Modern LLMs, arXiv:2603.23701v1 (March 24, 2026). The cost examples and implementation recommendations are my interpretation. Original PDF · HTML text · alphaXiv