All right, let's dive into something truly compelling today. In the world of large language models, pushing for cutting edge reasoning capabilities
often feels like, well, a constant tugofwar. We strive for higher accuracy. We employ these powerful techniques like self-consistency or um parallel
thinking, but this nearly always comes at a significant computational cost. It's a real bottleneck. So,
what if we could achieve that superior performance without the heavy overhead? What if we could, you know, drastically reduce the tokens generated for
these complex reasoning tasks? Today, we're doing a deep dive into a fascinating pre-print paper that directly tackles this very challenge. Our
mission for this deep dive is to thoroughly explore deep think with confidence or deep. It's a novel
method really designed to boost reasoning efficiency and performance right at test time. You've brought us a single but incredibly rich source. This
is pre-print from RQ7. We're going to methodically ex extract the core concepts, the uh ingenious methodologies, and the impressive datadriven results
presented within this research. >> Yeah. And what's immediately captivating here, especially if
you're kneedeep in LLM deployment, is how the authors approach leveraging model internal confidence signals. Think about it. Instead of just treating
every single generated reasoning trace as equally valid, deep con intelligently filters out or even more remarkably, it halts the generation of
lowquality traces. sometimes in real time. That's a significant paradigm shift. And crucially, it requires
no additional model training, no hyperparameter tuning, none of that hassle that makes it incredibly adaptable, really relevant for existing systems
you might already have running. It's basically about making the model smarter with its own thoughts, its own internal state. >> And the numbers they
present are genuinely eye opening. The researchers, they rigorously evaluated deepcom across multiple
models y including the latest open source series like Quinn 3 and GPToss. Right. And on exceptionally challenging benchmarks, too. >> Exactly. And get
this. On the AM 2025 data set, Deep Comp, using a budget of 512 traces, which sounds like a lot, but wait for it, achieves an astonishing accuracy of
up to 99.9%. Almost perfect. >> Wow. Okay. 99.9% is impressive, but what about the cost? That's the
whole point here, >> right? Here's the kicker. It simultaneously reduces the number of generated tokens by an astounding 84.7% compared to just doing
full parallel thinking without this smart filtering. >> 84.7%. >> Yeah, that's not just an incremental improvement. That's a massive leap in
efficiency for some of the most complex reasoning tasks out there. It's potentially a breakthrough for
optimizing large-scale LLM deployments. >> Hi, >> the challenge of test time scaling in LLMs. Okay, so to properly appreciate deoff's solution, let's
first set the stage. We need to understand the landscape. It's navigating. LLMs have shown these remarkable reasoning capabilities, especially when we
apply certain, you know, test time inference enhancements. The research points to a prominent
technique when we've touched on self-consistency. I know our audience is expert level, but maybe quickly remind us what self-consistency or parallel
thinking actually involves. >> Absolutely. So self-consistency or parallel thinking, it's fundamentally about letting the LLM think through a problem
multiple times. Each time generating potentially a slightly different reasoning path or what the
paper calls a trace. So instead of just relying on one single chain of thought, the model generates uh like an ensemble of diverse solutions to the
same problem. Once these multiple traces are generated fully completed, their final answers are then aggregated. Usually this is done through majority
voting. The answer that pops up most frequently is chosen as the final solution. This technique, it's
been shown to be incredibly effective, especially on challenging multi-step problems. It really boosts reasoning accuracy. It's kind of like
harnessing the wisdom of the crowd, but all within a single model, >> right? So, we get this clear boost in accuracy, which is fantastic for those
complex highstakes problems. Makes sense. But this benefit, as the paper details, comes with that significant
performance versus cost dilemma. What are the key elements of this trade-off that deep comp is trying to solve? >> Well, the primary challenge, and
this is really hammered home in the research, is the sheer computational overhead. It's the cost. While self-consistency undeniably improves accuracy,
generating dozens or sometimes even hundreds of these reasoning traces for every single query. Well,
it scales the inference cost linearly. for practical deployment especially if you care about latency or you're working in resource constraint
environments this becomes a major bottleneck >> and they give a concrete example right >> they do a very striking one to improve the pass at one
accuracy that's the accuracy from a single trace of a quen 38B model on the AAA 2025 benchmark to get it from 68%
up to 82% which is a nice 14 percentage point gain using standard majority voting you need to generate an additional 511 reasoning traces per
question. >> 511 extra traces. >> Yes. And that translates to, get this, 100 million additional tokens generated just for that improvement on that
data set. >> Wow. Okay. That really highlights the scalability issue. You're essentially paying a very, very
high price in compute for those extra thoughts. >> Exactly. And this brings up a critical question. Why does parallel thinking even with its accuracy
benefits often show diminishing returns as we add more and more traces? >> It feels a bit counterintuitive, right? You'd think more processing more
thoughts should always lead to better results. >> Yeah, that is the puzzle. Why doesn't just
generating more always help? >> Well, that's a crucial observation and it's a limitation that's explicitly addressed in related work and of course in
this deep comp paper. The core issue is that standard majority voting the way it's usually applied treats all reasoning traces as equally valid. It
doesn't really care about their internal quality. It doesn't differentiate between a robust, logically
sound trace and one that maybe, you know, went off the rails early on, made a mistake somewhere in the middle. >> So bad reasoning gets the same vote
as good reasoning. Precisely. This means that if a significant number of lowquality just plain erroneous traces are generated, they can dilute the
voting process or worse, they could even collectively converge on a wrong answer. In those scenarios,
adding more traces might actually lead to suboptimal performance or in some cases even a degradation in the overall accuracy compared to using a
smaller perhaps more carefully selected set of traces. >> It's like adding more noise essentially. >> Exactly. Adding bad data points to an
aggregation doesn't necessarily improve the signal. Now, previous work has definitely tried to tackle this. There
have been efforts. One promising direction involves leveraging uh next token distribution statistics to estimate the quality of a trace. The intuition
is pretty straightforward. When an LLM is highly confident about its next token prediction, the probability distribution over its vocabulary will be
very peaked. High probability for one token, low for others. >> Lower entropy, right? >> It's
exactly lower entropy, reduced uncertainty. These methods typically aggregate these token level statistics, whether it's entropy or direct confidence
scores, into a single global confidence measure for the entire reasoning trace. And this global measure is then used to filter out lowquality traces,
theoretically enhancing the majority voting process. Filter first, then vote. However, the deep comp
paper critically identifies two practical limitations with these existing global confidence measures. First, they tend to obscure local fluctuations
in confidence. Imagine a really long multi-step chain of thought, right? >> A critical reasoning error or maybe just a moment of high uncertainty
might happen at a specific intermediate step. But if you're only looking at an average or global
confidence score across the entire trace, a few high confidence tokens at the beginning or the end could easily mask these crucial early or mid-trace
errors. You lose that granular signal that tells you something went wrong when it went wrong. >> Right? the average hides the dips >> precisely. And
second, and this is maybe even more significant for efficiency, these global confidence measures
fundamentally require the generation of complete reasoning traces before they can even be calculated. >> Ah, so no early stopping. >> Exactly. No
early stopping. You can't terminate unpromising lowquality traces partway through. And this is precisely where the computational inefficiency really
bites. You're paying the full cost of generating a long ultimately flawed reasoning path before you can
even identify it as flawed and discard it. >> A new paradigm for confident to wear reasoning. >> Okay, this brings us directly to deepcom. The authors
introduce it as well while they say simple yet remarkably effective. A testime method that intelligently combines parallel thinking with this
confidence filtering. But the key differentiator, as you emphasize, seems to be its reliance on local
confidence measurements, not global ones. It's designed specifically to drastically reduce that unnecessary token generation while maintaining or in
some cases even improving the overall reasoning accuracy. Can you elaborate a bit on its dual operating modes? >> Exactly. The local aspect is key.
Deep operates in two primary scenarios, each tackling a different facet of efficiency and performance
optimization. The first is what they call offline mode. In this setup, you assume the reasoning traces have already been fully generated, completed
start to finish. Deep then comes in after this generation step to intelligently identify and discard the low confidence traces from this pre-existing
pool. It's about making a smarter selection from a given set of completed thoughts to improve the
final aggregated answer. Post-processing essentially. >> Okay, so filtering after the fact. What's the other mode? The second and arguably the more
groundbreaking mode is online mode. This is where Deep Comp dynamically identifies and discards low confidence reasoning traces during their
generation as they're being created token by token. This real-time control is what enables that crucial early
termination. It dramatically boosts computational efficiency by preventing the LLM from wasting time and resources completing a trace that's already
shown clear signs of being lowquality or errorprone. >> Like intervening mid-process. >> Exactly. It's the difference between reviewing a finished,
possibly flawed report versus intervening to stop a flawed project midway through, saving resources. So
for us as researchers and practitioners working with these models, what does this mean? It means we can potentially achieve superior reasoning
performance, often even surpassing traditional majority voting, but with significantly fewer generated tokens. >> Yeah. >> And the authors didn't just
stop at theoretical claims. They rigorously evaluated deep cough across a diverse set of really
challenging reasoning benchmarks >> like those math competition problems. >> Yeah, exactly. Highly difficult math problems like AIM 2024 and 2025,
Brumo 25, HMMT25 and also graduate level STEM reasoning tasks like GPQA diamond. So a good mix and they tested this on a range of open source models
from the you know 8 billion parameter deepseek up to the much larger 120 billion parameter GBT OSS. This
ensures its robustness across different model scales >> and the multiple repetitions give statistical significance. >> Right? Importantly, their
results are averaged across 64 repetitions per setting that provides a strong statistical foundation. This isn't just some interesting theoretical
idea. It looks like a practical deployable solution designed for realworld efficiency gains. >> Yeah. >>
Three, the mechanics of confidence in Deep Comp. >> Okay. So Deep Cop's real innovation seems to hinge on these fine- grained confidence measurements.
The research builds on prior work using internal model signals. Sure. But then it introduces several alternative metrics. Metrics specifically
designed to overcome those limitations of global averaging we just discussed. Can you walk us through how
they actually quantify confidence at this more granular level getting it straight from the model's internal workings? >> This is really the heart of
deep comp's effectiveness. These are model intrinsic signals, right? Meaning we don't need any external supervision. no extra labelled data to tell
good reasoning from bad. The model itself through its probabilistic outputs for each token it generates
gives us direct insight into its own certainty or lack thereof. Let's start with the base the token level metrics. First, there's token entropy which
they denote as a higher dollars. When a language model predicts the next token at position a dollar, it generates a probability distribution by pile
over its entire vocabulary. The entropy higher to dollars of this distribution quantifies its
uncertainty. Right. Low entropy means high certainty, peak distribution. High entropy means more uncertainty, flatter distribution. >> It's exactly
low entropy suggests the model is highly certain about the next token. Almost like a sure bet. High entropy reflects significant uncertainty, meaning
the model sees multiple plausible next tokens. Closely related is token confidence $6. This is defined
uh basically as the negative average log probability of the top U tokens at position of dollars. So it's essentially averaging the model's belief
strength for its top predictions. Similar to entropy, higher confidence here corresponds to those peak distributions. Greater model certainty. Lower
confidence points to a more diffuse distribution, more uncertainty. >> And they use the top K, not just
the single top token. >> Often yes, using the top K where a might be five or 10 helps capture the bulk of the model's predictive strength, not just
its single best guess. Now, these token level metrics are valuable, but as we said, previous approaches often just aggregated them into an average
trace confidence, six raven trace. It's simply the mean of the token confidences across all tokens in a
reasoning trace. And the paper acknowledges that this average trace confidence does show some effectiveness. It can help distinguish between correct
and incorrect reasoning paths. Generally, >> they show a figure illustrating that, right? >> Yes. Figure two in the paper illustrates how the
distributions for correct versus incorrect traces on the HMMT25 data are generally separated by this metric.
Higher average confidence tends to correlate with correctness. But again, this global aggregation has those critical limitations we discussed. It
obscures intermediate reasoning failures. A few high confidence tokens can mask many low confidence segments. It hides the local problems. And the big
one, it requires complete traces, preventing early termination. Still paying the full cost. >> Exactly.
You're still paying the full computational price even for traces that are ultimately going to be discarded. And this is precisely why Deep Coff
introduces its novel local confidence measurements. These are specifically engineered to capture that intermediate step quality more effectively and
overcome those limitations. The first of these new local metrics is group confidence, CGO day. Instead of
one average for the whole trace, group confidence provides a more localized, smoother signal. How? By averaging token confidence over overlapping
spans of the reasoning trace. Think of a sliding window. Each token's group confidence is calculated based on a window. Lever of the trippy tokens.
Say one deer could be 124 or 2048 tokens. The formula just averages the token confidence is within that
sliding window. >> So it's like a moving average of confidence >> pretty much. Yeah. The rationale here is to identify sharp drops or maybe repeated
sequences of low confidence tokens. The authors kind of liken it to the model internally saying wait h however or let me think again. Empirically
these often signal a disruption in the reasoning flow and typically precede errors. Building on this
group idea they propose bottom 10% group confidence cam noun 10. This metric determines a trace's overall confidence by taking the mean of only the
bottom 10% of all the group confences calculated within that trace. >> So focusing specifically on the weakest parts. >> Exactly. The authors found
empirically that focusing on this lowest 10% segment was really effective at capturing the most
problematic reasoning segments across different models and data sets. This metric proved particularly good at separating the distributions of correct
and incorrect traces. in their figure two. It suggests it's superior for quality estimation compared to just the simple mean confidence. It
essentially hones in on the weakest link or links in the chain of thought. Then there's lowest group
confidence. This is specialized kind of more aggressive version. It simply identifies the confidence of the single least confident group within the
entire reasoning trace. >> Just the absolute minimum. >> Yes. Its primary utility, as we'll see when we discuss the modes, is for improving reasoning
efficiency in online thinking scenarios. It provides a very direct immediate signal for potentially
stopping generation. If even one small segment shows extremely low confidence, maybe that trace isn't worth pursuing further. And finally, they
introduce tail confidence. This metric evaluates trace reliability by focusing specifically on the final portion of the trace. For instance, a fixed
number of tokens like the last 2048. It's defined as the average token confidence just over this tail
segment. >> Why the tail? >> It's motivated by the observation that reasoning quality often degrades towards the very end of long chains of thought.
And those final steps are absolutely critical for reaching correct conclusions, especially in math reasoning. Think about it. Traces that start strong
but then falter right at the conclusion are a common source of error. This metric helps catch those
latestage failures. Figure two also shows its effectiveness in separated the distributions deep conf in action offline and online thinking. >> Okay.
So with these refined more granular confidence measurements group bottom 10% lowest tail deep comp then operationalizes them into those two distinct
thinking modes offline and online. Each mode as you mentioned targets different aspects of improving
LLM reasoning but they both leverage this foundation of fine grained confidence. >> Precisely. Let's first dig into offline thinking with confidence.
Remember in this scenario, all the reasoning traces for a given problem have already been generated, usually to their full length. The core objective
here is to intelligently aggregate information from these multiple complete traces to arrive at the
best possible final answer. The baseline for this, the standard approach is standard majority voting. We talked about this. The final answer extracted
from each trace contributes equally. Count the votes for each answer. Pick the highest. Simple democracy for LLM thoughts. Okay, standard practice.
How does deep comp improve on that offline? >> Well, one way is with confidence weighted majority
voting. Instead of giving each trace an equal vote, each answer's vote is weighted by the confidence of the trace that produced it. So, if a trace
leading to answer A had very high confidence using one of those metrics we just discussed, its vote for A counts more heavily than a vote from a low
confidence trace leading to answer B. This method inherently favors answers supported by high confidence
reasoning. effectively downweing the influence of uncertain or likely flawed traces. It's like giving more weight to the experts opinion in a group
discussion. Then further refining this, Deep Coff introduces confidence filtering. This mechanism goes a step further. It selects only the top percent
of traces based on their confidence scores. Only these supposedly most reliable paths are then
allowed to contribute to the final answer, typically using weighted voting on just that filtered set. >> And they look at different percentages for
EDA, >> right? The paper primarily explores two options. Y equals 10% and DIY's 90%. The 10% option is quite aggressive. It focuses only on the very
highest confidence scores. This can be super effective when you think only a few traces will be truly
reliable, but as we'll see, it carries a risk of bias if the model happens to be overconfident on an incorrect problem. The 90% option is more
conservative, more balanced. It keeps more diversity by including a broader range of traces. This can reduce that potential model bias and ensure
alternative reasoning paths are still considered especially if the confidence scores are more spread out. >>
Makes sense. Filter then vote. >> Exactly. The algorithm details in the paper show the steps. Generate entraces, calculate their confidence CIA,
select the top percent and then compute the weighted vote VA on that subset. Figure 3 also provides a nice illustration of how these confidence
measurements play into this offline thinking process. Now let's switch gears to what seems like the really
efficiency boosting mode. Online thinking with confidence. This is where deep comp uses those confidence metrics during token generation in real time.
This allows for dynamic termination of unpromising traces which as you said is incredibly valuable for resource constrained environments or just
maximizing throughput. >> Precisely. This is where the big potential for savings comes in. And that
lowest group confidence metric, the one focusing on the weakest link in a trace as it's being built, is particularly well suited for this online
setting. The core idea is elegantly simple. If the group confidence of the currently generated tokens, that sliding window average falls below a
critical pre-established threshold. >> We just stop generating that trace. >> You halt the generation of that
particular trace right then and there. The reasoning is sound. Such a trace would with high probability be filtered out by an offline filter anyway.
So why waste compute finishing it? It's proactive pruning based on real-time self- assessment. The paper introduces two main variants for this online
mode. Deep conf. Both are algorithms based on this lowest group confidence metric to adaptively stop
generation. Now to make this work effectively, deep requires a brief offline warm-up phase. This isn't training. It's just running a few examples
first. The main purpose of this phase is to establish a robust stopping threshold that value social dollars which will be used during the subsequent
online generation. >> How do they set the threshold? >> It involves generating a small initial set of
non-oververse reasoning traces say 16 traces as used in their experiments. Then they look at the distribution of confidence scores like lowest group
confidence from these warm-up traces. The stopping threshold is then calculated as a specific percentile of these warm-up confidence scores. Deepcon
flow uses DL 10% for its filtering target. This corresponds to setting the threshold at the 90th
percentile of the warm-up trace confidences. Its aggressive traces will be stopped if their confidence falls below the level that would have kept only
the top 10% of the warm-up traces. Deep comp high, conversely, uses 90% for filtering. This means setting set of it at the 10th percentile of warm-up
confidences. It's much more conservative, only stopping traces whose confidence drops really low
below the level needed to keep the top 90% of warm-up traces. This threshold, once set based on the warm-up, acts as the gatekeeper during online
generation. Okay, so it sets a bar based on initial runs, but there's also adaptive sampling. It doesn't always generate the full budget of traces,
right? That's the other key part of the online efficiency. Beyond early stopping individual traces, DeepM
also employs adaptive sampling to adjust the total number of traces generated based on the problems perceived difficulty. How does it gauge
difficulty? By looking at the consensus among the traces generated so far. This consensus dvod is basically the fraction of votes going to the current
leading answer. >> So if most traces agree, it stops early. >> Exactly. If terabets falls below a preset
consensus threshold tow they used to 955 in the paper it suggests the model hasn't reached a strong agreement yet. So trace generation continues up to
a maximum budget battle dollars but if terabitet meets or exceeds that threshold towel meaning a a strong consensus has emerged. Trace generation
halts globally for that problem. The final answer is determined using the traces generated up to that
point. This cleverly prevents overgenerating traces for problems where a clear answer emerges relatively quickly among the initial traces. And the
elegant part ties back to the threshold. Any trace terminated early online because its group confidence drop below CN dollars is a trace that would
likely have been excluded by the corresponding offline filter. Anyway, so this online procedure
effectively approximates the offline lowest group confidence filtering policy, but achieves massive computational savings by not having to complete
those lowquality traces. >> Got it. Algorithm two in the paper details these steps. The warm-up, the online generation with early stopping per trace
based on confidence and the adaptive stopping for the whole problem based on consensus. And figure four
illustrates it. Very neat. V experimental validation deep comp's performance and efficiency. Okay, theory is great, algorithms are neat, but the proof
is in the pudding, right? The paper presents extensive experiments to back up Deep Comp's claims. For those of us focused on empirical evidence, this
is crucial. Let's look at how they set up their benchmarks and most importantly what those results
actually tell us about Deep Comp's realworld impact on accuracy and efficiency. >> Absolutely. This is where the rubber meets the road. Yeah. >> And
they've been quite meticulous in designing their experimental setup to show Deep Comp's robustness across diverse scenarios. This is critical if you
want a method to be broadly applicable. First, let's talk models. They picked five open source LLMs
from three different families. Deepseek 8B, then Quinn 38B, and the larger Quinn 332B, and finally GPT OSS 20B, and the very large GPT OSS120B. >> Why
the specific ones? >> They were chosen for a few reasons. They generally have strong mathematical reasoning capabilities and are good at long chain of
thought generation. Being open- source is key for reproducibility. And importantly, the range of
parameter scales from 8 billion up to 120 billion lets them test if deep works consistently across different model sizes and capacities. They even
provide the generation hyperparameters and prompt templates in the appendix which is great for transparency. For benchmarks they use five really
challenging data sets. You have AIM 2024 and 2025 Brumo 25 and HMMT25. These are all known for having very
difficult math competition problems. Plus they include a GPQA diamond which focuses on graduate level STEM reasoning tasks. So it's not just math.
These benchmarks are widely used featured on leaderboards like Math Arena, making them relevant for assessing state-of-the-art reasoning. The main
baseline they compared against was standard self-consistency using unweighted majority voting. This gives
a clear established reference point to measure deep comp's gains against. Now, the experimental settings were also carefully structured. They started
by pre-generating a large pool, 4,96 complete reasoning traces per problem. A huge data set. >> Why pre-generate so many? This large pool served as a
consistent foundation for the offline experiments. They would then resample a working set of size K
say K512 from this cool for each run, apply the voting method and record the result. This resampling controls for the variability in trace generation.
Online experiments also resampled a working set from the pool conceptually, but then simulated the on- the-fly generation with early stopping. Using
the pool ensures consistent sampling across different methods being compared for metrics. They
reported several key ones. Pass at one single trace accuracy cons at K standard unweed majority vote with K traces. Measure at K confidence weighted
majority vote and crucially measure plus top P% at K where they filter to keep the top P% traces by confidence before doing the weighted vote >> using
ADA of 10% and 90%. >> Exactly 10% and 90% for filtering. They also meticulously tracked the total
number of generated tokens, counting only the tokens produced before any early termination happened. And very importantly, all results were averaged
over 64 independent runs with fresh resampling to ensure statistical robustness. For the online specifics, deep low and deep high use that lowest
group confidence metric with a 2048 token window, the nois696 trace warm-up, and the tal 955 consensus
threshold. All these details are laid out making the experiments quite rigorous. >> Okay, let's dive into the offline evaluations first. Table one in
the paper seems key here. How did confidence aware waiting and filtering stack up against just plain old majority voting? >> The offline results
looking at K512 traces are pretty compelling. The big takeaway deep using confidence consistently often
notably outperforms standard majority voting across most settings. This strongly validates the core hypothesis. Incorporating confidence signals is
better than treating all traces equally. Specifically, that more aggressive filtering keeping only the top 10% of traces often yielded the largest
accuracy gains. >> Can you give an example? >> Sure. Deepseek 8B on AM25 jumped from 82.3% accuracy with
standard voting to 87.4% using the bottom 10% group confidence filter. That's a 5-point gain. Similarly, Quinn 332B on AME24 went from 85.3% to a
really impressive 90.8% 98% using tail confidence filtering. Perhaps the most striking result was with the huge GPT OSS120B model on AME25. Using tail
confidence filtering, it hit 99.9% accuracy, essentially saturating the benchmark. Compare that to
97.0% for standard majority voting and only 91.8% for a single trace. It really shows the power of filtering out the less reliable paths. >> So, both
local metrics like tail and bottom 10% and even the global average trace confidence showed promise. Yes, they all showed some benefit. However, the
paper points out that important trade-off. While aggressive filtering top 10% often maximizes
accuracy, it can sometimes hurt performance if the model is confidently wrong. >> Ah, the overconfidence issue. >> Exactly. In those cases, the more
conservative filtering top 90%. Provides a safer, more balanced approach, usually matching or slightly beating the baseline without the risk of sharp
drops. But overall, the substantial improvements over just using a single trace pass at one clearly
confirm the value of these confidence aware ensemble approaches. The paper also specifically highlights the effectiveness of lowest group confidence
in these offline settings, particularly using DeepSeek 8B data shown in figure 5. When retaining the top 10% of traces using this metric, they saw
consistent gains over the best majority voting accuracy, averaging a plus 5.27 percentage point
improvement. And compared to single trace accuracy, the average gain was huge, plus 14.30 percentage points. >> So focusing on that weakest link
really works for filtering. >> It seems so. Even with the more conservative top 90% filtering, lowest group confidence still matched or slightly
exceeded majority voting accuracy, average gain plus 29 points, and offers substantial gains over single trace
accuracy, average plus 9.31 points. The rationale holds up. Focusing on the least confident segment reliably identifies traces with localized
reasoning breakdowns, providing a strong signal for filtering. And crucially, this same signal is what enables the early stopping in the online mode.
Okay, that sets the stage for the online evaluations. This is where deep comp's real-time efficiency
should shine. How well did it actually balance maintaining that accuracy with achieving significant token reduction? Table two seems relevant here. >>
This is indeed where deep comp shows its most transformative potential for practical applications. The results for the adaptive sampling version of
Deep Cough, again at a budget of K512, are quite impressive. Let's look at Deep Cough Low first.
That's a more aggressive one using the top 10% filtering strategy equivalent. At K512, this method reduced the number of tokens generated by a
substantial 43% to 79% across the different math benchmarks compared to standard majority voting using 512 full traces. >> Wow. Up to 79% reduction
and accuracy. In most cases, it either matched or even improved accuracy compared to the baseline. For
example, DeepSeek 8B on AM24 saw its accuracy increase by plus 5.8% alongside those massive token savings. However, it's important to be balanced. In
a few specific settings, there were minor accuracy drops like Quen 332B on Brumo 25 saw a small mega0.9% decrease. So, the aggressive filtering isn't
always perfect. It can occasionally discard something useful. >> And what about the more
conservative deep comp high? Deep comp high using the top 90% filtering strategy equivalent also delivered significant token savings ranging from 18%
to 59% on these data sets. >> Yeah. >> And crucially, it managed this while maintaining nearly identical accuracy to the baseline or incurring only
minimal, often negligible degradation. This makes it a much safer choice if maintaining baseline
accuracy is paramount. It offers a really good balance of efficiency and reliability. >> They visualize this too, right, with the big GPOSs model. >>
Yes. Figure 6 shows the token reduction for GPTO OSS 12B. It visually confirms substantial computational savings, sometimes hitting that astounding
85.8% reduction mark while still preserving competitive accuracy levels. It's very compelling. And
figure 7 comparing deep comp and majority voting on deep segate B really underscores the efficiency advantage from another angle. It shows that to
achieve the same accuracy level as majority voting, deep compl needed on average 62.88% fewer tokens and deep comp needed 47.67% 67% fewer tokens. >>
So same result, much less work. >> Exactly. The online behavior largely mirrors the offline. Top 10%
filtering deep low generally gives the biggest accuracy gains but carries a small risk of drops while top 90% deep high is safer. Overall, these
robust results strongly validate the core ideas. Using the least confident segment to gate traces works for early termination and combining it with
the adaptive consensus stop effectively cuts down tokens without generally sacrificing the accuracy needed
for complex reasoning. It's optimizing smarter thinking, not just throwing more compute. >> Ablation studies and implementation detail. >> Okay,
impressive results. But the paper doesn't stop there. It offers detailed ablation studies which really help us understand how the different knobs and
components influence Deepcon's performance. For anyone actually thinking about implementing or adapting
this method, these insights are gold. >> Absolutely. These ablation studies are critical for truly understanding the mechanics and for guiding
practical deployment decisions. Let's walk through some key ones, mostly found in the appendices. First, the ablation on consensus thresholds appendix
B.1 table 3. They tested different values for that adaptive stopping threshold tal using Quinn 332b on
AM24. They found that tout 955 seemed to hit the sweet spot. It preserved the accuracy while delivering significant token savings. For DeepCom flow,
savings were 15.4% at a budget of 32 and 52.8% at 512. For Deepcom high, it was 22.0% and 54.7%. >> What if they made TAL more aggressive, like
stopping even earlier if consensus isn't strong? >> Yeah, a more aggressive tower could increase savings
further, but it sometimes led to accuracy drops for Dubcon flow. Deepcom high being more conservative overall showed greater robustness to changes in
tow. It's also interesting that token savings were more substantial at higher budgets like B512 versus B32 simply because the adaptive stopping has
more potential traces to cut short after the initial warm-up phase. Next, the ablation on warm-up
sampling size appendix B2 table 4. This looked at the impact of number of initial traces used to set the stopping threshold six. They compared using
8, 16, and 32 warm-up traces for deepcom flow. The main finding was that increasing note generally helps stabilize the confidence distribution you're
sampling from to set the threshold six. This tends to bring the online accuracy closer to the ideal
offline accuracy. The difference delta ACR usually got smaller. >> So more warm-up is always better. >> Not necessarily straightforward. The exact
relationship between the threshold and accuracy wasn't strictly monotonic. It depends on the model and data set. And total token usage also wasn't
monotonic because you have this trade-off. a higher fixed cost for more warm-up traces versus potentially
better early termination later. But empirically, the difference in accuracy between online and offline was generally small anyway, usually less than
one percentage point. So, they adopted 19 the $16 as a balanced default stable enough without excessive warm-up cost. Then there's the ablation on
filtering percent appendix B.3 table 5. This systematically varied that retention ratios from 90% down
to 10% for the lowest group competence metric in the offline setting. The trend was pretty clear. More aggressive filtering, meaning keeping smaller
top percentages like 10% or 20%, generally yielded higher accuracy in most cases. The top 10% frequently achieved the best performance, although the
absolute optimal dialers could vary a bit by data set. >> So being selective pays off offline. >> It
confirms the intuition. The filter preferentially throws out the low confidence, often incorrect traces, concentrating the final vote on the higher
confidence evidence, which boosts average accuracy. The ablation on confidence metrics appendix B4 table 69 is really comprehensive. It compared
various ways to aggregate confidence standard majority vote no confidence average trace confidence the
global baseline head confidence focusing on early tokens tail confidence final tokens lowest group confidence LGC and bottom percent confidence like B
10%. Both with and without filtering. >> What's out there? >> Well, headbased confidence seemed weak. It didn't correlate well with correctness.
sometimes even hurting performance compared to plain majority voting. Makes sense, right? Early tokens
are often just setup or planning. Tailbased and mean-based signals frequently yielded gains confirming their value. Crucially, lowest group
confidence, LGC, especially with a 2K window and 10% filtering was highly competitive. It averaged 84.4% accuracy across settings, right up there with
bottom 10%, 84.0% and tail 10%, 84.6%. The big takeaway here is that these local confidence signals tail
bottom lowest group are definitely not inferior to the global average trace confidence. On average, they deliver equal or even slightly better
accuracy and they enable online stopping. They also looked at scaling behavior for offline deep comp appendix C figure 8 across models, data sets, and
different voting budgets. Using top 90% filtering consistently matched or slightly outperformed unweed
majority voting, average gain plus 0.17% confirms it's a safe, reliable choice. Top 10% filtering yielded more notable gains on most data sets.
Average gain plus 1.22% but again with those occasional drops and confidently wrong cases. Both however substantially beat the single sample baseline
B1 reinforcing the value of confidence aware ensembles and the scaling behavior for online deep compend
table 10 is perhaps even more telling. They compared adaptive deep comp with the consensus stopping against a version that only did early stopping
based on budget and against standard majority voting. And the adaptive part really helped >> hugely at a matched budget K512 shown in table 10.
Deepcom flow achieved the largest cost reductions 43 84% fewer tokens while usually matching or improving
accuracy. Deepcom high was more conservative 16 59% savings with accuracy essentially unchanged. But figure 9 shows the paro frontier. Adaptive decom
consistently uses fewer tokens to achieve a comparable accuracy level compared to just running deep comp with early stopping but without the adaptive
consensus check. Adaptive deep hump dominates the token accuracy trade-off. Finally, the GPQA
diamond results appendix EXIX 1011 showed similar patterns on those graduate level stem problems. Offline top 10% filtering often beat the baseline on
Quen 3 models. Top 90% matched or exceeded it acting as the safer choice. Both beat single sample. Online, the adaptive policies again achieved
greater token savings. Deepcom 5i maintained accuracy. deep com low flow push for more savings sometimes
slightly underperforming the baseline on deepcate BB in this aggressive mode it reinforces the broad applicability of these strategies >> okay that's
a wealth of detail on performance but for those listening who are thinking how hard is this to actually implement the paper provides incredibly useful
implementation details that's often missing but super valuable for practical adoption >> indeed and
kudos to the authors for this they've gone to great lengths to provide practical guidance even showing how deep can be integrated with minimal changes
into VLLM which is a very popular LLM serving framework. They specify the exact VLM commit hash Python version CDA version they used which is
excellent for reproducibility. At a high level they state only two key modifications were needed in the VLM
codebase. First they had to extend a class called log props processor within VLM. This was to make it maintain that sliding window confidence
calculation and to expose a new method checkcom stop which returns true or false. Second, they inserted just a single check calling that new method
inside the main output processing loop output processor.py right before the output is finalized. >> That
sounds surprisingly minimal. >> It really does. It speaks to the elegance of the approach that it doesn't require deep surgery on the inference
engine. How do you turn it on? They show enabling it via an open AI compatible API call. Just pass some extra arguments in the request body like
enable contired window size for example 248 and the threshold for example 17 for early stopping. A key
requirement though is that you must request log probabilities log props true and specifically at least the top two log probabilities for the logic to
actually work because the confidence calculation needs those probability values. Getting slightly more technical for the log props processor. They
added a few fields to track the group confidence, the list of confidences, the window size, the
threshold. These get initialized from the request parameters. The new check con stop method performs the check against the threshold. And they
modified an existing method update sample log props to update these confidence values after each token is sampled. The confidence itself is calculated
as the moving average of the negative mean log probabilities of the candidate tokens over that fixed
window. And then in output processor.py, it's literally just calling restate.log props processor.con check con stop in the main decoding loop. If that
returns true, it sets the finish reason to test.py and adds a little stop reason tag like Jon of THR so you can see why a sequence was stopped early
in your logs. >> That's incredibly practical. It makes it seem much more feasible to try this out.
Seven future work and conclusion. >> The authors also very helpfully lay out some promising avenues for future research which points to the broader
impact and potential of this work beyond just optimizing inference as it stands now. One really interesting direction they highlight is extending deep
comp to reinforcement learning settings. Imagine using this confidence-based early stopping not just
at inference time, but during RL training. >> How would that work? >> You could potentially use it to guide policy exploration. Maybe prune less
promising exploration trajectories early on. If the model shows low confidence in its actions or predicted outcomes, this could significantly improve
sample efficiency, which is often a major bottleneck in RL. Another critical area they acknowledge is the
need for future work on addressing cases where models are confidently wrong. We saw hints of this in the results. The overconfidence issue, >> right?
The top 10% filter sometimes hurting accuracy. >> Exactly. Future research really needs to dig into more robust confidence calibration techniques. How
can we make the model's expressed confidence better align with its actual probability of being
correct? And exploring more advanced uncertainty quantification methods. The goal would be to better identify and perhaps mitigate these instances of
overconfident but erroneous predictions. Making LLM not just efficient but also more reliably self-aware of their own limitations is key for critical
applications. >> Okay, let's try and synthesize this. Here's where it gets really interesting for
those of us deep in the weeds of LLM deployment and optimization. What we've seen with deepcom is a powerful yet, as you said, remarkably elegant
solution for two really persistent problems in LLM reasoning. that high computational cost and the diminishing returns you get from just blindly
adding more reasoning choices. The big takeaway for me is maybe we should stop just throwing more compute at
the problem and start listening more carefully to the model's internal voice, its own signals. By carefully tuning into its own confidence signals,
particularly these local ones, we can fundamentally make the model a much more efficient and potentially a more reliable thinker. This isn't just
about scaling up model parameters anymore. It's about making the existing compute work smarter, more
surgically. >> And that really raises an important question for you, the listener, especially given your expert background in machine learning and
data science. How might the insights from Deep Comp, particularly this novel, focus on local confidence signals and the demonstrated power of
real-time trace pruning? How might that influence your own approaches when you're designing or deploying LLM
powered reasoning systems? And maybe even more concretely, considering those surprisingly minimal VLM edits they detailed, what specific technical
challenges or perhaps compelling new opportunities do you foresee in integrating this kind of fine grained control over the generation process into
your existing inference pipelines? Think about how this granular control could open up entirely new
optimization strategies or maybe even enable new classes of applications that were previously too costly or slow. Something worth pondering. Mhm. >>
And that wraps up our deep dive into Dethink with confidence. We've explored how a simple, conceptually elegant test time method can dramatically
enhance LLM reasoning performance and computational efficiency. It does this through intelligent,
confidence filtering and crucially early stopping based on the model's own internal signals. It's really a powerful testament to the potential that
lies in optimizing the inference process itself, not just endlessly scaling model parameters as we look towards the next generation of AI systems. If
this discussion has sparked some new ideas or maybe made you look at LLM inference with fresh eyes and
a renewed sense of possibility for efficiency, then our mission here is accomplished. Until next time, keep learning, keep questioning, and keep
exploring the fascinating, rapidly evolving world of artificial intelligence.