Juseon Do, Sangmin Bae, Jeesu Jung, Hwanjun Song
8 min
Abstract
LLM-based agentic systems are increasingly combined with CoT reasoning to solve tasks that require accurate reasoning, external knowledge, and multi-step deliberation. Yet, explicit CoT incurs substantial generation overhead, while implicit CoT, which replaces textual rationales with compact continuous tokens, does not naturally support the explicit communication interface required for agentic tool-use. We propose TAC-CoT, a tool-empowered adaptive implicit CoT framework that replaces explicit reasoning traces with compact latent reasoning segments while retaining textual tokens only for tool calls, observations, and final answers. This interleaved design enables agentic tool-use without verbalizing every intermediate reasoning step. Experiments on agentic math and QA benchmarks show that TAC-CoT improves accuracy by up to 8.8 points while reducing model forward passes by up to 6.8x. Furthermore, analyses show that TAC-CoT improves tool-use quality in both single- and multi-tool settings.
Sam: The number of segments, not the length within each segment. A query can finish after a short internal stretch. Another can continue through more tool interactions and more internal stretches. The overall budget is determined during inference rather than fixed for every query.
Alex: Give us a concrete case where the distinction between thinking and communicating actually matters.
Sam: The document includes an overtime-pay problem. The explicit agent correctly calculates regular pay and the overtime hourly rate. But then it applies that overtime rate to every hour worked, rather than just the overtime hours. The calculator faithfully computes the wrong expression.
Alex: So tool access does not fix choosing the wrong calculation. What does TAC-COT do in that example?
Sam: It keeps regular pay, overtime pay, and the final sum separate through calculator calls, and gets the correct answer. That illustrates the intended workflow. It is not, by itself, evidence that compact reasoning always produces better decomposition.
Alex: How do they teach the model to replace an explanation with something that is not language?
Sam: They start from complete trajectories containing written reasoning, tool calls, observations, and answers. For mathematics, they align existing reasoning examples with calculator annotations. For question answering, a stronger model generates tool-using solutions, and they retain those whose final answers match the reference.
Alex: Synthetic examples could carry the generating model’s habits. Is that shared across the comparison?
Sam: Yes. They construct forty thousand question-answering trajectories, and all the question-answering baselines use the same trajectories. That makes the relative comparison fairer, though it does not remove biases from the training data.
Alex: But correct final answers do not directly tell the model what to put inside those hidden segments.
Sam: Training adds a decoder that reconstructs written reasoning from the continuous representations. That decoder is removed for inference. They also use self-distillation: a teacher version of the same backbone supplies internal-state targets from explicit reasoning, while the student learns the compact route.
Alex: Let’s turn to the evidence. What is the main comparison, and how broad is it?
Sam: They compare against an explicit-reasoning agent trained on the same trajectories, across several model backbones. The tasks include mathematical word problems and knowledge-intensive question answering, both within and outside the training domain. The question-answering evaluations use a thousand randomly sampled examples per dataset.
Alex: That tests transfer beyond the training task. How much better were the answers?
Sam: Averaged across backbones, tool-enabled math accuracy improves by about six percentage points over explicit reasoning. The corresponding question-answering improvement is about two points. The direction of the gains holds across the tested backbones and evaluation domains.
Alex: Could the benefit simply come from deleting verbose prose, rather than learning useful internal reasoning?
Sam: Their controlled comparisons push against that explanation. TAC-COT beats a version that removes natural-language reasoning but retains arithmetic equations. It also beats a tool-enabled baseline that shortens written reasoning by pruning tokens. Those results suggest that simply saying less is not the whole explanation.
Alex: Do the tools themselves get used better, or does the model just spend less time between calls?
Sam: On question-answering tasks, the correct answer appears earlier in retrieved observations more often. In a separate tool benchmark, TAC-COT wins about fifty-nine percent of the multi-tool comparisons against the explicit agent. That result is statistically significant, but it comes from a language-model judge, not a human deployment study.
Alex: Now the cost claim. If I care about waiting for an answer, what does fewer computations actually buy me?
Sam: Their main cost measure is forward passes: individual computations through the model. On question answering, the reduction reaches nearly sevenfold relative to explicit reasoning. That isolates model computation better than elapsed time, but it does not include tool latency as model work.
Alex: So that is not a promise that a user gets an answer seven times sooner.
Sam: On their measured single-GPU setup, question answering runs about two-point-four times faster. External calls and implementation overhead narrow the gap. The paper explicitly warns against treating forward-pass reductions as equivalent wall-clock speedups.
Alex: What most limits how far you would carry these findings?
Sam: The authors identify longer interaction horizons as unfinished evaluation. These benchmarks support the approach, but not an assumption that it will stay reliable through much longer workflows. Routing is also only adaptive at segment boundaries, not after every internal token.
Alex: And the error-accumulation explanation is supported by diagnostic tests, rather than a direct window into the hidden reasoning.
Sam: That is how I would read it. Replacing more written segments improves accuracy in their tests, and excess reasoning hurts the compact method less. Those patterns support the authors’ explanation, without making the hidden reasoning readable.
Alex: Who should make time for the full document, and where should they start?
Sam: Researchers building tool-using agents or studying efficient reasoning should read it. Start with the inference procedure, then the controlled baseline comparisons. Read the wall-clock analysis before making performance projections, and keep the longer-workflow limitation in view.
Alex: What should everyone else remember?
Sam: A model can keep its reasoning compact while keeping its communication with tools explicit. In these tests, that separation improved both answer quality and computational efficiency.
Alex: Thanks for listening.