LLM-based agentic systems are increasingly combined with CoT reasoning to solve tasks that require accurate reasoning, external knowledge, and multi-step deliberation. Yet, explicit CoT incurs substantial generation overhead, while implicit CoT, which replaces textual rationales with compact continuous tokens, does not naturally support the explicit communication interface required for agentic tool-use. We propose TAC-CoT, a tool-empowered adaptive implicit CoT framework that replaces explicit reasoning traces with compact latent reasoning segments while retaining textual tokens only for tool calls, observations, and final answers. This interleaved design enables agentic tool-use without verbalizing every intermediate reasoning step. Experiments on agentic math and QA benchmarks show that TAC-CoT improves accuracy by up to 8.8 points while reducing model forward passes by up to 6.8x. Furthermore, analyses show that TAC-CoT improves tool-use quality in both single- and multi-tool settings.
Sam: A language model can use tools without writing out every step of its reasoning. In this study, replacing written steps with compact internal representations improved answers while reducing computation.
Alex: That sounds useful for agents that repeatedly search or calculate. Is the saving just shorter answers, or does the model solve the problem differently?
Sam: It changes the reasoning between tool calls, while keeping those calls readable and executable. This is a method paper and arXiv preprint by Juseon Do and colleagues at KAIST and Cluvion. They call the method TAC-COT.
Alex: So the audience is researchers trying to make tool-using models cheaper without making them less reliable.
Sam: And researchers asking whether reasoning must be written out to be useful. We’ll unpack the design, test what explains the gains, and separate reduced computation from actual speed.
Alex: Start with the tension. Written reasoning already works with tools, so what prevents researchers from simply compressing it?
Sam: Chain of thought means producing intermediate reasoning before an answer. In the explicit version, those steps are readable words. That fits naturally with an agent that reasons, calls a calculator or search tool, reads the result, and continues.
Alex: But writing every step becomes part of the workload. The model has to generate the explanation as well as the tool request.
Sam: The authors also point to errors accumulating across long written traces. An alternative is implicit reasoning: intermediate work represented by continuous vectors rather than readable words. Earlier methods, including CODI and SIM-CoT, generally put that work into a fixed-length block before answering.
Alex: Which leaves no place to ask the outside world a question halfway through. How does this method open that interface?
Sam: It breaks the internal work into short segments. After each segment, the model chooses whether to call a tool or produce its answer. A tool call is ordinary text; the returned observation also stays text. Then compact reasoning resumes.
So the novelty is not hiding the entire conversation inside the model. It hides the reasoning spans but preserves the communication.
Sam: That distinction makes the method work. Each continuous representation comes from the model’s internal state and is fed back as an input, without turning it into a vocabulary word. But external tools still receive explicit instructions they can execute.
Alex: The name also promises adaptive reasoning. What adapts if each internal segment has a fixed length?
Sam: The number of segments, not the length within each segment. A query can finish after a short internal stretch. Another can continue through more tool interactions and more internal stretches. The overall budget is determined during inference rather than fixed for every query.
Alex: Give us a concrete case where the distinction between thinking and communicating actually matters.
Sam: The document includes an overtime-pay problem. The explicit agent correctly calculates regular pay and the overtime hourly rate. But then it applies that overtime rate to every hour worked, rather than just the overtime hours. The calculator faithfully computes the wrong expression.
Alex: So tool access does not fix choosing the wrong calculation. What does TAC-COT do in that example?
Sam: It keeps regular pay, overtime pay, and the final sum separate through calculator calls, and gets the correct answer. That illustrates the intended workflow. It is not, by itself, evidence that compact reasoning always produces better decomposition.
Alex: How do they teach the model to replace an explanation with something that is not language?
Sam: They start from complete trajectories containing written reasoning, tool calls, observations, and answers. For mathematics, they align existing reasoning examples with calculator annotations. For question answering, a stronger model generates tool-using solutions, and they retain those whose final answers match the reference.
Alex: Synthetic examples could carry the generating model’s habits. Is that shared across the comparison?
Sam: Yes. They construct forty thousand question-answering trajectories, and all the question-answering baselines use the same trajectories. That makes the relative comparison fairer, though it does not remove biases from the training data.
Alex: But correct final answers do not directly tell the model what to put inside those hidden segments.
Sam: Training adds a decoder that reconstructs written reasoning from the continuous representations. That decoder is removed for inference. They also use self-distillation: a teacher version of the same backbone supplies internal-state targets from explicit reasoning, while the student learns the compact route.
Alex: Let’s turn to the evidence. What is the main comparison, and how broad is it?
Sam: They compare against an explicit-reasoning agent trained on the same trajectories, across several model backbones. The tasks include mathematical word problems and knowledge-intensive question answering, both within and outside the training domain. The question-answering evaluations use a thousand randomly sampled examples per dataset.
Alex: That tests transfer beyond the training task. How much better were the answers?
Sam: Averaged across backbones, tool-enabled math accuracy improves by about six percentage points over explicit reasoning. The corresponding question-answering improvement is about two points. The direction of the gains holds across the tested backbones and evaluation domains.
Alex: Could the benefit simply come from deleting verbose prose, rather than learning useful internal reasoning?
Sam: Their controlled comparisons push against that explanation. TAC-COT beats a version that removes natural-language reasoning but retains arithmetic equations. It also beats a tool-enabled baseline that shortens written reasoning by pruning tokens. Those results suggest that simply saying less is not the whole explanation.
Alex: Do the tools themselves get used better, or does the model just spend less time between calls?
Sam: On question-answering tasks, the correct answer appears earlier in retrieved observations more often. In a separate tool benchmark, TAC-COT wins about fifty-nine percent of the multi-tool comparisons against the explicit agent. That result is statistically significant, but it comes from a language-model judge, not a human deployment study.
Alex: Now the cost claim. If I care about waiting for an answer, what does fewer computations actually buy me?
Sam: Their main cost measure is forward passes: individual computations through the model. On question answering, the reduction reaches nearly sevenfold relative to explicit reasoning. That isolates model computation better than elapsed time, but it does not include tool latency as model work.
Alex: So that is not a promise that a user gets an answer seven times sooner.
Sam: On their measured single-GPU setup, question answering runs about two-point-four times faster. External calls and implementation overhead narrow the gap. The paper explicitly warns against treating forward-pass reductions as equivalent wall-clock speedups.
Alex: What most limits how far you would carry these findings?
Sam: The authors identify longer interaction horizons as unfinished evaluation. These benchmarks support the approach, but not an assumption that it will stay reliable through much longer workflows. Routing is also only adaptive at segment boundaries, not after every internal token.
Alex: And the error-accumulation explanation is supported by diagnostic tests, rather than a direct window into the hidden reasoning.
Sam: That is how I would read it. Replacing more written segments improves accuracy in their tests, and excess reasoning hurts the compact method less. Those patterns support the authors’ explanation, without making the hidden reasoning readable.
Alex: Who should make time for the full document, and where should they start?
Sam: Researchers building tool-using agents or studying efficient reasoning should read it. Start with the inference procedure, then the controlled baseline comparisons. Read the wall-clock analysis before making performance projections, and keep the longer-workflow limitation in view.
Alex: What should everyone else remember?
Sam: A model can keep its reasoning compact while keeping its communication with tools explicit. In these tests, that separation improved both answer quality and computational efficiency.
Alex: Thanks for listening.