Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.
Alex: Better answers do not prove that two language models communicated usefully. This study builds a connection between a large model and a smaller one, then tests whether the smaller model actually needs the information.
Sam: That distinction matters if you're paying for the larger model. Are the authors proposing a better system, or questioning what these systems achieve?
Alex: Both. This is a method paper and experimental study by Zhenyi Shen, Junru Lu, Wei Liu, and colleagues, available as an arXiv preprint. Their system improves benchmark performance, but several gains survive when useful information from the larger model is removed.
Sam: So this affects both efficient deployment and how researchers interpret collaboration between models.
Alex: Yes. If training the connection does the work, the larger model may be unnecessary for those gains. We'll follow how the system works, why its controls change the interpretation, and where useful communication still looks plausible.
Sam: Start with the division of labour. What does the larger model actually do, and when does it stop?
Alex: It reads the input prompt. That's prefilling: processing the input before generating a response. It passes internal representations to the smaller model, which then generates the entire answer without further help.
Sam: So the expensive model never writes an answer. The hope is to keep its contribution while avoiding its generation cost?
Alex: That's the aim. The authors call this Prefill-to-Autoregressive Cross-model Transfer, or PACT. Autoregressive means generating one token after another. The large model is the sender; the small model is the receiver.
Sam: This division already existed. What was missing from earlier approaches?
Alex: Mostik reported promising results, but remains closed-source. Open-source approaches included C2C and KV-Lingo, which connect models through cached attention information. The authors wanted an open framework supporting longer generation, large model-size gaps, and architectures that combine different attention mechanisms.
Sam: What is the one design choice that makes their framework different?
Alex: Their framework, PLatour, transfers hidden states: the internal numerical representations a model computes. It doesn't directly replace the receiver's attention cache. Instead, a trainable bridge converts sender representations into additions to the receiver's own processing.
Sam: An addition sounds less disruptive than replacing what the receiver already computed. How do they make that safe at the start?
Alex: They initialize the bridge weights to zero. Before training, the connection changes nothing, preserving the receiver's original behaviour. They connect layers at roughly corresponding depths, and changes introduced lower down flow through the receiver's later layers.
Sam: And what teaches the bridge which additions are helpful?
Alex: Reinforcement learning: training with rewards for generated answers. Both original models stay frozen; only the bridge changes. Unlike alignment or distillation approaches, the system isn't first taught to match internal representations or imitate the sender's answers.
Sam: So matching the models isn't the target. Getting the smaller model to solve the task is.
Alex: That's the distinction. The authors treat the sender's representations as tools the receiver might use. Their bridge uses simple linear projections, which transform those representations without a more elaborate network.
Sam: But a trainable connection is also extra machinery attached to the receiver. Couldn't that improve performance even if the sender contributes nothing useful?
Alex: That's the central problem. Freezing the original models doesn't eliminate adaptation. A trained bridge can change the receiver's computation, so an improvement doesn't automatically identify information transfer as the reason.
Sam: How do they separate those explanations rather than just acknowledge the ambiguity?
Alex: They introduce PACT-Check, with complementary controls. The first is dummy transfer. It keeps fixed sender features from the prompt template while removing question-dependent communication. The trained bridge stays unchanged, and the sender isn't called during this test.
Sam: That asks whether a reusable conditioning signal can substitute for reading the actual question. What does the second intervention test?
Alex: Irrelevant transfer supplies sender representations from an unrelated question, while the receiver still gets the original problem. A drop shows sensitivity to mismatched information. But sensitivity alone doesn't establish useful transfer: the wrong input could simply disrupt the system.
Sam: Those interventions test an already trained connection. What if another training route achieves the same improvement without an external sender?
Alex: That's why they also train separate controls. Self-communication uses the receiver as both sender and receiver. In a particular matching setup, this becomes a residual adapter: a learned adjustment to its own hidden states. The trained-receiver control adapts the receiver directly, without sender assistance.
Sam: So the tests ask both whether this system needs relevant information and whether its gains can be reproduced without it.
Alex: And PACT-Score makes the comparison explicit. It's the system's accuracy minus the strongest applicable control's accuracy. A positive margin supports a benefit beyond those controls, not a complete explanation of what information travelled.
Sam: Give us a case where the larger sender seems to contribute something.
Alex: On Chemistry questions requiring generated reasoning, the standalone receiver scored about forty-one percent. PLatour reached about sixty-nine percent. This was the Qwen model setting with a very large sender and a much smaller receiver.
Sam: That's a substantial improvement over the original receiver. How much survives comparison with adaptation?
Alex: The strongest control, direct receiver training, reached about sixty-six percent. PLatour therefore retained a modest advantage over every tested control. But Chemistry had only two hundred and ten test questions, so I'd keep that positive result tightly scoped.
Sam: And where does the headline improvement become less convincing evidence of communication?
Alex: On GSM8K, a mathematical word-problem benchmark, PLatour gained about eight and a half percentage points over the original receiver. Yet its margin over the strongest control was only about a quarter of a point. Unrelated sender information reproduced almost all its performance.
Sam: The system got better, but that result doesn't tell us the sender supplied useful problem-specific knowledge.
Alex: The harder mathematics and coding experiments reinforce that concern. Neither tested PACT system beat its compared trained-receiver baseline on domain-average accuracy. Fixed or irrelevant sender signals also reproduced much of the improvement.
Sam: Yet the authors also outperform Mostik. How should a listener weigh that comparison?
Alex: PLatour exceeded Mostik's reported results on shared mathematics and coding benchmarks using a sender with thirty-five billion parameters. Mostik used a sender with seven hundred and fifty-three billion. But its training data are undisclosed, and PLatour trains separate bridges for mathematics and coding.
Sam: So it's a useful reference result, not a controlled demonstration that this architecture is superior. Does the efficiency argument hold?
Alex: Their timing experiments show additional upfront cost from processing the prompt through both models. For longer outputs, that cost is spread over more generation, and total inference time approaches the receiver alone. That's an opportunity, not evidence that every task needs a sender.
Sam: What most limits the broader conclusion about communication?
Alex: The evidence remains specific to the tested controls, tasks, and models. The main experiments connect models within the same family, so transfer across unrelated model families remains open. They also observe training instability with larger senders; greater sender size didn't reliably improve results.
Sam: Who should read the full document, and where should they start?
Alex: Researchers building latent communication systems or efficient inference pipelines should start with PACT-Check. Then read the results asking whether gains depend on useful sender information, especially Chemistry versus mathematics. Read the bridge architecture afterward if you want to implement the open-source framework.
Sam: And what should everyone else carry into their next discussion of collaborating models?
Alex: Better answers show that the system improved. To show communication helped, test what happens when the information is gone.
Sam: Keep that distinction in mind.