Training-Free Token-Level Steering for LLM Personalized Co-Writing

As Large Language Models (LLMs) are increasingly deployed in specialized fields like medicine, law, and software development, they often struggle with domain-specific nuances and the need for personalization. Current methods like fine-tuning are computationally expensive and slow to update, while Retrieval-Augmented Generation (RAG) is generally limited to prompt-level guidance. This paper asks: can we achieve fine-grained, token-level personalized co-writing without the overhead of training or fine-tuning?

SteerWrite introduces a training-free framework that treats the LLM as a "shadow typist." It leverages the model's internal hidden states to perform token-level retrieval from a small, user-provided dataset. Instead of just injecting context into the prompt, SteerWrite uses Kernel Density Estimation (KDE) to approximate an ideal posterior probability distribution. This allows the model to adjust its next-token predictions at every step based on the retrieved relevant context. To ensure stability and prevent common issues like repetition loops, the authors implement three regularization techniques: rank-based calibration to filter noise, temporal momentum to maintain sequential continuity, and frequency scaling to penalize overly common tokens.

The researchers evaluated SteerWrite across four domain-specific datasets (clinical medicine, ultrasound reports, legal documents, and code) using multiple model sizes from the Qwen3 family. The results demonstrate that SteerWrite consistently outperforms seven training-free baselines, including standard RAG and kNN-LM. By providing granular, token-level steering, the framework significantly reduces human editing effort—measured by lower edit distances and fewer required keystrokes—while maintaining low latency suitable for real-time, stream-based co-writing.