Georgii Aparin, Tatiana Gaintseva
5 min
Abstract
Linear activation steering has gained popularity as a simple and empirically effective way to control language model behavior. More recently, spherical steering paradigms have been proposed to address limitations of additive interventions, often motivated by the assumption that hidden-state norm does not carry concept-relevant information. In this work, we revisit this assumption through a controlled empirical study designed to disentangle the roles of angular and radial components. We show that steering methods differ mainly in how they couple two geometric effects: changing a token's angular alignment with a concept direction and changing its hidden-state norm. Across seven language models, we find that concepts are represented primarily in angular structure, supporting the motivation for spherical methods, but that norm remains important for the stability and downstream effects of steering. Our results explain why interventions with similar concept-level effects can behave differently, and suggest that activation steering should be parameterized by interpretable angular and radial components of the intervention, rather than by a single additive coefficient that entangles these two effects.
Alex: How did they actually test this? What did the experiments look like?
Sam: They ran six different steering methods across seven different language models, measuring both how well the steering worked and how much the model's output quality degraded. The consistent pattern was this: methods that fixed the signal strength while changing the direction caused the most damage to fluency. Methods that allowed the strength to scale appropriately caused far less. It wasn't a marginal difference—the gap between stable and unstable outputs was meaningful, and it widened for larger models.
Alex: That last point is worth pausing on. Why would larger models be more sensitive to this?
Sam: The paper doesn't fully resolve that question, but the implication is that larger models have more finely calibrated internal signal strengths. Their signals are tuned more precisely to their training, so forcing a mismatch between direction and strength throws them off more severely. It's a bit like a more complex instrument being harder to play out of tune.
Alex: So the practical upshot is: don't just add a fixed value to the signal. Treat the direction and the strength as two separate dials that need to be adjusted together.
Sam: That's the key insight. The authors frame it as moving from a one-size-fits-all additive change to a two-parameter process. You choose your direction to encode the desired behavior, then you calibrate the strength to keep the model stable. The important thing is that stability isn't a lucky side effect—it's something you can actively control.
Alex: Is there a formula for how much to adjust the strength? Or is it still trial and error?
Sam: That's the central practical limitation the paper acknowledges. The optimal strength adjustment depends on the task and the specific model—there isn't yet a universal rule that works everywhere. It's currently an empirical process, meaning researchers have to run tests to find the right balance for each situation. The authors flag this as the logical next step: developing a way to predict the right adjustment automatically, without needing to run dozens of experiments each time.
Alex: So the paper's contribution isn't a finished tool—it's more of a framework. A clearer way of understanding why previous methods failed and what a better approach should look like.
Sam: That's a fair characterization. By separating the geometry—direction versus strength—the research gives the field a more principled foundation to build on. It reframes the problem from "how hard do we push the signal?" to "in which direction, and with how much energy?" Those sound similar, but treating them as independent variables opens up a much more precise approach to model control.
Alex: And that kind of conceptual clarity tends to be what moves a field forward. Thanks for walking through the mechanics of this, Sam. That's it for this look at activation steering and the geometry behind it. Thanks for listening to ResearchPod.