Takumi Shioda, Kohei Terashima, Tatsuo Nagai
4 min
This paper investigates whether reasoning-capable Large Language Models (LLMs) can effectively control multi-zone Variable Air Volume (VAV) HVAC systems. The authors evaluate two primary strategies: using a frontier reasoning model (GPT-5) without building-specific training, and using Reinforcement Fine-Tuning (RFT) to adapt a smaller, open-weight model (gpt-oss-20b) using a teacher policy (TD3) as a guide. The study uses a physics-based four-zone emulator to test these controllers against a standard Guideline 36 rule-based baseline.
The authors find that a frontier reasoning model can achieve competitive control, outperforming the Guideline 36 baseline in electricity reduction (6.2% vs 4.5% for TD3). However, the RFT process for the open-weight model proved unsuccessful. Despite using a deterministic rollout verifier to score actions, the model failed to show sustained improvement. An audit of the learned critic revealed that while it correlated well with long-term outcomes, it was unreliable at ranking specific actions within a single state. The authors conclude that exact rollout scores are insufficient for training because they do not provide the model with the necessary 'directional' knowledge—specifically, how an action changes the next state—which is required to improve policy performance.
This research highlights a critical bottleneck in applying reinforcement learning to complex physical systems via LLMs. While LLMs are excellent at reasoning, they often lack the internal world model required to predict the consequences of continuous control actions. The study suggests that simply providing scalar rewards (the 'what') is not enough; for effective fine-tuning, models must also learn the 'how'—the transition dynamics of the environment. This points toward a need for transition-focused supervised fine-tuning before attempting value-based reinforcement learning in control tasks.
Multi-zone variable-air-volume control must balance thermal comfort, indoor air quality, and electricity use across several continuous actuators. Model predictive control and reinforcement learning are widely studied, but deployment typically requires building-specific modeling or training, limiting scalability. We first test whether a frontier reasoning model (an LLM trained to use additional inference-time computation) can achieve competitive VAV control from text without building-specific training. With that capability established, we then test whether TD3-guided reinforcement fine-tuning (RFT) can transfer control knowledge into a locally deployable open-weight model. Five controllers are evaluated over three summer days in a physics-based four-zone emulator. Relative to a Guideline 36-based baseline, TD3 reduced HVAC electricity by 4.5% while improving temperature and CO$_2$ compliance. Without building-specific training, GPT-5 achieved the largest reduction (6.2%) but reduced the ventilation margin. For RFT, deterministic rollouts restore a saved state, apply one candidate, and follow TD3 to score each action. Auditing a learned critic against these rollouts exposed a failure hidden by its near-perfect across-time correlation ($r=0.9998$): within-state ranking was unreliable; the critic selected the rollout-best candidate in only 5 of 10 states. Even with the rollout verifier, 200 RFT steps produced no sustained improvement in sampled-action return; the open-weight controller used more electricity than the baseline before and after training, and its five-minute predictions remained worse than persistence. GPT-5 predicted transitions far better. Exact rollout scores rank sampled actions but reveal neither next-state effects nor an improvement direction. The unchanged transition errors motivate transition-focused supervised fine-tuning before value-based RFT.
Alex: That's precisely the limitation the paper identifies. There's a concept in physics and engineering—imagine a kind of map that tells you, for every knob you can turn, exactly how much the temperature or airflow changes in response. It describes the precise relationship between your action and the physical outcome. The researchers call this the "Jacobian." And the key finding is that reward scores alone don't give the model that map. Scores tell you which action ranked better. They don't tell you in which direction to adjust your actions to do better next time.
Sam: Oh, that's a meaningful distinction. It's like knowing that turning a steering wheel makes a car turn, but having no idea whether you need to turn it five degrees or fifty degrees to reach your lane. The score tells you "that was wrong," but not "here's how to correct it."
Alex: Exactly. And that directional knowledge—understanding how each action ripples through the interconnected systems of a building—is what the model consistently lacked. Even with access to a perfect simulation and accurate scores, its control didn't meaningfully improve over training.
Sam: So the conclusion is that "knowing the score" is a necessary condition, but not a sufficient one. You need to understand the underlying mechanism before the feedback becomes useful.
Alex: That's the paper's core argument. The researchers suggest that the right sequence is: first, teach the model to predict what the building's next state will be given an action—build that physical intuition—and only then use reward signals to refine its decision-making. Without that foundation, the rewards are essentially noise.
Sam: It's a useful reminder that these models are extraordinarily capable with language and reasoning, but a building doesn't care about language. It obeys thermodynamics. And you can't talk your way around the physics.
Alex: That's well put. This study contributes something specific to how we think about deploying AI in physical environments: the gap isn't just about having enough data or a good reward signal. It's about whether the model has a working model of the world it's trying to control. That turns out to matter quite a lot. Thanks for listening to ResearchPod.