Yanlun Tu, Huacan Wang, Ziyue Zhou, Jie Zhou, Ningyan Zhu, Ge Chen, Wangyi Chen, Tengfei Zhou, Yifan Zhou, Dasheng Yang, Xiaofeng Mou, Hui Zhang, Yi Xu
6 min
Abstract
Programmable logic controllers (PLCs) run industrial plants, and large language models can already generate independent program organization units (POUs) for them. Whether such logic integrates into an existing PLC project and then runs correctly has been checked only in limited tests. We present \textsc{SemaPLC}, a project-grounded and verification-gated agent harness assembled from conventional tools but governed by a strict completion rule. Rather than stopping when the model judges its own output adequate, \textsc{SemaPLC} declares a task complete only when logged external checks confirm it. Those checks cover the specification, the compilation, and the behavior on a live runtime. On 117 independent-POU tasks matching existing benchmarks, it attains the highest strict verified pass rate on all seven models (72.6\% mean). On a project-context track of 65 tasks whose generated logic must compile and run inside a real project, it attains the highest mean on integrated compilation, static behavior, and dynamic behavior. Of the three layers, dynamic behavior is the most revealing. We measure it by deploying the generated and the reference logic to a live PLC runtime and comparing their executed traces. All methods fall within 10 static points of one another, whereas dynamic scores separate them sharply, from 22.4 to 31.4 for the baselines against 52.2 for \textsc{SemaPLC}. Overall, our verification-gated harness raises the mean at every layer and most sharply at runtime. Execution, not static scoring, is the faithful test of whether generated control logic actually works. \textsc{SemaPLC} is open-sourced at https://github.com/midea-ai/SemaPLC.
Alex: So the errors aren't dead ends — they're clues that get fed back into the repair process.
Sam: Precisely. And the repair process has strict rules of its own. Each checking stage allows up to two repair attempts. But here's the part that really matters: any time the AI modifies the code — even a small change — every prior test result is immediately wiped out. The system starts the verification sequence from scratch.
Alex: Why is that important? Why not just re-run the specific test that failed?
Sam: Because a change in one part of the code can affect behavior somewhere else entirely. If you only re-check the thing you just fixed, you might miss a new problem you accidentally introduced. By resetting everything, the system guarantees that the final delivered program is the exact version that earned every single passing result — not a patched-together version where some tests apply to an older draft.
Alex: That prevents the AI from quietly claiming victory on a version of the code that was never fully tested.
Sam: That's the guarantee the framework is built around. And it matters because industrial control systems don't have much tolerance for partial correctness.
Alex: So does this actually work in practice? What did the tests show?
Sam: The results were meaningful. The runtime verification step — actually running the code and checking its behavior — caught failures that static compilation alone missed entirely. Programs that looked correct on paper turned out to behave incorrectly when deployed. The harness consistently produced higher reliability on the project-level tasks that most closely resemble real factory conditions.
Alex: Are there limits to what this approach can catch?
Sam: Yes, and the paper is clear about this. The dynamic checks only cover the test scenarios the engineers designed in advance. If a rare edge case wasn't included in the test set — an unusual plant condition, an unexpected sequence of inputs — the framework won't catch a failure there. The tests only find what the engineers thought to look for.
Alex: So it's not a guarantee of perfect safety. It's a significant improvement over the current standard, with known blind spots.
Sam: That's a fair summary. The paper also notes that the advantage of the framework narrows when using the most capable underlying AI models — so it's not a fixed margin of improvement across every situation. And there's a computational cost: the system makes substantially more requests to the AI model per task than a simpler approach would. Though the authors note that overall time stays comparable, because the checks run in a distributed way.
Alex: So you're trading more computational work for meaningfully higher reliability at runtime.
Sam: That's the trade-off. And the broader point the paper makes is this: static compilation — just checking whether code is grammatically correct — cannot reveal deep execution flaws in industrial control logic. True reliability only comes when the AI's completion is conditioned on external runtime evidence, not its own assessment of whether the job is done.
Alex: It's a bit like the difference between a student saying "I think my answer is right" versus actually checking it against the answer key and showing your work.
Sam: That's a good way to put it. And the authors suggest that future work in this area will likely move toward end-to-end development environments where writing, generating, and deploying code are all governed by this same verification gate — so the safety check isn't a separate step bolted on at the end, but built into the entire process from the start.
Alex: That seems like the logical direction. Thanks, Sam — this was a clear look at a genuinely important problem.
Sam: Thanks for listening to ResearchPod.