SemaPLC scores 52.2 versus baselines' 22.4 to 31.4 on live PLC runs

Programmable logic controllers (PLCs) run industrial plants, and large language models can already generate individual program organization units (POUs) for them. Whether that generated logic integrates into an existing PLC project and then runs correctly has so far been checked only in limited tests. The researchers present SemaPLC, a project-grounded, verification-gated agent harness assembled from conventional tools and governed by a strict completion rule: rather than stopping when the model itself judges its output adequate, SemaPLC declares a task complete only when logged external checks confirm it. Those checks cover the specification, the compilation, and the behavior on a live runtime.

On 117 independent-POU tasks matching existing benchmarks, SemaPLC attains the highest strict verified pass rate on all seven models tested, averaging 72.6%. On a separate project-context track of 65 tasks, where the generated logic must compile and run inside a real project rather than stand alone, SemaPLC again attains the highest mean score on integrated compilation, static behavior, and dynamic behavior.

Of the three evaluation layers on the project-context track, dynamic behavior turns out to be the most revealing. The researchers measure it by deploying both the generated logic and reference logic to a live PLC runtime and comparing their executed traces. All methods land within 10 static points of one another, so static checks alone barely distinguish them. Dynamic scores separate the methods sharply instead: the baseline methods range from 22.4 to 31.4, while SemaPLC reaches 52.2.

The authors conclude that their verification-gated harness raises the mean score at every evaluation layer, and most sharply at runtime, arguing that execution rather than static scoring is the faithful test of whether generated control logic actually works. SemaPLC is open-sourced at github.com/midea-ai/SemaPLC.

Key facts

SemaPLC only marks a task complete when logged external checks confirm specification, compilation, and live-runtime behavior, not when the model judges its own output adequate.
On 117 independent-POU tasks, SemaPLC attains the highest strict verified pass rate across all seven tested models, averaging 72.6%.
On a separate 65-task project-context track, where generated code must compile and run inside an existing project, SemaPLC again leads on compilation, static behavior, and dynamic behavior.
All methods score within 10 points of each other on static checks, but dynamic behavior, measured from traces on a live PLC runtime, separates them sharply: baselines score 22.4 to 31.4, SemaPLC scores 52.2.
The code is open-sourced on GitHub at midea-ai/SemaPLC.

Why it matters

PLCs run industrial plants, and LLMs can already generate individual program units for them, but whether that generated logic actually integrates into a real project and runs correctly has only been checked in limited tests. SemaPLC closes that gap by gating task completion on external checks rather than the model's own self-judgment, and its results show that static scoring alone can hide correctness problems that only appear once the code actually runs.

Who it affects

Teams building or evaluating LLM-based code generation for industrial control systems, and PLC engineers who might rely on such generated logic in a real plant. The findings also speak to researchers designing evaluation harnesses for generated code more broadly, since the paper argues static analysis can mask gaps that only show up at execution time.

How to use it

SemaPLC is open-sourced at github.com/midea-ai/SemaPLC. It is assembled from conventional tools rather than a new model, so it functions as a verification layer that can be applied around existing PLC code generation setups. The source text gives no pricing or licensing terms.

How solid is it

SemaPLC was tested on 117 independent-POU tasks matching existing benchmarks and on a separate 65-task project-context track, across seven different models, attaining the highest pass rate on every one. The sharpest evidence for the harness comes from the dynamic-behavior test: live-runtime traces separate methods far more clearly than static checks do, which the authors take as support for execution-based evaluation. The source text does not name the seven models or the baseline methods compared, and does not break out individual static or compilation scores per method.

Risks and caveats

The source does not disclose which seven models or which baseline harnesses were tested, nor does it explain how the 'logged external checks' are implemented beyond covering specification, compilation, and live-runtime behavior. No author names, institutional affiliation, or publication date appear in the extracted text, and the reported results come only from the paper's own benchmarks with no independent validation mentioned.