SemaPLC verifies LLM-generated PLC code against a live runtime

Programmable logic controllers, or PLCs, run industrial plants, and large language models can already generate individual program organization units, or POUs, for them. But whether that generated logic actually integrates into an existing PLC project and runs correctly has so far been checked only in limited tests.

Researchers introduce SemaPLC, an agent harness built from conventional tools but governed by a strict completion rule: instead of stopping once the underlying model judges its own output adequate, SemaPLC only marks a task complete when logged external checks confirm it. Those checks span three layers: whether the code matches the specification, whether it compiles, and how it behaves when actually run on a live PLC runtime.

On a set of 117 independent-POU tasks that match existing benchmarks, SemaPLC reaches the highest strict verified pass rate of any method tested, across all seven models used as generators, averaging 72.6%.

On a separate, harder track of 65 tasks, where the generated logic has to compile and run inside a real, existing project rather than stand alone, SemaPLC again scores highest, this time across three measured layers: integrated compilation, static behavior and dynamic behavior. Of the three, the researchers found dynamic behavior the most revealing. They deployed both the generated code and the reference logic to a live PLC runtime and compared the traces each one produced while running. Static behavior scores barely told the methods apart: every method compared landed within 10 points of every other on that measure. Dynamic behavior scores split the field sharply instead. The baseline methods scored between 22.4 and 31.4, while SemaPLC reached 52.2.

The authors' conclusion is that the verification-gated harness lifts the mean score at every layer, and by the widest margin at the runtime layer, and that actually executing generated control logic, rather than scoring it statically, is the trustworthy way to tell whether it works. SemaPLC's code is open source on GitHub.

Key facts

SemaPLC is an agent harness that only marks a PLC-code generation task complete once logged checks against a live runtime confirm the code actually works, not when the model itself judges the job done.
On 117 independent-POU tasks matching existing benchmarks, SemaPLC posts the highest strict verified pass rate across all seven tested models, averaging 72.6%.
On a separate 65-task project-context track, where generated logic must compile and run inside a real project, SemaPLC again scores highest on integrated compilation, static behavior and dynamic behavior.
Static behavior scores barely separate the compared methods, all within 10 points of each other, but dynamic behavior, measured by running the code on a live PLC and comparing execution traces, splits them sharply: baselines score 22.4 to 31.4 versus 52.2 for SemaPLC.
SemaPLC's code is open source on GitHub.

Why it matters

Large language models can already write individual PLC program units, but PLCs run physical industrial plants. A program that looks right, and even compiles, can still behave wrong once it runs against real inputs and timing. Before this work, whether generated logic actually integrates into an existing project and runs correctly had only been checked in limited tests. SemaPLC sets a stricter bar: do not take the model's word that a task is finished, and do not stop at static checks either. Confirm the behavior against a live runtime. The clearest result is as much about method as about SemaPLC itself: static analysis left every compared approach within 10 points of the others, while actually running the code separated them by tens of points. That gap suggests static scoring alone can hide large real differences in whether generated control code works.

Who it affects

This affects automation engineers who are deciding whether to let a language model touch PLC logic, and teams building code-generation harnesses that need a completion rule stricter than the model's own say-so. It also affects researchers benchmarking such systems, who now have a project-context track to test against rather than only isolated program units.

How to use it

SemaPLC's code is open source, published on GitHub by the authors, so a team can run the same verification-gated harness, checking specification, compilation and live-runtime behavior, on its own PLC code generation setup rather than building that checking pipeline from scratch. The source does not state a license or usage terms for the repository, and it does not name the seven backbone models or the baseline methods it was compared against, so reproducing the published numbers would mean choosing models and baselines independently.

How solid is it

The evaluation spans two tracks: 117 independent-POU tasks matching existing benchmarks, and a separate, harder 65-task project-context track, where generated logic has to compile and run inside a real project rather than stand alone. SemaPLC posts the highest strict verified pass rate across all seven tested models on the first track, averaging 72.6%, and the highest mean on all three measured layers, compilation, static behavior and dynamic behavior, on the second. The standout numbers are the dynamic behavior scores, taken by deploying both the generated and the reference logic to a live PLC runtime and comparing the traces each produces: the baseline methods score between 22.4 and 31.4, SemaPLC scores 52.2, against a static-behavior spread of under 10 points across every method tested. The source gives no raw compilation or static behavior scores beyond that tight clustering, and it does not name the seven models or the baseline methods, which limits how far an outside reader can check the result.

Risks and caveats

The text carries no author names, affiliations or institutions, no publication date, and no license or usage terms for the GitHub repository, so there is no way to tell from the source alone who ran the evaluation or on what terms the code can be reused. The identities of the seven models and of the baseline methods are not given either, which makes it hard to judge whether the result would hold on other systems. As with any self-reported benchmark, the numbers come from the authors' own evaluation, not an independent audit.

“Execution, not static scoring, is the faithful test of whether generated control logic actually works.”

— the SemaPLC paper