Researchers propose Locksmith Loop to validate AI-migrated COBOL-to-Java code
A paper titled "Agentic Method for Deterministic Validation of Legacy Code Migration", submitted to arXiv on 30 July 2026, tackles a problem specific to migrating legacy COBOL programs to Java: proving the translated code actually behaves the same as the original. The authors note that this kind of migration normally demands extensive testing, and that testing effort is often complicated by a lack of test data and by how hard it is to validate every corner case.
Their proposed method, called the "Locksmith Loop," starts by preparing two runtime environments side by side: the COBOL source and the generated Java target are each instrumented with mocks and run off-mainframe on ordinary commodity hardware. From there, an iterative agentic loop performs what the authors call a Witness Search over input mocks, searching for inputs that penetrate deeper into the program's branches, followed by parity-preserving mutations that vary those inputs while keeping COBOL and Java behavior comparable. When the search hits a routing boundary it cannot get past, an analyzer flags what the paper calls a Locked Paragraph, a condition that is preventing further exploration.
The authors tested the method across three COBOL-Java case studies: two open-source programs and one internal production-like COBOL program, ranging in size from 430 to 4,114 source lines. Locksmith consistently pushed coverage beyond the plateaus that plain input search reached on its own, achieving nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal production-like program. Across all accepted test cases, the generated Java matched the COBOL reference under deterministic parity checks. The authors describe this as, to the best of their knowledge, a novel way to validate agentic coding output using a deterministic oracle rather than relying on manual review or spot checks.
The abstract does not name the paper's authors, their institution, or which AI model or agent framework powers the Locksmith Loop. It also does not give a timeframe or cost for running the case studies, does not name the specific open-source and internal programs used, and does not report the exact coverage number reached on the two open-source programs beyond calling it "nearly complete." Despite the framing implied by the story's headline on Hacker News, the abstract itself does not describe any specific bugs found in the migrated code.
Key facts
- The paper proposes the "Locksmith Loop," an agentic test-synthesis method for validating COBOL-to-Java migrations, submitted to arXiv on 30 July 2026
- The method runs COBOL and Java side by side off-mainframe on commodity hardware, using Witness Search over input mocks and parity-preserving mutations to reach deeper program branches
- Tested on three case studies (two open-source, one internal production-like) ranging from 430 to 4,114 source lines
- Locksmith reached 91.90% branch coverage on the internal production-like program and nearly complete coverage on the two open-source programs
- Generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases
Why it matters
Migrating decades-old COBOL systems to Java is a recurring, high-stakes problem for banks, insurers and government agencies still running mainframe code, and the hardest part is not writing the new code but proving it behaves identically to the old one. Testing is usually hampered by missing test data and the sheer difficulty of covering every corner case a mainframe program can hit. The Locksmith Loop targets that verification gap directly, offering a deterministic way to check agentic coding output against a reference rather than trusting the migration on inspection alone.
Who it affects
The method is aimed at organizations undertaking legacy COBOL-to-Java migrations, and more broadly at teams evaluating AI-generated code migrations where a ground-truth reference implementation already exists. The paper's own case studies include one internal production-like COBOL program alongside two open-source ones, suggesting real production systems are already being used to test this kind of validation approach.
How to use it
The abstract describes the technique at a conceptual level: instrument both the COBOL source and generated Java target with mocks, run them off-mainframe on commodity hardware, then let an agentic loop search inputs to penetrate branches and apply parity-preserving mutations, flagging a Locked Paragraph when it hits a boundary it cannot get past. No code, tooling name, release, pricing or licensing details are given in the text, so there is nothing yet to adopt directly from this abstract alone.
How solid is it
The evidence comes from three case studies spanning 430 to 4,114 source lines, two open-source programs and one internal production-like program, with Locksmith reaching nearly complete coverage on the open-source cases and 91.90% branch coverage on the production-like one, and full parity between generated Java and the COBOL reference in all accepted test cases. That is a real, measured result rather than a claim, but it rests on three case studies from a single paper, and the abstract gives no author names, institution, underlying AI model, runtime or cost figures to independently weigh the work against.
Risks and caveats
The abstract does not disclose who wrote the paper, what organization they are affiliated with, or which AI model or agent framework generated the Java migrations being validated, all of which limit how far the results can be generalized. It also does not specify how long the case studies took or what they cost to run, does not name the specific programs tested, and does not give the exact coverage figure for the two open-source cases beyond "nearly complete." The Hacker News post title frames the story around bugs the AI-migrated code contained, but the abstract itself describes no specific bugs, only the coverage and parity-checking methodology used to find such problems.
“Locksmith consistently improved coverage beyond input-search plateaus, reaching nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal production-like COBOL program.”
— the paper's authors