New evaluation framework finds systematic semantic loss in legal ontology learning

Key facts

• The paper proposes an evaluation framework for ontology learning that compares a language model's task performance on the original source text against its performance on a transformed, structured representation of that text, with the gap between the two quantifying the semantic loss.
• It demonstrates the framework on legal merger agreement analysis, a domain chosen for its complex language and precise semantic requirements.
• The demonstration compares direct LLM application against three ontology learning methods, tested across six language models.
• The results show systematic semantic loss that varies significantly with the task's reasoning complexity and with which model and method are paired.
• The authors frame their second contribution as empirical evidence that semantic loss varies dramatically by model-method pairing, offering guidance for selecting configurations in legal knowledge systems.

Why it matters

Ontology learning turns unstructured text into structured representations that automated systems can reason over, but the paper's starting point is that this transformation can lose meaning even when the output looks structurally correct, and current evaluation methods have no way to catch that: they check for structural correctness, not whether meaning survived. The framework proposed here makes that loss measurable: run the same task through a language model twice, once on the original text and once on the transformed representation, and treat the gap in performance as the semantic loss. That turns an otherwise invisible property, how much meaning a given ontology learning method throws away, into a number that different methods and models can be compared on.

Who it affects

Anyone building or evaluating ontology learning pipelines for domains where getting the meaning right matters as much as getting the structure right. The paper's own demonstration domain is legal knowledge systems, specifically legal merger agreement analysis, chosen for its complex language and precise semantic requirements, so the framework is aimed most directly at that field. It also matters to anyone choosing which ontology learning method to pair with which language model, since the paper's central finding is that this pairing changes how much meaning survives.

How to use it

The paper's own demonstration is the template for applying the framework: pick a task and document domain, measure a language model's task performance directly on the source text as the baseline, then measure the same model's performance on the output of each ontology learning method under comparison, run across multiple language models, and compare the results to score each model-method pairing on semantic loss. For legal knowledge systems, the authors say the resulting evidence, that semantic loss varies dramatically by model-method pairing, offers guidance for selecting which configuration to deploy. The text gives no further procedural detail beyond that: no released code, tool, or dataset is mentioned.

How solid is it

The evidence is a single demonstration: legal merger agreement analysis, comparing direct LLM application against three ontology learning methods across six language models. The paper describes the resulting loss as systematic and its variation as significant, driven by reasoning complexity and by model-method interactions, but it gives no percentage, score, or effect size for how large the loss is, and does not define how reasoning complexity was measured or varied. It does not name the three ontology learning methods or the six language models, does not identify which model-method pairing performed best or worst, and states no sample size, such as how many merger agreements were analyzed, or evaluation metric beyond LLM task performance in general. No author names, institutional affiliation, venue, or review status appear in the text either, so the claims cannot be checked against a fuller methods section.

Risks and caveats

The demonstration covers one domain, legal merger agreement analysis, so whether the framework's findings generalize to other kinds of documents or ontology learning tasks is not addressed in the text. The paper's own terms for the size of the effect, systematic, significant, and dramatically, are qualitative rather than quantified, so a reader cannot judge from the text alone how large the practical impact is on a legal knowledge system built with a mismatched model-method pairing. Because neither the three ontology learning methods nor the six language models are named, the guidance for selecting optimal configurations that the authors describe cannot be acted on directly from what is available here.

“The results reveal systematic semantic loss with significant variation based on reasoning complexity and model-method interactions.”

— the paper's authors