CriticGen turns LLM evaluation into actionable rewrite feedback
A paper posted to arXiv proposes CriticGen, an evaluation framework built around a simple complaint: today's methods for judging large language model outputs are coarse and generic, so they hand back a score with an explanation that never tells the model what to actually change. CriticGen instead builds a rubric for each individual answer. It first generates sample-specific evaluation dimensions and scoring criteria, grouped under three high-level categories: subjective, objective, and self-derived constraints. That rubric then drives a single pass that jointly produces four things: a score, a reason for the score tied to the rubric's criteria, an executable suggestion for how to fix the answer, and the refined answer itself. The authors report that fine-grained, instance-specific rubrics beat generic ones on multiple fronts. Rubric quality rose on relevance from 3.33 to 3.97 and on coverage from 4.03 to 4.24. CriticGen's scores correlate closely with reference judgments, at 0.9556 Pearson and 0.9560 Spearman. The F1 score for reasons that are actually grounded in the stated criteria rose from 0.6369 to 0.7554, and the F1 for suggestions that are genuinely executable rose from 0.5994 to 0.7900. Most concretely, applying CriticGen's feedback improved 73.17% of the answers it touched, while 93.28% of answers were left no worse than before, meaning the refinement step rarely backfires.
Key facts
- CriticGen generates a sample-specific rubric for each answer instead of applying one fixed set of criteria to every case.
- The rubric's categories are subjective, objective, and self-derived constraints, and it drives a single pass that outputs a score, a reason, a fix suggestion, and a refined answer together.
- Rubric quality improved from 3.33/4.03 (relevance/coverage) to 3.97/4.24 with CriticGen.
- CriticGen's scores reach 0.9556 Pearson and 0.9560 Spearman correlation, and F1 for grounded reasons and executable suggestions rose from 0.6369/0.5994 to 0.7554/0.7900.
- Feeding CriticGen's suggestions back into the answer improved 73.17% of cases with a 93.28% non-degradation rate.
Why it matters
Most automated evaluation of LLM output stops at a number and a generic blurb, which is useless for actually fixing the answer because it is decoupled from how the answer was generated in the first place. CriticGen's pitch is to make evaluation double as control: the same pass that scores an answer also names what is wrong with it and rewrites it, closing the loop between judging output and improving it.
Who it affects
The intended audience is people building evaluation and self-improvement pipelines for language models: teams doing automated grading, RLHF-style feedback loops, or agent systems that need to critique and revise their own drafts without a human in the loop for every step.
How to use it
The method runs in two stages on each sample. First it generates dimensions and scoring criteria specific to that sample, sorted under subjective, objective, or self-derived constraints. Second, that rubric conditions a joint generation step that produces a score, a criterion-grounded reason, an executable suggestion, and the refined answer together, rather than as separate calls.
How solid is it
The abstract backs the method with several paired before-and-after numbers: rubric relevance and coverage, score correlation against a reference (Pearson 0.9556, Spearman 0.9560), F1 for grounded reasons and executable suggestions, and a downstream improvement rate on the refined answers. The abstract does not name the benchmark, dataset, or baseline system these figures were measured against, nor what the correlation numbers are correlated with.
Risks and caveats
The abstract names no authors or institutions and gives no publication venue or timeframe beyond the arXiv listing. It also does not spell out what falls under 'self-derived constraints' beyond naming it as one of the three rubric categories, so the exact mechanics of that piece are not verifiable from the source alone.
“fine-grained evaluation should be both instance-specific and actionable”
— CriticGen paper abstract