ReviewGrounder

Peer review is the primary mechanism through which the research community filters and improves new scientific work before publication. The rapid growth of submissions at major AI conferences (with counts at leading venues surpassing 10,000) has placed sustained pressure on peer review workflows. Meanwhile, recent advances in LLMs have spurred growing interest in using them to assist or complement the peer review workflow.

Despite these advances, prior work has highlighted notable shortcomings in existing LLM-based peer review frameworks: they produce routine, template-like critiques; accept authors' claimed novelty or limitations without thorough verification; and lack technical details, actionable suggestions, as well as justification grounded in the paper. These limitations can be traced to the underutilization of two crucial sources: (1) Reviewer Guidelines and Rubrics—top-tier venues provide well-established guidelines; and (2) Context from Existing Work—assessing novelty inherently requires situating a paper relative to existing work.

We introduce ReviewBench, a benchmark that leverages reviewer rubrics in an explicit and systematic manner, and ReviewGrounder, a rubric-guided, tool-integrated, multi-agent framework for producing grounded, content-rich reviews. ReviewGrounder decomposes reviewing into collaborating agents: the drafter produces an initial draft, and subsequent grounding agents (Literature Searcher, Insight Miner, Result Analyzer) refine it using tools. Experiments show that ReviewGrounder with a Phi-4-14B-based drafter and GPT-OSS-120B-based grounding consistently outperforms baselines including GPT-4.1 and DeepSeek-R1-670B across rubric-based dimensions and human-aligned metrics.

Similarity-based metrics and LLM-as-a-Judge approaches used by prior studies either fail to capture fine-grained review competencies or rely on ambiguous evaluation criteria. We introduce ReviewBench, a benchmark built on DeepReview-13K that augments each paper $p$ and its human reviews $\mathsf{H}_p$ with: (1) an aggregated reference review $r^*_p$; and (2) a set of paper-specific rubrics $\mathsf{R}^{\text{paper}}_p$. By leveraging these alongside an evaluator $\mathcal{E}$, ReviewBench enables accurate, multi-faceted assessment.

We define eight paper-agnostic meta-rubrics: Core Contribution Accuracy, Results Interpretation, Comparative Analysis, Evidence-Based Critique, Critique Clarity, Completeness Coverage, Constructive Tone, and False or Contradictory Claims (pitfall). Each meta-rubric is instantiated into paper-specific rubrics $\mathsf{R}^{\text{paper}}_{p,i}$ using the reference review and paper content. The overall content score is $S(p,\hat{r}_p) = \sum_{i=1}^{8} s_{p,i}$.

ReviewGrounder casts reviewing as a staged process that progressively refines an initial draft via targeted analysis, external evidence, and structured synthesis.

Stage I: Draft Review Generation. Given a paper $p$, a fine-tuned Drafter $\mathcal{P}$ generates an initial draft review $r^{(0)}$ that captures basic structure and stylistic conventions.

Stage II: Multi-dimensional Review Grounding. Three specialized agents collaboratively enrich the draft: Literature Searcher $\mathcal{S}$ situates the submission within contemporary literature via Semantic Scholar API and reranking; Insight Miner $\mathcal{M}$ consolidates conceptual understanding and refines method-focused critiques; Result Analyzer $\mathcal{A}$ strengthens empirical grounding by examining experiments, datasets, baselines, and quantitative evidence.

Stage III: Rubric-Guided Synthesis. The Aggregator $\mathcal{G}$ synthesizes the draft and grounded evidence $\mathsf{E}(p)$ with meta-rubrics $\mathsf{R}^{\text{meta}}$ to produce a coherent, accurate, and actionable final review. Paper-specific rubrics are not exposed at generation time, preventing evaluation leakage.

ReviewGrounder produces concise, evidence-grounded critiques with specific references (sections, equations, tables). DeepReviewer-14B tends to generate verbose, repetitive text that echoes prior reviewers without adding substantive, paper-specific insight. Toggle below to compare the two approaches.

We conduct evaluation on ReviewBench using two complementary metric families: (1) Rubric-based Evaluation—assesses textual quality across eight paper-specific rubric dimensions (Sec. 3.2); (2) Numeric-field Evaluation—measures predicted ratings with MSE/MAE and decisions with ACC/F1 (Sec. 3.3). Paper-specific rubrics and the evaluator are fixed across all methods; models access only the paper text at generation time.

Baselines: (1) Foundation Models: Qwen3-32B, QWQ-32B, GPT-4o, GPT-4.1; (2) Agentic Frameworks: AI Scientist, AgentReview (instantiated with GPT-4o/GPT-4.1); (3) Fine-tuned Reviewers: CycleReviewer-8B/70B, DeepReviewer-7B/14B. ReviewGrounder uses Phi-4-14B as Drafter and GPT-OSS-120B for grounding agents.

Table 1: Rubric-based Evaluation

Overall, ReviewGrounder consistently outperforms all baselines. Relative to the best foundation model (Qwen3-32B), +38%. Against AgentReview and AI Scientist (GPT-4o): +121% and +193%. Exceeds DeepReviewer-14B by 36%. Outperforms GPT-4o across all dimensions (+135%).

Higher scores indicate better performance. Contradict. is a pitfall dimension scored in {−2, −1, 0}; others in {0, 1, 2}. Core=CORE CONTRIBUTION ACCURACY, Res.=RESULTS INTERPRETATION, Comp.=COMPARATIVE ANALYSIS, EBC=EVIDENCE-BASED CRITIQUE, Clr.=CRITIQUE CLARITY, Cov.=COMPLETENESS COVERAGE, Tone=CONSTRUCTIVE TONE, Contradict.=FALSE OR CONTRADICTORY CLAIMS.

Table 2: Numeric-field Evaluation

Compared with all baselines, ReviewGrounder achieves the lowest rating error (MSE: 1.1607, MAE: 0.8597) and highest decision accuracy (ACC: 0.6809, F1: 0.6699). Relative to the strongest AI Scientist variant (Gemini-2.0-Flash-Thinking), improves ACC by 8% and reduces MSE by ~63%.

Table 3: Ablation Study

Impact of Drafter Backbones. When trained on the same SFT data, Qwen3-4B outperforms Phi-4-7B but remains inferior to Phi-4-14B. Smaller Drafters (e.g., Qwen3-4B: 10.6418) still benefit substantially from grounding and aggregation.

Impact of Grounding Agents. Omitting any agent (Searcher, Miner, Analyzer) degrades performance relative to the full model (10.7699), underscoring the importance of each component.

Additional Analyses

Hyperparameter Study (Literature Searcher). 10 reranked papers per keyword yields the highest overall score. OpenScholar-Reranker significantly outperforms BAAI-BGE Base/Large.

Defense Against Adversarial Attacks. With malicious instructions injected into input papers (500-sample subset): ReviewGrounder shows strong resilience (10.70→10.65, −0.05); DeepReviewer-14B drops from 7.70 to 7.30. Rubric-based evaluation prevents score inflation via instruction injection.

Human Evaluation. On 120 papers, expert raters (avg. 2,000 Google Scholar citations) closely align with the LLM-evaluator: Pearson r=0.8954, Spearman ρ=0.7923, MAE=0.0969, Pairwise Error=0.1494.