CSPaper: peer review sidekick

Dear CSPaper Review Users, After extensive preparation and foundational work, we’re thrilled to announce the release of CSPaper Review v1.2.0! [image: 1762547240915-screenshot-2025-11-07-at-12.55.20.png] Choose Your Favorite LLM We’ve introduced a new step after paper upload that allows you to select your preferred LLM for generating reviews. Currently supported models include: GPT-5, O3, O4-mini, GPT-4.1, Gemini-2.5-pro, and Gemini-2.5-flash. We plan to expand this list based on community feedback and our benchmarking capabilities across supported venues (conference + track). Guided by Benchmarking Results When selecting a model, CSPaper now displays recommended LLMs together with benchmarking results for the selected conference and track. We visualize comparative performance (measured by NMAE) along with standard deviation (STD) values to make an informed model choice. Normalized Mean Absolute Error (NMAE) measures how closely the predicted paper ratings align with ground-truth ratings, normalized to account for each venue’s rating scale. Lower NMAE values indicate better accuracy. [image: 1762549313265-screenshot-2025-11-07-at-22.01.16.png] Note: Benchmark results are venue-specific and continuously updated as: Our benchmark dataset (currently 150 annotated papers) expands. The review agent’s prompts and templates are refined, affecting LLM performance dynamics. LLMs might update their sub-versions. GenAI Text Analysis (Pilot) As part of our roadmap, we’re piloting a new feature — GenAI Content Analysis — for selected venues: TheWebConf 2025, KDD 2025, and CVPR 2025. Each review may now include a section titled “GenAI Content Analysis”, offering a qualitative assessment of AI-assisted writing likelihood: None / Minimal Partial / Moderate Extensive / Intensive If “Partial / Moderate” or “Extensive / Intensive” is detected, the agent will provide concise justifications with direct evidence, referencing specific sections, pages, paragraphs, or sentences. [image: 1762548511673-screenshot-2025-11-07-at-21.47.49.png] A Glimpse Into the Future We have fully refactored our agent architecture to enable the next generation of review intelligence: High-fidelity score calibration — ensuring review text and scores are more coherently aligned. Cross-review ranking — compare how your reviews rank among all generated reviews for the same venue. Custom review agents — create your own venue-specific agents with tailored review logic. Thank you for supporting CSPaper.org and contributing to our journey of building transparent, reliable, and intelligent academic review systems. — The CSPaper.org Team

In a recent LinkedIn post, Eric Xing — President of MBZUAI, Professor at Carnegie Mellon University, and Chief Scientist at GenBio — raised a pressing concern regarding the evolving peer review expectations in major AI and computer vision conferences. His post centers on a new clause in the CVPR 2026 Author Enrollment Form, which mandates that all qualified authors must commit to serve as reviewers, warning that failure to do so could affect the review process of their own submissions. This move, presumably designed to manage the explosive growth of submissions to top-tier conferences like CVPR, has stirred significant debate. Xing describes the approach as “a new tactic of imposing demands,” arguing that it risks excluding legitimate contributors who, for valid reasons, cannot serve as reviewers. These include senior academics and leaders who already shoulder immense administrative and mentoring duties, non-CV collaborators such as domain experts or data providers, and industry sponsors who meaningfully contribute but lack technical reviewing expertise. The broader implication of Xing’s post extends beyond CVPR. It highlights the sustainability crisis in the academic peer review system — a system strained by exponential submission growth, limited qualified reviewers, and increasing expectations of review quality. While the intention behind mandatory reviewing may be to distribute the workload more evenly, it inadvertently blurs the line between authorship and reviewer obligation, potentially discouraging interdisciplinary collaboration and senior involvement. Xing calls for a more thoughtful, long-term solution that ensures fairness, inclusivity, and sustainability in scientific publishing. Rather than relying on coercive mechanisms, the community might consider systemic innovations — from better incentive structures to leveraging technology that supports reviewer efficiency. One emerging direction involves AI-assisted review systems, such as CSPaper.org, which aim to streamline and partially automate aspects of peer review. Platforms like these could help alleviate the mounting pressure caused by the booming volume of submissions — a challenge that will only intensify as AI research continues its exponential trajectory.

TL;DR. In 2014, a so-called “international journal” accepted a ten-page manuscript whose entire content (text, figure captions, and even a scatter plot label) repeated seven words: “Get me off your fucking mailing list.” The authors declined to pay the $150 APC, so it never appeared online, yet the episode has aged into a compact case study in predatory publishing, reviewer automation, and the brittle parts of our peer-review culture. I re-read the paper, looked at the provenance, skimmed how folks on Hacker News processed it, and ran it through a modern desk-reject workflow (cspaper.org). Verdict: even the most bare-bones LLM-assisted triage would have outperformed the “review” that led to its acceptance — by a mile. [image: 1762295113107-screenshot-2025-11-04-at-23.22.41-resized.png] 1) The stunt, in one paragraph Back in 2005, David Mazières and Eddie Kohler composed a “paper” to reply to spammy conference solicitations: ten pages where the title, abstract, sections, and references are permutations of the exact same sentence. There’s even a flowchart on page 3 and a plot on page 10 whose only labels are, again, those seven words—weaponized minimalism at its funniest. In 2014, after receiving predatory solicitations, Australian computer scientist Peter Vamplew forwarded the PDF to the International Journal of Advanced Computer Technology (IJACT). The journal returned with glowing reviews and an acceptance, pending a $150 fee — the authors didn’t pay, so it wasn’t published. The paper remains a perfect negative-control sample for any review process. 2) What this reveals about predatory publishing (and why CS got caught in the blast radius) Asymmetric effort: Predatory venues invert the cost structure. Authors invest negligible effort (copy/paste seven words); the venue still “accepts,” betting on APC revenue. Reviewer theater: The “excellent” rating shows how some venues simulate peer review (checklists, auto-responses) without any reading. Brand camouflage: Grandiose titles (“International… Advanced… Technology”) plus generic scopes attract out-of-domain submissions and inflate perceived legitimacy. Spam-to-submission funnel: Mass email blasts are their growth engine; the stunt targeted that vector precisely—and exposed it. Archival pollution risk: Had the fee been paid, the paper would have entered the grey literature of indexes most readers mistake for the scientific record. 3) How a modern desk-reject should defeat a seven-word paper I ran the same PDF through a desk-rejection rubric (as used by cspaper.org). The screenshot (attached) shows: [image: 1762295095026-screenshot-2025-11-04-at-23.11.57.png] Even a lightweight LLM-assisted triage could flag: Lexical degeneracy: ≥95% repeated n-grams across the entire body. Structural vacuum: Missing standard rhetorical slots (problem, related work, method, experiments). Citation incoherence: References are RFCs that don’t support any claim. Figure semantics: Captions/axes contain no domain entities; the plot is visually present yet semantically null. If a “journal” can’t clear that bar, something is profoundly wrong with its editorial gatekeeping. 4) Why this episode won’t die (and why it still matters for CS) It’s a perfect meme with receipts. Anyone can open the PDF, skim ten seconds, and feel the absurdity. Email is the root cause. The HN discussion immediately veered into aliasing/relay defenses (“Hide My Email,” Firefox Relay, SimpleLogin, Postfix +addressing). That’s instructive: the paper is about spam as much as peer review. CS is the canary. Our field’s velocity and conference-first culture create pressure to publish-fast — terrain where predators thrive. Automation cuts both ways. Predatory venues automated acceptance; serious venues can (and should) automate rejection of nonsense while keeping humans for judgement calls. 5) Practical takeaways for authors, reviewers, and PC chairs For authors (especially students): Venue due diligence: Check editorial board credibility, indexing, APC policies, and transparency of the review process. COPE & ISSN sanity checks: A COPE logo is not membership; verify the ISSN actually exists in the ISSN portal. Watch the spam funnel: Unsolicited invites + fast acceptance + low APC = red flags. For reviewers: Refuse review for suspicious venues. Your time legitimizes their theater. Encourage institutional training: Teach how to spot predatory features in grad seminars/onboarding. For program chairs/editors: Automate triage, not judgement. Degeneracy/boilerplate detectors and structure-aware checks. Bib sanity (out-of-scope references, self-citation barns). Figure/caption semantic mismatch checks. Show your work: Publish desk-reject reasons (anonymized) to build trust and calibrate the community. 6) A 60-second “stupidity firewall” (a suggested checklist) Scope match? One sentence explains relevance to the call. Core slots present? Problem, method, evidence, limitations. Text originality? No high-degeneracy copy, no nonsense word salad. Figure semantics? Captions and axes reference real entities/units. Reference fit? Citations actually support claims, not just exist. Author intent? No spammy metadata, no mass-submission artifacts. This kind of rubric is where LLMs shine as assistants—labeling obvious failures and freeing humans to read the borderline cases carefully. 7) Why this still makes me laugh (and wince) The paper is satire that doubles as a unit test for editorial integrity. Any venue that “passes” it has failed the most basic invariant of peer review: someone must read the paper. On the bright side, our tools and norms are better today. The cspaper desk-reject outcome shows that even a simple, transparent rubric—augmented by LLM checks—can protect serious tracks from time-wasters and protect authors from predatory traps.

Impl. based on nr0034je9.zip . Table A: Model Performance on NLP Benchmarks Model SST-2 (Acc) MNLI (Acc) QNLI (Acc) CoLA (Matthews) Avg Score BERT-Base 91.2 84.6 90.1 58.2 81.0 RoBERTa-Base 92.3 87.4 91.8 63.1 83.7 GPT-3 (175B) 94.1 88.9 93.0 66.4 85.6 Our Method 94.8 89.7 93.5 68.9 86.7 Table B: Ablation Study on Model Components (Evaluated on MNLI) Configuration Attention Mechanism Pretraining Corpus MNLI (Acc) Full Model Multi-head Self-Attn Custom + Public 89.7 – w/o Custom Corpus Multi-head Self-Attn Public Only 87.1 – w/o Attention Refinement Block Basic Self-Attn Custom + Public 86.5 – w/o Positional Embeddings Multi-head Self-Attn Custom + Public 85.2 – Random Initialization — — 72.4