When Crowds Disagree: Using AI to Build Fact-Checking Consensus
New research shows how generative AI can help achieve consensus across diverse fact-check viewpoints, offering a blueprint for more reliable AI systems.
A research paper published by Proceedings of the ACM Web Conference 2025 (WWW ’25), April 28–May 2, 2025, Sydney, NSW, Australia, titled Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking, proposes an innovative AI framework designed to bridge divides in public knowledge. The authors — Soham De and Martin Saveski (University of Washington), Jay Baxter (X’s Community Notes), and Michiel A. Bakker (MIT and Google DeepMind) — introduce “Supernotes,” AI-generated fact-checks that synthesize crowd-written contributions to achieve consensus, clarity, and credibility.
Crowdsourced Fact-Checking: A Modern Strategy with a Familiar Challenge
Crowdsourced fact-checking empowers online communities to collectively assess and clarify questionable or ambiguous claims. Platforms like Community Notes on X (formerly Birdwatch) enable users to propose explanatory notes on posts and rate the helpfulness of others' contributions.
The strength of this system lies in its diversity — anyone can participate. Yet this is also its central limitation: for 91% of posts with proposed notes, none get displayed. The issue? Even helpful notes often fail to gain wide support. Many are fragmented, biased, or don’t resonate across differing perspectives. Achieving consensus is the bottleneck.
The Supernote Framework: Synthesis, Scoring, and Selection
To address this, the paper’s authors built a framework that uses AI to create consensus-friendly notes — called Supernotes — by synthesizing multiple human-written contributions and rigorously filtering them.
The process unfolds in two stages:
Candidate Generation: A large language model (LLM) generates many Supernote drafts by combining content from existing community notes and structuring them around clarity, balance, and relevance.
Candidate Scoring: A second model, trained on millions of prior Community Notes ratings, simulates how a diverse pool of users (a “synthetic jury”) would respond to each draft. The model evaluates helpfulness and filters out any notes that break platform guidelines (e.g., opinionated tone, speculative claims, broken or unverified links).
This second component — the Personalized Helpfulness Model — is central to the system’s operations. It predicts how users with different perspectives would rate a note, ensuring that only candidates likely to satisfy a broad spectrum of readers are selected.
Tested and Preferred: Supernotes Win Over Participants
The researchers ran a double-blind study in which 42 participants evaluated Supernotes alongside the best existing human-written note for 100 posts. The findings were compelling:
75.2% of participants preferred the Supernote when choosing which note was more helpful.
Supernotes outscored human-written notes on clarity, coverage, source quality, neutrality, and contextual depth.
They also received higher overall “helpfulness” scores, including when evaluated using the Community Notes’ own scoring algorithm.
A follow-up experiment confirmed the value of the Candidate Scoring component: participants chose full Supernotes over standalone LLM-generated summaries 61.5% of the time, underscoring the importance of consensus filtering in producing better outputs.
Implications: Trustworthy AI in an Uncertain Information Landscape
While this research focuses on improving public annotations online, its implications reach into broader AI development — particularly the challenge of AI “hallucinations” or confidently incorrect statements generated by language models.
These hallucinations typically occur because LLMs generate language based on probabilities, not verification. The Supernote model hints at a viable solution: rather than relying on a single output, AI systems could generate multiple responses and use a consensus-based scoring framework — much like the synthetic jury — to select the most broadly acceptable and trustworthy one.
This approach could help AI systems deliver more accurate answers in fields like science, medicine, history, or current events, where factual grounding and neutrality are essential.
Practical Tips for Reliable AI Use Today
While this consensus-based model is promising, it’s not widely deployed yet. For anyone using AI tools for research or writing, here are a few techniques to improve information accuracy in the meantime:
Request sources explicitly. Ask the AI to cite references or explain where it got the information.
Cross-check details. Look up names, dates, statistics, or technical claims using trusted databases, libraries, or journalistic sources.
Test multiple phrasings. Rephrase your question in different ways to compare AI outputs and spot inconsistencies.
Use community-reviewed fact resources. Platforms like Wikipedia, Community Notes, and domain-specific repositories often highlight what is broadly agreed upon — and what is still disputed.
The Supernote research suggests a future where AI doesn't just produce information — it curates, harmonizes, and elevates it through the wisdom of the crowd. By blending collective insight with generative precision, we move closer to AI that not only answers questions — but earns trust.