Every Wikipedia edit, judged live by AI

Will it revert? streams every edit to English Wikipedia the moment it happens. About half a second later, Jev, a decision model from TypeSafe, has judged it: is it damaging, was it made in good faith, and will another editor undo it within the hour? Then Wikipedia's own editors decide, in the same live stream, whether Jev was right.

What you are looking at

Each row is one real edit to English Wikipedia, usually a few seconds old. The row appears as soon as Wikimedia publishes the edit. A moment later Jev's verdict lands on the same row:

• Intent, the most likely reason for the edit out of ten, drawn as a bar of all ten probabilities.
• Damaging, the probability that an experienced patroller would want the edit reverted.
• Good faith and revert probabilities, a four-level severity, a topic, and a warning when the edit adds a risky claim about a living person.

The scatter plot on the right places each edit by damage and good faith. Clumsy but sincere newcomers land in one corner and vandals in the other. Edits Jev rates at least 90% damaging go on the Watching list for 15 minutes. If a human editor or an anti-vandalism bot reverts one, it moves to Caught. The Calibration tab shows whether Jev's probabilities mean what they say.

The seven questions Jev answers for every edit

Jev does not write text. It takes a description of the situation, here the page, the edit summary, the editor and their track record (edit count, account age, user rights) and the actual diff with the changed words marked, plus a set of typed questions, and returns a probability for every possible answer. All seven questions go out in one call:

1. Damaging (yes/no): would an experienced Wikipedia patroller want this edit reverted, regardless of motive?
2. Good faith (yes/no): is the editor trying to help, even if the result is clumsy?
3. Will be reverted (yes/no): will another editor revert this within the next hour?
4. BLP risk (yes/no): does it add negative, contentious or unsourced claims about a living person?
5. Intent (one of ten): vandalism, test edit, POV push, unsourced claim, spam link, copyedit, content added, content removed, revert, or maintenance.
6. Severity (ordered score): none, low, medium or high damage if the edit stays.
7. Topic (one of eight): politics, sports, science, entertainment, biography, geography, history or other.

Because every answer comes from a fixed list, the model cannot return anything outside it. That is what makes it usable as a live filter: the output can be compared, thresholded and charted without parsing.

How the ground truth works

Wikipedia is one of the few places where the right answer shows up on its own, fast and in public. Patrollers and bots such as ClueBot NG revert bad edits within minutes, and those reverts arrive in the same recent-changes stream as the edits themselves. The live view matches each revert to the edit it undid, by revision id when the edit summary names one and otherwise by article and editor, so a catch shows up within seconds. Every flagged edit that gets reverted is a catch you can click through and check on Wikipedia.

Live matching misses reverts that come later or say nothing in their summary, so the numbers use Wikipedia's own label instead: MediaWiki tags a revision mw-reverted when a later edit undoes it, by any method and at any time. The site reads that tag back one hour and one day after every edit.

How good is it?

Backtest on 3,989 English Wikipedia edits from 22 and 23 September, labelled by the mw-reverted tag (4.7% were reverted), with Wikimedia's own production model revert risk scored on the same edits:

• Of the 1% of edits Jev rates most likely to be reverted, 74% were. For revert risk it was 38%.
• At a precision of one in two, Jev finds 25% of all reverted edits; revert risk finds 4%.
• Across all edits the two rank about equally well (ROC AUC 0.78 for Jev, 0.77 for revert risk). Reverts of plausible-looking edits, such as a disputed number or an unsourced claim, are hard for both.

Jev's raw probabilities run high in the middle of the range: edits it gave a raw 15% were reverted about 2% of the time, while its answers above 80% held up. The revert chance on each row is Jev's answer mapped through a calibration curve fitted on 5,583 labelled edits. The Calibration tab shows that curve on held-out folds, then switches to live edits once a day of labels has built up.

Until 23 September the site had a bug: the diff never reached Jev, so it judged edits from the title and summary alone. Verdicts are versioned, and the numbers above are for the fixed setup.

What else you can build with Jev

Wikipedia is a showcase because it is public, fast and scored by humans in real time. The same pattern fits any decision where the possible answers are known in advance, a careful person could decide quickly, and the volume or speed makes a large language model the wrong tool:

• Content moderation and trust and safety: triage comments, reviews, listings or user edits, and send only the uncertain ones to people.
• Guardrails for AI agents: before an agent runs a tool, ask whether the action is read-only, reversible or destructive, and require confirmation for the risky branch.
• Model routing: decide whether a request needs a frontier model or whether a cheap one will do.
• Agent monitoring: is the agent repeating itself, did the tests pass, does the output follow policy.
• Classification at volume: support ticket intent and urgency, document labels, spam, lead quality.
• Retrieval filtering: check whether a retrieved passage actually answers the question before paying for a long LLM call.

The working rule is the same as on this page. Act automatically when confidence is high and the stakes are low, ask a person or a stronger model when it is in the middle, and keep the thresholds in code where you can tune them against your own data.

Jev is available from TypeSafe and on OpenRouter as typesafe/jev-1.13 through its decisions endpoint. A single call looks like this:

POST https://openrouter.ai/api/alpha/decisions
{
  "model": "typesafe/jev-1.13",
  "state": { "title": "…", "summary": "…", "diff": "…" },
  "questions": {
    "damaging": { "type": "noul",
      "instructions": "Would an experienced patroller want this edit reverted?" },
    "intent": { "type": "choice",
      "criteria": { "vandalism": "…", "copyedit": "…", "content_add": "…" } }
  }
}

Open API

Everything on the page is available as read-only JSON, open to any origin. No key needed.

curl -N https://willitrevert.com/events

Questions