Open benchmarks

The detector industry advertises single accuracy figures (99%, 99.9%, "1 in 10,000 false positives") that almost never reproduce on independent test sets. We publish three orthogonal benchmarks instead. They measure different things and fail in different ways, which is the point: together they show where the Slop Score holds and where it breaks.

1 · Calibration anchors (CI-pinned)

The hard gate. Six reference texts pinned in our test suite. Any engine change that pushes one out of band fails the build and can't merge. The published anchor texts and bands are on the calibration page.

2 · Zhang & Gosline 4-paradigm experimental anchors

A separate, looser set inspired by Zhang & Gosline 2023 (the MIT/Berkeley paper) that tested four content-generation paradigms in 1,203-participant blind ratings of persuasive copy. We use those paradigms as a stress test for the messy middle of our score, where AI and human contributions mix. Validated 2026-05-28: the falsifiable claim that our score produces a monotonic ordering across the four Z&G paradigms holds. Bands below are observed values ±10. Not in the strict gate; revalidated when the engines change.

Read it as a sanity check on the architectural claim. If the four paradigms stop ordering monotonically (human_only < augmented_human < augmented_ai < ai_only), the engine change broke something and the build shouldn't ship.

3 · Paraphrase resilience

A well-documented failure mode in classifier-based AI detection is paraphrase: Krishna et al. 2023 dropped DetectGPT's true-positive rate from 70.3% to 4.6% with one DIPPER round trip; GPTZero's own published data shows its accuracy halving on text rewritten through a paraphraser. The theoretical context is in the limits of AI detection. We run the same kind of exercise on our own high-slop anchors with two prompts:

• Surface paraphrase. Rephrase but preserve register, tone, and voicelessness. This is the analogue of QuillBot / DIPPER: a stylistic-only rewrite that should NOT fool a register-based detector.
• Substantive rewrite. Actively improve voice (vary rhythm, drop tricolons, add named examples). This is what our own rewriter does and should drop the score by 25+ points.

Across three high-slop anchors, surface paraphrase shifts the score by −17 points on average (1/3 stayed in their original band), while substantive rewrite shifts it by −33 points (0/3 stayed in band). The surface delta is roughly half the substantive delta. The score isn't paraphrase-proof. It is roughly twice as resilient as a score that gets fooled by a stylistic rewrite, which is the bar most detectors fail.

What we don't measure yet

The known gaps. If you spot another, the waitlist email reaches us.

• True cross-model robustness. Our calibration anchors use output from Claude models; we don't have a benchmark that runs GPT-4o / Gemini / Llama outputs through the same engine and confirms they all land in the AI band. Architectural reasoning says the score is register-based and should generalise (the deterministic engines are pattern-based rather than perplexity-based, so a single-model bias would be surprising). We still owe you the measurement.
• Languages beyond English, German, and Spanish. Those three have full lexical dictionaries, structural regex banks, and Sonnet judge prompts; the score is meaningful and the highlights point at the right spans. Other languages run through the engines but the numbers are noise. Adding a new locale takes the same authoring pipeline. See known limitations.
• Iterated paraphrase chain. One round of paraphrase is the test above. We don't know how the score behaves after 3–5 rounds of successive rewriting, which is closer to how a determined user would actually try to evade detection.
• Short-text floor. Below ~150 words the deterministic engines don't have enough sample for their per-500-word normalisations to be stable. We have anecdotal evidence the score is unreliable below the floor; we don't have a published threshold.
• L2-English false-positive rate. The Stanford 2023 finding on non-native English speakers (61% false-positive on competitor detectors) is why we publish per-engine sub-scores and added the L2-register interpretation banner. We haven't run an L2 corpus through our engine to measure our own FPR. We should.

How to reproduce

The benchmarks are open and re-runnable in the GitHub repo. Commands are in the README; if a number on this page doesn't reproduce, that's a bug, tell us.