Five axes, one score, a blind human vote at its centre, all measured against writing that provably predates ChatGPT. No LLM grading another LLM. Here is exactly how the number is built.
Everyone can feel AI slop when they read it: the throat-clearing opener, the tidy rule-of-three, the "it's not just X, it's Y," the closing offer to help further. But feeling it isn't measuring it. The Slop Index turns that instinct into a reproducible number by asking a single question, does this read like a person, or like a chatbot?, and answering it two independent ways.
Every model gets a Human Score from 0 to 100, where 100 means it writes like a human and 0 means maximally like AI. The score is a weighted blend of five axes. The biggest single input is other humans.
We put the arena vote at 40% deliberately. Mechanical axes catch the tells you can count; people catch the ones you can't. Weighting them together means a model can't win by gaming one rule, it has to actually read as human to both a measurement and a crowd.
A live Elo from the arena, where people flag the sloppier of two blind samples one pair at a time. The least-flagged model rises. This is the only axis humans control directly, and it is the largest.
Does the model resist length inflation? We compare each response's length to what a human wrote for the same task. Padding a two-line reply into six paragraphs is one of the most reliable tells.
Reuse of openers and skeletons across unrelated prompts. Humans vary how they start; models fall back to the same scaffolds, "I hope this finds you well," "Here are a few thoughts," the same three-beat structure.
Natural variance in paragraph and sentence length. Human writing is lumpy, a one-word line next to a long one. AI writing tends toward an even, metronomic cadence that reads smooth and lifeless.
Over-used AI vocabulary and constructions measured against their pre-AI base rate: "delve," "tapestry," "it's not just X — it's Y," the reflexive em-dash, the closing "let me know if you'd like…"
A slop score is only meaningful relative to something. Ours is genuine human writing collected before generative AI existed, everything predates October 2022, so no model could have trained on it, shaped it, or been tuned to imitate it. Each domain has its own human baseline:
| Domain | Human baseline | Vintage |
|---|---|---|
| The Enron email corpus, real workplace mail | 2000–2002 | |
| Essays | Blog-authorship corpus + student essay sets | pre-2012 |
| Social | Archived public tweets | pre-2022 |
| Chat | Public Discord / IM conversation logs | pre-2022 |
For each domain we build the distribution of human behaviour, how long replies run, how starts vary, how cadence moves, and measure how far each model drifts from it. Drift toward the human distribution scores high; drift toward the AI extreme scores low.
Every model faced the identical 112 scenarios across the four domains, five samples each, for
19,928 real, unedited generations in total. Each scenario is a hand-written brief with a persona, a recipient
and the prior context of the exchange, so the model writes to a real situation (a specific email to answer, a launch
post to draft, an essay prompt) instead of a bare instruction. Models ran at their default settings (reasoning effort
disclosed per model), and we never picked the best of a batch. What the model produced is what got scored, and every
scenario, prompt and generation is published in the repo (runs/full-merged/outputs.jsonl).
The four mechanical axes are computed per-sample and averaged; each is then min-max normalised across the 18 models so the cleanest becomes 100 and the sloppiest 0 on that axis. This is what gives every axis real spread instead of a cluster of near-identical numbers, you can see exactly where a model wins or loses.
The arena shows two responses to the same prompt, from two different models, with the labels hidden. You flag the one that feels more like AI slop (or mark both). Each vote is a pairwise comparison, and we turn the stream of comparisons into an Elo rating with draws (a standard Bradley-Terry update, K=32): the model you left alone gains, the one you flagged loses, and marking both a draw converges the two toward each other. The least-flagged model floats to the top, and that ranking is fed back in as the 40% human axis in real time.
Votes are defended without getting in your way: a proof-of-human check, a minimum reading dwell so reflex-clicks don't count, a per-visitor daily cap, and a raw append-only log for auditing. The goal is a signal that reflects people reading carefully, not scripts.
Benchmarks earn distrust when they saturate, when they can be gamed, and when a synthetic score has nothing to do with real work. We tried to build against all three, and we are upfront about the limits:
Where the two lenses agree, the mechanical index and the human vote landing on the same model, the signal is strong. Where they diverge is often the most interesting thing on the board, and it is all there to inspect.
Everything is public: the harness, the 112 scenarios, the scoring code, and all 19,928 raw generations. The repo
ships an agent playbook (AGENTS.md) that walks an agent, or you, through sourcing the datasets and
reproducing every number end to end.
# clone and reproduce the board
git clone https://github.com/hgaddipati1118/slop-index
cd slop-index
pip install -r requirements.txt
python harness/score.py runs/full-merged/outputs.jsonl
Browse the code & data on GitHub →
The method is the boring part. The rankings are where it gets interesting, including the most expensive model that writes some of the sloppiest prose.
See the rankings → Play the arena →