Fresh tests for new models

Every benchmark stops working. So write a new one.

Labs train on the tests everyone watches. Here anyone writes a fresh vibe test, today's models answer it side by side, and the community votes up the tests that still separate them.

Goodhart's law: When a measure becomes a target, it ceases to be a good measure. How we try to stay ahead of it

0live tests
0models in the roster
0blind judgements
0tests retired

The freshest tests

Ranked by votes, with age decay.

All tests

Write a test

A prompt, an output type and a note on what a good answer looks like. It goes live at once, and at 3 upvotes it runs free on every model in the roster.

Judge blind

Two answers to the same test, no names. Pick the better one, then see who made it. Ratings come only from blind judgements.

Watch the gap

Every model has a training cutoff. Tests written after it are unseen. A model that rates far higher on old tests than unseen ones is showing you its training data.

Leaderboard

Overall, next to unseen tests only. The gap column is the point.

Full leaderboard
#ModelRatingJudgementsWin rateUnseenGap

No judgements yet

Ratings appear after the first blind votes in the arena.

Email me when a new model is tested

A short weekly note: new models, and the fresh tests that separated them. Unsubscribe in one click.

For labs. Sealed holdout sets that have never been published, licensed for evaluation. Talk to us

Tell us what broke, or what is missing