Every benchmark stops working. So write a new one.
Labs train on the tests everyone watches. Here anyone writes a fresh vibe test, today's models answer it side by side, and the community votes up the tests that still separate them.
Goodhart's law: When a measure becomes a target, it ceases to be a good measure.
How we try to stay ahead of it
Write a test
A prompt, an output type and a note on what a good answer looks like. It goes live at once, and at 3 upvotes it runs free on every model in the roster.
Judge blind
Two answers to the same test, no names. Pick the better one, then see who made it. Ratings come only from blind judgements.
Watch the gap
Every model has a training cutoff. Tests written after it are unseen. A model that rates far higher on old tests than unseen ones is showing you its training data.
Leaderboard
Overall, next to unseen tests only. The gap column is the point.
| # | Model | Rating | Unseen | |||
|---|---|---|---|---|---|---|
No judgements yetRatings appear after the first blind votes in the arena. | ||||||