Pricing
Free to write and judge. Pay only to run.
Upvoted tests run on the roster for free. To run any test on any model right now, buy credits: provider cost plus 30%, quoted to the cent before anything runs.
Free
$0
For everyone.
- Write public tests
- Vote and judge blind
- Free roster run at 3 upvotes
- Leaderboards, charts, sharing
Pro
Most useful$12 / month
For people who test models for a living.
- $0.00 of run credit every month
- Private tests, visible only to you
- Sealed tests: prompt hidden until a reveal date
- 25 tests a day instead of 5
- JSON and CSV export
- Charts of how a test aged across releases
Cancel any time; it runs to the end of the month you paid for.
Credits
from $10
Pay as you go. Never expire.
1 credit = 1 US cent of runs. Minimum $0.02 per model per run.
What a run costs
Running a pelican riding a bicycle on all 11 roster models costs $0.28 right now: from $0.02 on the cheapest model to $0.05 on the most expensive. Prices move with the providers' list prices, refreshed hourly.
- You see the price first. Pick models and samples, and the run page shows every model's price and the total before you confirm.
- Never more than quoted. If a run costs us less than expected, you pay less. If it costs more, we eat the difference.
- Prepaid only. Runs draw from your balance; nothing is billed afterwards and a balance never goes negative.
- No surprise refunds or cancellations. We only refund or cancel when you ask. Write to us any time.
Running evaluations for a lab? Sealed holdout sets are licensed separately. For labs