The pelican test, and why every famous test eventually stops working
A short history of the most famous informal AI benchmark, the ones that came before it, and the law that retires them all.
A prompt nobody could have trained on
On 25 October 2024 Simon Willison published "Pelicans on a bicycle". He ran one prompt across 16 models from OpenAI, Anthropic, Google Gemini and Meta:
Generate an SVG of a pelican riding a bicycle
His reasoning was about training data. He likes pelicans, and he was fairly sure there were no pelican-on-a-bicycle SVG files online for a model to have memorised: "I'm pretty sure there aren't any pelican on a bicycle SVG files floating around (yet)". The prompt and results live in his pelican-bicycle repository, described as an "LLM benchmark".
It is also simply hard. In his June 2025 write-up he points out that "pelicans can't ride bicycles. They're the wrong shape!", calls it "an unreasonably difficult test for them", and notes that the comments models leave in their SVG show what they were trying to draw.
From joke to yardstick
At the AI Engineer World's Fair in San Francisco in June 2025 he gave a keynote, "The last six months in LLMs, illustrated by pelicans on bicycles", saying the test "started as a joke but is beginning to show itself to actually be a little bit useful!"
For that talk he ran a tournament: 34 pelican SVGs, paired every possible way for 560 matches, each judged from screenshots by gpt-4.1-mini with the prompt "Pick the best illustration of a pelican riding a bicycle." The whole run cost about 18 cents. Elo ratings started at 1500; gemini-2.5-pro-preview-05-06 finished first at 1800.4 and llama-3.3-70b-instruct last at 1196.2. That pairwise idea, scaled to many tests and human judges, is what our arena does.
His pelican-riding-a-bicycle tag now has well over a hundred posts, because he runs it on nearly every major model release.
Then the labs noticed
In the same June 2025 talk he described a pelican on a bicycle appearing briefly in the Google I/O keynote in May 2025: "a blink and you'll miss it moment! ... They're on to me." His 20 May 2025 post runs the prompt on Gemini 2.5 Flash "as seen in the Google I/O keynote". He had already said the test "should stay useful for a long time... provided none of the big AI labs catch on."
That is the moment every famous test reaches. On 13 November 2025 he asked directly, "What happens if AI labs train for pelicans riding bicycles?" His answer was that they would get caught: he would test a strong new model on "all manner of creatures riding all sorts of transportation devices." The same post quotes OpenAI's Aidan McLaughlin denying it: "we do not hill climb on svg art".
In July 2026 he covered a study by Dylan Castillo that checked properly, in "Are AI labs pelicanmaxxing?". Castillo's method: 8 animals times 6 vehicles, each prompt run 3 times through 7 models. The finding: no evidence of pelicanmaxxing, "Pelicans aren't drawn any better than other animals." GLM-5.2 showed the largest boost, but "the effect is small and not significant."
So, as far as anyone has measured, the pelican still works. But notice what the defence is: variations. A walrus on a unicycle. An octopus at a piano. New tests, written after the fact.
The tests that came before
How many r's in strawberry
An August 2024 post describes a viral Mastodon post in which ChatGPT insisted "strawberry" has two r's, and reproduces it on GPT-4o. TechCrunch reported GPT-4o and Claude both answering "twice", and attributed it to tokenisation: models see chunks of words, not letters. TechCrunch also noted the irony of reports that OpenAI had a project codenamed Strawberry; Wikipedia records that o1 was known inside OpenAI as "Strawberry" and that o1-preview shipped on 12 September 2024. Today's models answer it, which is why our copy of it is in the graveyard.
Is 9.11 bigger than 9.9?
In July 2024 the South China Morning Post reported that this comparison went viral after posts by Bill Yuchen Lin and Riley Goodside, with GPT-4o, Claude 3.5 Sonnet and Mistral AI models claiming 9.11 is larger.
Alice's brothers and sisters
A June 2024 paper, "Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models" (Nezhurina, Cipolina-Kun, Cherti and Jitsev), asked: "Alice has N brothers and she also has M sisters. How many sisters does Alice's brother have?" The answer is M+1. The authors reported a dramatic breakdown in GPT-4 and Claude 3 Opus, with big swings when only the numbers changed.
Why they all stop working
In 1975 the economist Charles Goodhart wrote that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." The version everyone quotes, "When a measure becomes a target, it ceases to be a good measure", is the anthropologist Marilyn Strathern's 1997 phrasing, which she credited to Keith Hoskin (all per Wikipedia's account).
A famous test is a measure that has become a target. Once it is quoted in launch posts and screenshots, it is in the training data, or in the product team's checklist, or both. It may still be measuring something real; you can no longer be sure. Researchers have a partial defence: BIG-bench put a canary string in every task file, "to prevent benchmark tasks from leaking into web-scraped training data", so a lab can filter it out. We do the same on every test page and export.
The only complete defence is the one Willison reached for: keep writing new tests. That is what this site is for. Every test records when it was written, every model records its training cutoff, and the unseen leaderboard counts only the tests a model could not have seen.
Our own pelicans
Every roster model on the original prompt, run by us. Vote on the test, judge the drawings blind, or run it on a model we skipped.
A walrus balancing on a unicycle, written after every roster cutoff. Write the next one.
The best defence against Goodhart's law is a test nobody has seen.