Can frontier vision-language models identify Pokémon from official artwork and silhouettes? WTP-Bench evaluates visual recognition across 1,160 Pokémon — including regional forms, Mega Evolutions, and Gigantamax forms — from exact form matching to the iconic "Who's That Pokémon?" shadow challenge.
Pokémon recognition requires fine-grained visual understanding — distinguishing between 1,160 entries across 8 generations — including regional forms, Mega Evolutions, and Gigantamax variants — that share similar body shapes, color palettes, and evolutionary lineages. WTP-Bench scores models across four evaluation tiers, from exact form identification to the ultimate silhouette challenge.
The model nails it — correct species, form, regional variant, and all. Distinguishing Alolan Vulpix from Kantonian Vulpix or Mega Charizard X from Y. A perfect throw.
The model identifies the correct Pokémon by name but misses the specific form, region, or variant. It said "Vulpix" but didn't specify Alolan — close, but not a critical capture.
The answer lands in the right evolutionary family — guessing Charmeleon when it's Charizard, or Pichu instead of Pikachu. The model sees the family resemblance but picks the wrong stage.
The star of the show. Inspired by the iconic anime segment, models face pure silhouettes with all color stripped away. Only shape and outline remain — the true "Who's That Pokémon?" test.
See how frontier models respond to WTP-Bench queries across both visual modes — hover to pause the scroll.
Accuracy scores across WTP-Bench evaluation tiers — from exact form matching on official artwork to the silhouette-only Shadow Ball challenge.
| # | Model | Master Ball ↓ | Ultra Ball | Great Ball | Shadow Ball (Who's That Pokémon?) |
|---|
Our evaluation across 27 vision-language models reveals systematic patterns in how VLMs handle Pokémon recognition.
The top 7 models are all proprietary — GPT-4.1 leads Master Ball at 79%. The best open-weight model (Qwen3-VL 4B) manages only 13.4%. But relaxing to Great Ball, Gemini 2.5 Pro reaches 91.4%, showing most errors are form/variant confusion rather than total misidentification.
Stripping color devastates performance. GPT-4.1 drops from 79% to 49%, and Gemini 2.5 Pro plummets from 78% to 23%. Only Claude 4.5 Sonnet actually leads on silhouettes at 53%, flipping the HQ leaderboard on its head.
Claude 4.5 Sonnet ranks 4th on HQ images (65%) but takes the crown on silhouettes (53%) — the only model where shadow performance exceeds half. Claude Haiku 4.5 shows near-identical scores on both modes (~27%), suggesting a uniquely robust shape understanding.
Proprietary models average 73.4% on Gen 2 but only 39.2% on Gen 8 — a 34-point gap. Gen 6 is a surprising weak spot at 48.4%, likely due to many visually similar small Pokémon. Open-weight models collapse after Gen 1, falling to under 2% from Gen 3 onward.
When open-weight models guess wrong, 57% of guesses on HQ and 79% on silhouettes are Gen 1 Pokémon — despite Gen 1 being only 19.6% of the dataset. These models have heavily memorized iconic Pokémon like Pikachu and Charizard and default to them when uncertain. Proprietary models show only mild bias (~24%).
Most models are nearly blind to variant forms. Claude 4.5 Sonnet hits 82.8% on base forms but 2.1% on Megas and 0% on Gigantamax. Only GPT-4.1 handles Megas (87.5%) and regional forms (63.6%). 162 Pokémon get zero correct across all 27 models — nearly all are variant forms like Pikachu caps and Deoxys formes.
Among proprietary models, Fire-type Pokémon are identified most accurately at 65.7%, while Electric types trail at 51.0% — a 14.7-point spread. Dragon (51.5%) and Bug (53.0%) types also prove difficult, possibly because these categories contain many visually similar species. The type ranking stays consistent between proprietary and open-weight models.
Some Pokémon are 100% on HQ but 0% as silhouettes — Magmortar's fiery details vanish into a generic blob. Slugma, Galvantula, and Alcremie all drop 85+ points. These Pokémon are recognized by color and texture, not shape — exposing a fundamental reliance on surface-level features.
A few Pokémon are better recognized as silhouettes than as full artwork. Absol jumps from 85.7% to 100% — its distinctive scythe-horn outline is unmistakable. Scizor and Mega Charizard X also gain +14 points, suggesting their silhouettes are more iconic than their colored forms.
WTP-Bench is lightweight and straightforward. Clone the repo, plug in your model, and run evaluation in minutes.
# Clone the repository git clone https://github.com/eternal-f1ame/WTP-Bench.git cd WTP-Bench && pip install -r requirements.txt # Run inference with your model python src/scripts/vlm_infer_template.py \ --input-csv data/metadata.csv \ --output-csv predictions/my_model_hq.csv \ --prompt "Who's that Pokemon?" \ --model-id your-model-id \ --base-dir data # Evaluate predictions PYTHONPATH=src python src/scripts/eval_predictions.py \ --pred-csv predictions/my_model_hq.csv
@misc{baranwal2026wtpbench,
title = {WTP-Bench: Who's That Pok\'{e}mon? A Visual
Recognition Benchmark for Multimodal Models},
author = {Baranwal, Aaditya},
year = {2026},
howpublished = {\url{https://github.com/eternal-f1ame/WTP-Bench}},
note = {University of Central Florida, CRCV}
}