Euchre Reasoning Arena

Most LLM evals score correctness on questions that have a known answer. This one scores calibration under uncertainty. Does the model’s stated confidence match its actual hit rate, and does it know when to reach for a tool?

Origin

This started in August 2025 as a side project: an “Euchre AI Lab” in Ruby + Sinatra + Vue, built because I wanted to play Euchre against LLMs and watch how they handled a partial-information card game. By mid-November 2025 it had a working game engine, multi-provider model support (Anthropic, OpenAI, Ollama), and 130+ RSpec tests. It was fun, I could play against LLMs or just watch four play against each other.

Then Vercel announced the AI Gateway Game Hackathon with a Model Eval category. The “play Euchre against LLMs” substrate was already there; what it needed was the eval angle. I reused the Euchre game engine to build euchre-reasoning-arena in TypeScript + Nuxt + Vercel AI SDK. This project added a calibration layer: confidence ratings on every decision, tools at point cost, Brier scores, post-hand analysis.

The substrate

Euchre is a four-player card game with hidden information. You can’t see your partner’s or opponents’ hands. Coordination has to happen without direct communication. That makes it a useful environment for asking do you know what you don’t know, because the right answer is sometimes “I’m not sure, let me consult.”

How a decision works

Each turn, an agent chooses: pass on calling trump, call a suit, or play a card. The flow is two-phase.

1. The agent sees its hand, game state, prior plays, and any partner-relevant signals. It returns a tentative decision plus a 0 to 100 confidence score and, optionally, a request for one of the lifeline tools.
2. If the agent requested a tool, the platform runs it, hands the result back to the agent, and asks for the final decision (which may differ from the tentative one).

Reasoning streams from the model as it happens (token by token over Server-Sent Events) so a viewer can watch a model talk itself into or out of a play. The whole exchange flows through the same event stream, and the analysis page can replay it after the hand.

Tools

Three lifelines, each with a fixed point cost. Using one means accepting a deduction in exchange for help.

• Situation Lookup (1 point). Returns reference plays for the current decision type. Closest to “look up the convention.”
• Ask Audience (2 points). Polls a simulated audience for collective wisdom on the right move.
• 50/50 (3 points). Eliminates roughly half of the legal-but-wrong options, narrowing the choice.

Score per decision is confidence × correctness − tool cost. A confident-and-right decision scores most; a confident-and-wrong decision scores least; reaching for a tool only pays off when it flips a wrong answer to a right one.

What gets measured

• Calibration. Brier scores over the run, plus reliability diagrams that show where each model is over- or under-confident.
• Tool spend efficiency. Did the tool change the decision? Did it change it toward the correct answer? A model that uses tools rarely but well is showing real metacognition. A model that uses them constantly or never, regardless of difficulty, is showing poor self-knowledge.
• Hand strength. Each hand is scored numerically (trump: Right Bower 12, Left Bower 11, A 10, K 9, …; off-suit: A 5, K 4, …). Round 2 trump decisions get a strength matrix across all four potential trump suits, so it’s visible whether the model picked the strongest suit or left value on the table.
• Strategy adherence. Runs can be configured with different aggressiveness profiles or no guidance at all, which isolates the effect of the prompt versus the model.

The analysis surface

Live: a four-seat table showing each agent’s reasoning stream, declared confidence, and any tool calls, color-coded by team, with a running activity log of plays.

After each hand: an analysis page with a performance scoreboard (per-player accuracy, calibration, tool spend), hand-strength rankings or a matrix depending on the round, a tool-usage panel, and an AI-generated post-hand insight that calls out interesting moments: over-confident misses, recoveries, tool decisions that paid off or didn’t.

Status

Shipped for the Vercel AI Gateway Game Hackathon, Model Eval category. The next iteration is in active planning, building on the existing platform.

What’s next

A few directions in active planning:

Trace-level observability and a golden dataset. Wire Langfuse around the agent loop so every decision becomes a trace span with structured metadata (decision type, hand strength, position features). Curate a stable dataset of interesting positions and replay it against new models or prompt versions, so iteration becomes measurable instead of vibe-driven.

Better confidence elicitation. Recent work (Rescaling Confidence, 2026) shows LLMs cluster 0 to 100 confidence on round numbers, and a coarser 0 to 20 scale produces measurably better metacognitive efficiency. The same paper argues against ECE and Brier as headline metrics because they’re bias-confounded; Signal Detection Theory decomposition (sensitivity vs criterion) is the cleaner read. Existing data already supports the analysis change.

Tool refresh. The current three (Ask Audience, Situation Lookup, 50/50) are creative but uneven. 50/50 doesn’t map cleanly to Euchre’s variable legal-move count, and “audience” is a Who-Wants-to-be-a-Millionaire framing that doesn’t quite fit a card game. Planned replacement covers more metacognition dimensions: Card Counter (memory), Convention Lookup (reference), Ask Partner (theory of mind, the most interesting), Rollout Sampler (forward simulation), Confidence Audit (pure self-reflection).

Causal tool ROI. Capturing both the pre-tool tentative decision and the post-tool final decision gives a cleaner read on whether tools cause better outcomes or merely co-occur with them. The pattern shows up across the cost-aware tool-use literature (CostBench, 2025).

Ground truth for hard positions. A Monte Carlo determinization oracle (sample plausible unseen-card distributions, solve each as perfect information, average expected value per move) gives a defensible “correct answer” for the harder dataset positions. Similar to what bridge bots use. Expensive per-position; only run on the curated set.

Tech

TypeScript, Vite, custom game engine, Vercel AI Gateway for multi-provider routing, ~570 tests at 98% coverage.