How we measure SpellRack's accuracy

What we measure

Four metrics. Bracket detection is live and reported on /accuracy today. The other three are filed for Q2 2026 — listed here for full transparency, not yet on the headline page.

How we benchmark

• Bracket detection. Every WotC-published precon (after a 30-day post-release cutoff to let the card index settle) is scored by our bracket engine and compared to the WotC ground truth: every official precon is Bracket 2 / Core. Anything else counts as a miss.
• Combo coverage (Q2 2026). Cross-reference detected combos against the Commander Spellbook combo database. We'll publish recall and precision side by side.
• Mulligan keep accuracy (Q2 2026). 500 synthetic 7-card hands, hand-labeled keep/mulligan by experienced Commander players. We report agreement rate.
• Manabase quality (Q2 2026). For each deck, count colored sources per pip and compare to Frank Karsten's formula. We report the share of decks within ±1 source per pip.

Reproducibility

The bracket benchmark is a single script: scripts/benchmark-bracket-accuracy.ts. It loads every settled precon from the database, scores each with the same estimateBracketAsync function used by the live deck pages, and writes a JSON report to data/audits/bracket-accuracy-YYYY-MM-DD.json.

The SpellRack repo is currently private — once we open-source it (Q4 2026 plan), the script link will go live here. In the meantime we publish the latest report contents on /accuracy itself, and the JSON file is the same shape any third party would produce.

Update cadence

The headline number on /accuracy is cached for 24 hours and regenerated on the next page request after the cache expires. A weekly auto-rerun on a cron worker is filed for Q2 2026 — until then the benchmark also runs inline if the cache is missing, so the live page is never stale by more than a day.

Scoring rules

• Exact match. A deck WotC ships as Bracket 2 must score Bracket 2. We do notgrade ourselves on ±1 bracket; that's not a useful answer to a player asking “what bracket is this deck?”.
• No partial credit. Anything other than the expected bracket = miss. We log the miss distribution per set (where we predicted instead) so we can spot systematic drift.
• Skip rule. A precon with an incomplete decklist (under 99 cards in the loaded list) is skipped, not counted as a miss. The /accuracy page shows skip count next to the headline.
• 30-day settle window. We exclude precons released in the last 30 days because card data (oracle text, Game Changers list inclusion) can still be shifting. WotC has historically tweaked the Game Changers list 1-2 times per year; the settle window prevents short-term churn from polluting the headline number.

Comparison to competitors

Every published accuracy claim we're aware of in the Commander deckbuilding space:

If you publish an accuracy benchmark for a competing tool and want to be listed here, email [email protected].

Limitations

• Auto-built decks lack user-intent signals. A user can build a Bracket 4 deck that looks like Bracket 3 on paper if they avoid Game Changers but still play turbo combos. The benchmark grades against precon ground truth where intent IS known; live user decks don't expose intent.
• Bracket 1 (Exhibition) requires explicit user tagging. Exhibition is a flavor / rule-zero tier — we cannot detect it from cards alone, so users must opt in. Auto-scoring never assigns Bracket 1.
• Precon-aware softening. A few WotC precons include a Game Changer or two for flavor (the face commander's flagship card) but are still ground-truth Bracket 2. Our Q2 2026 roadmap adds a precon-aware softening rule so face-card flavor inclusions don't bump the bracket. Until then those decks are honest misses.
• Game Changers list churn. WotC updates the list 1-2× / year. We rebuild and re-run the benchmark when the list changes; the 30-day settle window mitigates short-term shifts but cannot eliminate them entirely.