Rogue Report – Dear WotC: Open the (Data) Vaults



Being a Magic player means a lot of things, including that you only get one day a week to sleep in: Sunday. Not that I’m complaining – I love getting up early on Saturday, gathering my things, and making my way to the tournament hall. The whole PTQ experience has become something I look forward to, but last Saturday I was a little burned out. I was getting nowhere in the format, so I eventually just gave up and played Five-Color Control. My heart wasn’t in the deck, so it wasn’t in the tournament. I did exactly how I expected, dropping after round five at 3-2. I’m glad I have a break before the next PTQ season to get my head straight. The Brian Baker Memorial Tournament is coming up, and it’s standard, so I’m going to try to play something fun. (See you there!) Maybe it’s time I actually play Time Sieve in a tournament myself.

Meanwhile, I’ve started reading again. I used to read a lot, but lately it’s been an activity relegated to travel. School can really zap the will out of you when it comes to reading for fun, at least when you’re taking classes you don’t like. Luckily, flying to Kansas City, then Seattle, then Montana, then Indianapolis, then back to Seattle in the span of three weeks gave me a lot of time to read. I remembered how much I liked doing it, so now I find myself laying in bed and reading.

First it was Self Made Man. A female journalist dressed as a man for about a year-and-a-half and wrote about her experiences dating, working, and interacting with “other” men. A solid read. Then came Freakonomics, a book I’ve heard a lot about. I enjoyed it, and while I think it’s a great book and I learned from it, I found it largely overrated. Next was Moneyball. Riki Hayashi must be pulling his hair out because he told me to read the book years ago, but after the urging of a friend – “Trust me, it isn’t really about Baseball” – I finally read it. I loved this book and found myself applying a lot of it to Magic. I thought I had found the holy grail of books in Moneyball, until I was handed Outliers. Now THAT’S a book. I couldn’t put it down.

[True story: if not for Moneyball, you would not be reading articles by Jon, Zaiem Beg, and Jeremy Fuentes here on ChannelFireball. -Riki, Beane-ingly]

By telling you the books I’ve been reading over the last few weeks I’m trying to provide some background on what I was doing while I came up with what I’m about to talk about. Ok, so I didn’t actually come up with this idea myself, I first heard it from my roommate Brian Wong. Brian was talking about how awesome it would be to have stats for Magic Online. Imagine, he said, what you could learn by seeing how often the person going first in Shards of Alara Limited actually won. Or how many nonbasic lands successful decks are playing. Or how often a first-pick [card]Tower Gargoyle[/card] led to a win. We think we know these answers, but we have such a limited amount of feedback to draw from, usually only our own games, or articles where people tell us about their own games.

To find real, concrete answers, we need to look at the data. I was thinking about writing this article a few weeks ago, when all of a sudden William Spaniel rose from the dead and stole my topic. (I’ll get you, Spaniel!) I put off the article until the Standard season ended, assuming I’d need a topic, so here we are. Spaniel’s article just scratched the surface; he wrote about the importance of information and how helpful it would be to WotC. I want to dig a little deeper and start asking some questions.

Imagine if WotC had a big computer that only stored data. This computer wouldn’t do any reasoning or make calculations or infer anything based on what data it has. It wouldn’t make charts or write articles. All I’m asking this computer to do is store some data and let me access that data easily. I’m going to take you through what data I want and what kind of questions I would want to ask.

The Games

Zaiem warns me about being results oriented, and in his last article he talked about how damaging it can be. You kept a two-land hand and you lost, thus you should mulligan two-land hands. This is a bad mindset to have because you’re not drawing from actual statistics, you’re just drawing from your own personal experiences – a small sample size. What if, however, you could actually be results oriented? What if you could look at how often two-land hands actually got there compared to mulling to six?

After two players finish their match, I want Magic Online to send the following game data to our database:

This one is easy: which player won the match? However simple this statistic is, when our goal is to win, it’s the most important. This is the statistic that all our other statistics will be compared to. This is the cold heartless number that doesn’t care about how often you hit your fourth land drop, or how many counterspells they drew, or how few creatures you drew. All we care about is whether or not a certain player won a game, and from there we can find the important factors that led to that result.

We also need to see who won each game, not just the match. It’s easy to see how this applies to sideboarding, but I’m also interested to see if it has any other applications. How often does a player come back from losing game one? How often does a player win game one and lose the match? Each game – one, two, and three – is very different, and they must be treated as part of a match and not just dumped into a data bucket full of single games.

We also need to take this a step higher and look at the overall tournament placement. Just as each game is different, each round is different, and the top eight is a beast all its own. You know that guy you go to when you need to know if you can draw into top 8 or not? That new guy is our database. How often does a player lose round one and still make top eight? Is a player’s top eight performance independent of their swiss performance? Once in top eight, does the guy who lost round one only to make it to top eight win the event less often than the guy who drew in at X-0-2?

I could ask the data the win percentage of a person who scoops compared to somebody that dies naturally, and use that to prove my point on scooping.

Whenever I actually think about what turn a game ended on I’m usually surprised, especially in Limited. We hear of fast formats and slow formats, but our database can rank them precisely. Mike Flores (at least I think it was him) talks about critical turns and the different phases of the game. By tracking activity based on turns, we can view this data in tangible figures. The importance of turns is something that’s hard for us to track naturally, and it’s a method of analysis that would break wide open with the introduction of this system.

Magic Online provides a unique opportunity with the clock system to actually track how often a players has priority. I’m not sure what to learn from this data, but I have a feeling it’s useful. When all the data is organized and analyzed, I wouldn’t be surprised to see patters in priority emerge.

This one is easy, right? Just send our database the life totals at the end of each game. That’s bound to hold a lot of useful information on its own, but life is so much more meaningful than that. New players value their life much too highly, we know this, but our database can prove it. It can show us just how valuable life really is. Imagine tracking life point changes by turn. By spell type. By amount. Our database would need to know when a player’s life total changed (down to the step of the turn), how much it changed, and what made it change.

Die Roll & The Play
Who won the die roll? Who won game one? Who won the match? Who won the tournament? Does the winner of the tournament have a higher die-win percentage than average? Winning the die roll doesn’t necessarily mean going first, so we would want that piece of data as well. Choosing to play or draw is usually a topic of conversation for a new format, and judging by the split response in the “play or draw” question asked to the pros for the M10 Sealed format, it’s one we never really have a definite answer to. Data can give us that answer.

Easy: who mulliganed and to what? Also, what was in their hand when they mulliganed? What was in their hand when they kept? Here’s a theory: the winner of a tournament, on average, has a lower mulligan percentage than they normally do. They were “running hot” that day. Maybe they had the aura. Data could prove this theory wrong or right, but without it, it’s really hard to know. Data might tell us that this theory is true, but that the winner of a tournament, on average, mulligans more than the average person, meaning that we don’t mulligan enough.

How many cards were sideboarded? Which cards were put in, and which ones came out? I think this would be very interesting when it comes to Limited. It’s safe to assume that winning Limited players sideboard more than losing players (though data could tell us for sure) but how many cards are they actually sideboarding, and what are they bringing in? For Constructed you could track exactly how useful Pithing Needle is as a sideboard card, and how often players that have it win compared to those that don’t.

Cards & Plays
Now we hit probably the biggest chunk of information: what cards were used in each game, and how they were used. This is the hardest part of the data collection, but is the biggest window into gameplay. Somebody smart would need to find the most efficient way to relay the actions taken in a game – once we have that, we’re good to go. Once we have the data we can track card advantage, attack phases, land drops, mana used, creatures destroyed, flyers cast the list goes on. You could start to identify the best spells in your deck. How many games does this deck win when it doesn’t draw Bloodbraid Elf compared to when it does? How important is drawing a Volcanic Fallout against Faeries? When is playing the second Bitterblossom correct? How much mana is typically spent each game? There are a millions questions to be asked. This section could be an article all its own.

What’s the best color in standard? What’s the best card in Standard? We could find out.

The Decks

James Lee, a local and highly respected judge, recently mentioned his desire for a deck builder and database. I find it weird that if I want to build a deck I usually open up Magic Workstation. Shouldn’t there be a tool online for doing this? I think this is a service WotC should provide, much like they provide the Gatherer. They’ve already got Magic Online players building decks, and they could all draw from the same database. Magic Online can do for a deck database is incredible: it can tie a lot of decks to a lot of results. Our real life data of decks and results is severely limited, but Magic Online has heaps of it. These are the things I would look for:

Deck Makeup
How many lands do winning decks play? I’ve found it odd that over the last year or two we’ve been playing decks with 25, 26, or 27 lands without batting an eye. I found myself wondering if 26 lands was too few in my Nationals deck, and it wasn’t playing anything that cost over 6. Six years ago, however, when I started playing Magic, most decks ran around 24 lands. Have we changed our deckbuilding theory, or are decks fundamentally different now than they used to be? Do they have a higher curve now? There is a lot of talk about curves in Magic theory, so what makes a good curve? How bad is 61 cards really? We can assume that 60 is better than 61 (and again, data can prove this) but how much are you hurting yourself by running 61 cards? Or 41 in limited?

Deck Type & Metagame
Once we have a database of decklists it will be important to have a tool that can identify those decks and classify them. Once a certain class of decks is identified, like Five-Color Control, Red Aggro, or Faeries, we can start to examine the metagame. Who really wins in Kithkin against Faeries? Once we start tying all our data points together we can ask really interesting questions. Which deck has the highest mulligan percentage? Which deck mulligans the best (wins the most on a mulligan to six)? Apply this to Limited, and we can find out what shard is actually the best, the worst, and by how much.

The Players

Tying decks to results is one thing, but tying the results to a player is just as important. Trends that appear among worse players might be completely different amongst the better players.

Without a better tool to measure a player’s skill, this is what we’ve got. In fact, by checking out the win percentage of players based on their rating, we can find out how much rating actually matters, instead of how much we perceive it to matter (or not matter). Once you take rating into account you can start looking at the data in a more personal way. Take the Faeries vs. Kithkin matchup. How different are the win percentages for each deck when played by low-rated players compared to when they are played by high rated players? Does the rating of your opponent effect whether or not you should mulligan?
Since MODO is anonymous, and WotC has no real way of knowing exactly who is behind the username, it’s important to track what we do know, like when a user created their account. I’m not sure what this could mean, but much like priority, once things are organized a trend might start to appear. How often is a player logged on to their account? What types of events do they play in, and how many? Even what is in their collection could tell us something. Are the better players parts of clans? How much does being in a good clan help?

The Draft

Drafting provides its own unique set of data, and a pretty simple one: what cards were in the packs, and how high were they taken. We can take this information and see what the chances are you’ll get a useful Borderpost in pack three. Where this gets really interesting is when you start to cross this data with all the other pieces we’ve collected. How good is first picking a tri-land like Seaside Citadel? Not only that, but how important is that tri-land to actually winning the draft? Maybe winning players tend to have more tri-lands than losing players, but only because they value them higher, not because they are actually worth it.

Could we devise a formula to give each card a to-the-decimal-point pick number? Once that’s been done, you could actually rate the power level of the whole draft, proving or disproving comments like “my draft pod was weak.” You could even compare the power level of packs opened to a player’s left and see how much of a factor luck in opening is. You could even analyze signals, a topic highly questioned for its relevance. This is one of those points where I think the Magic community would do something surprising, we’ve just got to give them the data.

Lastly, The You

Now we come to why this actually matters for you. If you’ve played any serious online poker, you’re familiar with a stat tracker. (Reading up on Poker was another thing I was doing leading up to the creation of this article.) These track the players you’re gambling against – their raise percentage, fold percentage, and a bunch of numbers I’m not good enough to even be able to describe to you – but they also track your own stats. Being able to look at your own patterns and see that you’re not raising nearly enough, or that you’re folding too often, is paramount in your personal improvement as a player.

I want this as a Magic player. I want to be able to track my mulligans, and see how I compare to the ideal. I want to be able to track the consistency of my manabases. I could track my own results and see what works for me. We have our own notions of what kind of player we are – combo, aggro, control, etc. – but we’re basing our assumptions largely off of our feelings. This thing could tell me which deck will give me the best chance of winning an event, weighing my own personal preferences against the metagame.


I’m not asking for WotC to do any of the heavy lifting of data analysis. All I want is the data, and we’ll do the rest. Seeing how impacting this data would be on me as a player I can only imagine what WotC R&D would do with this information. As a guy that’s interested in game design, being able to see exactly what goes in to winning a game of Magic would be incredible. I have no doubts that the quality of Magic design and theory would go up immensely. That’s why I’m calling this the most important article I’ve ever written.

Thanks for reading,
Jonathon Loucks
Loucksj @ gmail.com
JonLoucks on twitter

P.S. I’m going to be at PAX this weekend! I’m demoing Arcane Legions again, so find me at the Wells Expeditions booth and say hi.

Scroll to Top