Rogue Report – Dear WotC: Open the (Data) Vaults



Being a Magic player means a lot of things, including that you only get one day a week to sleep in: Sunday. Not that I’m complaining – I love getting up early on Saturday, gathering my things, and making my way to the tournament hall. The whole PTQ experience has become something I look forward to, but last Saturday I was a little burned out. I was getting nowhere in the format, so I eventually just gave up and played Five-Color Control. My heart wasn’t in the deck, so it wasn’t in the tournament. I did exactly how I expected, dropping after round five at 3-2. I’m glad I have a break before the next PTQ season to get my head straight. The Brian Baker Memorial Tournament is coming up, and it’s standard, so I’m going to try to play something fun. (See you there!) Maybe it’s time I actually play Time Sieve in a tournament myself.

Meanwhile, I’ve started reading again. I used to read a lot, but lately it’s been an activity relegated to travel. School can really zap the will out of you when it comes to reading for fun, at least when you’re taking classes you don’t like. Luckily, flying to Kansas City, then Seattle, then Montana, then Indianapolis, then back to Seattle in the span of three weeks gave me a lot of time to read. I remembered how much I liked doing it, so now I find myself laying in bed and reading.

First it was Self Made Man. A female journalist dressed as a man for about a year-and-a-half and wrote about her experiences dating, working, and interacting with “other” men. A solid read. Then came Freakonomics, a book I’ve heard a lot about. I enjoyed it, and while I think it’s a great book and I learned from it, I found it largely overrated. Next was Moneyball. Riki Hayashi must be pulling his hair out because he told me to read the book years ago, but after the urging of a friend – “Trust me, it isn’t really about Baseball” – I finally read it. I loved this book and found myself applying a lot of it to Magic. I thought I had found the holy grail of books in Moneyball, until I was handed Outliers. Now THAT’S a book. I couldn’t put it down.

[True story: if not for Moneyball, you would not be reading articles by Jon, Zaiem Beg, and Jeremy Fuentes here on ChannelFireball. -Riki, Beane-ingly]

By telling you the books I’ve been reading over the last few weeks I’m trying to provide some background on what I was doing while I came up with what I’m about to talk about. Ok, so I didn’t actually come up with this idea myself, I first heard it from my roommate Brian Wong. Brian was talking about how awesome it would be to have stats for Magic Online. Imagine, he said, what you could learn by seeing how often the person going first in Shards of Alara Limited actually won. Or how many nonbasic lands successful decks are playing. Or how often a first-pick [card]Tower Gargoyle[/card] led to a win. We think we know these answers, but we have such a limited amount of feedback to draw from, usually only our own games, or articles where people tell us about their own games.

To find real, concrete answers, we need to look at the data. I was thinking about writing this article a few weeks ago, when all of a sudden William Spaniel rose from the dead and stole my topic. (I’ll get you, Spaniel!) I put off the article until the Standard season ended, assuming I’d need a topic, so here we are. Spaniel’s article just scratched the surface; he wrote about the importance of information and how helpful it would be to WotC. I want to dig a little deeper and start asking some questions.

Imagine if WotC had a big computer that only stored data. This computer wouldn’t do any reasoning or make calculations or infer anything based on what data it has. It wouldn’t make charts or write articles. All I’m asking this computer to do is store some data and let me access that data easily. I’m going to take you through what data I want and what kind of questions I would want to ask.

The Games

Zaiem warns me about being results oriented, and in his last article he talked about how damaging it can be. You kept a two-land hand and you lost, thus you should mulligan two-land hands. This is a bad mindset to have because you’re not drawing from actual statistics, you’re just drawing from your own personal experiences – a small sample size. What if, however, you could actually be results oriented? What if you could look at how often two-land hands actually got there compared to mulling to six?

After two players finish their match, I want Magic Online to send the following game data to our database:

This one is easy: which player won the match? However simple this statistic is, when our goal is to win, it’s the most important. This is the statistic that all our other statistics will be compared to. This is the cold heartless number that doesn’t care about how often you hit your fourth land drop, or how many counterspells they drew, or how few creatures you drew. All we care about is whether or not a certain player won a game, and from there we can find the important factors that led to that result.

We also need to see who won each game, not just the match. It’s easy to see how this applies to sideboarding, but I’m also interested to see if it has any other applications. How often does a player come back from losing game one? How often does a player win game one and lose the match? Each game – one, two, and three – is very different, and they must be treated as part of a match and not just dumped into a data bucket full of single games.

We also need to take this a step higher and look at the overall tournament placement. Just as each game is different, each round is different, and the top eight is a beast all its own. You know that guy you go to when you need to know if you can draw into top 8 or not? That new guy is our database. How often does a player lose round one and still make top eight? Is a player’s top eight performance independent of their swiss performance? Once in top eight, does the guy who lost round one only to make it to top eight win the event less often than the guy who drew in at X-0-2?

I could ask the data the win percentage of a person who scoops compared to somebody that dies naturally, and use that to prove my point on scooping.

Whenever I actually think about what turn a game ended on I’m usually surprised, especially in Limited. We hear of fast formats and slow formats, but our database can rank them precisely. Mike Flores (at least I think it was him) talks about critical turns and the different phases of the game. By tracking activity based on turns, we can view this data in tangible figures. The importance of turns is something that’s hard for us to track naturally, and it’s a method of analysis that would break wide open with the introduction of this system.

Magic Online provides a unique opportunity with the clock system to actually track how often a players has priority. I’m not sure what to learn from this data, but I have a feeling it’s useful. When all the data is organized and analyzed, I wouldn’t be surprised to see patters in priority emerge.

This one is easy, right? Just send our database the life totals at the end of each game. That’s bound to hold a lot of useful information on its own, but life is so much more meaningful than that. New players value their life much too highly, we know this, but our database can prove it. It can show us just how valuable life really is. Imagine tracking life point changes by turn. By spell type. By amount. Our database would need to know when a player’s life total changed (down to the step of the turn), how much it changed, and what made it change.

Die Roll & The Play
Who won the die roll? Who won game one? Who won the match? Who won the tournament? Does the winner of the tournament have a higher die-win percentage than average? Winning the die roll doesn’t necessarily mean going first, so we would want that piece of data as well. Choosing to play or draw is usually a topic of conversation for a new format, and judging by the split response in the “play or draw” question asked to the pros for the M10 Sealed format, it’s one we never really have a definite answer to. Data can give us that answer.

Easy: who mulliganed and to what? Also, what was in their hand when they mulliganed? What was in their hand when they kept? Here’s a theory: the winner of a tournament, on average, has a lower mulligan percentage than they normally do. They were “running hot” that day. Maybe they had the aura. Data could prove this theory wrong or right, but without it, it’s really hard to know. Data might tell us that this theory is true, but that the winner of a tournament, on average, mulligans more than the average person, meaning that we don’t mulligan enough.

How many cards were sideboarded? Which cards were put in, and which ones came out? I think this would be very interesting when it comes to Limited. It’s safe to assume that winning Limited players sideboard more than losing players (though data could tell us for sure) but how many cards are they actually sideboarding, and what are they bringing in? For Constructed you could track exactly how useful Pithing Needle is as a sideboard card, and how often players that have it win compared to those that don’t.

Cards & Plays
Now we hit probably the biggest chunk of information: what cards were used in each game, and how they were used. This is the hardest part of the data collection, but is the biggest window into gameplay. Somebody smart would need to find the most efficient way to relay the actions taken in a game – once we have that, we’re good to go. Once we have the data we can track card advantage, attack phases, land drops, mana used, creatures destroyed, flyers cast the list goes on. You could start to identify the best spells in your deck. How many games does this deck win when it doesn’t draw Bloodbraid Elf compared to when it does? How important is drawing a Volcanic Fallout against Faeries? When is playing the second Bitterblossom correct? How much mana is typically spent each game? There are a millions questions to be asked. This section could be an article all its own.

What’s the best color in standard? What’s the best card in Standard? We could find out.

The Decks

James Lee, a local and highly respected judge, recently mentioned his desire for a deck builder and database. I find it weird that if I want to build a deck I usually open up Magic Workstation. Shouldn’t there be a tool online for doing this? I think this is a service WotC should provide, much like they provide the Gatherer. They’ve already got Magic Online players building decks, and they could all draw from the same database. Magic Online can do for a deck database is incredible: it can tie a lot of decks to a lot of results. Our real life data of decks and results is severely limited, but Magic Online has heaps of it. These are the things I would look for:

Deck Makeup
How many lands do winning decks play? I’ve found it odd that over the last year or two we’ve been playing decks with 25, 26, or 27 lands without batting an eye. I found myself wondering if 26 lands was too few in my Nationals deck, and it wasn’t playing anything that cost over 6. Six years ago, however, when I started playing Magic, most decks ran around 24 lands. Have we changed our deckbuilding theory, or are decks fundamentally different now than they used to be? Do they have a higher curve now? There is a lot of talk about curves in Magic theory, so what makes a good curve? How bad is 61 cards really? We can assume that 60 is better than 61 (and again, data can prove this) but how much are you hurting yourself by running 61 cards? Or 41 in limited?

Deck Type & Metagame
Once we have a database of decklists it will be important to have a tool that can identify those decks and classify them. Once a certain class of decks is identified, like Five-Color Control, Red Aggro, or Faeries, we can start to examine the metagame. Who really wins in Kithkin against Faeries? Once we start tying all our data points together we can ask really interesting questions. Which deck has the highest mulligan percentage? Which deck mulligans the best (wins the most on a mulligan to six)? Apply this to Limited, and we can find out what shard is actually the best, the worst, and by how much.

The Players

Tying decks to results is one thing, but tying the results to a player is just as important. Trends that appear among worse players might be completely different amongst the better players.

Without a better tool to measure a player’s skill, this is what we’ve got. In fact, by checking out the win percentage of players based on their rating, we can find out how much rating actually matters, instead of how much we perceive it to matter (or not matter). Once you take rating into account you can start looking at the data in a more personal way. Take the Faeries vs. Kithkin matchup. How different are the win percentages for each deck when played by low-rated players compared to when they are played by high rated players? Does the rating of your opponent effect whether or not you should mulligan?
Since MODO is anonymous, and WotC has no real way of knowing exactly who is behind the username, it’s important to track what we do know, like when a user created their account. I’m not sure what this could mean, but much like priority, once things are organized a trend might start to appear. How often is a player logged on to their account? What types of events do they play in, and how many? Even what is in their collection could tell us something. Are the better players parts of clans? How much does being in a good clan help?

The Draft

Drafting provides its own unique set of data, and a pretty simple one: what cards were in the packs, and how high were they taken. We can take this information and see what the chances are you’ll get a useful Borderpost in pack three. Where this gets really interesting is when you start to cross this data with all the other pieces we’ve collected. How good is first picking a tri-land like Seaside Citadel? Not only that, but how important is that tri-land to actually winning the draft? Maybe winning players tend to have more tri-lands than losing players, but only because they value them higher, not because they are actually worth it.

Could we devise a formula to give each card a to-the-decimal-point pick number? Once that’s been done, you could actually rate the power level of the whole draft, proving or disproving comments like “my draft pod was weak.” You could even compare the power level of packs opened to a player’s left and see how much of a factor luck in opening is. You could even analyze signals, a topic highly questioned for its relevance. This is one of those points where I think the Magic community would do something surprising, we’ve just got to give them the data.

Lastly, The You

Now we come to why this actually matters for you. If you’ve played any serious online poker, you’re familiar with a stat tracker. (Reading up on Poker was another thing I was doing leading up to the creation of this article.) These track the players you’re gambling against – their raise percentage, fold percentage, and a bunch of numbers I’m not good enough to even be able to describe to you – but they also track your own stats. Being able to look at your own patterns and see that you’re not raising nearly enough, or that you’re folding too often, is paramount in your personal improvement as a player.

I want this as a Magic player. I want to be able to track my mulligans, and see how I compare to the ideal. I want to be able to track the consistency of my manabases. I could track my own results and see what works for me. We have our own notions of what kind of player we are – combo, aggro, control, etc. – but we’re basing our assumptions largely off of our feelings. This thing could tell me which deck will give me the best chance of winning an event, weighing my own personal preferences against the metagame.


I’m not asking for WotC to do any of the heavy lifting of data analysis. All I want is the data, and we’ll do the rest. Seeing how impacting this data would be on me as a player I can only imagine what WotC R&D would do with this information. As a guy that’s interested in game design, being able to see exactly what goes in to winning a game of Magic would be incredible. I have no doubts that the quality of Magic design and theory would go up immensely. That’s why I’m calling this the most important article I’ve ever written.

Thanks for reading,
Jonathon Loucks
Loucksj @ gmail.com
JonLoucks on twitter

P.S. I’m going to be at PAX this weekend! I’m demoing Arcane Legions again, so find me at the Wells Expeditions booth and say hi.

27 thoughts on “Rogue Report – Dear WotC: Open the (Data) Vaults”

  1. Nice idea, but unfeasable. The sheer amount of data and servertime needed to store these statistics make it prohibitive for WotC.
    There’s cost involved without return.
    Not to mention, it’ll probably affect the performance of MODO, running slower and crashing more.
    Also, you have to be filter out matches when reviewing the stats, because you don’t want the info on casual room games, non-relevant drafts, classic, commander, Vanguard, etc.. Do you remember how long they worked on the new Gatherer? Now imagine the same with a larger, everchanging database.

    Plus, I’ld be bored to tears reading an article analyzing all these stats.

  2. Pingback: MTGBattlefield

  3. Part of me wonders if this would kill the fun. But a bigger part is too curious and wants to get its grubby paws on those delicious data!

    It makes me wonder how useful it would be though. First picking a Tower Gargoyle is great in the world where it offers you the best chance to win the draft and few know it, but is that first-pick as good in a world where everyone knows it’s the best first pick?

    My point is that this would be a massive change to the game Magic, probably on the scale of net-decking. Most of so-called Magic theory is based on anecdotal evidence. To have real data to back it up would be huge. It is interesting that having access to the results would change the game, thus making early collected data less meaningful.

    Of course, analyzing such data would be a monumental task. One I’m sure the nerds like me who play this game would willingly undertake, but huge none-the-less.

  4. As a CS grad, having access to data like that would be so much fun to play with. Probably more interesting to a more math-oriented person, but you could certainly find some very interesting stuff regardless. The issues with implementation are valid though, which is probably enough to hold it back.

  5. I am a numbers geek, and I would be tempted to spend waaaaay too much time with a database like this. However, like the above poster, I wonder if this would kill the game a little. We know that rogue deckbuilders are swimming against the tide, and Netdeckers have advantages of so much more match data and playtesting, but with the data set you are talking about, I wonder if the min-maxing would just be more out of control then it is.

    Another example, for those of you that played WoW at the progression raiding level, know that data parsing has become a fact of life, and while it certainly improved efficiency, did it make the game more fun? I wonder, it certainly became another tool to e-peen with and berate lackluster raiders.

    I am just not sure if it would actually make the game better.

    Be that as it may, I would love to see that data too.

  6. If you really think this is important you should be working to accomplish your goals without WotC having to do anything.

    A feasible approach to the problem would be to build a client application that a user could run on their computer that kept track of what happened in MTGO and upload it to a common server. The server could then aggregate and provide the (anonymized) data to those users that had provided data above a certain amount. A system like this could probably be scaled up in a fairly rational way – start by tracking the most important simple statistics – something like win percentage, land percentage, and average CMC in a deck, and then add on statistics as you continue to develop both the site and the client app.

    Far from easy to do, but much more likely to happen if you’re really dedicated and want to make it happen than simply hoping that WotC will decide to undertake a large IT investment for questionable (for them) benefit.

  7. You currently cannot see the rating of your current online opponent. This was done consciously by WotC. Do you really think there is a chance in hell actual statistics will be released, even if they were recorded?

  8. For those that play Magic like a job, I’m sure this would be a useful tool.
    For those that play it for a game, it’s a terrible idea! You’re basically answering a bunch of decisions. No more “what to do?”, just do the “right” play.

    It even reduces the amount of skill required (even if it improves the overall adeptness of players).

    It’s a nice idea, but it would be a sad, sad day if this ever happened.

  9. There are already replays on modo so tracking and storing is not a going to change core functionality. The work required depends on how their infrastructure is implemented, but feeding a timestamp, decklists, playerratings(not names), k-value and replay data into a database for correlation should be quite possible and would not have to strain modo’s performance significantly. When this work is done by wizards I think a lot of people would be happy to mirror this database and create the reports and analyse and interpret and all that good stuff.

    I don’t see why WotC would provide this service for free, though. You generally don’t want to expose your intellectual property without some kind return. They could possibly make it a subscription thing. After creating the initial interface and subscription system, maintance would probably be rather cheap which would allow the system to generate profit over time. Still .. pretty unlikely. Part of the game is retaining the lore/RP feeling and cold cold math is hard to spin i think. Phyrexian Silo? 😀

  10. @joe – A system like that would be rather easy to manipulate. A few players could benefit greatly by skewing the stats and thus making other players chose bad decks or make the wrong plays.

  11. Adam: The reason they gave for disabling ratings was due to abuse. 1600 player makes an account, 1600 player beats an 1800 player due to a combination of good draws, bad draws, and opening ridiculous bombs…and the 1800 player is hounding the 1600 player with messages like, “Do you feel powerful?”

    Since rating on MODO means nothing – there are no GP byes or PT invites based on rating, it was really doing nothing positive while encouraging poor behavior.

    Not that I necessarily agree with WotC’s decision to disable rating, but the reason for doing so was not to keep proprietary information close to the chest, but rather to limit abuse of the players. I don’t think the rating limitation indicates one way or another what Wizards would do regarding the release of anonymous data.

  12. thomazalexandre on MTGO

    Nice article. As a graduated and working statystician, I hope they one day hire competent people to process the data, because I really doubt they would release the raw data files.

    @Adam: the real reason I think is the invaluable nature of the info, and not confidentiality, beacause you can easily organize the rrgistries without naming individual players in games. The rating stuff was really more of a reaction to players’ bad behaviour, and let’s face it, MTGO ratings are actually useless.

  13. You want them to build a data mart for you. Nice idea, but never gonna happen. You say that the ‘heavy lifting’ will be done by the user anyway, but fact is what you describe is not the ‘heavy lifting’ – doing the actual analysis needs ‘only’ smart application of data mining techniques and/or OLTP queries. The really expensive part is building the data mart, buying the hardware, paying for the network traffic, … Additionally, some privacy issues would come (at least here in Europe, don’t know about the US 🙂 ). Solving all those issues is absolutely feasible and state of the art, just /very/ expensive. My gut feeling would be that this data mart would be the most expensive MTGO component to develop and run by far. For this, the ROI is just too little for WotC.

  14. Having worked at Firms which are keeping track of much more and a lot more valuable data, this isn’t really unfeasible. Whether its worth the cost is more debatable. And for all we know Wizards does collect Data and just doesn’t want to release it.

    My guess is the fixed cost for something like this might be expensive but after its up and running it would be relatively cheap. Afterall replays basically take all the data John wants and saves it for some temporary amount of time. All you would need is the code altered to save the data permanently (Which I would bet they do anyways).

    However, the one thing that John and most people are unfamiliar with is that you can’t just store data. Especially if there is an unreasonable amount of it. Lets say you have a few terabytes (maybe an underestimate) which is all your raw data for 6 months. You can’t exactly just ship that over and over to whoever asks. And while people might not need every piece up there it costs programming time to collect specific amounts for individuals.

    Return’s to Wizards probably has to come in at Development. They can actually get factual ideas about whether a set is less skill instensive (do Higher Rated Players lost more). About whether mana screw ends more games in limited (do people concede earlier and with more cards in hands). etc..

  15. I feel like this article is like selling me on the idea that a flying car would be awesome. I think very few people disagree that flying cars and this database would be awesome. But, we don’t have flying cars not because no one would like one.

    I guess what I’m saying is that the hard part isn’t convincing us that someone with this information would have a tremendous advantage over those who didn’t. The hard part is making it happen.

    On the other hand, not a bad article. The value of statistics cannot be overstated. The NFL actually does analysis exactly like this. For examples, “If a team has a 100 yard rusher, how often do they win the game? If a team has a punt return for a touchdown, how often do they win the game? If a team has a at least 30 rush attempts, how often do they win the game?” That data guides management when making personnel decisions. It would be significantly more useful in magic, though, where execution is virtually nonexistent.

  16. Information is juicy, I have to admit, but there IS an element of skill lost when you make every game viewable and statistically recorded. It seems like net-decking on a higher level. Personally, I don’t like the privacy part.

  17. I don’t think this will ever be done. Far more feasible would be some kind of MTG tracker that one could use on their own games. Not even sure that would work though since poker tracker usually works off hand summaries that all poker websites already provide. Just having personal stats would be awesome.

  18. This kind of statistical analysis is very availible for MMO’s (since its easy to parse combat logs et) and it results in a lot more have/have not complaints and cry babies. “Oh you won that arena match because you’ve got x glyph” sounds a lot like “you won because you opened Baneslayer”.

    Data like this is a 2 edged sword. Programs like RAWR which endlessly crunch gear templates tell MMO players which items to roll on in upgraded dungeons. There is no happy medium when you can get one item but someone else got the one that is 0.02% better, or Mages do 3% less damage on this fight so lets sit them all out and bring rogues.

    Before too long, we’ll have a calculator telling us based on casting costs and ratios how many lands to include and whether or not our deck will win before we even play it.

    It’s like a bad case of MTG genomics weeding out only the best and while it improves player quality in the same way that mana curves, tempo, and the G.U.T have, it also destroys the fun in the game. You’d sit down and draft and then watch half the draft drop out so not to lose points with thier sub-par deck while the rest fight for packs.

    No thanks!

  19. Great idea and if there’s a viable, efficient, and rather easy way to implement this idea, then I think it would be great. And remember folks, similar to how science fiction has inspired real science, what seems impossible today, may be within reach tomorrow.

  20. I think the idea is a good one, and that short term the affects of such a thing would be absolutely fascinating, but long term it’s just too detrimental to the game. Remember, this is a game and that while the pieces are changing, for long stretches of time the pieces in any given format are fixed until new cards are released. The most fascinating thing about competitive magic is the give and take of the metagame which is what would be greatly threatened by such a creation.
    The confusing thing is, is why would any Pro level player or an aspiring Pro want to release this information to the average PTQer. Information like this would be a huge leveling of the playing field, making it much more difficult for Pros to beat an average player.
    Maybe I’m overreacting on this, but I don’t see any long term benefits to this. Short term, I definitely believe it would be quite fascinating to understand the underpinnings of the Magic Tournament “cosmos”, but long term the metagame would devolve into who has access to the information and who doesn’t, which is not good.

  21. I like Joe’s idea that kind of got lost in the shuffle…

    Why hasn’t someone built something to track modo games and send the data to a central pool?

  22. Jon, if you liked Outliers be sure you read the rest of Malcolm Gladwell’s books. The Tipping Point is particularly compelling as well.

  23. Some of the simpler points are definitely interesting. I noticed the change when I came back to yahoo fantasy baseball this year: each player has a percentage by his name for “the percent of fantasy teams that own this player”. Yahoo keeps track of stats similar to what you were saying about drafting, the average spot that a player is drafted based on all yahoo drafts – a huge data pool. It’s oversimplistic to correlate “decks with ~this~ in them” with “win percentage”, like you have outlined. It’s much more important to see how a card gets used.

    I am definitely a casual player, but I do read the MTG articles regularly to pick up tricks and see how some of my cards are better than I first expected. Just to see the data and know, “Hey look! I have that card that ‘parses’ in the top 10!” would be worth it. Plus, I would definitely respect the card all the more.

  24. Pingback: September 3 MTGO Metagame Report « Scrubbin' In Fort Wayne

Comments are closed.

Scroll to Top