fbpx

Quantifying the Card Evaluation Metagame

For those not familiar with 17Lands.com, we provide a tracker for MTG Arena with a goal of helping players understand their own data as well as generate insights from the larger pool of data from our community of users. I’m the main developer for the site, and today I’m happy to share with you some of our newest findings.

We’ve seen a pretty big growth in our users in the last few months, and with more users comes more data. This growth in data means we can get more accurate estimations on things like how strong a given color pair is or how fast a format is. More interesting, though, is that we can now get enough data in a short period of time to get a clear picture of how the metagame shifts. This is something people have been able to get a qualitative feel of for a long time, but today, we’re going to be able to quantify shifts in the card-evaluation metagame by looking at draft data.

One of the things that 17Lands tracks for our users is their draft history, revealing the cards they see for every pack in their draft. With enough packs from enough drafters, we can get an estimate of how highly the overall draft community values each card using an “average last seen at” (ALSA) metric. To illustrate this metric, let’s look at a few examples. If there were a colorless card that was better than everything else and people always took it first pick, we would only see it pick one of each pack; no one would ever pass it. For this card, the ALSA would be 1.0, as we never see it later than pick one. If there were a card that people always took as the third pick out of every pack, slightly less intuitively, it would have an ALSA of 2.0 because there’s an equal chance you see it at picks one, two, or three. When a card “wheels” (we see it when the pack comes around again), we only count the final time we see the card. As a result, a card that is always last picked out of a 15 card pack would have an ALSA of 11.5 (because there’s an equal chance we last see the card at picks 8 through 15). There is some non-intuitive behavior in how ALSA behaves that prevents us from using it to compare across rarities, but the general understanding you should have is: the lower the ALSA, the more the community values the card. Note: below, we’ll use a moving average over three days to smooth out the ALSA variability.

Now that we have this background in mind, let’s take a look at some of the interesting things we can see in the data!

Card Evaluation Consensus

With all of the great content available today and the ease of jamming many drafts in MTGO and Arena, it’s easy to assume that card evaluation will converge pretty quickly. As much as we like to pore over the spoilers and talk about cards before the set comes out, no one will get their initial ratings right. So much depends on the dynamics of the set that you just can’t learn until playing with the cards in context, seeing how they line up with what everyone else is playing. As such, there has to be some period for shuffling card ratings. But when the initial impressions episodes of Limited Resources and Lords of Limited come out, does the limited community converge on card ratings quickly? From the data we’ve seen, it looks like not!

Let’s take Drowsing Tyrannodon as an example. People knew it was a pretty good card in set review season, but many put it around the 4th best green common. Within a week or so, though, once people started playing and seeing how many ways there were to get it to four power, many content creators were seriously considering whether it might even be ahead of Llanowar Visionary as the best green common. While there’s still some debate about where Drowsing Tyrannodon, Llanowar Visionary, and Hunter’s Edge fall, we can see in the graph below it took almost a month for the overall Arena drafting community to settle on Tyrannodon at an ALSA of about 4.0.

We can see another example of this with a pair of white spells: Feat of Resistance and Secure the Scene. People are generally too high on clunky removal spells at the beginning of a format, and the aggression of white in M21 made Secure the Scene even worse than it might be in another set. On the other end of the spectrum, even though many had experience with it in its appearance in Khans of Tarkir, the overall community started fairly low on Feat of Resistance. Both cards started with an ALSA in the 5 to 5.5 range, but we can see it took around a month for their rankings to settle.

In some cases, it takes even more than a month for evaluations to converge. Anointed Chorister is a prime example of this in M21. Given that 1-drops were pretty bad in limited sets for a number of years, you can’t really fault people for writing them off in spoiler season. But when you start so low on a card that it’s not even on your radar, it takes a while – in this case, about 7 weeks – to catch on to it being one of the top commons in white.

Misevaluation happens at other rarities as well. Clearly there are some weird rares that are just tough to evaluate without seeing them in action (Akroan War from THB, for example), and you can expect it to take a while before everyone gets to see how good they are. But even some powerful uncommons can take a long time to jump up in people’s pick orders or for them to see them as signals to latch onto a color in the draft. An example from M21 is Seasoned Hallowblade. As early as a week into the format, some were wondering if it was better than Baneslayer Angel, but you can see it still took almost a month for the overall community to see it in consideration for the best white uncommon.

So what should we learn from this? First, by all means do exploit early misevaluations. Don’t be the drafter who looks at card ratings from before the set was released and hold onto them for weeks into the format. Instead, use what you’re seeing for yourself and hearing from others to look for those cards that are top commons that are undervalued. Bump up cards like Pridemalkin and Setessan Training knowing that you can pick up more Drowsing Tyrannodons than you should be able to in the first few weeks.

At the same time, you’ll need to juggle a bit more in your head during this period of misevaluation. Don’t fall into the trap of misinterpreting signals as a result. Just because you get a card like Drowsing Tyrannodon seventh pick doesn’t mean that green is actually open; It also just might mean the green drafter to your right hasn’t seen how good it is yet. If you’re drafting the hard way, you’ll want to keep an eye out for the consensus top commons to make the decision to move in on the open color.

Color Openness

While it’s good to know how long you can take advantage of certain cards being picked too late, perhaps even more important is knowing what colors are more likely to be open. One way to gain an edge early in a format is to learn what colors are underdrafted and to bias towards them early in a draft. This can lead you to drafting a better deck on average than completely drafting the hard way, as you’re likely to find your correct lane a couple of picks sooner.

There’s no exact way to tell how open a color is in any given draft – if there were, drafting the hard way wouldn’t be so hard. Given a few drafts, though, you can start to get a sense of generally how open a color is. For example, looking back to Theros Beyond Death, it took most people just a few days to realize they weren’t getting as many mid-tier black cards like Omen of the Dead or Soulreaper of Mogis as they would expect in later packs after settling into black. This was due to how good black was overall, with many top-tier uncommons that drew many people into drafting it.

In our case, we have a lot more than a few drafts – we see hundreds per day from our users. Given this data, there are a few different ways we can estimate how open a color is. The metric we’ve found that seems the best is to look at the ALSA of the top three commons in that color. This heuristic certainly isn’t perfect. Sometimes a color might have a few very strong commons that get picked early, but poor commons after that. Alternatively, a color might have nothing that stands out as substantially better, but its whole roster of commons can still be pretty deep. However, so far, we’ve found it matches the sentiment we’ve heard from the community. Below, you can see that value (using a three-day rolling average) for the five colors in M21:

This pattern demonstrates an equilibrium given what people have felt anecdotally – that the Naya colors are the best, and that black is the worst. Given that black is not as deep as the other colors, you expect fewer people to want to draft it and thus its best commons wind up going later. Many people are drafting with a preference to be in an aggressive WG, WR, or RG deck, so their best commons disappear from packs early. This lends a lot of credibility to the fact that draft is self-correcting.

With this information in hand, if you know early on that white, red, and green are the best colors and you can see that people aren’t ending up in them as much as they should, you can bias your early picks towards them and really capitalize if they are truly open. You’ll also be more likely to reap the rewards in pack two if you’re able to push your neighbor off if they don’t have the same information you do.

You can also use a metric like this to get a better sense of when you might want to jump ship on a color. Going back to the example of black in Theros Beyond Death, if you could see that its top commons were going incredibly early, you might know to try to avoid it even though it’s the best color. You don’t want to go too far overboard, but you can also consider taking things to the next level – knowing that a particular color or archetype is overly prominent, you can start thinking whether that should increase the value of other cards that might combat that archetype.

Looking Forward

With M21 in the rear view mirror, let’s recap in order to be able to take advantage of this in the early days of Zendikar Rising.

  • Don’t be afraid to take advice from pros or limited grinders even if you see their article or listen to their podcast a few days late – it takes weeks for the overall community to catch up!
  • Take content that came out before a set’s release with a grain of salt. Content creators often have pretty good initial evaluations, but nothing compares to seeing the cards in context on the battlefield.
  • Don’t misinterpret signals early on. Someone passing to you might have taken something over what you think is a top common in the same color, as initial evaluations are all over the place.
  • If you know a color is good, lean into it. Early into a format, drafting with preferences can yield its best results.

You might also be wondering how you can get a sense of the current card evaluation metagame to determine what’s still undervalued. You can check out a new Card Evaluation Metagame page that we just launched today to see where things stand! And if you want to help 17Lands get more data for new insights like this, you can see how to get started with our Arena tracker here.

Discussion

Scroll to Top