As anyone who’s invested enough to read an article with that title knows, the ranked system in Hearthstone changed significantly at the start of this Standard year. Prior to April 1st, players would queue into each other on ladder purely based on their position on the ladder; if you’re at rank 4, 3 stars, the matchmaking system would try to match you up against someone else at rank 4, 3 stars, then it would look through the rest of rank 4, then out to ranks 3 and 5, and so on, until it found another player close to your position. At the start of this Standard year, the matchmaking changed to favor MMR over ladder position. MMR is a score that Hearthstone maintains for each player behind the scenes that roughly correlates to player skill. As you win, your MMR increases, and as you lose, it decreases. As a result, players at different positions on ladder would get queued against each other, but since they are at equivalent skill level according to their MMR, the games should still be fairly matched. To cement the point that rank doesn’t matter for matchmaking, the dev team removed the ability to see your opponent’s rank until a player reaches legend.
A funny thing started to happen once legend players started seeing their opponents’ ranks, though. A number of people noticed they were queueing into a lot of players at Diamond ranks and below. To some extent, this is expected given the emphasis on MMR matchmaking; sometimes, players would be of equal skill to legend players but stuck at lower ranks due to having less time to play or just a rough patch of luck. But it seemed to be happening way more than one would expect given that the MMR of players should be relatively equal.
The legend climb seemed slower, too. Once one got slotted into legend, the climb seemed to require a win rate significantly higher than 50% to generate forward progress. As Sidisi767 posted on Twitter after tracking his daily win rate, it seemed to need an absurdly high win rate, above 65%, to break out of low legend ranks. I had been feeling the same lack of progression myself, even after sessions with 60% win rates. To be clear, I’m not particularly invested in my rank once I hit legend; obviously, I’d like to hit high ranks as much as anyone else, but mainly just as a way to measure my own improvement in the game. That said, I couldn’t shake the feeling that my lack of progression felt different this time.
So, mostly to remove those excuses and get back to focusing on my play, I leaned into my natural instincts that I use at my day job as a data architect - I collected data. I decided to manually log every game I played on Monday, June 29th, with starting rank, ending rank, opponent rank, and whether I won or lost. The spreadsheet with my results can be found here; if you want to track your own results, feel free to make a copy and just clear columns A through E. (Most of these games were recorded on stream as well.) The results actually surprised me; while this is admittedly a small sample size, the results within this sample were consistent enough that I think it’s worth raising as an issue for further investigation.
Before I discuss the results, a note on MMR and how it’s supposed to behave. While the actual MMR calculation for Hearthstone is not public, it does generally abide by common behavior for MMR scores in multiplayer games, meaning that a player’s MMR should change by a set amount, with adjustments for discrepancies in the two players’ starting scores. Simply put, if the higher player wins, their MMR is expected to increase less, because they are “expected” to win that game by the algorithm. The lower player should lose fewer points for the same reason. Conversely, if the lower player wins, they should see a large jump in their score, as they beat a player who was significantly better than them (as far as the MMR score is concerned), and the higher player gets punished with a larger MMR drop for losing to the weaker player. Therefore, all things being equal, if Hearthstone’s matchmaking is working as described, each game should result in a roughly similar change in MMR, and therefore Legend rank, because the players matched should always be relatively close in score. (Note this isn’t strictly true, because Legend ranking is a leaderboard ordered by MMR, and is affected by other players’ results along with new players entering Legend, but late in the month, the rankings should be roughly settled, and rank is the only public metric available, so we’ll work with what we have.)
As I logged my games, a couple of things became clear. One, I was being matched against a lot of Diamond players. Over a 24 game sample set, 12 were against other Legend players, 11 were against players at Diamond rank (between Diamond 5 and Diamond 8), and one was against an opponent set to appear offline. During this sample, nearly half of all my games were against non-Legend players! The frequency of games against non-Legend players seemed high, but half, especially this late in the season, seemed higher than I expected, even going in with a suspicion that there were a lot of matches against non-Legend players. That would be fine, though, if the MMR for those players is equivalent to mine, on average.
Unfortunately, that seems to not be the case. Again, using rank as a proxy for MMR, some trends from this sample became quickly apparent. On average, when paired against other Legend players, the number of ranks gained and lost were roughly even, which one would expect with equivalent MMR scores. On average, I gained 272 ranking places for a win against Legend players, and lost 285 places for a loss. Even when paired against players more than 500 ranks apart, the change in rank stayed in the 270-300 range fairly consistently.
Data collected June 29th, 2020. All games were on Standard Ranked ladder.
When paired against Diamond players, however, things varied much more wildly. Wins against Diamond players gained 222 places on average, 50 fewer than against Legend players. Losses against these players, however, were devastating, losing 344 places on average. Every loss to a Diamond player was more punishing than the worst loss to another Legend player, and the most lucrative win against a Diamond player was equal to the least lucrative win to a Legend player. Over this run, I went 8-4 against Legend players and gained over 1000 places over those games, but that was countered by a 6-5 record against Diamond players, that lost me 345 places despite the slightly positive win rate. That seems less consistent with matchmaking by equal MMR, and more consistent with being matched against players with a much lower MMR. Combined with the frequency of Legend players matching against non-Legend players, the end result will be a winrate of between 55% and 60% just to break even, and significantly higher than that to move up. Again, this is a small sample size, but the results are internally consistent enough that I believe it merits further scrutiny.
And just to be clear, I don’t think that if this was fixed, I’d magically be rocketed up into 11x multiplier levels of MMR. If, say, the change that was in place in the last ladder system that prevented most Legend players from queueing against non-Legend players was reinstated, all Legend players would be on an equal playing field, and I’d be as far from there as I am now. However, it would go a long way to make your legend rank feel correlated to your performance in ladder matches, which is enough to make ladder ranking feel less arbitrary. If one is unlucky enough to queue into more non-Legend players than Legend players, they will quickly get to a point where they will need three wins to cancel out two losses in terms of MMR contribution, meaning that they would need a minimum 60% win rate just to see forward progress. That’s demotivating under the best of circumstances, and it has the added effect of discouraging experimentation, since that often involves losing a fair number of games while trying a new deck or tweaking a new deck list. If you lose too many games, your MMR could get stuck in the depths of Legend, and that lower MMR will now carry over to subsequent months, as a result of other changes implemented in April. That’s not good for the meta, and by extension the health of the game.
In conclusion, matchmaking at Legend seems in need of attention. If this is working as designed, then so be it, but given the lack of public information and my decision on a whim to try to quantify the feeling that “ladder feels different”, I would be remiss if I didn’t at least try to bring this to the dev team's attention, and to the community's attention as well.
Thanks for listening.