Matchmaking for the AI Arena StarCraft II Ladder

AI Arena runs its StarCraft II bot ladder in rounds: bots are split into divisions by rating, and in each round every bot plays every other bot of its division once. This page asks whether a matchmaker without rounds and without divisions could serve the ladder better, and how to tell. It describes a model of the real ladder fitted to its match history, a simulator that replays the ladder under any matchmaker, four matchmakers, and how they compare.

The page is generated by analysis/analysis.py from analysis/report.md; every number in the text comes from the data, the fitted model or the simulation runs.

Overview

Goals

A new matchmaker should make sure that all bots

• play a fair number of matches;
• play mostly against opponents of similar skill;
• play varied opponents within that skill range;
• play matches spread fairly evenly over time.

Divisions may still be shown on the website to report progress, but they would no longer decide who plays whom.

Approach

Trying matchmakers on the live ladder is slow and risky, so we compare them in simulation:

1. Data. We take the ladder's match history from the AI Arena Recap database.
2. Ladder model. From it we fit a model that gives, for any pair of bots, the probability of each result and the distribution of game durations, falling back on the bots' ELO ratings where a pair has rarely or never played.
3. Simulation. A simulator plays out a ladder of the same bots on the same number of servers. Whenever a server is free, it asks a matchmaker who should play next, draws the result from the model, and updates the bots' ratings as AI Arena does.
4. Matchmakers. We compare the current round-robin system with a random baseline and two alternatives.
5. Results. Because the simulator draws every result from the model, it knows how strong each bot really is. We score each matchmaker against that truth, not only against the ratings it produces itself.

Data

The data is AI Arena's 2026 Season 1, read from the AI Arena Recap database with data/from_recap.py. It covers the 165,297 matches from 115 rounds played between 2026-05-01 and 2026-08-23 (about 1,437 a day), up to the end of the season.

The season started on 4 March, but its first two months are left out. Every bot starts a season at a rating of 1600, and the ratings took five to six weeks to spread out; match-ups also changed fastest in those first months, as bots were updated (see Limitations).

Of the 257 bots that took part, 173 finished the season in one of its 7 divisions; the simulation uses these. 105 of them have bot data enabled: they keep files between matches, so AI Arena runs each of them in at most one match at a time.

Each result falls into one of the categories the model uses:

Durations are game time, computed from the number of game steps at 22.4 steps per second.

Ladder model

The simulator needs, for any pair of bots, how likely each result is and how long the game takes. The model (model/fit.py) has to cover two regimes:

• Data-rich match-ups, where the two bots have played each other often. Of the 14,878 pairs of active bots, 3,211 have played at least 10 games.
• Data-poor match-ups, where we must fall back on the bots' ratings. 7,764 pairs of active bots have never played each other; most of them are far apart in rating, so they rarely share a division.

The model blends the two with Bayesian inference: the rating-based prediction is the prior, and a pair's own games update it.

ELO ratings

AI Arena rates bots with the ELO system. The expected score of bot A against bot B, with ratings \(R_\text{A}\) and \(R_\text{B}\), is

and after a game A's rating changes by

where \(S_\text{A}\) is A's score (1 for a win, ½ for a draw, 0 for a loss) and \(K = 16\) on AI Arena. The simulator updates ratings with the same rule.

ELO assumes that one number per bot describes every match-up, and that strength is transitive: if A usually beats B and B usually beats C, then A usually beats C. Bot match-ups often break this; a bot may reliably beat stronger bots with one strategy and still lose to weaker ones. Of the 3,545 pairs with at least 10 wins and losses between them, 31% were won at least 90% of the time by the same side. If every pair played exactly to its ELO expectation, chance alone would make that 7%. Where a pair has played, its own results are therefore a better guide than the ratings.

Game categories

Each simulated match is drawn in two stages. First its category: normal, time limit or abnormal. Ladder-wide, 93.0% of valid games are normal, 3.4% reach the time limit and 3.6% end abnormally, but abnormal games are mostly down to a few bots: 8 active bots crash or stop responding in more than 10% of their games (up to 58%), while the median active bot does in 0.1%.

So each bot gets its own crash rate \(c_i\): the ladder-wide rate of \(\bar{c}\) = 1.8% crashes per bot and game as prior, updated with the bot's \(n_i\) games and \(k_i\) crashes,

The prior strength, \(n_0\) = 5 games, is the same for crash rates, game categories and match outcomes: once there are that many games, they weigh as much as the prior.

A game between A and B ends abnormally if either bot crashes, which has probability \(p_\text{crash} = 1 - (1 - c_\text{A})(1 - c_\text{B})\). Otherwise it reaches the time limit at the ladder-wide rate among games without a crash, \(\bar{p}'_\text{tl} = \bar{p}_\text{timelimit} / (\bar{p}_\text{normal} + \bar{p}_\text{timelimit})\). This is the prior for the pair's category probabilities \((p_\text{normal}, p_\text{timelimit}, p_\text{abnormal})\), a Dirichlet distribution updated with the pair's own counts \(N\):

Then, depending on the category:

• Normal: the result and the duration are drawn from the pair's distributions below.
• Time limit: the result is a draw and the game lasts 60 minutes.
• Abnormal: the bot that crashed loses, and the duration is drawn from a ladder-wide log-normal distribution. Which bot crashed follows a Beta distribution with the two crash rates as prior, updated with the \(C_\text{A}\) and \(C_\text{B}\) games in which each crashed against the other:

Match outcomes

For a normal game, the probabilities of A winning, drawing and losing follow another Dirichlet distribution. Its prior is the ELO prediction, with the ladder's draw rate in normal games, \(d\) = 1.8%:

updated with the pair's normal-game wins \(W\), draws \(D\) and losses \(L\):

The ratings in the prior are the bots' final AI Arena ratings of the season.

For each match, the simulator draws the probabilities from these Dirichlet distributions and then the result. For a single match that is the same as drawing the result from the posterior means, \(\alpha_k / \sum_j \alpha_j\), so in effect every pair plays with fixed probabilities: its posterior means.

The plot compares predicted with observed win rates for pairs with at least 10 normal games, from both bots' points of view; larger circles are pairs with more games. On the left, the prediction is the ELO expectation \(E_\text{A}\): many pairs lie far from the diagonal, which is the non-transitivity described above. On the right, the prediction is the posterior mean. It is fitted to the same games it is compared with, so its closeness to the diagonal shows how strongly a pair's own games outweigh the prior, not how well it would predict future games.

Match durations

Durations of normal games are positive and skewed to the right. We model them as log-normal, truncated at the 60-minute limit (which the time-limit category covers):

Over all normal games, the log-durations have mean \(\mu_0\) = 2.41 (a median game of 11.1 minutes) and standard deviation \(\sigma\) = 0.51. Part of that spread lies between match-ups (some pairs play long macro games, others end in an early rush) and part between games of the same match-up. We split \(\sigma^2\) into a within-pair variance \(\sigma_\text{w}^2\), pooled over all pairs (\(\sigma_\text{w}\) = 0.36), and a between-pair variance \(\sigma^2 - \sigma_\text{w}^2\), which is the prior for a pair's mean:

That prior is worth \(n_0^\text{dur} = \sigma_\text{w}^2 / (\sigma^2 - \sigma_\text{w}^2)\) = 1.0 games, so a pair's own durations soon dominate. For a pair with \(n\) normal games of mean log-duration \(\bar{x}\), the posterior mean is

and a simulated game's log-duration is drawn with the posterior predictive spread

the scatter within the pair plus the remaining uncertainty about its mean. For a pair without games, this is the overall \(\sigma\).

Abnormal games get a separate log-normal distribution with ladder-wide parameters only, since crashes are too rare per pair to fit.

Left: the log-durations of all real normal games, with the fitted normal distribution. Right: the same games in minutes, with the normal games of the simulation, in which each pair's games follow its own distribution.

Simulation

The simulated ladder

The simulator (sim/ladder_sim.py) is event-driven. The ladder has 12 server slots; AI Arena ran 11 matches at a time on average in this period. Whenever slots are free, the simulator asks the matchmaker for a pair of bots, draws the match's category, result and duration from the model, and starts it, until every slot is busy or the matchmaker declines. Then it jumps to the next match to finish, updates both bots' ratings with the ELO rule, and fills the free slot again.

Bots with bot data play only one match at a time, the others at most 4 at once. The real data agrees: 92% of the bots with bot data never had two matches running at once, and bots without bot data spent 99.98% of their busy time in at most 4 matches.

The simulator's clock runs on game time. In the real data, a match occupies its server for a median of 0.53 times its game time, so a simulated day corresponds to about that fraction of a real day. This scales every matchmaker's throughput alike, but absolute numbers per day are off by that factor.

Experiments

Each matchmaker runs 8 times with different random seeds. A run plays 50,000 matches, about 578 per bot. Ratings start from the bots' final AI Arena ratings of the season, as they would if the ladder switched matchmakers then. The first 10,000 matches of every run are a burn-in, during which the ratings adjust to the new matchmaker; all metrics except the rating trajectories leave them out. Tables show the mean over runs and its 95% confidence interval.

Ground truth and metrics

Because every result is drawn from the model, the simulator knows each pair's true expected score: the probability of a win plus half the probability of a draw, counting time-limit games as draws and abnormal games as losses for the bot that crashed. From these we compute each bot's true rating: the ELO-scale ratings that best explain all true expected scores, with every pair weighted equally. As match-ups aren't transitive, no set of ratings reproduces every pair, but the true ratings are the best single number per bot. Their standard deviation is 206 ELO points.

The metrics are:

• Favourite's true expected score: for each match, the stronger side's true expected score, where 0.5 is a coin flip and 1 a certain win. The lower its average, the closer the matches. A match is lopsided if it is at least 0.9.
• |ΔELO| at dispatch: the rating difference between the two bots when the match starts, on the simulation's own ratings. This is what rating-based matchmakers try to keep small, but it is measured on ratings that the matchmaker itself shapes, and ELO predicts bot matches poorly, so it isn't a goal in itself.
• Rating accuracy: how well the simulation's ratings match the true ratings, as a rank correlation (Spearman ρ), a root-mean-square error in ELO points, and the ratio of the two spreads. A spread ratio above 1 means the ratings are stretched: strong bots overrated, weak bots underrated.
• Fairness: the number of matches each bot plays, through the least-played bot and the standard deviation across bots.
• Variety: how many distinct opponents a bot meets, and how soon it meets the same one again.
• Server time in use: the share of the servers' time spent playing matches, which drops when a matchmaker leaves slots idle.

Matchmakers

Each matchmaker (matchmakers/*/matchmaker.py) is asked for one pair at a time. It sees the bots' current ratings, the matches of the last 24 hours, and which bots are free, and it returns a pair, or nothing to leave the remaining slots idle for now.

Random

Picks two free bots uniformly at random. It has no memory and ignores skill: it is the baseline that every other matchmaker should beat on closeness, and it shows how much variety comes for free.

Round-robin

Modelled on AI Arena's current system. Bots are sorted by rating into 7 divisions of equal size, as many as the real ladder had at the end of the season, and in each round every pair of bots within a division plays once. Each round re-assigns the divisions from the ratings at its start.

Rounds run as they do on AI Arena (sim/rounds.py). Within a round, matches between two bots with bot data start first, then those with one, then the rest, in random order within each group. That is the order in the season's data, where the three groups started on average 21%, 58% and 88% of the way through their round. As soon as no match of the current round can start, because the rest have started or wait for busy bots, the next round starts, while the last matches of the previous one still run. In the data, 100% of rounds started before the previous one had finished, and never more than 2 rounds ran at once; the simulation allows 2 as well.

Every bot gets the same number of matches per round, but nobody plays outside their division.

Rung

Keeps rounds but drops fixed divisions. At the start of each round, every bot picks 8 opponents at random from its rung, the 20 bots nearest to it in rating, and 2 wildcards at random from the rest of the ladder. The round's schedule is the set of all pairs picked, so each bot plays at least 10 matches per round, and more when other bots pick it. Bots in the middle of the ladder sit in more rungs than those at either end, so they get picked more often. Rounds run as for round-robin.

Stochastic

The proposal for a ladder without rounds or divisions. Whenever a slot is free, it scores every pair of free bots and samples a pair, favouring high scores. The score combines three of the four goals:

The fourth goal, matches spread evenly over time, is covered by the fairness term: a bot cannot sit idle for long without falling behind on games.

Bookkeeping at dispatch

Several slots are filled at once, before any of the new matches finishes, and bots without bot data can play several matches in parallel. Everything that steers the next choice (games since two bots last met, games in the fairness window, rematch blocks) is therefore updated the moment a match starts, not when it finishes. Otherwise the same pair, or the same underplayed bot, would win every free slot in a row.

Rematch prevention

Before scoring, a pair \((A, B)\) is excluded if it is being played right now, or if either bot's most recent match was against the other. This prevents back-to-back rematches regardless of score: a bot must play someone else before it can face the same opponent again. The rule is asymmetric: after \(A\) plays \(B\) and then \(C\), the pair \((A, B)\) stays blocked until \(B\) has also played someone else. This avoids a failure mode in which two isolated top bots build up the same fairness debt and keep being paired on that alone.

Skill matching

Matches between similarly ranked bots are preferred. Rather than the rating difference, the score penalises the rank difference, with a Cauchy (Lorentzian) kernel:

where ranks follow the current ratings (best = 0) and \(\tau\) is measured in rank positions.

Why ranks, not ratings. Ranks make the skill tolerance adapt to how densely the ladder is populated. Where it is dense, adjacent ranks are close in rating; where it is sparse (say, an outlier at the bottom), adjacent ranks may be far apart in rating, but there is nothing closer to play. A penalty on the rating difference would instead push the outlier to play its single nearest neighbour over and over.

Why a Cauchy kernel. It has a flat top, so adjacent ranks (1, 2, 3, …) count as roughly equal and the top bot isn't locked into always playing rank 2; it is smooth, with no cusp that would favour a difference of one rank over two or three; and its heavy tail keeps rank differences of 10 to 30 distinguishable from cross-ladder pairings. A Gaussian decays too fast, leaving far-apart pairs with a skill score near zero and letting variety dominate; an exponential has a cusp at zero. Values lie in \([0, 1]\).

Fairness

Bots that have played fewer games are prioritised. Let \(g_i\) be the number of games bot \(i\) completed in the last 24 hours plus those it is playing right now, and \(\bar{g}\) the mean over all bots. Each bot's normalised deficit is \(d_i = \max(\bar{g} - g_i, 0) / \bar{g}\), and

Clipping at zero means a bot above the average neither adds to nor subtracts from a pairing's fairness score. The root mean square rewards pairing two underplayed bots more than pairing one, and dividing by \(\bar{g}\) keeps the term on a scale comparable to the others. Bots that have just joined the ladder get a temporary boost, since their count in the window is low. Values lie in \([0, 1]\).

Opponent variety

Repeated match-ups are discouraged. Let \(a_{AB}\) be the number of games \(A\) has played since it last faced \(B\) (all of \(A\)'s games, if they have never met), and \(b_{AB}\) the same for \(B\):

where \(\lambda\) sets how quickly a past match-up is forgotten. The geometric mean gives partial credit when one bot has moved on and the other hasn't, while still requiring both to have played others. Values lie in \([0, 1]\).

Softmax sampling

Rather than always taking the highest-scoring pair, the matchmaker samples a pair with probability

where \(T\) is the temperature: \(T \to 0\) recovers the pure maximum, larger \(T\) adds randomness. This has two benefits. With a pure maximum over thousands of candidate pairs, whichever pair edges ahead by the slightest margin wins, so a small change in the weights can flip which kind of pair wins every time, and that flip is amplified over thousands of matches; sampling lets near-ties be decided by chance. And it lets every near-optimal pairing happen sometimes, which adds variety without extra weight on the variety term. \(T\) should be small compared with the score differences that matter: large enough that near-ties (\(\Delta\text{score} \lesssim T\)) are randomised, but not so large that far-apart pairs with high variety compete with close ones.

Parameters

The components have different scales, so the weights also normalise them and can't be compared with each other directly. The skill weight sets how much closeness counts against fairness and variety anywhere on the ladder, while \(\tau\) sets which rank differences count as close; lowering the weight is the more direct way to trade closeness for the other goals. The matchmaker was designed with a skill weight of 0.15. Lowering it to 0.1 gave up some closeness for ratings that settle instead of stretching, more variety and more even match counts, so that is the setting shown here.

Higher temperature

stochastic-temp-0.02 uses the same parameters but a higher temperature, \(T =\) 0.02, so pairs that score a little below the best are picked more often. It shows what more randomness buys and what it costs.

Results

Summary

The table keeps one metric per goal, plus both views of rating accuracy, which pull against each other: matching bots more closely sorts them better but stretches their ratings. The other metrics are in the sections below. Each row is coloured by where a matchmaker's value lies between the row's best (green) and worst (red), so values that are close get similar colours.

Findings

• Skill matching makes matches closer, but much less than the rating gap suggests. With random pairing, the favourite's true expected score averages 0.779; round-robin, rung and stochastic bring it down to 0.682, 0.701 and 0.728. On the simulation's own ratings, the gap between paired bots shrinks far more, from 243 to 74 ELO points. The ladder itself sets a floor: even bots whose true ratings are within 50 points of each other give the favourite 0.65 on average, where ELO would predict at most 0.57, because so many match-ups are one-sided.
• Pairing by rating stretches the ratings. Because neighbouring bots beat each other more decisively than their rating gap predicts, ratings learned mostly from close pairings spread out too far. Under rung and stochastic, the ELO spread levels off, at 1.23 and 1.20 times the true spread (random: 1.04). Under round-robin, whose matches are the closest, it is still growing at the end of the run, at 1.43, and so is its rating error. The real AI Arena ratings are stretched too, 1.21 times the true spread; since the model's prior is built from those ratings, this fits but doesn't prove the picture.
• Every matchmaker keeps the servers busy. Because the next round starts while the last matches of the previous one finish, as on AI Arena, round-robin and rung keep the servers as busy as random and stochastic do, 99.98% of the time. Random completes the most matches per simulated day (1,208, against 1,187 for stochastic), but only because its lopsided games end sooner: they last 14.3 minutes on average, against 14.6.
• Rung spreads matches unevenly. Bots at either end of the ladder are in fewer rungs: the least-played bot gets 330 matches, against an average of 462.
• The stochastic matchmaker keeps match counts nearly even (standard deviation 10.5) and lets each bot meet 153 distinct opponents on average, without rounds. Its matches are less close than round-robin's and rung's (favourite's expected score 0.728, against 0.682 and 0.701), but its ratings are the most accurate of the three (error 57 ELO, against 81 and 60).
• A little more randomness pays off. Raising the temperature from 0.01 to 0.02 (stochastic-temp-0.02) makes matches slightly less close (favourite's expected score 0.728 → 0.741). In return, ratings get more accurate (error 57 → 48 ELO, spread 1.20 → 1.14 times the true spread), opponents more varied (153 → 162 distinct opponents) and match counts more even (standard deviation 10.5 → 8.7). In a sweep up to \(T = 0.1\) (not shown), higher temperatures moved closeness and rating accuracy further towards random pairing's, while match counts became less even again.

How close are the matches?

The favourite's true expected score over all matches after the burn-in, all runs pooled. Mass near 0.5 means close matches; the dashed line marks the lopsided threshold.

Rating gap of matched pairs

The rating difference between paired bots at the start of each match, after the burn-in, all runs pooled. This is what rating-based matchmakers aim to minimise, measured on their own ratings; compare it with the true expected scores above to see how much of it turns into closer matches.

Rating accuracy over time

How well the simulation's ratings match the true ratings over each run: the RMSE, the rank correlation, and the ratio of the ratings' spread to the true spread. Lines are means across runs, bands 95% confidence intervals, and the dashed line marks the end of the burn-in. Ratings keep fluctuating from game to game (\(K = 16\)), so the RMSE cannot reach zero. A curve that keeps rising instead of levelling off means the ratings drift away from the truth; Spearman ρ is insensitive to the ratings being stretched, the RMSE and the spread ratio are not.

Matches per bot

Each dot is one bot in the first run, after the burn-in: the number of matches it played, and their average duration. A narrow cluster means matches are spread evenly across bots; a wide one means some bots are dispatched more often than others. A difference in height between matchmakers would hint that one favours short or long match-ups.

Opponent variety

How many different opponents a bot meets: the distinct opponents it has played, against the number of games since the burn-in. Lines are the median over bots and runs, bands the interquartile range. A line that flattens out means bots keep meeting the same group of opponents; under round-robin, that group is mostly the bot's own division. Random pairing climbs fastest, but by pairing bots of any strength (see How close are the matches?).

How soon a bot meets the same opponent again. For every repeat meeting, we count the games the bot played since the two last met; each line shows the share of repeat meetings that came within that many games (note the logarithmic axis). Round-robin's steep rise is the length of a round: two bots in the same division meet once per round. The stochastic matchmaker never pairs two bots that have just played each other, and its variety term pushes repeats further out.

Who plays whom

How often each pair of bots played in the first run, after the burn-in. Rows and columns are sorted by true rating, strongest first; the number in brackets is a bot's true rating relative to the ladder average. A band along the diagonal means skill-matched pairs, a uniform colour that everyone plays everyone.

Limitations

• Game time, not server time. Matches occupy a server for their game time, while in reality they take about 0.53 times as long (see above).
• Fixed probabilities. Every pair plays with its posterior mean probabilities; the uncertainty of the fit doesn't carry over into the comparison. Drawing one set of probabilities per run would carry it over.
• A snapshot of the ladder. The model treats the bots as unchanged over the 115 days of data, although bots get updated: between the first and the second half of the data, pairs' win rates change 2.5 times as much as chance alone would explain. A shorter window drifts less but leaves fewer games per pair; data/from_recap.py --since sets it.