By Blackmead · Development milestone:
A search bot handles the fights. A reinforcement-learning deck builder handles the choices between them. Here is how automated runs help us find balance problems, without pretending they can tell us what is fun.
Send in the tireless volunteers
A rule changes. A number moves. Somewhere, a combination of cards has just become a much better idea than it was yesterday. Finding that by hand takes time, especially when six heroes have different ways of getting through a run.
Our baseline solo sweep plays 200 runs for each of the six heroes: 1,200 runs before the extra party, mastery and oath checks. The point is to get more evidence about what changed, rather than asking one memorable victory or one miserable defeat to speak for the entire game.
The work is split between two kinds of decisions: playing a fight and building the deck between fights. The video above walks through both.
Search inside the fight
The combat bot uses search to choose its plays. Crucially, it does not peek at the draw pile. A test that wins by knowing hidden information would give us a misleading picture of the decisions available to a player.
It takes a shortlist of legal plays, tries each on a copy of the fight, resolves the enemies' announced attacks and looks another turn ahead. Then it makes one choice in the real fight and searches again. The copies shuffle hidden draws rather than handing it tomorrow's cards.
That search bot is one part of the testing setup. It is not the same thing as the reinforcement-learning deck builder. Keeping the two roles clear matters: a good turn and a good reward choice solve different problems.
Learning between fights
We also have a hand-written deck builder, with rules we chose ourselves. The learned deck builder uses reinforcement learning to make the decisions between fights.
Training starts by imitating choices from the hand-written planner. Reinforcement learning then improves the policy through whole runs, using the score, islands cleared and finishing the saga as its reward. Its job includes reward picks and skips, forge work, shopping, events and the Saga Stone.
Comparing those approaches helps expose the assumptions inside our own rules. A card or hero that looks weak under one way of building a deck can perform differently when the builder learns another approach.
The video follows what the learned builder does differently. The useful result is a new question to investigate: did the rule change alter the game, or did the agent become better at navigating it?
When the tester is still learning
Two findings in the demonstration make that distinction concrete. The testing exposed a gap between heroes that the old numbers had hidden. It also showed a hero that only looked broken because the bot had not finished learning how to play it.
That hero was the Pyromancer. In one evaluation on the same 200 held-out seeds and the same game rules, its full-saga completion rate climbed from 1% at training update 200 to 67.5% at update 360. More training changed the verdict without a balance patch. These are bot results from that test snapshot, not promised player win rates.
Those lead to different responses. One points towards a balance question. The other warns against changing the game to compensate for an unfinished tester.
A low score is evidence to inspect, not an instruction to buff a hero immediately. We need to understand the decisions behind it before deciding what the game should do differently.
The cards it refuses matter too
The learned builders do not simply take every reward. In the recorded evaluation, the learned Vanguard skipped 41% of card offers, compared with 10% for the hand-written planner. It also spent more Hearth visits on forge work and fewer on resting.
That is a useful clue about what the rules reward. It is not a commandment for players, and it does not mean every hero should draft the same way. The point is to examine decisions that produce a result, rather than staring only at the final score.
We evaluate on seeds kept separate from training and compare builders on matching rules. When the game changes, we must also check whether the tester has learned the new decisions. A fair comparison needs both a known game and a known player.
People still make the call
The bots help tell us what changed and by how much. They do not decide whether a turn was satisfying, a card was clear or a difficult fight was worth another attempt.
That part comes from people playing. Your first run, the explanation you needed, the moment a party plan clicked: those are different kinds of evidence from an automated result.
Watch the testing walkthrough, then try the public Windows build. The tireless volunteers can keep running. We still want to hear from the people holding the cards.