How Total War AI is evolving to be more fun – and how we’re stopping Skynet

Total War without opponents – someone, indeed, to wage total war against – really would not be the same game. While multiplayer is a host of delights, we don’t all always have a friend available to be an Orcish punching bag at two in the morning after a couple too many grogs. That’s where artificial intelligence steps in.

We sat down with Dr. Tim Gosling, one of our AI top dogs at Creative Assembly, to talk about the goals and evolution of AI in Total War. When should it be hard, when should it be fun, and when has it tried to overthrow humanity?

CA: Hi, Tim! First of all, could you please introduce yourself?

Tim Gosling: Hi! I’m Dr Tim Gosling, the AI Technical Director for Total War. I head up the team that develops the AI systems that are your opponents in both the Campaign and Battle parts of a Total War game. I started at Creative Assembly in 2006 working on the new Campaign AI for Empire: Total War, so I have been at this for a while now!

Before anything else, I must emphasise the team effort involved in developing the AI for Total War. Total War as a series relies heavily on its AI systems and I’m lucky to work with a dedicated and capable group of programmers, designers and QA specialists to deliver that AI. They do an amazing job in the face of a challenging problem.

What do you mean when you talk about “AI” in the context of Total War games?

AI can mean a lot of things to different people, but in the context of Total War I’m talking about AI that plays the game much as a human player does. So, a form of behavioural AI.

A Total War campaign will often have dozens of factions – human players control only a small number of these, so the majority of factions are controlled by an AI. Getting the AI right for all those factions is really important for how the campaign plays out and the overall experience in the game. In Battle, unless you are playing multiplayer, you’ll be playing against an AI and you want a challenging and fun opponent.

So, when I am talking about AI, it is mostly going to be about how to play a Campaign or Battle in a way that is interesting to the player. Things like pathfinding, which are important, are more of a service we draw on rather than part of the AI itself in this context. There are a whole host of further AI topics that are relevant for game development, such as various forms of procedural content generation, but that mostly isn’t what I deal with. Still, happy to talk about it at length though, perhaps another time!

Tell us a bit about the AI used in the campaign view of Total War games – what does it do and how does it do it?

Sure, so the first thing to say is that this is a really challenging problem for an AI. There are a lot of moving parts to a Total War campaign game and there is a lot of interconnection between different systems. We often talk about ‘branching factor’ in AI, essentially the number of options a player has at a given point in time. With a Total War campaign, even at the start, with all these systems and interconnections this can be pretty huge.

For comparison, this branching factor is far higher than for chess or Go. In the latter case AI researchers backed by lots of specialist hardware were able to finally beat the best human players a few years ago with the Alpha Go system. This was a big deal as they managed it sooner than anyone thought possible. Go is played by two players on a 19 by 19 board, where both players can see everything, and each player takes turns to place black or white stones. This sounds simple enough but is notoriously complex to actually play and there is a lot of subtlety and long-term planning required to do well (some of my team play).

A Total War campaign, by comparison, may have millions of positions, hundreds of settlements, armies (of various compositions) and agents and dozens of factions playing, each with various diplomatic relationships between them and each able to make multiple moves per turn, all this under a cloud of uncertainty about the outcome of some moves or even where things are. To be fair, the importance of one position over another for an army in Total War will typically be less than for a given stone in Go but from a computational point of view the AI must still consider those options, if only briefly, and there are a lot of them, and that’s just the army positions.

Tackling this using regular desktop hardware is interesting. The Campaign AI’s job is to make sense of all this and play reasonable moves for each faction – and it has to do this quickly as we don’t want to keep the human players waiting. Beyond making generally reasonable moves we want it to also be interesting and fun. So different factions should have different playstyles, or personalities, which can shift between (or during!) campaigns and should give the player different sorts of challenges. We also don’t want to be unfair, either by cheating (we try and play the same game as the player as much as possible), or by just having everyone gang up on the players (which is typically not fun for most people).

So how do we tackle this massively complex problem in a timely manner while retaining some personality and providing a challenge? Divide and conquer!

We essentially break the problem up in to a set of core sub-problems and deal with these separately and then provide a coordination layer on top in terms of contextual awareness and budgeting. So, each individual system is pulling in the same direction with respect to what is going on in the world and operates within its budgets but pretty much focusses on its one area of expertise – with the occasional additional hint passed between them.

By carefully ordering which sub-systems get to decide what to do first, we can have a single faction operate effectively without having to have every choice cross-referenced against every other choice. This reduces the overall complexity enormously. The individual sub-systems handle things like diplomacy, constructing buildings and recruiting and moving forces, anything that requires a player choice, and each one is free to solve its particular sub-problem in its own way.

So, for instance, we have a template-like approach for building construction, but we use a detailed search for figuring out where and how to move forces. The individual sub-systems respond to what is going on within the game but their response is mediated by control data configured by our AI designers. The design team is able to specify various AI personalities and these are set up such that the different sub-systems play in certain ways for those personalities given the game context. It is a big task and crucial for getting the feel of an overall campaign and individual factions right.

I’ve mentioned fun and personality a few times and it is important to re-emphasise that unlike a lot of academic AI or competition AI, the AI’s role as a player is not so much about winning as it is about entertaining. The AI can’t really win, only players can do that, so the AI’s job is to make that journey interesting – it might even beat the player (the AI can’t win but players can lose) but hopefully that is entertaining too.

This can lead us to what might seem like unusual choices. On the whole, the AI can’t be as good as a really good human player, partly because of the computational constraints, but also because some types of reasoning are hard for current AI systems (and are often active areas of AI research). However, in some instances it absolutely can outplay a human because some problems are just easier for a computer to analyse. In these cases though, because a player can’t outplay the AI, it is typically super frustrating to be beaten in this way – we then have to consider deliberately making mistakes to make the AI feel more human and believable and less frustrating to play against. This is just as true in the Battle as the Campaign, we’ll get back to that later.

From a further technical perspective there is a constant balancing act between AI turn times, efficacy, memory use, and so on. One of the big wins for us as a team was the move to 64bit which opened up more available memory that we could then use to help speed up our processing, a large path distance approximation cache helped enormously.

Conversely, multi-core processors have had less of an impact than you might imagine – we can do some amount of processing in parallel, and this has helped, but a lot of interactions we deal with are inherently linear which limits how much benefit we see. We are always looking for opportunities to improve things though, and we do make gains as demonstrated with the improvements we made to the Warhammer II end-turn times during the Shadow and Blade update.

What about the battle AI? How does it differ?

Much like the Campaign AI, the Battle AI is responsible for playing like a human player. To this end it controls its available set of units, issuing orders and trying to win the battle (the Battle AI can actually win!). The movements of individual soldiers fall below the Battle AIs control and are considered part of the simulated environment.

Compared with the Campaign AI, the Battle AI has a very different sort of challenge. Instead of lots of factions vying for control over a large map with lots of assets, turn by turn, the situation is more constrained. However, now it is highly dynamic and real-time, with lots to consider in terms of unit positions, dispositions, types, and abilities alongside concerns about terrain and so on. You need decent strategic and tactical awareness to be able to play well. I talked about the Campaign AI having a large number of options to deal with, the same is true for the Battle AI, though it might not be as obvious, and timing is often as much a factor as anything else.

The nature of a Battle and how it is played means there are some advantages here, if you have a relatively expensive calculation you can in principle have this run over several seconds ‘time-slicing’ it to get an answer a bit later while still doing other things. This won’t work very well for something you need an answer to immediately, but it can help for decisions that can be made more slowly or where you can make a decision incrementally. Other things are more difficult though, the Battle AI has to constantly be on the look-out to counter its opponent’s moves – the Campaign AI doesn’t have to worry about an enemy army moving while it is taking its turn.

The way a battle can be constantly shifting also makes things like caching information more tricky as the validity of any decision, path or so on is constantly subject to change. Where decisions can’t be time-sliced we have to be super aware of the efficiency of what we are doing and keep computational costs down so as to not hold up the flow of play at all.

The way the Battle AI works is therefore significantly different to the Campaign AI, with an emphasis placed on slicing up decision-making in to the different strategic and tactical layers, from really high-level strategic decisions working down to individual choices about what units should be doing when they face off against other units. The layers in between break down those top-level decisions in to various objectives and then group units up appropriately to complete them. Additional sub-systems take on some specific responsibilities though, a bit like with the Campaign AI, with specialist components looking at things like special abilities or siege equipment utilisation.

There are further parallels that can be drawn too, with problems such as resource assignment. How you allocate a unit to a task is different to how you allocate a Campaign army to a task, but there are important similarities and useful cross-learning to be had in how to do that effectively. There are various other unique challenges too, things like how do you keep an effective battle line in an open field and how do you balance the look of how the AI is playing versus what might be most tactically expedient? There is an element of communicating and enabling gameplay through unit positioning with the Battle AI (and systems below the AI) that is less pronounced in the Campaign.

In some respects the Battle AI has had to evolve more aggressively than the Campaign AI systems – the introduction of more complex settlements, other types of battles and a wider diversity of unit types has meant a lot of work to add appropriate system layers to handle these things. The settlements in particular have required complex additional layers to understand and respond to threat locations, breaches in defences, fall back positions and so on. Correspondingly, the AI has to understand how to get into and capture settlements so needs to figure out how to effectively distribute or concentrate its forces to best effect and to use what units when, where, and how.

How has the AI evolved since you have been here?

I started at CA working on Total War: Empire so I don’t have a huge insight in to the AI for games prior to that. With Empire we started with a pretty clean slate and one of the big differences was to more strictly separate the AI out from the rest of this game. Prior to this it was common for AI behaviour and information to sit directly alongside the gameplay elements – this was very efficient but had the downside of making it easier to trip over dependencies on either side while making changes. Less of a problem with a smaller game, code base, and team but more of a problem as things got more ambitious, and we were being ambitious with Empire.

That separation allowed teams to operate more independently with less of a risk of disrupting each other, though you of course must still work together closely. We continue with this practice today and it has paid dividends.

Since then, and in general, the complexity of the AI systems has increased with the increase in complexity of the game itself and as our understanding of the experience we are trying to deliver has matured.

There are a lot of different changes I could talk about but for the Campaign AI possibly the biggest thing, since Empire, was the shift on Rome 2 to using Monte Carlo Tree Searches (MCTS) coupled with a task system to move around forces and agents. An early version of this was necessitated by gameplay changes in the Shogun 2 – Fall Of The Samurai DLC which more tightly coupled land and naval gameplay.

When Rome 2 came around, we expanded on this and the newly minted Task Management System (TMS) became the core of a much more effective way of managing our campaign map movement. Other systems have been added or improved as well but that was a big shift. Alongside the purely technical changes has been a desire to provide ever increasing expressive power to the design team, allowing them to more effectively craft how the AI plays. So, a substantial effort has gone in to trying to define ways in which to do that alongside the techniques we want to use to solve the various AI problems.

This has led to the AI Personalities system which lets us define sets of control data for an AI at different levels of resolution depending on what is more or less important in each game. For Warhammer III, we have another new system that is supplying a lot of expressive power to the design team and is letting us interact more effectively with some of the new gameplay features. Moving forwards, we plan to increase that expressive power further while adding to and improving the technical solutions available.

The Battle AI has likewise had to adapt to major shifts in gameplay. Empire introduced battles at sea, for instance, which effectively necessitated a new AI variant and as mentioned before, more complex settlements have also required additional layers of support to understand that environment and respond accordingly.

For settlements it’s not just navigation – we also have to consider things like if it is worth defending the walls? If we decide yes, when should we abandon them? And so on. Things like this add a whole host of complexity. The introduction of out-field bombardment again added challenges and the introduction of magic and flying creatures with Warhammer mixed things up further.

The latter presents an extra challenge as the type of movement is just entirely different to what the system was dealing with before and interacts in new ways with existing features. Those walls we were worrying about before might suddenly mean less if things can just fly right over them and similarly having those capabilities gives new tactical flexibility.

So, the addition, removal or change of any given gameplay feature is not isolated to itself, it is also about how it interacts with everything else – that’s challenging from a purely gameplay perspective and tracking that with respect to the AI is likewise pretty difficult.

What are some of the biggest challenges when it comes to working on the AI for Total War?

I talked before about balancing efficiency, speed, memory consumption, and fun, beyond simply ‘how do we play’ – which isn’t trivial, it’s the biggest challenge.

There are always plenty of questions to answer – how much delay to the end turn time do we accept in the Campaign AI to improve the challenge for the player? Do we allow the Battle AI to react instantly to perfectly counter a player’s special move? What experience are we actually trying to deliver with respect to a given character, unit, faction, feature or campaign? What level and type of control does the design team need to let them express those differences in play and still keep the AI effective? Can we afford more memory for a cache of solutions to a problem versus using that memory for something else? What amount of logging do we need to understand what the AI is actually doing and why?

There are lots of day-to-day questions like this and it is a constant balancing act. Then, yes, there are questions about what sort of technique is appropriate to solve this type of problem. Do we use some sort of search? Could it just be purely random? So, it is good to know what options are available and upcoming, one of the reasons we keep an eye on the academic literature and participate in conferences!

Are there any examples of the AI looking silly but actually being very clever?

There are absolutely times when the AI is making reasonable moves but the player might not see why, things like walking calvary through spears to avoid charge reflection damage for instance.

There are also quite a few occasions where the AI is making deliberate mistakes, and these will seem more or less obvious depending on the experience of the player. This is usually driven by the trying to be fun. One example of this is movement ranges, one thing the Campaign AI is good at is knowing where everything can move to precisely – this means it can do things like place an army exactly outside your attack radius. This is incredibly frustrating for players, an example of where the AI can outplay a player.

So, entirely to relieve this annoyance, we had to add in an error factor, forcing the AI to sometimes make mistakes and allow itself to be caught. Lots of positive feedback from doing this but from an AI perspective not really a sensible move (also, as a team, we tend to root for the AI so we were a bit sad about this despite it being the right thing for the game).

Similarly, for Three Kingdoms and with reference to talking about challenges, we had a problem in the Battle AI where it could perfectly counter arrow fire by using appropriate formations. Is this a problem? Well, it is if you’d like your players to feel like they are being effective but also want them to feel like the AI knows what it is doing. The answer in this case was that on easier difficulty levels the AI wouldn’t shield properly from arrow fire giving players a “hah, I got you!” moment.

On higher difficulty levels, those more experienced players would tend to think “daft AI, why didn’t it counter my arrow fire?” So, on those difficulty levels, that is what we did. That gives both groups of players a more satisfying experience but does mean that in the grey zone between them people can be confused or annoyed. We have the same things the other way around by the way, with the AI shooting at units which are defensive on lower difficulty settings (‘cool, I picked the right stance’) versus not doing so on harder settings (‘why is the AI shooting at me when I turtle?’).

Do you have any particularly funny behind-the-scenes stories on making the AI do what it’s meant to?

We’ve had a few funny moments when the AI went a bit rogue and did things it shouldn’t. An example would be making use of features that were not meant to be there and there was no UI for – the AI while playing the same game as the player doesn’t literally hit the UI buttons, it has its own interface to the game. Been a few times we’ve not properly tidied up that interface and the AI has found itself with options the player doesn’t have, certainly surprised QA!

We’ve also had it decide the player needed to do more on their turn and take over – people started wondering why their armies had moved. Put a quick stop to that, no going full Skynet for us. We also had a case where it played one feature so well it ended up convincing the design team to take it out as it would be too powerful for everyone.

We can also be a bit of a funny bunch as a team. On one occasion everyone decided to bake cakes for the team at the weekend but not tell anyone else – not the healthiest start to the week, tasty though. We also have a reputation for being heavily armed – but it is only one or two nerf guns each, the axe was a gift, and the swords and armour are usually kept at home (this is normal for a company working on historical war games, right?). We also have a habit of using sci-fi quotes to catch people’s attention in error messages, this has worked pretty well!

In all seriousness, it’s a great group and they’ve been really supportive of one another over the lockdown – from bringing groceries to those who’ve needed them to taking socially distanced walks together when allowed. I look forward to getting back to our wide-ranging morning tea chats in person.

What areas of game AI excite you the most in relation to how they could be applied to Total War?

I think AI in general is an exciting place to be right now – there is increased awareness of its potential and games are just one part of that. I’m obviously biased here, but I think AI is a crucial frontier for games. We have ever-improving graphical fidelity in games which make them look great, but I think how things behave within those games is the next thing. You want believable characters that respond reasonably to a wider range of situations. What are the techniques we can use to support that objective?

I come from a machine learning background and it is great to see that getting more attention. However, our ability to use machine learning is somewhat constrained by the dynamism of game development and the difference in objective, we don’t just want the AI to win, we want it to be fun. Fun is a bit nebulous so having a system learn how to do that is tricky, though there is research in this area! Still, all that said, I’m very much interested in machine learning as an area along with any other techniques for helping the AI play a better or more interesting game.

Beyond in-game behaviour there is a lot of fascinating work in what might be considered tooling or utilities. Procedural content generation can help artists and designers by acting as a huge lever to their productivity. Similarly, systems for facilitating QA or other areas are super useful. There is also a lot of interesting work going on in speeding up a lot of in-game tasks by using machine learning to develop cribs for doing things such as physics more quickly. This all leads to a richer experience for players.

Thanks for your time! Any final words?

No problem, been great to have an opportunity to talk about our AI systems!

Also, if anyone reading this has an interest in AI and games we are always on the lookout for new team members, so keep an eye on our recruitment page!