You may be surprised to hear that not all studios reuse their AI code from one game to another. Low-level systems are increasingly being reused, but when it comes to decision making and game-specific logic, that often gets thrown out. One way to approach this strategically is to update the AI post-release with the next game in mind...
In this 1h30 interview with Mike Robbins, Gameplay Engineer at Gas Powered games, you'll hear about the AI of SUPREME COMMANDER 2 and its many updates since release. As well as learning about all the different systems of the RTS AI, you'll hear how Mike took into account players' feedback, and polished the strategy of the opponents. You'll also discover about the new neural networks that make the decisions for the game's units!
Mike Robbins is Gameplay Engineer at Gas Powered Games, currently Lead AI on KINGS & CASTLES. He worked as Content Engineer on SUPREME COMMANDER 2, and was responsible for improving the AI of the game post-release via numerous patches. He was previously known as "Sorian" in the mod community, famous for his AI mod of SUPREME COMMANDER 1.
Alex This evening we have a real pleasure of welcoming Mike Robbins. So Mike’s an associate content engineer at Gas Powered Games where he works on Supreme Commander 2 and I guess another title in the pipeline at GPG. So welcome Mike it’s a real pleasure to have you here.
Mike Thanks, thanks for having me
Alex So in this interview we’re going to go over all the details about Supreme Commander 2’s AI. Now Mike joined the team at Gas Powered Games after the release, and he was involved in improving the AI; so we’re going to focus this interview on how he improved it after the launch.
So for those of you that aren’t familiar with the game, Supreme Commander is like this fast-pace kind of action-oriented strategy game; I bet it’s got some elements of RPG style, a bit like DemiGod in a sense because you have this Avatar here that represents you in the game. So there are some elements that reminded me a lot of DemiGod. So it’s a really interesting hybrid there.
There’s a first question Mike I’d like to ask you, it’s not very common to do post-release updates on the AI in the games industry these days; so what makes Supreme Commander 2 different and what convinced Gas Powered Games to do this?
Mike Well, the AI when it was released was fairly broken. If you interrupted the AI while it was doing its initial building well it would just completely stop, I mean, it was up to par. So they decided to bring me in to fix the bugs with the AI, then they basically gave me “carte blanche" to do whatever I wanted above and beyond that to make the AI better.
Alex I find actually most RTS games at launch have an AI that’s not necessarily up to par, according to the AI programmers, they would happily admit that! But it’s quite rare to see a team that’s willing to put that amount of dedication into the AI; so we’re going to go into that in a bit more detail.
So tell us exactly what you wanted to achieve with the AI. What were your goals, and what did you want to improve when you were given that carte blanche as you said?
Mike Well I wanted to make the AI more responsive to the player. If you started building nuclear launchers then the AI would realize that and start building anti-nukes to counter it.
If you start building the artillery then the AI should start building shields so that you’re not decimating its base with artillery shells that it’s doing nothing about; just make the AI feel more responsive and as human-like as possible. While fixing all the various bugs that may cause the AI to not behave.
Alex That’s something I noticed on the release updates of Steam - there’s like a long list of little fixes, and a long list of requests and features and various little fixes, all those things that come from the community, from you, from your producers, your testers. Who decides on what you put into the updates of the AI every month, or every update?
Mike While I’m in the game engine fixing other things, then when I go to test it or run matches with the AI, I’ll notice little things here and there, and when I fix them, I’ll bring them up to our manager; as long as they’re fairly small, not something that’s going to completely break everything if it goes wrong, they’ll usually let me go ahead and sneak it in.
Other fixes come from comments from the community, comments from the testers because they’re constantly running against the AI, either for fun or for testing purposes. So the fixes come from all over, and as long as it doesn’t take too much time and is not very risky, then the producers don’t have a problem with me adding them in.
Alex From an AI programmer's prospective, it’s a really wonderful position to be in when you have the rest of the game engine that’s kind of stable, and it works, and you can just focus on just improving the AI and tweaking that. It must have been quite an exciting few months working on this!
Mike Yeah, kid in the candy store syndrome totally.
Alex So for those of you who don’t know, Mike came from a quite heavy moding, he did lots of mods for the original Supreme Commander and the extension to that; so he built up quite a following within the community for his work on most, original two games.
That’s why Gas Powered Games brought him on board to work on the AI for Supreme Commander 2. So he had a lot of experience on polishing the AI techniques, which those of you that are looking for a good way into the game industry; that sounds like a really good way to approach it.
So in this interview we’re going to talk about four different things : first we’ll start off with an overview of the high level architecture, then we’ll go into the managers, let’s say the different components at the high level from a more strategic perspective –
then we’ll go into the lower levels; stuff at the platoon level and we’ll find out more about this famous neural network that you talked about in that promotional video that you released recently; then finally we’ll talk about little things, workflow, and that will be a good time to ask Mike any questions.
Ok let’s dig straight in! Part one is about the architecture. So why don’t you explain to us the system that is currently in place for the AI in Supreme commander, and then we’ll look at the differences that you brought into the architecture and what was there originally. So let’s start with where it is now.
Mike It’s very similar to what it was before, there was a strategic manager before, there was a task manager before; everything is still there, just a lot of the code has been improved, the way the AI makes decisions has been improved, the way the AI identifies threats and that sort of thing has definitely been improved.
On top of that we also added the team manager, which allows the AIs to communicate with each other, call for help if there’s a big blob of units coming and they don’t have enough defense there to handle it; then they’ll call their allies for help and ask them to send units over to take care of the threat or assist them in defending their base.
If they start getting hit with artillery they’ll let all the other AIs know “hey, these guys have lots of artillery, we should all build some shields in our base" and that sort of thing.
But the strategic manager is basically at the heart of the way that the AI works: it manages the economy manager, the team manager, the event manager, the condition manager, which is what’s responsible for deciding what the AI does and when.
That’s all still there; there’s just been a lot of improvements made in a way the AI makes all its decisions.
Alex Let’s go into a bit more detail about each of those managers within the top layer, like the strategy layer, let’s talk about them one by one. Why don’t you give us like an overview of each one, like a sentence or so from left to right, to give people a bit of context of what each of these managers does?
Mike Ok, the economy manager is responsible for keeping track of the AI’s economy, it knows how much resources it has, and how much goes towards which types of buildings; experimental land units, air units, it decides how much it’s going to spend on each one of those.
The team manager is responsible for managing communication between all the team members, any AIs that exist on the same team, it makes sure that all the AIs are attacking the same enemy, that sort of thing.
Event manager is for events that fire off during the game like, “I have more than a certain amount of resources" or “my enemy has this much land threat" that sort of thing. And it reacts to that condition.
The condition managers; that is for all the various build conditions. So if I have more than this much resources then I want to focus on building this type of unit, or, I don’t want to build this type of unit if it’s a water map..
And then we have all the base managers. They are what control each of the AI’s individual bases. We can have specific behavior for the AI’s main base, or the expansion bases depending on where they located, whether there’s water nearby, that sort of thing.
And they keep track of their own units, and send out platoons or whatever they decide to build, they manage their own engineers... all that.
Alex I guess the base managers are the ones that have the most functionality or responsibility it seems. So we’ll talk about those in more detail a bit later.
But is it fare to say that all these different managers at the top level are all running in parallel, so every time you update the AI then you’ll update each of these managers, and if so, how often do you update them per frame?
Mike We run through the update per tick, which is 1/10 second, then we’ll have various behaviors that would happen less often – you know like the AI’s not going to decide who it’s going to attack every 1/10 of a second.
We don’t want to do that because the AI would constantly be bouncing between different enemies. We want him to focus on somebody for a while and really inflict some damage.
So that only happens every 5 minutes, or if the enemy they’re focusing on dies. But most of everything else is every tenth of a second, sometimes we’ll slow it down if it’s like an intensive thing, we’ll do it like every second or two, or whatever.
Low Level building blocks
Alex Ok let’s drop down to the bottom level. We’re actually going to go into each of those blocks one by one using screen shots that Mike prepared, and some screen shots from my earlier attempts at beating the normal AI.
You’ll be able to see some details there. Let’s talk about the low level stuff – the more tactical, the more micro-oriented stuff. How is that build?
Mike Ok we have an overall platoon class, every time the AI wants to give units a particular behavior, they form them up into a platoon; the task manager then finds out then what they’ve been assigned to do, and assigns onto the appropriate scripted task.
For land units, it’s a neural-land attack task. What they’ll so is find a place to attack, they’ll move to that location and if they encounter resistance along the way, then they’ll send the information about their platoon and the enemy’s in the area to the neural network, then it’ll make decision based on what the neural net tells them to do.
Same thing for engineers. If a engineer has a behavior, then it’s got an item that it’s supposed to build, that item gets passed into the script, they find a place to build it, they move there, they build it. That’s what the task tells them to do.
Same thing for the Nukes, when you have a platoon of nuclear launchers, they check to make sure “ hey do I have enough resources to build a nuke, if I do build one, if I’ve got somewhere to fire it then launch the darn thing"
Alex So I presume very much like the top level, all these things are running in parallel, I guess that’s what the task manager does, it manages all of these separate platoon behaviors, is that correct?
Mike Yeah it manages all the behaviors, and then when the behavior’s done, it contacts back and the base manager says, hey, my task is done, give me another one!
Alex You mentioned that it started out quite differently, and certain components didn’t exist. Let’s go through that briefly again. You mentioned that the team manager didn’t exist at all when you launched, and all the other systems you improved.
Which one did you spend the most time on, which ones required the most tweaking since you’ve been on board with the project?
Mike Probably the economy manager. The AI’s economy was absolutely horrid when it was released. I would run tests and towards the end game, the AI would have like 10 000 mass saved up, and like untold amount of energy, but the resource manager and economy manager was telling them that you can’t build anything.
The economy manger is split up so that there’s several buckets that the AI's resources go into, for land building, air building, experimental’s artillery, that’s how we determine what the AI builds.
You know, if you’re air-based AI, you have a lot of spending in air, not so much in land or navy. This opposite is true for land, they’d have more land spending. >
But they also have a slush bucket, in which anything that doesn’t go into any other of the main buckets, goes into the slush bucket; now what would happen is, late game, they’d have all these resources in the slush bucket, and nothing in any of the other buckets.
The economy manager was saying “no you can’t build that land unit; because you don’t have anything in your land bucket". So I had to make changes in the economy manager to account for that.
Every so often the AI will go through its slush bucket, and if it has tons of resources in the slush bucket, that gets distributed to each one of its other buckets – so that’s it’s constantly feeding resources into its buckets if it’s not spending anything out of slush.
Events and Conditions
Alex So what about these events and conditions, I guess these are helping the AI handle the special cases that you mentioned like “I'm currently being attacked by artillery", or various special cases that I guess your players or testers would report;
did a lot of work go into these two components to add new events or new conditions that made the AI deal with these special cases better?
Mike Yes there’s lots of new conditions and stuff that the AI looks for, like the AI wasn’t building anti-nukes in its expansion bases;
which early game you don’t want to do, but I added a condition that actually looks at the value of all the units that the AI has on its expansion base, and if it's more than the cost of a nuke, then it’s worthwhile to build an anti-nuke expansion base.
If you’ve got tons of artillery, you want to protect them, so that’s one of the new conditions I added. There’s several others.
Same thing with the event manager, we actually didn’t have any events for if the unit that AI unit controls gets damaged, so I had to add an event for that, you know, if I got hit by artillery, then I know they have artillery, even if I don’t have Intel on it, I know they have it because they just hit me with it.
Same thing with the Nuke. The AI was really restricted before, because it had to have Intel on something before it could react to it, now if it takes a damage from something, they can react to it. If my team mate got hit by a Nuke, I know I need to build an anti-Nuke.
Alex Actually, that brings up a question I was meaning to ask in terms of the overall architecture. Does the AI cheat, and how much information does it have?
You mentioned intel and gathering information about the player, can it actually see what’s going on, or does it somehow induce what the player is doing just based on the fog of war and what it can see?
Mike No the AI does not cheat at all, unless you tell it that it can. There is an option you can turn on where it has full map radar, or full map-line-of-sight if you rather have the AI know everything. Otherwise no, it doesn’t.
The only thing the AI knows at the start of the game is where your starting location is. That’s it, other than that it’s all from Intel.
Alex That sounds exactly like the kind of information you’d get as a player, because you know where people are going to spawn on the map, if you know that the map, it’s not cheating anymore than the player is.
Ok we’re going to dig into each of these components shortly. Let’s talk about pathfinding just a little, because you sent these debug screenshots, so let’s get that out of the way – what can we see here in these screen shot?
Mike That’s looks like the naval path graph on that map, essentially, all it is is a bunch of marker locations on the map.
We use it for the quick A* pathfinding, you know “can I get to my destination" that’s essentially all it’s used for- “can I get from point A to point B" and then what it’ll do is basically generate a path between all of those nodes, and take step by step, going from here to there following those nodes and then our GPnavigation takes over from there.
Alex This is like the same screen shot? Is that the same thing going on here? And if so, what are these circles in the bottom left here that are overlaid onto the map?
Mike The circle on the bottom left are actually the intel lines, I forgot to turn those off when I was taking screen shots.
So that just shows the radius of intel that the AI has on the map. It’s essentially the same thing, just a different path graph; this one looks like it’s for amphibious units – so units that can go on the surface of the water and on land.
Alex Yes you can see that in the middle because here there’s a bunch of nodes that are on land, whereas here it doesn’t seem to be any in the previous slide.
I think that covers the screen shots that you sent about pathfinding. So let’s dig into these manager components that you were talking about, these blobs within the top level strategy.
Can I start with, what seems to me the simplest, but I don’t know how simple it is under the hood, but tell us how the event manager works and what its role is?
Mike Ok, the event manager is basically just a set of conditions that the AI is looking for. Does my enemy have more than a certain amount of threat of land units, if they do then these are the actions I should take to combat that – like, you know, building more point defense turrets.
If they have more than a certain threat of air units then maybe I should put up some anti-air. If I start getting hit by artillery, or I see them building an artillery, then I build shields to compensate.
Then there’s some other fun ones in there, like if the enemy as more that a certain amount of shields and defensive units, then they’re turtelling, so I should build nukes and artillery to combat it.
So this seems like the component that you would have to change to add more reactiveness into the AI, because you mentioned originally that your plan was to make the AI respond more to the player.
Is that where all this logic went into, and if this is the case, then do this, and adding all of these different rules to make the AI a little more adaptive?
Mike Yeah, that’s where a lot of the changes were for a lot of the reactiveness.
There’s other ones like, rather than reaction there’s what the AI is allowed to build and when, there’s a lot of blueprints that say what the AI can build and what situations have to occur for them to be able to build;
most of those being “I have to have a place to put it, I have to have the economy to support it"... that sort of thing. But there’s also other ones like, you know, “I don’t want to build this, point defense, if they don’t have any land units at all."
So there’s other conditions like that where “I’m not going to bother to build it if I have nothing to do about it" they don’t want to build land units on a naval map, because they’re not going to be able to get to their destination – I can build amphibious units that’s fine, but don’t build tanks, because they don’t float, they sink!
Scripting or C++
Alex So is all of this stuff implemented in C++? Seems like it’s kind of a rule based system, so I’m wondering if this was scripted or if it was in the low level code, how did you approach the implementation of this event manager.
Mike Like I said the event manager existed when I started, it’s all implemented in C++, but there is a script side to it where all the events are all script side, you specify all you want to do, and then that gets parced and read, and used by the event manager which all exists in code.
Alex So is this scripting language Lua? Because you’ve got a history of using Lua at Gas Powered Games; Demigod was based on Lua, actually most of the AI was written in Lua for Demigod, so is the team still using Lua for Supreme Commander?
Mike It’s still Lua, we tried to move quite a bit of it out of Lua into code for performance reasons, but for the AI having that all in Lua and tweakable does make quite a bit faster to make changes and that sort of thing.
Alex One thing that I was wondering while preparing these slides is that the subtleties between the differences in the event manager and the condition manager, because you mentioned that the event manager is there to detect conditions in the game; what’s the difference between the two?
Mike The condition manager is for built conditions, like “what can I build and when" sort of thing.
It’s for units, it’s also for forming platoons, because the platoon manager actually decides how big of a platoon to build based on certain conditions, and what can comprise that platoon, like I said, you don’t want to build a bunch of land units and send them out if it’s a naval map it’s just not going to work.
But for the form manager like, if my enemy has more defenses than a certain threshold, then I don’t want to send a little three tank platoon – I want to send a big group of like 20 units at him. And that’s all controlled by the condition manager.
The other thing that that does is, if different builders share conditions, we only have to check that condition once. We don’t have to check it for every single one of them.
If it’s the exact same conditions, same parameters, same everything, we only have to check it once, and then when the builder goes to see whether that condition has been met, we can check it for multiple builders with just that one condition.
Alex Ok I think that’s covers this one. There’s a couple more managers that seem more meaty, and I’ll get into those right now, because you said you spent a lot of time on the economy manager.
So tell us how this works in practice.
Mike Well the economy manager, it has several buckets that the AI takes its resources and puts them into, like a land AI or rush AI might put 35% of its resources into the land unit bucket.
When the AI decides it wants to build a land unit, it’ll check that bucket to see if it has enough resources in that bucket to build that land unit.
It also has the slush bucket which is for emergency buildings; and also now, it will actually take resources out of that slush bucket and put it into its other buckets if there’s more than a certain threshold of resources stored in slush, because we don’t want them to get lost there.
Also some builders like shields - if they have artillery, I want to build shields, we don’t care what bucket it comes out of; so some builders can specify to take resources out of all of the buckets.
And that’s one of the other things I optimized for one of the recent patches was, what would end up happening is that the AI would end up taking out of the first 2 or 3 buckets, the same buckets every single time, and if you’re taking resources out of the land bucket in your resh AI, then you’re not going to build any land units.
So now the AI has been changed so that it’ll look through all the different buckets, take from all of them individually, based on how full they are.
That way, we’re not always hitting in the same bucket, therefore, trying to build emergency stuff and not being able to build anything else.
Alex So, from your explanations, it sounds like the economy manager’s both responsible for unit production as well as technology. Does this all come under the category of economy manager?
Mike The technology is different, it takes research points, not mass and energy; there’s no build time, none of that, that’s all handled somewhere else. Now the economy manager only handles the mass and energy part of it.
Alex So who pays for the research facilities, if that has like a common shared research you have to build these research buildings or facilities. Which budget does that come out of, is it like one of those examples of using the 'all' bucket, all specifiers, as the bucket type?
Mike Yeah, the research buildings are a little too important just to have it just as one separate bucket. It comes out of everything.
Alex So by tweaking the ratios between the land, the air, the experimental and the defense, do you get very different profiles in the way that the AI plays out different styles?
Is that a good way to control the overall behavior of the AI without having to change anything else, or if you change those ratios, do you have to change the rest of the AI as well?
Mike No actually that’s one of those the ways we tune how the AI works, like a typical land AI might use 25% of its resources for land units, I don’t know what the actual percentage is because I don’t have it in front of me but, resh AI might take 10 or 15% more than that.
Which makes it pump out more land units, has less focus on defenses, which makes the AI more aggressive without having to touch anything else.
Alex That’s a really cool trick to have if you have all these different buckets you can affect the different character types or the different profiles of your AI just with a bunch of floating point numbers.
Ok I think we’ve gone through all of that. Now you’ve sent us a screen shot for the economy manager, talk us through this.
Mike That is probably the worst debug hood in the world. Basically that's just a screen shot showing the economy information for the AI that I’m currently focused on, which looks like I’m looking at Blue here.
Basically it shows how many resources the AI currently believes it has, which may not be the same as what the UI says, because the UI is updated less often than my debug hood there.
It also shows how much resources are currently reserved; when an engineer gets a build order, it actually pre-reserves the resources, because it has to travel to get there, and we don’t want the engineer to get there and have the resources spent before it actually arrives. So it shows me that, because there was a bug with that one time. So I needed that information.
And then it shows all the individual buckets; how much they’re currently putting into each bucket, how much that bucket has.
That can change on the fly. We do have events in there that will increase experimental spending once they actually research an experimental unit.
Alex So you mentioned the turtling of the player, and that’s something you countered. I presume this was done using an event and adjusting those budgets?
Mike Actually I don’t think I adjusted the budgets so much; it’s just we tell the AI you know, “if you don’t have nukes researched, then research them" and “if you don’t have artillery researched, research it" and the AI will build it to counter the player.
And that was actually done because one of the guys that was actually testing the AI in the office was turtling like crazy and the AI was like sending unit after unit after unit at him! So it’s like, I’m going to stop that!
Alex Ok, part 2 is about the base management; it seems like in most RTS this is quite an important part of the game, and the AI certainly as well.
So I presume a lot of work went into this as well; so why don’t you give us a quick overview of the different components that go into the base manager.
Mike Ok, the AI has a base manager for each one of its bases. It is responsible for the platoon form manager, the engineer manager and the factory manager; those managers are the ones who control all the various behaviors of the units inside of their own base.
And they’ll have different builders, for like expansion bases they’ll typically build more defense because they’re closer to the front lines, so they actually have different builders specified for the expansion bases.
The form manager is the one responsible for forming the platoons and sending them out to attack something. The engineer manager manages all the engineers assigned to that base, because once an engineer is assigned to that base, they stay there, they’re just at that base and follow what the base manager tells them to do.
And same thing with factory managers, they’re responsible for building their own units and they follow the same general guide lines that the main base does.
Alex Here’s an example screen shot of the base I’m building up; it’s quite in early game, just before I got my first big defeat against the normal AI; I was feeling a bit overly confident after beating the easy AI.
But what you can see here is a bunch of mining facilities, which basically extract resources from the ground. so they are typically what you start out with, and then there’s a bunch of research buildings you can build; and I believe this is an energy creation, energy factory building.
And what you can see in the middle is an engineer here. So if you haven’t played the game, it should give you a quick overview.
So you mentioned the engineer management, and these guys at the bottom are both engineers.
To what extent does the management of what should be built should go into this manager, or is the manager figuring out where to place the buildings in terms of the logistics of the base building, but without thinking about the strategy?
Mike Well, in the script slide there’s a lot of blueprints specifying what the AI can build, then there’s a bunch of conditions for when the AI can build that.
There’s also a priority system; each one of those build options has a priority associated with it, that tells the AI what it should build at a higher priority than something else, like late-game artillery when you have it researched and you have an enemy firing range and you have the resources and somewhere to put it, it’s a good idea to build an artillery.
So it’ll have a high priority than saying, like building something else, like a point defense. So what will happen is that the engineer manager will go through all of those, all of the available build options in priority order and find out which one with the highest priority has all of the conditions met.
Once it finds something that it can build, then it’ll assign an engineer to build that item. And then the task manager takes over from there.
When the task is done, they’ll call back in to the base manager, and let the engineer manager know that they’ve finished their task and they need to be assigned something else to do.
Alex So is this logic that you describe separate from the engineer manager, or is this the actual engineer manager that does all of this with the blueprints and prioritized conditions specified on the script slide?
Mike No no, that’s all the engineer manager. The engineer manager knows all of the blueprints that it’s able to build, and it goes from there; it’ll contact the condition manager and see if all the conditions are met, and handles all of that.
Alex So do these blueprints specify the position of the buildings in space, so you don’t have to work out exactly how to fit these different buildings in space or how the buildings are laid out –
because I was going into an enemy base at some point, and it was so nicely packed together, it was almost Manhattan with nice regular grids in these different directions, and I was wondering how you managed to pull that off?
Mike Now I wish it worked that way, but it really can’t, because if we specify like a specific location, it might work great on one map, but on another, it might be in the middle of a mountain and that’s it, it just doesn’t work.
For most everything, it just does a spiral search, searching for an empty location that is more than a certain distance away from anything else so that they’re not blocking themselves in.
For some things it gets a little more complicated, like shields, the AI tries to space its shields out, so that there’s not like two shields right next to each other, because that just doesn’t work well; so you want to spread the shields out, so that everything gets protected.
But later game, if the enemy has artillery, that gets relaxed somewhat, they want to build them closer together. But for most of everything else, it’s just you find a place to put it down.
Alex Ok, let’s move on to the factory manager. So this is one of my factories. It’s in the same game. And you can see one of my tanks coming out, I think it was one of my first ones. So what does the factory manager do? What’s its responsibility?
Mike Essentially the same thing as the engineer manager; it also has a set of blueprints that can be built with conditions.
Once it finds one that has a high enough priority, then all the conditions are met, it’ll assign a factory to build that unit, or add on, or whatever, and then, once it’s done, it contacts back and says “find something else for me to do"
Alex So are each of these factories requesting stuff to build from the manager, or is the manager dispatching stuff out – I guess it depends on which way you look at the problem, is it more event driven or top down management from the factory manager’s perspective?
Mike No it’s more event driven. Once the factory’s created it gets added to the base, then it gets assigned to the factory manager, then the factory manager finds something for it to do, and once it’s finished, then it contacts back “I’m done" and the factory manager will try to find something else for it to do. So it’s really event driven.
Alex That covers the factory manager as well, oh, here’s another one, the experimental units, this is one of the fat boys I think, the big tank, and it’s a really cool unit to have.
Is this experimental factory managed by the factory manager in the same way as you described?
Mike The exact same way.
Alex Then we can move onto the next one: the form manager. So how does the form manager fit onto the base?
Mike It uses pretty much the same logic as the other two managers. It also has a list of blueprints that it goes through, does the exact same thing, checks the different conditions, except here there’s fewer of them; it’s mostly “do I have enough units" or “does the enemy not have more than a threshold of this amount of threat" that’s just to insure that the platoon that I’m sending out is big enough to handle whatever the player has.
I’m not sending 2 or 3 tanks when it’s 20 min into the game and they’re just going to die before ever getting within firing range.
That’s essentially the same thing, the only difference is that, once they’re assigned to a platoon and to a task, they don’t really contact back and check in for another task; they just do whatever they were told to do, until they die and that’s it.
Alex So you mentioned that all these 3 builders within the base have the same design and same structure to them, is the code actually shared among those builders or is that variations on a theme but not shared code.
Mike There’s a main manager class, and the form engineer and factory manager all inherit from it. So that’s why they all act the same, there are some minor variations, but for the most part it’s all the same.
Managers and Low Level Connections
Alex To what extent do these managers within the base take input from the higher level or the big picture, or the big strategy, is that done entirely via those conditions that you mentioned? Is there any other kind of over-rule/overall guidance that these low level base managers are given?
Mike No it’s mostly just from the various conditions for all the blueprints that basically tells them what they’re supposed to build and under what conditions that they’re supposed to build it.
Alex Let’s talk about the more strategic side of things in terms of attacking and so on; tell us, first the expansions, how does the logic work for figuring out when and where to expand?
Mike Well the AI tries to expand into a place that doesn’t currently have enemies in it, and they also try to expand out basically from closest to expansion bases on, so there’s not a big bunch of gaps.
So like you can see the screen shot, they’ve got their main base indicated in green, this debug shot shows me how many mass extractors are assigned to that base and how many of those have been built on, either by the AI or somebody else; so how many are currently covered.
And how many engineers are assigned to the base out of how many are supposed to be there. You can see the E = 2/2, and there’s two engineers, and there's supposed to be a max of 2 engineers assigned to that base.
This orangish yellowish color edone, means that the AI is currently expanded there, and there is enough of the mass extractors built on there to allow them to expand again.
We didn’t want the AI to get runaway happy with expanding, they actually have a set amount that they have to have a certain percentage of their mass extractors built on before they move onto another one.
And this blue circle, that’s where the Ai wants to expand to next, but it’s currently occupied by the enemy. So that’s where all the AI's platoons are going to go to, to try to take out whatever resistances are there, so that they can expand there.
Alex So is it this blue the stand alone blue circle, or the circle that’s second layer around the original green circle?
Mike I see what you’re talking about, no, that’s the intel ring again I think. No it’s the stand alone little blue circle over to the right, that’s where they’re trying to expand to.
Alex And so how is this point picked if there are multiple different bases you could expand to? Who decides which one goes first or which one is the priority?
Mike That’s the job of the strategic manager, it determines the reason why they’re not going further down to that other area, one or two things (I’m focused on the blue AI right now.) So either the red AI has decided to expand down there, and they don’t want to fight for the same spot, because they’re allied in the screenshot, or it’s simply because the one that’s outline in the blue circle is actually closer physically to blue’s base than any of the other ones.
And since all of the other ones are occupied, because there’s an AI in each of the other two starting locations on the right hand side, and the AI knows that.
The only other expansion base Is the one that's circled in blue, and the one below that miring/nearing the one colored in orange in the screen shot.
Alex You sent in another screenshot that consists of the same thing, but from the perspective of the opposing team –
Mike Yes, I believe this is the same game but from the opposing team’s perspective and you can see, I believe this is actually later in the game, I am focused on green AI in the screenshot, and they’ve got their main base, and you can see that, just outside their base, they’ve got one in red, that means that they don’t actually have enough mass extractors built on there to actually warrant another expansion.
Even though they’ve got that one colored in blue, they probably made that determination before they got attacked. So now they won’t expand because it looks like that expansion base is getting attacked pretty good.
Alex Sounds like the strategic manager is very good at evaluating different things and different options for expanding, does it do the same thing for attacking, if so, how does it work?
Mike It does have an algorithm for where to attack. It’s based on the threat map, where the influence map that the AI uses, it tries to find out where the economy threat is; which is basically just an evaluation of how worth while those units are – mass extractors, power generators, they have a high econ threat, the commander has a really high econ threat because if he dies the game ends.
And other things that are worthwhile to destroy also have an econ threat. So the AI tries to determine what area has a good econ threat, and also doesn’t have a lot of anti-surface, anti-air threat in that area, because that means it’s not defended well at all; so that’s where the AI tries to attack.
And then it tries to determine where to attack based on the platoon that’s asking, find me some place to attack; if it’s a small platoon, then we don’t want to send it to really defended area, we want to send it somewhere to kill some outlying stuff. If it’s a really great big platoon with a lot of experimental units, then we want to get right on into the highly defended areas.
Alex So does this involve tweaking one big heuristic equation of some kind that specifies the value of different targets ?
Mike Yes there is a little sorting algorithm that sorts them by how threatening they are, how valuable they are, and how far away they are; if they’re closer targets then they’re more valuable because they closer to us, because they’re faster to get to.
Air units don’t care, they can go anywhere they want and don’t care about distance, but like land units that makes a big difference
Alex I can see why you were so interested in working on the planet wars bots for the Google AI contest, because it seems like very much the same kind of stuff that goes on here.
Mike Yes that’s right down my alley
Alex Is there anything else we missed for the strategic attacks; or is that calculate the scores, search through the targets then rank them based on the platoons that’s requesting that, did we miss anything here, are there any subtleties that we’re not aware of?
Mike The only other thing that we look at is path-ability; if the land units can’t get there, then it’s not worth while to even send them out to attack it. The big change that change recently was for naval units.
Before, there wasn’t good checking for naval units and they’d be told to attack something that was so far out of their firing range from water, that they’d go over there along the shore and really be ineffective.
Now we’re actually checking well, is there a water location that is within firing range of that place, so that way the navies have become a lot more effective in finding places to attack without just going to the shore line and being completely out of firing range.
Alex Ok, let’s move onto the debug screenshot, I think this is a debug screen shot of the attacking. So talk us through this; what can we see here?
Mike This is another one of those debug huds that are useful but it could be better. The blue circle on here indicates what platoon is requesting to find an attack location; over here it looks like it’s a group of cyber ships that have the land-walking ability, so they actually came up out of the water and are walking across the land.
To the upper right we see a bunch of numbers there, that’s basically showing which way that those locations were sorted. it’ll actually have what number it was in the sort, then it’ll have the threat value, the econ value, and the distance value to show me what numbers came up for those.
And the one highlighted in green is where the strategic manager actually chose to have that platoon attack and then you see they’re going to be making their way over to attack it.
Alex How often is this process run to evaluate where you are going to focus your attack energy on; is that whenever there’s a platoon that’s been built by the engineer manager?
Mike It’s whenever a platoon is requesting a place to attack, which happens when it’s finished built, it also happens every so often when we send a platoon out, something may have changed since it got sent out.
So every once in while it’ll check back and say “do you still want me to attack here, or has something else popped up that you'd rather have me attack, that’s better and more valuable for some reason; either because it’s closer to the base and it’s a big threat, or an ally is requesting assistance" then they’ll peal off and go there. Otherwise it’s just when a platoon is built, you know “where am I supposed to go?"
Alex Ok, I guess finally in terms of the strategy, that scouting, it sounds like it’s very much the same process as the other two; are there any subtleties or differences in terms of sending out scouts or is it the same approach that you use?
Mike Well the scout system basically keeps track of all the different map markers, and we have expansion plant markers, path markers, it adds those all to a list, and rates them based on whether or not there’s been enemies there;
and then it’ll just send out a scout every so often, you know if this place hasn’t been scouted in a while, they’ll send a scout out to see if there’s anything there. If it’s a high priority spot, like a place where there’s been enemies, then it’ll get scouted more often to try to keep an eye on what the player’s doing.
Alex Is this also using shared code for these three different functions like deciding where to expand, deciding where to tag, deciding where to scout; is there a common underlying framework that you use to make these kind of decisions, or is it using the same idea, but different code?
Mike It’s using the same idea but different code, the scouting locations doesn’t require as much detail, we just want to know if there’s been something sighted there, and how long it’s been since it’s been last scouted; so it doesn’t really require that much code.
Deciding where to attack, that’s a big function actually, it takes a lot of information to make that decision.
Where to defend, that one is a little different because it’s not only how much enemy activity is in the area, it’s also how much defenses do I have; you know if there’s three tanks coming and I’ve got a bunch of point defense, I really don’t care because the defenses will take care of it. They only really care if there’s a big platoon coming and there’s not enough defenses there to handle it.
Alex Let’s move on to the next section: teams. So this is a team manager that you introduced after the release, so tell us what you did there.
Mike Well the team manager is there to allow allied AIs to communicate with each other. They all ensure that they’re attacking the same target, the same enemy. They’re focusing all their attacks on that one person, whoever that might be; and that decision is made based on who’s the bigger threat, who’s closest. In certain circumstances it’s not who the biggest threat is, it’s who the weak link is.
If you turn the AI difficulty up for that, then if one of the enemies ends up lagging way behind, then they’ll find themselves the focus of the attacks to take advantage of that fact. It also allows the AIs to gather their units into one big platoon, which the AI has never done that before; one of the features that was requested in my AI mod for a long long time; and I haven’t been able to put it in until now.
The AI actually takes its units, put them together then you’ll have these big several hundred unit platoons coming at you, it’s really fun to watch.
It allows the AIs to communicate to each other about any sort of dangers that they found; if they start taking artillery fire and what not, then they’ll let the other AIs know on their team that, they’ve got artillery whatever.
They also try to make sure that they’re not taking expansion bases from each other, they will let each other know, “hey, I’m taking this spot" and then somebody else will take another expansion base.
So they’re not trying to take mass extractors from each other and stuff, because what would happen is that you would have one that would expand really fast, and the other Ais would lag behind because they didn’t get as many mass extractors as the other AIs. So it tries to alleviate that.
And it also allows them to call for help. If they find a danger that they can’t handle coming towards one of their bases, they’ll mark it in team manager as a place where they need assistance, and the other AI guys will send platoons to help.
Alex So Miko has a question in general about taking orders, or receiving information from players; so you mention here that there’s a possibility to call for help. But can the AIs communicate with each other beyond that as well?
Mike At the time? no, I know my Forged alliance AI mod allowed that. You could give orders to the AI, you could tell them to build an expansion based somewhere, tell them to focus their attack somewhere, tell them to switch targets with a particular enemy whatever.
It was a feature that was tossed around but we never really got it into Supreme Commander 2, maybe the next project?!
Alex Tell us about this screen shot then.
Mike Ok, this screenshot is actually debug information for the team manager, it tells me who the team leader is on a particular team, and who the members are. In this screen shot there’s only one team member.
And then it tells me debug information showing, who they’re currently focusing attacks on, and they’re currently focusing on this red player, because his threat is highlighted in red.
Essentially that threat number it’s just an evaluation of how strong there are versus how far away they are; even though the blue one has a higher threat amount, they probably chose to focus on the red one first, and they won’t break off of that target unless blue becomes a much bigger threat.
This stops the AI from flip-flopping between two targets that have similar threats; they’ll focus on one until the other one becomes a big enough threat to warrant them to switch.
Alex I guess that prevents the oscillation and the looking indecisive. Ok, I think that covers all the AI managers there; did we miss anything for this top strategic level or is there anything else you wanted to talk about?
Mike No, I think we covered everything.
Platoons and Neural Networks
How Neural Networks fit in
Alex Let’s move on to the next part about the platoons and the use Neural Networks. So you sent another bunch of debug screenshots for the platoons here; tell us what the Neural Networks do and, while using these screenshots I’ll annotate it. So explain to use where the Neural Networks fit in for the platoon.
Mike The Neural Networks are used mostly by the bulk of the AI’s attacking force. All of its land attacks, air attacks, naval attacks, they all use a Neural network to control their fighter flight mechanic.
What happens is that once they’re moving out and they encounter resistance, they will gather information about their platoon, about allied units in the area, and about all the enemy units in that area.
And they’ll take all that information feed it into the Neural network; and the Neural Network will spit out a bunch of data about what action they should take.
Should they attack the highest value unit, should they attack the weakest unit, should they attack this whatever from range, if they attack it from structure should they attack it from range rather than running up to it and attacking it.
If it can’t find any good things to do then that means run away; because whatever you’ve encountered you can’t handle it. so that’s what the Neural Networks do.
Alex Let’s use an example from a screenshot as an illustration; so I presume this group here in the middle is one platoon, and presumably under threat from these guys at the top. So what kind of specific inputs would they be getting in this case through the Neural network?
Mike They would be getting essentially here it looks like there’s only a couple of units, so they’d just be getting input about those two enemy units, they would also be getting input from the influence map about any threats in the area, and then they’d be getting inputs about their platoon.
And here it looks like they’ve been told to attack the highest value target, which , since they’re about the same unit, it’s basically again a default to which one is closest.
Alex So in this case, presumably they pick the highest value because they were clearly going to win anyway, so there was no risk in going for anything else but the highest value target.
Ok there are a few questions about the actual underlying Neural network structure, so we’ve covered the purpose of fight of flight mechanics, and this telling the platoon what to attack first and whether to attack from a range; but tell us more about the structure and the types of the Neural Networks that you use in Supreme Commander 2.
Mike It’s a three layer network, it’s got the input layer, the hidden layer and the output layer. We have a Neural network for the different attack types; the Neural network used by the land platoons, the one used by the naval platoons;
then we have a Neural Network for air to air combat, and also air to ground – because they need different information - air to air units, they care more about other air units they can attack that sort of thing. They had to be trained separately .
They use 36 different inputs, which is a lot, they look at how many units there are, how much health total those units have, how much shielding do they have, what’s their short, middle and long range DPS like... all those different inputs are fed into the Neural network.
We also get those same inputs from the enemy units and the platoon. It’s fed into the Neural Network and then spits out basically what decision they need to make in that instance.
And each one of the outputs corresponds to a different action that the platoon can take. Either attack highest, attack weakest, attack strongest, or some of the other ones, attack structure from range, attack resource if they want to attack a mass extractor power generator first, attacking shields, that’s another thing they may want to do; and of course attacking commander which is my favorite.
Alex So how many outputs do you have, do you remember how many actions are available for each platoon; and how many nodes do you have in the hidden layer in the middle?
Mike I believe, the air units have 12 or 14 outputs, I think it’s 16 or 18 for the naval, and for the land units, because they have different stuff, air units can attack from range, so the land units have the at-range option for some of their actions so they have more.
The hidden layer, I think it’s somewhere in the neighborhood of 100 or 150 nodes in the hidden layer, I think. So there’s like 36 input, 150 hidden, yeah, that’s 16 or 18 output.
Alex So those of you looking for a book on the subject I think these types of Neural Networks are called multi-layered perceptroms, and there’s a lot of tutorials out there on the subject. And depending on which book you read, some of them might call them two layer Neural Networks , because they don’t count the inputs, but some refer to them as three layers, so don’t worry if it sounds like a different solution.
Ok, I have a lot of questions about this; how do you train these Neural Networks, and do you train them before the game starts or do you train them whilst they're/I'm playing against the AI.
Mike Actually they’re trained in my office actually; I have a dedicated computer that I have command line functions that I can start the game that will actually tell the AI that it’s in training mode, so it behaves a little differently.
And then the AI just plays against itself, either teamed, not teamed, I prefer not teamed; I give them full map vision so that they get all the intel information when they’re making their decisions in training, which they don’t get that information in the game, but I want their decisions to be perfect during training; they need to know everything.
And they play against themselves; When the game’s over or it’s gone an hour, then they’ll save up and on a data, and restart the game and start again. They just get iteration after iteration by just playing.
And I have a debug hud that shows me the average error rate for each of the outputs, so I can see how they’re learning and how well they’re learning and how close they are to getting to being 0 errors;
Alex One thing that seems difficult to do is figuring out the actual impact of your action; let’s say if you’re in a really big battle with multiple platoons, and your different platoons are doing different things, how do you know if one platoon’s being successful or if it’s the other platoon that contributed to that platoon’s success;
it’s seems like a quite a noisy environment to get some good data from; so I’m wondering if that’s caused any problems or if you had to design specific scenarios to get around this kind of problem.
Mike Well in training the AI only cares about its own platoon, it doesn’t care about any of the other platoons, because we only want to know if the decision was good for that platoon.
You can get some noise from the other platoons doing their own thing, but I generally put them on small enough maps so that doesn’t happen; usually there’s just that one platoon and they attack.
The other thing is when two platoons from the AI get near each other, they merge. They’ll merge the two platoons into one, so if they get close enough to each other to actuallly influence their Neural Net data, then generally they’re going to turn into one platoon anyway. And that will reset everything, and they’ll start from scratch with a fresh platoon with fresh Neural net data.
Alex Ok I have a bunch of questions about this Neural network, and the decisions that you made.
Why choose that architecture
Leo asks, why did you use different Neural Networks and was this done for design reasons, or, why didn’t you go for one Neural network that was maybe a bit more general?
Mike We had a Neural Network that did all air units, and it didn’t seem to be all that effective; because the data that the Ai needs for air to air combat is different form air to ground. Air to air, they don’t care about ground targets.
They only care about the other air targets, they don’t care about anything else. Air to ground they care about a lot more; they want to know what stuff they can attack down there, is more worthwhile for them to take out anti-surface structures, they can attack shields, air to air combat units they can’t take out enemy shields, so it ended up being way too noisy and too difficult to have just those two Neural Networks sharing data.
I couldn’t imagine trying to make all four of them into one Neural network, the data would just be way out there.
The other issue is, the air to air, they have a different number of outputs than the air to ground, or naval and the land, and some of them, the outputs correspond to different things. Air to air, they don’t have an attack ACU option, they can’t hit him because he’s on the ground. So trying to merge them into just one I think would not work out well.
Alex Ok, ATijea asks if you could create a simple level just to test out different platoons, and to avoid that kind of noisy situation I mentioned in terms of a full-game, just create a test environment that would allow the AI to run through the different possible decisions in a more simplified environment that’ s more Neural Network friendly.
Mike I probably could have taken a DevMap; the problem is creating a simple level in our game is not so simple; they’re all 3D, they’re created in MYA 3D studio; the amount of effort it would have taken just to create a Devlevel probably would have been too much.
I could’ve tried to do scenarios, but then I would’ve had to sit down and say, well, what kind of scenarios are going to show up in a real game environment? Having the AI play against each other seemed like a better option at the time. And it seems to have produced pretty decent results for just doing it the way I did it.
NNs Supervised Learning?
Alex There’s a question by Mike who’s asking if this was supervised learning; so it sounds like it was supervised learning, because you gathered lots of data and just fed it into the Neural Network and used it as a function approximator.
So maybe you could tell us a little more about how you gathered this data, you must have written a program that monitors the situations to get these little pieces of information; how did that work and how long did it take to get working?
Mike Well the Neural network exists in for the N-code/ fully in code except for the Neural network data that’s actually stored in a Lua file.
The Neural Networks are loaded at the start of the match and as the game progresses, every time the AI platoon learns and it does this by looking at the platoon and then looking at the enemy; getting that information then feeding it to the Neural Network, it chooses a behavior at random, so it doesn’t even look at the output data, it just chooses one of the available actions at random.
When it chooses an action, when the enemy is dead, or the platoon dies or whatever, then it’ll determine how well it did. It stores all of the units that were in the area when the decision was made, and all the platoon units that were involved when the decision was made. And then it runs the same set of information that it got when it made the decision; how many units, how much help... all that information is gathered again, and then we see what the difference is.
If they took more damage than us, then obviously it was a good decision, but if they only took a little more damage than us, then it was a marginally good decision, if we got absolutely wasted then it was a bad decision. And that information is sent back to the Neural network and the weights are adjusted accordingly.
Alex So is damage the major factor you are measuring here in terms of measuring effectiveness; that’s the damage that becomes the biggest indicator of how effective a platoon is being?
Mike The way the multi-layer perceptroms works is that it kind of determines on its own what the most determining factor is. We just feed it the information; there’s like 14 different inputs just for platoon data per platoon.
Then there’s the influence map, how much threat is still there versus how much threat was there before, but it looks at everything; DPS, It looks at the shield health, and how much mass that those values and energy that those units were worth.
So you know if I’ve got a bunch of tanks, they may only kill one unit, but if that one unit was a fat boy, it’s worth a heck of a lot more mass, and it’s a much better kill than however many tanks you took on.
Learning Curve of Neural Networks
Alex So Neural Networks historically have scared game developers’ a lot, because they can be difficult to work with and tweak, they can be a bit like a black box.
So I was reading your blog about this, and it’s been interesting seeing your insights and how you went from sort of treating it as a black box to then getting more into it. How has your learning process been for working with these Neural Networks?
Mike Actually, I think it’s been generally good. I have definitely gone into the core code and made some changes, because the learning function that was in the Neural Networks that release all the way up until two weeks ago, was completely and utterly broken.
Apparently the person who coded before me forgot to actually ... well they fed all the inputs into the network, but they forgot to feed it forward before we got decided to go back and adjust weights. So we were adjusting weights based on the wrong outputs.
So that was pretty bad; basically it invalidated all of the Neural net training that I had done the previous month.
But I have a much better handle on how it works, I’ve added more debug hud information so that I can actually see how the Neural Networks are learning. I can see what their average error rate is, you can see the screenshot right there. And basically I have a much better understanding on how it all works and how it all works together.
Review of Debug Screenshot
Alex So you can zoom into this screenshot as well to get a list of floating point numbers. I presume this is the output of the Neural Networks that is currently being simulated; is that what’s going on, or is it the error levels?
Mike No no, that’s the average error rate. It shows basically how close to perfect decisions the Neural network are making. And the closer to 0, the better.
If you see a bunch of higher low numbers, that means that the decisions that the AI was making did not match the desired output that was determined after we re-evaluated the Neural Networks. If they’re close to 0, then that means they were making decisions and the output was what was expected to happen.
Community Access to NNs
Alex Mathia has a question about modifying the Neural Networks; is that something you’re doing over at the offices at Gas Powered Games, but that’s something that modern community can also modify and train, or is it something too difficult for someone to get their hands on and sink their teeth into?
Mike Well because it all exists in code, no it’s not really modable, I mean I guess they can go through and try to adjust weights; but that’s just not going to work out, it really isn’t.
Without having access to all the debug information they would need a special build, because all that code is compiled out when we make the retail builds, it would take a lot of effort to get a special build for people to actually retrain the Neural network system.
and then because it all exists in code, there’s not a whole lot that anybody’s really going to be able to change; you won’t be able to change the inputs, you won’t be able to change the outputs, and that’s really what’s going to make any sort of difference as far as the Neural network is concerned. The only thing you’ll be able to do is to run more training scenarios which, that’s about it.
Alex In terms of the inputs and the outputs, how often have those changed while you’ve been working on them; did you change them on a regular bases, or they remained relatively stable since you’ve been updating them?
Mike As little as possible. The only bad thing I have to say about working with Neural Networks is that any time you want to change the inputs or outputs, you have to retrain. Which is an absolute... well it’s not a pain, but it’s time consuming.
Like if I want to add in one more input, I really got to think about it; because it means it’s going to take that many more iterations of training, because I had to dump completely everything that the Neural network thought it knew, because if the inputs and outputs don’t match, then your Neural Net data is completely useless. You have to completely dump it and start from scratch.
So I added more inputs at the beginning, I added some other stuff later on, where the AI ... because if it found a unit under construction, it would actually give it the full mass value, the full DPS value, I added something else so that if the unit was under construction, then it would only add a portion; like if it was only 50% completed then it ‘d have 50% of its DPS capability, 50% of its mass value, that sort of thing.
When I did that I had to start from scratch again, so I do that as little as possible.
Alex You mention that it takes a while to retrain the Neural Networks, could you give us an indication of how long it takes, is it something you run over night or you let it run for a couple of days; how long does this process take?
Mike That thing it runs for a while. My general is, I’d like to at least give each individual network at the bare minimum 2 or 3 days, just to run iterations. That’s why I added the debug hud there, so that I could see how well they’re training.
The land units and the naval units don’t really take that much time actually I found they really don’t take that much time. The air attack doesn’t take that long. The one that really takes a while is the air vs ground one, which is I think on there as the bomber attack platoon.
The reason that that one takes a while is because you’re using fighters and gunships. I can get DPS values for fighters, but the problem is that I can’t take into account how long it takes for the fighter to come back and make another pass.
Which makes all of the inputs that go into the bomber platoon or the bomber Neural network; they’re a little skewed because you know, gunships can get on target and stay there, put constant DPS on the target, bombers just don’t.
So that one is the one that takes the absolute longest to train to get it to a decent level. The other ones are really not that bad.
Alex So as soon as you have a change in the balancing, a change in the units, change in the behavior of the bombers as it circles around to hit its target, presumably you have to retrain these Networks as well?
Mike To a point, our balance engineer, he goes by several different formulas whenever he’s making balance changes. As long as he basically sticks to that formula it’ll be fine, because the AI doesn’t look at specific units.
It doesn’t care that it’s a UEF fighter; all it cares about is “what is my DPS capability, how much health do I have, how much shield do I have" whatever... “If all of a sudden I have less shields, the Neural Networks will adapt to that because now I have less shielding, which makes me more vulnerable."
The only thing that would really kill it is if, I don’t know; even if you take weapons off it’s going to take away the DPS amount and that sort of thing, so; I can’t really think of any major balancing changes that could happen that would really throw the Neural Networks off, unless he starts using a different formula.
Like if all of a sudden, kingcryptors had tones of mass and not very much DPS, that might throw a wrench at things, but I think the Neural network would adapt to that too.
Alex There was a question about turning of the graphics and just running the game in a headless simulation so you don’t have to keep the graphics card running and maybe update the game at a faster rate, is that something you can do when you’re training a Neural Networks, or does it run at the same speed as normally wave graphics?
Mike I had done that at one point, I set up a renderer to only run render one frame every 5 seconds, and I didn’t see any speed improvement because, in our game, the UI and graphics run on a whole separate thread.
And I have a 6 core machine that I’m running this on, so it’s just going to max out one core doing the rendering, then it’s going to max out another core doing the simulation stuff.
So having it render doesn’t affect the speed at all. So it’s not any faster to turn off the rendering, and it’s a heck of a lot more fun to watch.
Alex There’s a couple of extra questions. I presume that the low level targeting that’s done for each unit is done, well by the game - I don’t have to micro-manage my units to attack the enemy, I presume that’s not done using a Neural Network, based on your explanation so far?
Mike No, Neural Networks are only used by the AI.
Alex Ok. Last part, we’re going to talk about workflow; if you have any questions, now’s probably the best time for them, so please feel free to type them in.
I’m going to ask you about, this was a release made in October, October 8 actually just a couple days ago, and there’s this long list of things; you mentioned earlier where you got your task list from about things you wanted to fix, things that your testers wanted to fix;
how did you go through these, did you just take them one by one and then send the build off to the tester, and iterate over it, tell us a little bit about how you went through this process of implementing all these fixes?
Mike I’ve been working really close with our test team, I usually meet with them on a daily bases at the very least. They actually got kind of spooked because I ended up showing up just as they were running a bug to send to me, so they thought I was reading their mind or something.
They end up playing with the AI a lot, so they have a lot of get inside on - where the bugs are and that sort of thing; so if they found something that was relatively easy to fix, or something that was related to some other bug that I was fixing, then I can go in there and fix it like building stuff in an ally player’s base; that got really annoying.
Then there was other bugs they found where the AI would have this big hunk of units in its base and wouldn’t send them out, and for a long time I couldn’t figure out why, I mean they should be sending them out!
And I found out it was just a bug in the influence map, the AI thought that there was a unit near its base and it was trying to attack it but it didn’t exist. So I made influence map changes; if the AI has intel on a unit when it dies, it’s immediately removed from the influence map, which makes more sense.
Then there’s the community. They have all sorts of opinions on what needs to get fixed and when, and what has priority over what. But they’re great for finding bugs like that. They want the AI to behave a different way, then if it’s something easy to fix I can go in to change it.
My manager has been really good about that, if it’s something that’s relatively simple or is not very risky, and I’ve got the time to do it, then they let me make AI updates. The community has been really positive about that. They really like the updated AI, they really like playing against it.
I’ve got messages from people saying like, the real high level players, the ones that never touch the AI, are actually playing against it and enjoying themselves. Which is a Huge compliment.
Alex It’s got to be one of the few real-time strategy games where good players, experienced players can play the AI and have fun. I can’t remember many other RTS games reaching this level.
There’s a couple questions that I picked up from earlier. One from Nicolas, about the SDK, does Supreme Commander 2 have an SDK, and how easy is it to get into?
Because you came from a moding background, so I presume you must have used the SDK in supreme commander 1, and started from there, are there opportunities for students or enthusiasts to do the same with supreme commander 2?
Mike There wasn’t really an SDK for Supreme Commander or Forged Alliance. There was a mod manager, but the way I was moding the AI, wasn’t even compatible with their mod-manager. I actually had to look at all the Lua code, because that was all available and wasn’t compiled or anything.
And basically I went from there, there were several other mods that did the same thing, they didn’t go through the mod-manager because it just didn’t support what we were trying to do.
Simple mods like changing blueprint values or that sort of thing, that worked through the mod-manager just fine, but anything more complex than that, you had to basically replace the files that the game was using with your own versions of them.
Supreme Commander 2 is the same way, you can replace files if you would like to change behaviors, change blueprint values, that sort of thing you can do that; but there isn’t like a moding SDK that you can download and use to make mods for the game.
Actions Per Minute
Alex Ok, there was another question by Nicolas, who was asking about actions per minute, because you mentioned the AI doesn’t cheat at all in terms of its sensory input, but do you have any kind of throttle or limit of how many actions it can do at the same time as well?
Mike There really is no limit to the actions per minute. There used to be for a Perf reason, but it really didn’t need to be there, and it was having issues anyway. So I basically just scrapped it, because you’d end up getting units that were just sitting there doing nothing; it was just causing too many issues - that was gotten rid off, and it didn’t need to be there anyway. So, no there isn’t like a actions per minute limit on the AI, although adding one probably wouldn’t be that hard.
Alex There’s a question about how long it took for the AI to reach its current state in terms of the major features since you started working on it; are we talking about weeks, months?
Mike I’ve been working on the AI steadily since they had me come back and work on it shortly after release. So what has that been now, it was released in March, so we’re on 7 months now.
Total work hours is obviously a lot less because I’ve been fixing a lot of bugs along with doing AI stuff, so I mean what; like 2 or 3 months in addition to whatever time was dedicated to it during development.
Alex There’s a couple of good questions; how did you adjust the difficulty levels of the AI. So there are like a bunch of difficulty rankings available when you select which AI you want to play against; how did you tweak those difficulty values?
Mike Actually those have been the same since release, so I wasn’t responsible for tweaking those values. I made minor minor adjustments in post release just to fix some issues that the AI would have.
Because if you set like its build cheat to the same as its economy cheat, then it sometimes kind of stall at the economy; or if you set the build cheat higher, then it’s essentially building faster than the economy can handle so it was having issues.
And then we added the AI options menu. So really the difficulty levels there, are just there for legacy, so people can choose those that they want to.
If you go down and choose the custom AI option, you actually get to adjust those cheating values yourself, to whatever skill level you want to set it to. If you want them to super cheat, you can set them to super cheat.
If you want them to be even easier than the easy AI, then you can do that too. or if you want something in between normal and hard, you can just adjust the slider to whatever you want. If you’re doing like 4 people versus 1 AI, you slide those cheat values up a little to give you guys a challenge.
Alex Ok. Now is a good time to wrap up with something you mentioned by e-mail; you said the AI platoons are a tad too timid; and this is an example of one of your platoons kicking my other platoon.
So I’m just wondering, how would you go about tweaking a platoon to do that kind of stuff; if you think it’s too timid, how would you approach the process of making it less timid?
Mike The reason I believe it’s too timid is because I think the Neural Nets have learned the game a little too well. and the way the Neural nets works, is they use that sideway function; it’s either a good decision or a bad decision, there’s not grey area.
So if the AI think it's going to lose the encounter, it’s not going to engage. So if you outnumber them by just a little bit, then they don’t want to commit. So my plan is to give all of the platoons that small yappy dog syndrome; and make them think that they’re a lot bigger than they are by just adjusting the inputs a little on the retail side.
If the AI thinks it has 100 DPS, then we tell the platoon that we have more than 100 DPS, like 125; that way the AI is still getting good information back from the Neuralnet, so that if they are really badly outnumbered they still won’t engage, but it’ll give the feeling to the player that the AI is being more aggressive because it’s being more forward’ with its units.
It’s more willing to take chances and that sort of thing. And I think that’s going to be an easy and effective way to do that.
Alex It brings up a really interesting topic in terms of making an AI competitive, certainly makes it fun for different players at different ranks. But there’s also always that slight difference of making an AI competitive and one that's fun to play against.
It’s a fine line to balance, and for example I was watching a video that you released about the AI changes, and in that I think your testers were talking about the nukes, and they were saying that the AI’s learn to fire its nukes all three at a time, so you have much less chance of surviving, whereas the original AI just sent one nuke; so you had a bit of warning.
But I was thinking, well having one nuke sent beforehand is kind of good to have from a gameplay perspective because it does give you a bit of warning, so it sort of accompanies the beginners through the game in a little more friendly way.
Whereas firing three nukes at the same time, is more optimal but it’s maybe not necessarily as fun for a person that’s getting into the game.
So my question is, how do you balance these two things, is it a constant to and fro; are you constantly tweaking these things accordingly?
Mike I actually put up a poll last night up on our community forums asking the community what type of AI they like to play against. I gave them 3 options; one being an intelligent but difficult AI, one that’s less intelligent but less difficult, or one that’s dumb and cheats to make up for it.
And actually I have gotten only one vote on the less intelligent, everybody has voted for intelligent but difficult. They would rather, from their posts and their replies, they would actually rather have a more intelligent AI and to be able to control that difficulty through how badly the AI cheats or maybe have an option to adjust the AI’s aggressiveness or that sort of thing.
They would rather see intelligent behavior and have the control over how aggressive or how hard the AI is through other means.
Alex That’s really interesting to hear, I presume it’s the same kind of polls we get from AI Gamedev if we did a survey there, we’d get the same kind of results because the audience is in a way a little self-selecting. So I don’t know how much we can read into that.
But I think your patches are catering to this audience so it makes a lot of sense for your patches to address these exact issues.
Finally a last question for wrapping up; do you think this entire infrastructure of the AI that you extended for Supreme commander 2, do you think that’s reusable to a different game of the same genre?
Mike Yes I really do. I just got done integrating all of my recent AI changes to our next game Kings and Castles, it’ll use the same core, but it’s obviously going to be quite a bit different.
I’m going to make some changes like I did by adding the team manager; I’m going to make other changes to make the AI more of a strategic character than a tactical one; so it can make decision in advance and it makes better decisions on where to attack and that sort of thing, and has more of a pre-planned attack,
Not so much of this reactionary and that sort of thing. It’s just making tactical decision at the moment, I’d like to see something that makes the AI feel more human honestly.
Alex Well to finish off, this is a screenshot of me loosing that final battle loosing against your platoons that were a little too timid... but do you have any final words you’d like to wrap up with, any final thoughts that you’d like to share with people?
Mike Yes, I can’t wait for Kings and Castles, because that’s going to be all me, I’m going to be the lead AI engineer on that, so that AI is going to be all me!
Alex I’m looking forward to that game! and looking forward to the follow-up interview that’s going to be awesome! Mike I’d like to thank you so much for your time, it’s been really fascinating to hear about all the stuff you’ve done on the game, and congratulations again on the patches. I’ve very much enjoyed playing the game, and I recommend it to everyone as well.
Mike Thanks for having me, it was a lot of fun!
Alex See you next week! Ciao!
Here's a specialized format for you to download and play offline via a portable player:
Post-Release AI Improvements in SUPREME COMMANDER 2 — Video Mike Robbins, Alex J. Champandard Download MOV (QuickTime) or WMV (Media Player)
The MP3 file below is better quality than the streaming video above, and is a perfect candidate for listening to via a portable player (96 Kbps). The OGG file is the highest quality of all (128 KBps). You can download them both here:
Post-Release AI Improvements in SUPREME COMMANDER 2 — Audio Mike Robbins, Alex J. Champandard Download MP3 or OGG
The slides used for the presentation itself can be downloaded from here for further study:
Post-Release AI Improvements in SUPREME COMMANDER 2 — Slides Mike Robbins, Alex J. Champandard Download PPT or PDF
If you have any questions feel free to post them below, or in the forum thread associated with this post!