Article
i/2008/08/tf_demo

Play First… Code AI Later

Dave Mark on August 13, 2008

This week, Dave Mark tries to take a break from thinking about AI, but ends up doing it anyway! Do you play your games this extensively before implementing the AI? Join the discussion below by posting a comment.

KA-BOOoooom!

An interesting thing happened to me on the way to the second capture point the other day… I was playing Team Fortress 2 as I have been inclined to do when I am burnt out and need a break from AI, coding, writing my book, writing this column or whatever. However, I have found that it is not that easy to shut of my AI-inclined brain. As I am racing about the various maps in an elaborate dance that is a peculiar mixture of repetitiveness and instantaneous reaction, I find that I am not just thinking about what I am doing… I’m also thinking about how an AI agent would “think” about what I am doing.

This may seem a triviality — and, in a way, a sort of skill set that is necessarily endemic to game AI designers and programmers. However, there was a different cast to what I was doing. When I play single player games, for example, I am always analyzing not only my own play, but that of the enemies. I even write (irregularly) about my observations on one of my blogs, Post-Play’em. In a non-symmetrical game such as Halo (no Damian, I haven’t written about it yet), these two streams of thought are somewhat disjoint. I can analyze what I do and what they do. In a reasonably symmetrical game such as Civ 4 (No Soren, I haven’t written about it yet), however, those two analyses tend to merge somewhat. What I am doing (or should be doing) is something that can be mapped over rather well to what they are doing (or should be doing). This, of course, gives me enormous insight into the possibilities of how the AI was designed and coded.

“If the player looses, he should at least know why and learn from it.”

Using Civ 4 as an example (OK fine, Soren… now I’m writing about it, aren’t I?) (Note to everyone else, he told me that he has been patiently waiting… so I’m tweaking him a little here.), I can look at things like the influence maps that are very exposed in the game interface and see how the AI may be making decisions on troop dispositions. I can look at all the different values for food, trade, and production that are being generated and see how the AI may be making decisions on what to do with certain squares in the city radius. Thankfully, the AI seems to be making reasonable decisions in this manner — which allows me to let the city managers do the same for me for the most part. Still, it is interesting to watch the decisions change over time… allowing me to infer formulaic thresholds and whatnot. The point is, I can learn a lot… not only about the AI’s design but about my own play as well. (Soren quote: “If the player looses, he should at least know why and learn from it.” Apparently my education is ongoing. *sigh*)

On-the-Job Training

Back to TF2… as I mentioned, I have found myself noting a lot of things as I play. For starters, as I originally started playing a few months back, I had to look to other people to learn some of the tactics that were specific to the different maps and the different classes that you can play. I have been more successful with some classes than others. I still can’t get a handle on being a good Spy, for example. I don’t know why. Being a good Scout still seems to evade me — so to speak. As you can see from my Steam ID, I tend to alternate between Engineer and Demoman depending on whether my team is offense or defense at the time. (Upon careful observation, you can also determine that I’m not very good at the twitchy stuff. My teenagers tell me it’s because I’m old… usually when they are pwning me on Halo 3.) Now that I’m a bit more familiar with the game and I know the maps (2Fort, Dustbowl and Goldrush being my usually playgrounds), I can definitely speak to tactics, however.

And that’s where I get annoyed. This time, however, it is not at a computer AI… it is with real people. Some of the things that people do completely fascinate me in their ineptness and, really, lack of comprehension. This is nothing new, though. I sometimes get the same feeling driving my car or waiting at a fast food counter. What really gave me the epiphany in this whole process was the fact that the types of errors that I was watching live people making looked like the errors that AI agents would make. And that made the whole process of trying to determine how to “solve” the problem a lot more challenging and interesting.

Allow me to toss out a few anecdotes (which will, admittedly, read like a rant or a list of pet peeves):

Fun with ballistic trajectories!

Shooting Blind

As I mentioned, I play as a demoman a lot. I mean a LOT. Why is that? Because I like the indirect nature of the attacks. In fact, I excel at that. What confuses me is that other people do not even consider it. Unlike most of the other classes in the game… or many other games, for that matter… the demoman’s weapons don’t go in a straight line. They go up and come down. Welcome to Gravity 101. What this means is that I can shoot over stuff like walls and even entire buildings. If I know there are bad dudes on the others side of that wall, I will just stand back and lob pipe bombs over the top. As visually pleasing as animated carnage is at times, I don’t need to see dude die. I will be dutifully notified by the indicator in the upper right corner of the screen. I have used this tactic in so many situations on so many maps, it is silly. On the last level of Goldrush, for example, as you make the final turn to the base, there is a building with an elevated path behind it. From there, I can lob over the building all the way into the base area. What’s more, if your team is fighting around that corner (and anyone knows you can kill 15 minutes there), none of the defenders are likely to pester you back there. Get a dispenser, and you are in business… just keep spraying stuff back and forth like a killer lawn sprinkler! But how come no one seems to do that?

“It seems that there is this subconscious need to see what they are shooting at.”

On the other hand, I watch demomen keep walking around corners and pitching bombs down alleys just like this one when they could have safely lobbed those same bombs over the corner of a wall. It seems that there is this subconscious need to see what they are shooting at. Sure, if you are trying to take down a sentry, it helps to be accurate. But if you are just trying to make a mess of the guys that are milling about, accuracy is overrated… by the time your bomb gets there, they are going to be gone. So why do you need, to see what you are shooting? Especially if it is quite likely that it will shoot back?

Taking that to AI, I realized that this is a similar problem that we face when deciding when to shoot. Our natural instinct is to cast a ray and see if there is a LOS to the target… just like the human player is doing. Calculating the ability to hit someone in ballistic fashion, is a bit more complicated, of course. What’s more, we also have to put in the logic that we are quite willing to take a “blind barrage” approach rather than a “shoot to kill” approach.

Yep… I can shoot around corners!

Pipe Bomb in the Side Pocket

The same can be said for the fun ability to shoot around corners. If we were creating a pool (i.e. billiards… not swimming) simulation, we would be very cognizant of the fact that we have to take bank shots into account. However, if we were creating the AI for the demoman, this would be a little more obscure. Of course, if you have played a demoman for anything length of time, you have likely realized the power of the bounce. Not only off of walls, but even off of posts and railings. But, surprisingly, many people don’t use it. Is it because they don’t even think in those terms? Is it because they can’t intuitively calculate where they need to shoot in order to hit the desired area?

Again, this would be a similar problem in AI. Calculating where to shoot is trivial for a computer. It’s a simple exercise in geometry and physics. However, having an AI agent determine that those calculations should even be started is another matter entirely. When is it reasonable? This is similar to the problem above.

Dude… give it up already!

Why Don’t You Just Paint a Target on that Sentry

Next pet peeve… engineers who put sentries out in the open — especially when there is a long distance shot right at the sentry. I’ve often voiced the point that “Uh… demomen and soldiers can shoot a lot farther than your sentry, folks!” But what are the calculations for this? Obviously, there is a control radius for a sentry and you don’t want the enemy to have a straight shot at you from beyond that area. It sounds simple, but I can see it getting kinda funky.

Sticking with engineers, it is amazing how powerful a network of cross-facing sentries are. This is especially helpful in dealing with spies that enjoy wrecking machinery. As the spy comes visible next to one sentry, the 2nd one can take him out. And yet people seem to not realize the value of this.

I’ll scratch your spy, you scratch mine.

From an AI design standpoint, both of these problems could be solved somewhat by cheating… that is, tagging the map with desired sentry locations. However, that really isn’t AI, is it? Part of being an intelligent agent is that determination process of “what should work” and “what doesn’t work”. It would seem that there needs to be a calculation that would involve the good place for a sentry when paired with another sentry.

“One level 3 sentry is better than three level 1 sentries!”

One last note on Engies… one level 3 sentry is better than three level 1 sentries. Sheesh… it’s math, people! And yet, many guys are blissfully pounding away on their own little sentry rather than using the valuable resource of metal and, more importantly, time to work together and build sentries one at a time. (BTW, 5 engies working on the same sentry can be built to level 3 in about 8 seconds, I believe.) And yet how many times have we seen AI agents doing the same thing? No awareness of each other’s projects that may be futile when taken individually.

Now is Not the Time…

Another thing that fascinates me is watching how an engineer will build a dispenser or a teleporter when there is less than a minute to go in the round. What is it you are trying to accomplish here? It’s like how some cultures would bury food with a dead body. Ya know, the thought is nice, but it probably is going to go uneaten.

AI Solution? A quick fix is a check of the timer… if the time remaining is less than a particular threshold, don’t build a certain unit. However, is there more of a comprehensive way of approaching this? If you were to combine the above notion with another observation, we may have an answer. Read on…

If you are pinned down defending the final capture point on a level, you and your buds are probably not going to be running very far. Sometimes, it’s all of 20 steps from the spawn door to where the action is. Why the heck would someone be building a teleporter entrance? Sure, they are handy to use to get up to sniper platforms or something, but not as urgent as some other things. And I certainly wouldn’t leave my sentry and dispenser combo unattended to go and accomplish this. So maybe there would have to be a general “what is the utility of a teleporter” value. If you aren’t going to have time to use it or if it won’t be worth using it to go all of 30 feet, then don’t bother.

… the Time is NOW

“It’s not just about killing people… it’s about killing the right people!”

I have often been heard saying two things. When there is about 2 minutes left in a level and my team is defending, I will often remind people to “tighten up… let them come to us”. Sure, I’m channeling Sun Tzu to an extent, but it makes sense. The worst thing that can happen is for your team members to go wandering off, over-extend, maybe get killed off, and therefore let people through the cracks to capture the control point, push the cart or whatever. I can’t tell you how many times I have seen a massive push in the last 30 seconds because people were out running around trying to be heroes and, when 2/3 of our team is dead, there is no one left to defend the base. On the other team, if you need to push the cart with 20 seconds left, everyone should be on that cart… snipers, spies, engineers… they can’t kill ‘em all. And yet I see people standing around doing their usual gig figuring someone else will capture the point or whatever.

On a related note, I have also been heard after a spectacular loss saying “It’s not just about killing people… it’s about killing the right people!” This can be applied to a lot of situations. If there are two people about to capture a control point, the sniper back in the hall is not terribly important. If a single Soldier or Demoman is wreaking havoc on all your Engy’s equipment from some perch (1st stage of Goldrush is notorious for this), he is a high priority assignment. Take him down. If there is a Medic/Heavy combo cruising around… don’t try and kill the Heavy while he is being killed… kill the Medic first — that Heavy will be a lot easier when he isn’t receiving a constant stream of steroids.

Going up?

You would think all of this is obvious, but people just don’t seem to get it. But how would I do it in AI? Obviously, the proximity issue is something that can be solved by proximity… especially when combined with time. If the clock is almost out, building that teleporter or sapping a remote dispenser isn’t high priority. So all of the values that we would normally assign to a task need to be indexed to the remaining time somehow. The AI for a spy would normally have “pushing the cart” as his lowest priority… as the timer clicked down to the end of the round, however, it would suddenly be a serious consideration.

Cheers, mate!

A similar sort of value system would be put into place for killing enemies. That Soldier killing all the sentries from up on the balcony? Never mind the dude you are chasing around all over the place, lob some grenades up there, would ya? The guy on the cart gets a priority boost over the guy on the other side of the map. The Medic healing the Heavy is a lot higher priority than the Heavy himself… even though said Medic isn’t damaging people like the Heavy is.

My point is, people talk about how the playing online is much more engaging and challenging than playing against AI agents. But yet, with all of the observations that we can make about the online players (or even how we play the games ourselves), we should have a massive amount of information on how we can go about making some of the more abstract strategic or tactical decisions. The only layer missing is how to convert those observations into a representation of knowledge and then code to make decisions based on it.

Is that really all that hard? On the other hand… as long as people keep trying to attack ubered enemies, it makes our existing AI look brilliant.

BTW, if you want to have some fun, there will be a TF2 bot API released soon! In the mean time, in the words of the demoman… “Cheers, mate!”

Discussion 12 Comments

zoombapup on August 14th, 2008

Nice post Dave, Weirdly enough, when I play online games like TF2 etc. Sometimes I deliberately WANT to play like an idiot. Thats kind of part of the fun for me, figure out what the team is doing and sometimes choose to take part, sometimes choose to do something completely off the wall. I find that occasionally that off-the-wall thing actually does better. People are by and large pretty predictable. Predictable to the point where other people learn to out predict each other, they start using lame tactics like bunny hopping round corners, or concussion jumping, or flash rushing. And truth be told, if you have slower reflexes like us old guys, then you're not going to be able to win in the predicting game. So you have to try and win by not being predictable. Sometimes of course, I do it just for the comedy effect.

Dave Mark on August 14th, 2008

[quote=zoombapup;4245]Sometimes of course, I do it just for the comedy effect.[/quote] That was [I]you?!?![/I] :-)

Ian Morrison on August 14th, 2008

Sometimes the comedy is part of normal play, though... especially with TF2. I swear, the pyro can't do ANYTHING that isn't hilarious. Especially since the latest update, where punting enemies off high ledges with the compressed air blast is one of the greatest joys the game has to offer... I second Zoombapup with the "off the wall" comment. Nothing causes an enemy to make mistakes more than throwing them off their routine, especially as most classes in TF2 are built such that they have a few ways to throw people off... engineers can choose unconventional sentry positions off the beaten path, soldiers rocket jump, pyros are custom built for creating anarchy, and spies are practically built for the purpose of throwing a monkey wrench into the well oiled machinery of the other team. It's probably a good thing to think about on the subject, actually... finding the patterns and pre-cooked strategies for the map are crucial to upsetting them, and predicting typical enemy actions. There's no way a reactionary agent could handle them though... some kind of "memory" is required, be it machine learning, data crunching, or mapper assistance. Back on topic, I suppose. I think your article really points out the difference between simulating players and simulating good players in bots. A lot is often said about making bots make human mistakes instead of just toning down their reaction times and accuracy, and this kind of boneheadedness is exactly what we need to be simulating (and then removing on higher difficulties!)

Dave Mark on August 15th, 2008

[quote=Ian Morrison;4270]Back on topic, I suppose. I think your article really points out the difference between simulating players and simulating good players in bots. A lot is often said about making bots make human mistakes instead of just toning down their reaction times and accuracy, and this kind of boneheadedness is exactly what we need to be simulating (and then removing on higher difficulties!)[/quote] Keep in mind, I'm [URL="http://aigamedev.com/discussion/mistakes-look-smart"]a big proponent of simulating mistakes[/URL].

Andrew on August 15th, 2008

Neat reflections on some of the tactics in TF2. I think many games have better user-made AI simply because these kinds of things can be easily thought of during gameplay. No doubt there is a big advantage to doing it in a symetrical game too - many maps in TF2 are symetrical, and the sides can have the same abilities (to a degree, with these unlockables changing things a bit). I'm all for mistakes, but yes, the general feeling from me is the bottom third of a team generally don't make mistakes, they simly fail to grasp how to play the game at times, doing the same thing over and over again, or not getting on with anything at all :) We'll see what the TF2 AI does, but it *looks* like it is using designer-placed things from the demo videos. I hope it won't entirely be stuck with that, it'll cause a lot of monotony - which is one thing TF2 avoids in some cases, since Dustbowl, one of the most popular maps, simply has a lot of different ways to attack and defend - if the engineers and defenders all place things in the same places all the time, it'll be the same game again and again due to them having such a big affect on the opposing team's tactics. It does help initially, but it gets easy to beat something that does the same thing over and over :) A lot of random > predictable in some cases works well!

Dave Mark on August 15th, 2008

Agreed... a lot of times I will put a sentry in a non-standard location just to take people by surprise. Is it random? No. Is it thought out? Yes. Is it effective? Sometimes. As to your comment about the difference between mistakes and not being effective... that's the overall point I was trying to make about our AI agents. Many times, we have agents that would fall into that bottom 1/3rd. Not from a learning standpoint, but just from the standpoint of not acting [I]as if[/I] they had been playing for a while and had gotten the tactics down for the most part. The trick is, look at the people that are playing the game and their differences and determine "what calculations are the better people making that the poorer ones are not?"

PaulTozour on August 16th, 2008

Great article, Dave. When Valve decides they need AI bots for this, you can write the "Demoman AI," and I'll sign up for the "Heavy AI." :) I've found in the past that I've learned a tremendous amount about AI from playing a game I was very good at (in this case, the Desert Combat mod for Battlefield 1942), recording videos using FRAPS, and then finding myself using all sorts of advanced tactics I hadn't even realized I was using. Once I really sat down and figured out everything I was taking into consideration each time I pwned a n00b, I gained a lot of insights into game AI and how to write a system that thinks like I do.

zoombapup on August 17th, 2008

Yeah, thats an important point Paul. We're not trying to create "intelligence" we are trying to mimic its result. We are trying to create behavior that follows the rules created by intelligence, rather than try and create intelligence for its own sake. I think that is the distinction between "regular" AI and game AI. Having said that, some people think otherwise. But its my take on things :)

Dave Mark on August 17th, 2008

[quote=PaulTozour;4301]When Valve decides they need AI bots for this, you can write the "Demoman AI," and I'll sign up for the "Heavy AI." :) [/quote] Oh great... "stand in the middle of the place and spin slowly while shooting non-stop". You really are taking on a big load with the Heavy, Paul. Sure you can hack it? ;-)

Dave Mark on August 17th, 2008

[quote=zoombapup;4303]Yeah, thats an important point Paul. We're not trying to create "intelligence" we are trying to mimic its result. We are trying to create behavior that follows the rules created by intelligence, rather than try and create intelligence for its own sake. I think that is the distinction between "regular" AI and game AI. [/quote] Absolutely. The choice is... where do you put that layer of abstraction necessary as a starting spot? Something like scripting puts it pretty close to the top layer (right under cut scenes) where an agent is simply told what to do. At the absolute bottom is all the jazz that is trying to mimic biology. Someplace in the middle is where we get world knowledge representation and algorithmic decision making processes. It's low level enough to be flexible and deep, but high level enough that we aren't trying to do the e-Frankenstein bit. As I explain in my book on a couple of places, if we encounter a place in mimicing decisions where it would be difficult to create agents whose collective decisions would form a bell curve, then we simply need to use the bell curve as the [I]basis[/I] for the decisions. In the end, it looks the same.

anglewyrm on August 24th, 2008

[QUOTE=Dave Mark;4293]... a lot of times I will put a sentry in a non-standard location just to take people by surprise. [/QUOTE] This seems like the subjective experience of a mind using a [url="http://en.wikipedia.org/wiki/Strategy_(game_theory)#Pure_and_mixed_strategies"]mixed strategy[/url]. A model of this might be: Knowing that good opponants will anticipate the standard locations, the set of possible locations is increased to include less optimal choices, and then randomly selected from the larger set. The probability of using a given location from the set of eligible locations might be from memory by an accumulated win/loss ratio of that location. Player "Skill" could then be defined as how well the set of win/loss ratios match the game world. A noobie might have an idealistic view that includes a small set of obvious choices, with perhaps some color added from forum browsing. A more skilled player might give extra weight the latest results of the current game. And a masterful player might also have a memory that distinguishes the win/loss ratios of the individual opponants.

Dave Mark on August 24th, 2008

[quote=anglewyrm;4444]This seems like the subjective experience of a mind using a [URL="http://en.wikipedia.org/wiki/Strategy_(game_theory)#Pure_and_mixed_strategies"]mixed strategy[/URL]. A model of this might be: Knowing that good opponants will anticipate the standard locations, the set of possible locations is increased to include less optimal choices, and then randomly selected from the larger set.[/quote] Yep. Very much so. Good call.

If you'd like to add a comment or question on this page, simply log-in to the site. You can create an account from the sign-up page if necessary... It takes less than a minute!