When your job is to build an intelligent system, the more time you spend testing and tweaking the better the results will be. Conversely, if you waste half of your working day waiting for something to happen, then you’re not at the peak of productivity! These problems don’t only occur on big budget games, but also creep up quickly in smaller prototypes as well.
In practice, there are many things you can do to remove bottlenecks during the development. In fact, generally speaking, they fit into three categories:
Building — Tips 1-7 will help you spend less time compiling and linking your game code, and the necessary assets.
Loading — Items from 8-11 cover the aspect of running the game and loading in the data to setup situations that you want to test.
Testing — Tips 12-17 look into the interaction with the game and the AI, particularly how you can gain time here too.
The following tips are grouped, but in no particular order — except the first tip which is at the top for emphasis…
Automated testing of classes at a low-level has many benefits (including helping solve hard problems), but one of the most underrated is probably the speed of development. Adding new features backed by tests is very fast, and you can check the behavior of your code almost immediately.
Tip: The secret is to identify bits of code that are not plumbing connections between parts of the game, but instead core algorithms and data-structures. Develop these with tests, then once it works you’re ready to integrate into the game. With experience, you can end up spending very little time in the game when the first integration works without hiccups.
On large games, build times really kill any notions of interactive development. Despite using good coding practices, it’s not uncommon for just the engine to take 45 minutes or even over an hour to compile and link.
Tip: If you have a network of machines or a server farm, then investing in distributed build software will pay off very quickly. The compilation (and sometimes the linking) can be spread over multiple machines to bring full build times down to around 5-10 minutes.
Edit and Continue
Of course, avoiding full builds is paramount; incremental building and linking is enabled by default. You can even use Edit & Continue (in Visual Studio) to speed things up even more without having to restart the game. However, some distributed builders don’t support this feature, or it can be buggy.
Tip: Even if it involves building the whole game normally (during breakfast/coffee or via an automated job), make sure that you can use Edit & Continue no matter what. Often, as long as you don’t change headers, you can easily hot-swap code in a live game.
Libraries and Branches
Parts of the game, like the core libraries, don’t change very often. When building AI, it’s also not entirely necessary to stay up to date with the very latest version of other libraries, like graphics. Especially if your studio is developing the rendering engine or the asset pipeline, things can become unstable at times and keeping up to date takes time.
Tip: Consider isolating the AI and its logic from other parts of the game that are not entirely necessary. Either use a separate branch in source control, or build dependent libraries once and for all to reduce build times even further.
Mixing Build Types
It’s nice to be able to see inside the code you’re working on using the debugger. A release build gives you very little information about the stack or the content of variables in memory. And compiling the whole games in debug mode is rarely feasible as it gets too slow at runtime.
Tip: Try having a separate configuration that uses release libraries from the rest of the engine, and keeps the AI logic in debug mode. This provides the best of both worlds, since the game loads faster and the important part can be debugged.
Dynamically Linked Libraries
Another way to keep the AI code simple is to store it in a DLL. If your engine is robust enough, you can let it stay up and running and hot-swap parts of the engine in and out.
Tip: Put the AI logic into a separate dynamic library, then allow your engine to unload and reload new libraries at runtime. However, beware of memory and objects usage to prevent crashes once the new DLL has been loaded.
Only certain parts of the game and technology are suitable for being compiled to a DLL. Other parts might be more suited to a dynamically interpreted language which has much faster turn around times by avoiding the compile cycle.
Tip: Don’t hesitate to implement logic and behaviors as scripts — or even potentially prototype low-level algorithms as scripts too! Simply implement hot-reloading of scripts into your engine from the start to speed up development by an order of magnitude.
Screenshot 1: Distributed builds help speed up compilation times.
Low Graphics Test Level
Most of your game’s production levels may take a fair bit of memory and time to load. Even if this only a handful of seconds, it’s still wasted time on a regular basis.
Tip: Get your level designers to create simple levels without textures, only vector graphics and no lighting. Make sure your graphics team supports this rendering codepath! Then create many different levels for the situations you need to build your AI for; individual behavior levels, squad test levels, general sandboxes.
When you’re working in the sandbox, you can further reduce the time required to reach the specific situation you’re interested in by using scripts.
Tip: Setup situations exactly as you need to test them and write them as scripts. That way, immediately after entering the game you’ll see the results you’re interested in.
Similarly to the previous tip, you can use save games to store situations — but this time in full production levels.
Tip: Allow your game to load and save arbitrary situations, and restore them in a full level to reproduce bugs or improve behaviors in the final game environment. Feel free to use save games from your level designers or testers also if they have problems.
No matter how simple your in game menu is, it’s probably not the most efficient way to setup a specific situation that you want to test. You can combine the use of command lines with any of the tricks above to speed things up even further.
Tip: Build the game application such that it recognizes command line parameters, and set these up via your project settings. Then, when you launch the game, it can go directly into the level, script or save-game you choose. For bonus points, make it easy to send commands to the game via a socket or pipe.
Screenshot 2: Sandboxes with simplified graphics speed up load time.
Customized Game Logic
When you’re testing AI that integrates tightly into the game logic itself, it can take a while for you to reproduce the situation that you’re interested in.
Tip: Don’t hesitate to tweak the starting state of the game, make it easy to fake changing the game state, and set parameters such that you don’t have to wait too long (e.g. short intermediate states, cut-scenes skipped by default, etc.)
Debug Tools and Commands
While setting up regular situations quickly is important, it also helps a lot if you can play with the game simulation at runtime.
Tip: Build yourself an in-game debug menu with options that can be selected recursively using single quick-access keys. Also consider using an interactive console for the special one-off commands that you may need.
Stored Settings and Parameters
Even if you’re testing various situations and fixing different bugs, specific views of the game and debug rendering always come in handy for AI developers (e.g. drawing the navigation mesh, character statistics, etc.)
Tip: Have a simple configuration (.cfg) or initialization (.ini) file for your various debug settings, and support saving them from the game when necessary. They could even be saved automatically when the game shuts down.
The obvious solution to computer speed problems is to not use them! Paper is always available and requires no waiting.
Tip: Whenever you have a moment of pause, scribble down ideas, notes, the next steps of things to do. Use the time you can’t play with the game constructively.
Interleaved Editing & Testing
When the game is running and you’re waiting for something to happen, you can still be doing edits to the source code or scripts at the same time.
Tip: If you don’t want to use Edit & Continue, set the compiler to ignore your new changes while the game is running and don’t build the game until the testing is done. Consider using the test framework while the game is running.
Active Focus on Removing Bottlenecks
When you get bogged down with implementation details, it’s easy to get into a routine and forget to take a step back and remember what’s important.
Tip: Keep track of how your time is being spent, and actively work towards reducing the biggest costs. This pays off tremendously in the long run — especially if you share your tricks with your colleagues.
Do you have any tricks that you like to use to speed up your prototyping, testing or tweaking of game AI?