Game Assets: Compilation

Many things go into making a game besides just the code; textures, icons, sound clips, music, models, and animations are just a few of the things you might need to create in order to have a complete game. Once created, the assets need to get into the actual game itself. One of the problems facing game developers of all scales is creating a content pipeline, such that artists and content producers can create and edit their assets with rich editing capabilities and fast iteration times while the game itself can load a stripped down and efficient representation at runtime to reduce memory consumption and load times.

I was surprised when a quick search showed few tools to help facilitate this process for smaller developers who don’t have the resources to spend creating a full featured content pipeline. If you’re using XNA or Unity3D, you get decent support out of the box. Otherwise, you’re left to your own devices, and I imagine many developers simply go the route of ignoring the content pipeline completely. This can save time up-front, but often leads to inefficiencies down the road when trying to rapidly iterate on pieces of your game. Thus, I decided to write my own content build tool, which I’ve called Ampere. You can find it on GitHub here.

What does this tool do, and why is it useful? The asset compiler is similar in idea to the many source code build systems available; you give it a build script, it follows dependency chains and runs builds when necessary, avoiding a build if the source assets haven’t changed. The major difference with a content build system, as opposed to code, is that code project builds are very specific, so each source file is listed explicitly in the script. Additionally, there’s really only a few things you can do with source code, compiling and linking being the major ones. In contrast, game assets have wildly varying needs based on their type, and you often want to run several tools in a chain against a whole *class* of content, rather that on individual files. You don’t much care to only run your texture processor on ground1.png, you want to run it on all ground textures in your game at once.

Working with Ampere involves writing a build script in C# that looks something like this:

// --- Test build script --- //
#r "Plugins.dll"
using Plugins;

// set up the build environment
Env.InputPath = "./";
Env.OutputPath = "../Products/bin/Debug/Data/";
Env.InputResolver = Resolvers.Flatten(CreateDirectoryCache(Env.InputPath));
Env.CreateOutputDirectory = true;

// create processors
var materialProcessor = new MaterialProcessor();
var shaderProcessor = new ShaderProcessor();

// parse materials into binary blob
Build("*.material")
    .Using(materialProcessor.Run)
    .From("$1.xml");

// compiling shaders
Build("*.*.*.shader")
    .Using(shaderProcessor.Run)
    .Run("%WinSdkDir%/fxc.exe", "/nologo /E $2 /T $3 /Zi /Od /Fd $(TempName).pdb /Fo $(TempName).fxo $(Input[0])", "$(TempName).fxo")
    .From("$1.hlsl");

// start builds
Start("default.material");
Start("sprite.material");

// notify of new items and run again
Notify("127.0.0.1:9001");
RunAgain();

When run, the tool finds “root” assets, listed above in the Start() calls, and kicks off content builds for them based on their name and the build rules defined for your various asset types. Each asset build is run asynchronously, and kicks off further builds based on any dependencies it may have. Additionally, the tool tracks changes to source data, so it avoids rebuilding assets that haven’t been changed since the last run.

Finally, when a build completes, Ampere can optionally connect to a TCP port and send off a list of all the assets that were built. This can be used to notify a running game about content changes, prompting a hot-swap of assets on the fly. This can greatly reduces the time taken to test content changes in your game, and in turn lets you avoid writing a whole class of “visualizer” type tools, since the game itself can act as a perfect visualizer for much of your content.

I made the tool with my own needs in mind, but hopefully others will find it useful as well. If you do end up using it, or if you have suggestions or bug reports, let me know. I’d also be interested in uploading any content processor plugins you develop so that others can find ready-made solutions for common game asset types.