XNATools

Once again, I go for a significant length of time without any postings. The primary cause is work; been swamped, blah blah. Anyway, during my rare downtime I’ve been thinking that the content processing toolkit needs to make the leap into the modern era and be written as a web app. Queue learning of many new technologies and languages. Java, Ruby on Rails, etc. Not happy with any of them; don’t like having to constantly switch between languages, don’t like having to take two weeks to just get simple shit done with Java, blah blah the usual. Then comes the realization. The days of the ‘traditional’ programmer are very, very limited.

What I mean is that knowing just the traditional languages and paradigms isn’t going to cut it anymore. If you’re thinking in terms of traditional concepts like applications, clients, servers, and so on, pretty soon you’re going to be out of a job. The future is all about the production, manipulation and distribution of data. Useful functionality is built up is small bits, and every single one of those bits is connected. Networking functionality is absolutely core to everything. And it’s obvious that traditional languages and development techniques simply aren’t going to cut it.

This isn’t to say that being a proficient C/C++/C#/Java/whatever programmer isn’t useful – those skills are still essential. But it won’t be enough to save your ass in the coming years. You also need to be proficient in web technologies. JavaScript will be key. The browser isn’t the operating system of the future, but there isn’t going to be any distinction between a ‘browser’ and an ‘operating system’ – it’s all one and the same. I don’t know that the specific implementation of node.js is where it’s at, but I know it’s at least heading in the right direction. And it’s nice because I only can use the same language for ‘client’, ‘server’ and ‘tools’ (though there isn’t much distinction between them.) Time to get on with re-inventing myself and my skill set.

Step 1: Install nvm from here.
Step 2: Install one or more versions of Node.js using nvm.
Step 3: Select the version of node to use.
Step 4: Install npm from here (nvm install does this actually.)
Step 5: Install Sublime Text 2 from here.
Step 6: Install Sublime Text 2 Package Control from here.

OpenCP 1.0.0 is now released. It is available at:
OpenCP-src-1.0.0.zip

To the extent possible under law, Russell Klenk has waived all copyright and related or neighboring rights to OpenCP. This work is published from: United States.

As mentioned previously, OpenCP is structured around the concept of a content database. The content database is constructed during the content build process for a specific project and target platform combination. It maintains four different types of records:

Source content records are constructed by the build tool at the beginning of a build; source file records, target asset records and target file records are created by content conditioners during the build process. Links between the records are also constructed during the content build process. For example, a source content record maintains references to the source files used as input and the target assets produced as output. The content database allows an application to ask the following questions:

Maintaining the links between records can be a bit tricky, so OpenCP attempts to simplify the process. An application can query the content database for source content and target content by name or index, and can query for source and target file references by path or by index.

The nice thing about OpenCP is that it makes this information available to any application that wishes to use it. The OpenCP library includes a .NET implementation; a C/C++ implementation is coming soon.

The following diagram gives a basic overview of the OpenCP build process. The blocks and arrows in red are application-defined portions, while the blocks and arrows in blue are things implemented by the OpenCP library.

The process begins with application-defined source content, composed of source files and possibly (likely) some metadata. The files and metadata are in application-defined formats; OpenCP places no restrictions on the data or the file formats. This differs from previous iterations, where the content pipeline library defined the metadata format.

The build tool takes this data and transforms it into a series of one or more OpenCP source content definitions, which are persisted in the OpenCP content database. Each source content definition has a unique content ID (a simple string name) within the database, zero or more references to source content files, and some optional metadata, represented in a generic key-value form (both keys and values are strings.) Content processor implementations can access this metadata during the build, providing a mechanism for passing variable parameter data to control the content build process.

Each source content file reference maintains a hash of the file contents, last write timestamp, size in bytes, and path (relative to the application-defined project root – this allows projects to be easily rebased.) OpenCP uses this information to optimize the build process, so that only modified content gets rebuilt. If a piece of source content has not changed (and all of the associated target files still exists, and are unmodified) then the source content is not rebuilt, and any output is simply reconstructed from information stored in the content database.

The actual content build process is not controlled by OpenCP anymore. Instead, the application-defined build tool can load the content database for the build target platform, and query the database to determine whether a particular piece of source content needs to be rebuilt. Again, this allows the project to fully define the content build pipeline; OpenCP just takes care of the parts common to all content build pipelines.

Content processors in OpenCP serve largely the same purpose as they did in previous iterations; they receive a source content definition and produce zero or more target assets (each with zero or more target files.) Different content processor implementations can be specified for different target platforms, so it is now possible to specify a content processor that outputs textures using DXT compression for OpenGL and Direct3D-based platforms, but outputs textures in PNG format for the Flash platform (as an example.) The same content processor implementation can also be supplied for multiple target platforms.

Deferred content processors are a new feature. Deferred content processors are intended to perform any ‘global optimization’ portions of the build process. A great example is building texture atlases or sprite sheets, where information about multiple source content items is required to produce output textures with the least amount of wasted space. Execution of deferred content processors is an optional step in the build process.

The content database is the key to enabling all of this functionality, and the information it tracks will be the topic of the next post.

Tomorrow, I’ll be posting the first release of OpenCP, the next iteration of the xnatools content processing toolset. Work on the OpenCP library was completed in November of 2010 (why does everything seem to get done in November?). There are quite a few major differences between OpenCP and previous iterations, not the least of which is that it is cross-platform, running on the Microsoft .NET Framework on Windows, and Mono on Linux and OS X. It still targets the .NET 2.0 runtime. Since it has absolutely nothing to do with XNA, I figured it’s time to drop the XNATools association.

Unlike previous releases, OpenCP provides only the foundation for a content pipeline. I realized that for the library to be useful, people are going to need to provide their own content definition formats and customize the build system to their workflow. OpenCP provides the basic content build framework, including dependency and reference tracking, minimal rebuilds, pluggable content conditioners, target platforms, and so on.

The biggest change is the the switch to focusing on a content database model. Upcoming posts will provide more information on the content database and how it is used by OpenCP and build systems. The major benefit is that a content database can be loaded and dependency, reference, input and output information can be used by a number of tools, which when combined make up the full content pipeline. For example, the content packaging process can now be implemented entirely independently of the content build process.

Edit: OpenCP 1.0.0 is now released. It is available at:
OpenCP-src-1.0.0.zip

To the extent possible under law, Russell Klenk has waived all copyright and related or neighboring rights to OpenCP. This work is published from: United States.

I occasionally have to look this up at inopportune times (like when pulling all-nighters) so I’m just going to post the relevant info here so I don’t have to search for it. To enable CRT heap debugging, add the following to the top of the source file – order is important:

#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>


To track down heap corruptions, add _ASSERTE(_CrtCheckMemory()) before and after the suspect code.

Memory leaks: _CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF|_CRTDBG_LEAK_CHECK_DF).

Adding flag _CRTDBG_CHECK_ALWAYS_DF will add a call to _CrtCheckMemory() after every malloc and free.

In the previous revision of the content toolset, the base Conditioner class was essentially just an interface and provided no useful functionality; it was just there so we had a common base class to search for when locating content conditioners. The Conditioner class is still used to locate content conditioners, but it now has a number of helper methods that take care of most of the work when implementing a new content conditioner.

Most of the helper methods you wouldn’t use directly. They are used by the base Conditioner class to determine whether a rebuild is actually required, and if not, to re-generate the list of output assets (see the previous post for more on this). All of this is handled by the public content conditioner entry point, implemented by the ‘Process’ method.

The user-level entry point of your content conditioner is the private ‘Rebuild’ method. This method is called by the Process method only if it determines that the source asset must be rebuilt for some reason. A source asset would need to be rebuilt if:

The simplest possible implementation of Rebuild should perform the following actions:

Take a look at the no-op content conditioner in ContentPipeline/PassThroughConditioner.cs for an example implementation that adheres to the guidelines above. Assuming your conditioner actually does something (probably a safe bet), you will most likely end up adding additional source content items using Conditioner.AddSourceContent. You may also add additional output assets using Conditioner.AddOrUpdateAsset; there need not be a one-to-one correspondence between source assets and output assets.

To give a more real-world example, let’s say we implement a conditioner to generate a sprite sheet from one or more series of sequential image files. We have a single source content item, referred to as the ‘primary source content’, that is referenced by the asset metadata. This item would likely define the set of sprite animation sequences present on the sprite sheet, with each animation sequence consisting of a reference to one or more image files, a name, the playback rate in frames-per-second, and so on. A source content reference would be generated for each one of the referenced image files, and two output assets would be generated – one for the metadata (frame rectangles, etc. for each animation sequence), and one representing the sprite sheet image file.

What about dependencies? There will usually be assets that need other assets to be processed before they can be built. When processing these assets, you need to do two things. First, after creating the asset surrogate, your conditioner should add the asset ID’s of the referenced assets to the surrogate using the ‘AddDependency’ method of the Surrogate class. This allows the pipeline to determine whether any dependencies have been changed, in which case your conditioner’s Rebuild method will be called. The final step is to call the ‘BuildDependencyList’ method of the ContentPackage class, after you’ve updated the dependencies on the asset surrogate but before you continue processing the source asset. This ensures that all referenced assets are up-to-date before proceeding with your conditioner’s processing.

I think that the concept of an asset surrogate requires a bit more explanation. The term is a bit of a misnomer in the current system, but in the previous revision of the toolset, it was a 128-byte “stand-in” value for an source asset. In the current system, it stores considerably more data, but the intention is similar – this data helps speed up the build process.

Each asset definition file has a corresponding surrogate cache file, which stores one or more asset surrogates per-source asset. The surrogate cache file is generated during the build process by the content pipeline. Each surrogate stores the following information:

Each source content item represents a single input file, by storing a relative path to the file, the file content CRC, and the last write date/time of the file. Each output asset stores the asset ID, asset type, relative file path of the output file, and CRC of the output file content. A single source asset can generate zero or more output assets.

All of this information allows the pipeline to determine with certainty that a source asset does or does not need to be rebuilt by a specific conditioner, and if no rebuild is required, the pipeline can just re-generate the output assets using the saved data. This can speed up the build process considerably in most cases.

One last thing – if a source asset doesn’t specify any content conditioners that process it, it is still sent to a special conditioner (PassThroughConditioner.cs in the ContentPipeline project) that ensures any changes are detected and reported properly.

There are a couple of features from the previous version of the content processing toolset that didn’t make it into this new release:

First, there is no source control integration, and I’m not sure that I’ll add that capability back in. On the one hand, being able to pull directly from source control is nice if you’re using an automated process to build your final content packages for release or a major demo. On the other, it’s a pain in the ass and (IMO) not really desirable during development because you are building frequently, and every single source file that is referenced directly or indirectly by the content package must (potentially) be updated. With the svn plugin for the old system, I mitigated this somewhat by updating an entire directory at a time, but it was still a hassle and a slowdown. The new system just uses package-relative paths (so that everything remains relocatable) but leaves it up to the user to determine the specific file revisions they want beforehand.

Second, I’ve removed the multi-threaded build system and made it single-threaded. This was more to keep the initial (re)implementation simple, so I’ll probably add this back in soon. Instead of creating one content conditioner instance per-package, I’ll probably switch to creating one per-build thread to remove the need for most conditioners to be thread-safe.

Finally, the package file generation process has been completely removed, since there is no command-line or GUI compiler distributed with the release. I’ll post one as an example, but it’s quite trivial to build one that outputs files in your own custom package file format.