Archive for the Software Development Category
The following is a raw dump of my notes on the conditioning pipeline, minus most formatting (I’ll update if I can figure out how to indent properly). First some terminology:
Asset Definition File: A file containing one or more pieces of asset metadata. There are zero or more asset definition files per source content directory.
Asset Metadata: A single piece of metadata defining a source asset. The metadata specifies the location of the primary source file, the asset ID, and the list of content conditioners that should process the asset and (optionally) generate one or more pieces of output content.
Asset Surrogate: A value that specifies information about the files and metadata that are associated with a source asset and content conditioner pairing. An asset surrogate also stores all of the information necessary to regenerate the output content items if the source asset doesn’t need to be rebuilt.
To start, load all directory-level asset definition files. This produces a set of asset metadata instances. Each directory-level asset definition file also has a surrogate cache file; these should be loaded or created as well. The asset metadata uses the ‘location’ tag to specify the ‘primary source file’.
The role of the asset surrogate is expanded. Now, one surrogate object is defined for every asset+conditioner combination. The surrogate object contains the following data:
The asset ID.
The content conditioner name.
The content conditioner version.
The CRC of the content metadata.
A list of referenced asset ID’s (dependency list).
A list of all source files that contribute to the final output file.
The path is defined relative to the ‘primary source file’.
The last modification date/time of the file must be stored.
The CRC of each source file content must be stored.
A list of all output files generated by the content conditioner.
Paths are defined relative to the ‘primary source file’.
An asset must be fully rebuilt if:
No surrogate exists in the cache for the asset+conditioner combination.
The content conditioner version is different.
The asset metadata CRC is different.
Any of the output files do not exist.
Any of the contributing source files have a different mod time OR CRC.
Or they don’t exist.
Check the modification time first – less expensive than CRC.
In any case, when the package is mounted, every single piece of asset metadata must be passed through the content pipeline. The build process begins with an empty collection of assets. The conditioners populate this collection as they execute.
The list of conditioners is first extracted from the asset metadata’s ‘conditioners’ tag.
If the asset has no conditioners, the ‘primary source file’ is assumed to be the primary output file, and the build process completes.
If the asset has one or more conditioners, the following process occurs for each conditioner:
The surrogate is requested from the surrogate cache.
The full rebuild status is determined as described above.
The following data are passed to the content conditioner:
Asset metadata object.
Content package root path (accessible on metadata?).
Full rebuild status.
Asset metadata surrogate & cache (accessible on metadata?).
Content package asset collection.
If the ‘full rebuild’ flag is not set, dependencies are updated.
Use the reference list stored with the surrogate.
Pass this list to the BuildDependencyList method.
The return value is the new status of the ‘full rebuild’ flag.
A value of true indicates that the referenced assets changed.
A value of false indicates that referenced assets were current.
The content conditioner executes its build process, if necessary:
The content conditioner generates a list of referenced assets.
The BuildDependencyList method is called as above.
Primary processing is performed on the source content.
The content conditioner generates a surrogate and updates the cache.
The content conditioner generates zero or more output Asset instances.
Each asset instance has only the required properties defined on it.
Asset type.
Media type.
Production status.
Primary source file location (relative to package root path).
The conditioner first checks the collection of assets for the asset.
It may have been generated by a previous conditioner.
If it exists, the location tag is updated.
Once this process has been completed for all input asset metadata, the final set of output assets is generated from the asset collection.
This process should handle dependencies implicitly, since the full set of input source files is stored as part of the surrogate. If referenced assets have not been built yet, the full rebuild flag will be set since the surrogate will be invalid.
A complete collection of source files can be determined from the location tags of the asset instances in the collection of output assets.
At this point, the content package load/reload process is complete. In case the pipeline is being run in bundle builder mode, the asset collection can be set to the bundler process. Otherwise, the location tags can be used to load the data directly from the local file system. The final set of assets can be passed to the game, and all game-ready content is present.
 No Comments »
The next few posts cover some major changes to the content conditioning toolset. After considerable thought, I felt these issues needed to be addressed before proceeding with the release.
The primary issue is that the toolset in its current form only solves a very small portion of the problem. I was so focused on the actual content build process that I totally missed the chance to address real problem, which is decreasing iteration times, and making the process transparent to the user so that content can be refined without programmer assistance.
The secondary issue is that it was a major pain in the ass to write a content conditioner. It took significant effort to parse the XML metadata for complex schemas, and that’s before even getting to the meat of the conditioner.
To address these issues, I’ve gone back to the drawing board. The XML requirement is no more – I’ve switched to a simple key-value pair format for metadata. This was actually used by the previous toolset for specifying resource types; I’ve just expanded its use throughout the entire pipeline. The second major change allows the content pipeline to be integrated into the runtime engine using a simple sockets-based protocol.
The next post will give a brief outline of the content build process. You’ll note the lack of “solution” or “project” files, and the general simplicity of the whole thing…and so I don’t repeat the constant “not quite ready” cycle from last time, the code is all done now.
No Comments »
Life gets in the way…I haven’t even had a chance to touch the code since the last post. So, obviously there is no beta yet, and no corresponding site relaunch. As soon as I can, I’ll post the latest code, so it’s at least available, and get back to work on it. There are some bugs, for sure, but everything works as it should.
No Comments »
I’ve finished adding the final features to the main projects, but there’s still a lot to be done to make a real release – finishing up the base set of content conditioners (texture processing based on NVIDIA Texture Tools, geometry from COLLADA files), as well as installers for everything, some decent testing, and documentation.
To avoid a bunch of postings like this one, I’m targeting early September for the release, accompanied by a reboot of the site. Check back September 12th.
Russ
No Comments »
I was supposed to post an incremental update this weekend, but after some review and a lot of late-night work, I’ve decided that version 1.0 is feature-complete. So, instead of the incremental release, I’m going to spend the next week or so performing some testing and review of features, and release 1.0 beta 1 of the content tools software. The release will include source code as well as binary installers for Windows x86 and x64. Check back next weekend for the release.
No Comments »
I forgot to mention this in the previous post, but the use of mixed-mode assemblies like SharpSVN has required everything to be build for specific CPU’s (x86 and x64) instead of the generic AnyCPU. This required creating x86 and x64 configurations for each existing project. Additionally, since SharpSVN supplies both 32- and 64-bit binaries, the project file for the Subversion URI resolver had to be edited by hand. It wasn’t difficult, but it would have been nice if different assemblies could be referenced for different platforms in the Visual Studio UI.
If you build the source, you need to make sure that everything is being built for the same platform. Post-build events have been added to all projects to copy their output into a ‘.dist’ directory (in the same directory as the ContentTools.sln file) where they are organized by platform.
When the code stabilizes a bit more, and has been proven out by implementing various URI resolvers and content conditioners, I’ll start releasing x86 and x64 binary packages (MSI installers) for setting up client machines.
No Comments »
The next source update fixes some (significant) issues with loading of assemblies for URI resolvers, package builders and content conditioners. Additionally, it will include an example package builder, the Subversion URI resolver, and Content Studio will be updated with integrated build support.
Most of the work is already completed. I discovered the issues with assembly loading while working on the Subversion integration. There were a few major issues. First, there were a number of places where I was using Assembly.LoadFile() instead of Assembly.LoadFrom(). You can read about the differences on MSDN (here for example), but the important difference is that using Assembly.LoadFile() doesn’t search for dependencies in the same path as the assembly being loaded, whereas Assembly.LoadFrom() properly handles dependency loads.
I decided to use SharpSVN to help with the Subversion integration. This exposed the second major issue with the content pipeline’s handling of assemblies. Content Studio copies any referenced assemblies into a directory (Binaries) local to the solution. I was using reflection to load the assembly, and copy any assemblies referenced by the assembly into the Binaries directory as well. The problem is that a significant portion of SharpSVN is implemented as a mixed-mode assembly (mixed CLR and native code), and part as (I believe) a native-code exe. The reflection mechanism doesn’t report these references, and as a result, the Subversion URI resolver would fail to load because referenced DLL’s weren’t copied into the local Binaries directory. This would also be an issue with native code DLL’s that were P/Invoked, or native code command-line tools. All of these things are common occurrences during content processing, since you’ll often have an existing toolset you want to use, or it may just be easier to implement a portion of your pipeline in C/C++.
My solution was to allow the user to specify a set of additional files they want to be copied into the local binaries path when selecting the main assembly file. It’s not ideal, but it’s the only thing that’s (reasonably) guaranteed to work.
The third major issue has to do with different versions of referenced assemblies. For example, say you have assembly A and assembly B, both reference ContentPipeline.dll, but assembly A was built against version 1.0.0.0 of the DLL, while assembly B was built against version 2.1.0.0 of the DLL. Assembly B may not be compatible with version 1.0 of the DLL, but using the previous directory structure, only one version of the DLL could exist, likely breaking either assembly A or assembly B. The solution to this is pretty simple – each logical assembly group now gets its own subdirectory inside of the Binaries directory.
The intention behind copying all assemblies, executables, etc. to the local solution directory is to (a) allow the solution to be moved from machine to machine and (b) allow the project to be rebuilt with relative ease on different machines without having to install a bunch of different programs and utilities – everything is in one place, and the solution is self-contained.
The final major change is that the command-line content compiler has been split into a DLL and EXE. The DLL contains the meat of the compiler, while the EXE basically just parses the command line, fills out a parameter structure and passes it onto the DLL, which implements the actual build process. This allows the compiler to be embedded into applications (Content Studio) without having to spawn an external process.
As a side note, the next release also has the search functionality implemented in the Asset Definition File editor.
The update should be posted by the end of the week.
No Comments »
The next two major components in the content processing system are targets and packages.
Each content solution must define at least one target and at least one package in order to build anything. A content package is pretty simple – it’s just a collection of references to asset definition files (discussed later). Content target files are a bit more interesting.
Each content target definition specifies the information necessary to process and package assets for a specific target platform. A target platform would be something like “Windows PC DirectX-9″ or “Xbox 360″. It defines two critical pieces of information:
Package Builder: The package builder is executed as the final phase in the build process. It runs after the content conditioners have been run on all assets. Usually, they are used to package all of the compiled asset data up into a single resource file for optimal loading, but they aren’t required to do anything like that. For example, the ContentProject project defines a package builder class that places all compiled asset data into a directory.
Conditioner Assemblies: A list of references to conditioner assembly DLL’s that are responsible for processing assets into the format appropriate for the target platform. Since the optimal output format may vary from platform to platform, and things like endianess may be different, the set of conditioner assemblies is specified on a per-target basis.
The next post will talk about the asset definition files that specify the set of assets and the parameters associated with them that are used during processing.
No Comments »
The content compiler code has been updated (minor updates have also been made to the ConditioningEngine class in the ContentPipeline project.) The content compiler should now print out full log information. Text is stored in an application resource file, and can be localized if necessary. The new code is available here.
No Comments »
The content solution is the top-level container in the content processing system. It defines four major items:
URI Resolver: The URI resolver is responsible for taking the URI’s used to reference content files and transforming them into locations on the local file system. By default, file system paths, UNC paths, and http/https URL’s are supported. If necessary, you can implement your own URL resolver in order to do something like interfacing with a source control provider. URI’s may be specified as absolute URI’s, or as relative URI’s. The location of the UriResolver DLL (and the specific class) used by the content processing system for all packages in the solution is specified in the solution file, and may be edited in the Solution Editor pane inside of Content Studio.
Content Targets: The content solution maintains a list of content target files. References to these files are stored relative to the content solution file so that the entire directory structure can be moved easily. Content targets will be described in more detail in future posts.
Content Packages: The content solution maintains a list of content package files. Like the content target files, these references are stored relative to the content solution file.
Schema References: The content solution maintains a list of references to XML schema files (*.xsd) that it uses to validate and process asset definition files. There is one schema file, FrameworkTypes.xsd, that defines the base framework types (AssetDefinition, etc.) This file is embedded in ContentPipeline.dll as an application resource. Any other schema files are defined by the user and are specific to their content pipeline.
The raw XML of a content solution file looks like this:
<?xml version="1.0"?>
<ContentSolution
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
SolutionName="Test"
OutputPathName="Output"
UriResolver="XNATools.DefaultUriResolvers.BasicUriResolver">
<LocalSchemas>
<SchemaLocation>.\Schemas\AudioTypes.xsd</SchemaLocation>
</LocalSchemas>
<Targets>
<TargetDefinition>.\Targets\WINx86_DX9.cctgt</TargetDefinition>
</Targets>
<Packages>
<PackageDefinition>.\Packages\Level01.ccpkg</PackageDefinition>
</Packages>
<UriResolverAssemblyLocation>
.\Binaries\XNAToolsDefaultUriResolvers.dll
</UriResolverAssemblyLocation>
</ContentSolution>
When you create a new content solution in Content Studio, it creates the following directory structure for you (values in parentheses have user-configurable names):
Binaries: This folder will contain all DLL’s referenced by the content solution and its targets (except for unmanaged code DLL’s and DLL’s in the GAC.) This includes the UriResolver DLL, the packager DLL’s, and all content conditioner DLL’s (and any DLL’s they reference).
(Output): This folder contains the final output (usually package files) of the build process. The exact contents depends on the selected content packager implementations.
Packages: This folder contains all content package files (*.ccpkg), along with one subdirectory for each content package. These subdirectories contain the asset definition files associated with each package, along with the surrogate cache files (*.cache) used to reduce build times.
Schemas: Content Studio copies all referenced XML schema files (used to validate and process asset definition files) to this directory.
Targets: All content target files (*.cctgt) are placed in this directory.
(SolutionName).ccsln The content solution file (XML).
Next time, we’ll take a look at content targets and content packages.
No Comments »
|
Entries (RSS)