C# SDKMesh Loader

SDKMESH Format

The SDKMESH file format is a very simple binary format for storing 3D models, generally used in some of the moderately old DirectX sample code. Quoting from the documentation provided with the June 2010 version of the DirectX SDK:

The SDK mesh format used in the samples is a binary format created for the sole purpose of providing an easy way to get mesh data into the SDK sample and tutorial applications. Because of this, it is a very simple format that will not meet the needs of most game developers.

The file consists of multiple fixed sized structures that contain offsets to any variable sized data that may be associated with the structure. For example, the main file header contains a count for the number of materials stored in the file as well as an offset from the beginning of the file to the first material.

Effectively, this format is just a raw binary serialization of the data structure used in the DXUT library. There are some tricky bits, involved however, as there are a few places where padding is included to keep the structure aligned on 32-bit byte boundaries, which is not completely obvious from the diagram accompanying the documentation, shown below.

I've put together a somewhat more complete diagram of the file format as I was working on the C# parsing code, which shows the byte-sizes of the fields in each structure element, along with the padding, and the relations between the fixed-size headers and the variable-sized stream data. You can download a pdf version of this diagram here, if you'd like a higher-resolution version.

We're going to look at each of these structures in turn shortly, when we examine the code for parsing the model file, so I won't go into more detail here.

The SdkMesh Class

    internal class SdkMesh {
    private SdkMeshHeader _header;
    internal readonly List<SdkMeshVertexBuffer> VertexBuffers = new List<SdkMeshVertexBufferHeader>();
    internal readonly List<SdkMeshIndexBuffer> IndexBuffers = new List<SdkMeshIndexBufferHeader>();
    internal readonly List<SdkMeshMesh> Meshes = new List<SdkMeshMesh>();
    internal readonly List<SdkMeshSubset> Subsets = new List<SdkMeshSubset>();
    internal readonly List<SdkMeshFrame> Frames = new List<SdkMeshFrame>();
    internal readonly List<SdkMeshMaterial> Materials = new List<SdkMeshMaterial>();

The SdkMesh class is a fairly straightforward mapping of the structure of the file format to a C# data structure. Each of the HEADER elements in the SDKMESH file has a related C# structure corresponding. I have declared these structures as internal, because this code will only be used in process of creating a BasicModel instance, through a factory method that we will add to that class shortly. If you use this code outside of my framework, you'll probably want to make all of these structures public.

For the most part, I have consolidated the static header elements with the stream data that they refer to, for ease of parsing and use. Thus, the SdkMeshVertexBuffer elements contain both the data from the SDKMESH_VERTEX_BUFFER_HEADER portions of the file and the actual vertex data that the header describes.

For debugging purposes, I overloaded the ToString() method on each of these structures, but I'm going to leave that out here since the implementations are trivial.

Our constructor for the SdkMesh takes a single parameter, the filename of the SDKMESH model file to parse. We then use a BinaryReader to open and read from the filestream, and read each portion of the file in order, beginning with the file header. Once the file header has been read, we know how many vertex buffers, index buffers, meshes, subsets, frames and materials to read, and rely on the fact that these elements are laid out sequentially in the file. This allows us to simply pass the BinaryReader instance into the constructors for each structure - so long as the constructor for the structure correctly consumes all of the bytes making up the record, the stream should be positioned to begin reading the next record.

public SdkMesh(string filename) {
    using (var reader = new BinaryReader(new FileStream(filename, FileMode.Open))) {
        _header = new SdkMeshHeader(reader);
        for (int i = 0; i < _header.NumVertexBuffers; i++) {
            VertexBuffers.Add(new SdkMeshVertexBuffer(reader));
        }
        for (int i = 0; i < _header.NumIndexBuffers; i++) {
            IndexBuffers.Add(new SdkMeshIndexBuffer(reader));
        }
        for (int i = 0; i < _header.NumMeshes; i++) {
            Meshes.Add(new SdkMeshMesh(reader));
        }
        for (int i = 0; i < _header.NumTotalSubsets; i++) {
            Subsets.Add(new SdkMeshSubset(reader));
        }
        for (int i = 0; i < _header.NumFrames; i++) {
            Frames.Add(new SdkMeshFrame(reader));
        }
        for (int i = 0; i < _header.NumMaterials; i++) {
            Materials.Add(new SdkMeshMaterial(reader));
        }
    }
}

SdkMeshHeader Struct

[StructLayout(LayoutKind.Sequential)]
public struct SdkMeshHeader {
    public readonly uint Version;
    public readonly byte IsBigEndian;
    public readonly UInt64 HeaderSize;
    public readonly UInt64 NonBufferDataSize;
    public readonly UInt64 BufferDataSize;
    public readonly uint NumVertexBuffers;
    public readonly uint NumIndexBuffers;
    public readonly uint NumMeshes;
    public readonly uint NumTotalSubsets;
    public readonly uint NumFrames;
    public readonly uint NumMaterials;
    public readonly UInt64 VertexStreamHeaderOffset;
    public readonly UInt64 IndexStreamHeaderOffset;
    public readonly UInt64 MeshDataOffset;
    public readonly UInt64 SubsetDataOffset;
    public readonly UInt64 FrameDataOffset;
    public readonly UInt64 MaterialDataOffset;

    public SdkMeshHeader(BinaryReader reader) {
        Version = reader.ReadUInt32();
        IsBigEndian = reader.ReadByte();
        reader.ReadBytes(3); // allow for padding
        HeaderSize = reader.ReadUInt64();
        NonBufferDataSize = reader.ReadUInt64();
        BufferDataSize = reader.ReadUInt64();
        NumVertexBuffers = reader.ReadUInt32();
        NumIndexBuffers = reader.ReadUInt32();
        NumMeshes = reader.ReadUInt32();
        NumTotalSubsets = reader.ReadUInt32();
        NumFrames = reader.ReadUInt32();
        NumMaterials = reader.ReadUInt32();
        VertexStreamHeaderOffset = reader.ReadUInt64();
        IndexStreamHeaderOffset = reader.ReadUInt64();
        MeshDataOffset = reader.ReadUInt64();
        SubsetDataOffset = reader.ReadUInt64();
        FrameDataOffset = reader.ReadUInt64();
        MaterialDataOffset = reader.ReadUInt64();
    }
}

The SdkMeshHeader structure is relatively straightforward, although there are some fields which are not totally clear, due to the spotty documentation I have found.

1. Version: This indicates which version of the SDKMesh format the file supports. I have not tested an extensive number of meshes, but so far I have only seen a value of 101 here.
2. IsBigEndian: Presumably indicates if the file was saved using a big or little endian byte order. The vast majority of Windows machines are Intel-compatible, little-endian, so I did not bother to take account of this.
3. HeaderSize: Unknown...

SdkMeshVertexBuffer Struct

After parsing the file header structure, we need to parse the vertex buffer description headers. In this example, we only have one vertex buffer in our model, but there can be more than one.

internal struct SdkMeshVertexBuffer {
    private const int MaxVertexElements = 32;

    public readonly UInt64 NumVertices;
    public readonly UInt64 SizeBytes;
    public readonly UInt64 StrideBytes;
    public readonly List<VertexElement> Decl;
    public readonly UInt64 DataOffset;

    public SdkMeshVertexBuffer(BinaryReader reader) {

        NumVertices = reader.ReadUInt64();
        SizeBytes = reader.ReadUInt64();
        StrideBytes = reader.ReadUInt64();
        Decl = new List<VertexElement>();
        var processElem = true;
        for (int j = 0; j < MaxVertexElements; j++) {
            var stream = reader.ReadUInt16();
            var offset = reader.ReadUInt16();
            var type = reader.ReadByte();
            var method = reader.ReadByte();
            var usage = reader.ReadByte();
            var usageIndex = reader.ReadByte();
            if (stream < 16 && processElem) {
                var element = new VertexElement((short)stream, (short)offset, (DeclarationType)type, (DeclarationMethod)method, (DeclarationUsage)usage, usageIndex);
                Decl.Add(element);
            } else {
                processElem = false;
            }
        }
        DataOffset = reader.ReadUInt64();
        Vertices = new List<PosNormalTexTan>();
        if (SizeBytes > 0) {
            ReadVertices(reader);
        }
    }

In this case, the fields are less mysterious, since they map almost directly to DirectX 9 vertex buffer concepts.

1. NumVertices: The number of vertices in the buffer.
2. SizeBytes: The size, in bytes, of the vertex data.
3. StrideBytes: The size, in bytes, of an individual vertex in the vertex data.
4. Decl: This is an array of D3DVERTEXELEMENT9 structures, which is used to describe the members of a vertex structure. This is saved as a fixed-size array of 32 elements.
5. DataOffset: This is the position in the file where the vertex data for this vertex buffer begins.

For my own convenience, I decided to also read the vertex data while parsing the vertex buffer header. This is performed by the ReadVertices function.

private void ReadVertices(BinaryReader reader) {
    var curPos = reader.BaseStream.Position;
    reader.BaseStream.Seek((long) DataOffset, SeekOrigin.Begin);
    for (ulong i = 0; i < NumVertices; i++) {
        var vertex = new PosNormalTexTan();
        foreach (var element in Decl) {
            switch (element.Type) {
                case DeclarationType.Float3:
                    var v3 = new Vector3(reader.ReadSingle(), reader.ReadSingle(), reader.ReadSingle());
                    switch (element.Usage) {
                        case DeclarationUsage.Position:
                            vertex.Pos = v3;
                            break;
                        case DeclarationUsage.Normal:
                            vertex.Normal = v3;
                            break;
                        case DeclarationUsage.Tangent:
                            vertex.Tan = v3;
                            break;
                    }
                    //Console.WriteLine("{0} - {1}", element.Usage, v3);
                    break;
                case DeclarationType.Float2:
                    var v2 = new Vector2(reader.ReadSingle(), reader.ReadSingle());
                    switch (element.Usage) {
                        case DeclarationUsage.TextureCoordinate:
                            vertex.Tex = v2;
                            break;
                    }
                    //Console.WriteLine("{0} - {1}", element.Usage, v2);
                    break;
            }
        }
        Vertices.Add(vertex);
    }
    reader.BaseStream.Position = curPos;
}

The first step is to save the current file position of the BinaryReader's stream in a temporary variable. We'll then skip ahead in the file to the position we read into the DataOffset member. This is the start of the vertex data for this vertex buffer. Then we will read the number of vertices indicated by NumVertices.

I'm using my PosNormalTexTan vertex structure here, as a generally useful vertex structure. For most cases, this is a good choice; depending on how the mesh file you are loading was generated, there may be additional vertex properties that are defined in the model. Note that the vertex parsing code here is not exhaustive - I am only handling 2D and 3D vector vertex elements, as those were the only element types I encountered in the sample meshes that I needed to load.

We read each vertex by looping over the VertexElements in the vertex buffer's Decl list, and relying on each VertexElement's Type and Usage values to assign the vertex data read to the appropriate fields of our PosNormalTexTan vertex structure.

Finally, after reading all the vertices and populating the Vertices list, we reset the BinaryReader stream position to the saved position, so that the Reader is ready to be used to read the next header chunk of the file.

SdkMeshIndexBuffer Struct

The next block of the SDKMESH file, following the vertex buffer declaration headers, is the index buffer declarations. This structure is quite similar to the SdkMeshVertexBuffer structure. The only tricky bit is the IndexType field, which determines the byte-size of each index value. If IndexType == 0, then the indices are 16-bit short integers, otherwise they are 32-bit integers. There is also a four-byte chunk of padding following this value, before the DataOffset value.

[StructLayout(LayoutKind.Sequential)]
internal struct SdkMeshIndexBuffer {
    public readonly UInt64 NumIndices;
    public readonly UInt64 SizeBytes;
    public readonly uint IndexType;
    public readonly UInt64 DataOffset;
    public readonly List<int> Indices;
    public SdkMeshIndexBuffer(BinaryReader reader) {

        NumIndices = reader.ReadUInt64();
        SizeBytes = reader.ReadUInt64();
        IndexType = reader.ReadUInt32();
        reader.ReadUInt32(); // padding
        DataOffset = reader.ReadUInt64();

        Indices = new List<int>();
        if (SizeBytes > 0) {
            ReadIndices(reader);
        }
    }

    private void ReadIndices(BinaryReader reader) {
        var curPos = reader.BaseStream.Position;
        reader.BaseStream.Seek((long) DataOffset, SeekOrigin.Begin);
        for (ulong i = 0; i < NumIndices; i++) {
            int idx;
            if (IndexType == 0) {
                idx = reader.ReadUInt16();
                Indices.Add(idx);
            } else {
                idx = reader.ReadInt32();
                Indices.Add(idx);
            }
        }
        reader.BaseStream.Position = curPos;
    }
}

We use the same pattern as before to jump into the stream data and read the indices for the index buffer.

SdkMeshMesh Struct

After the index buffers, the next portion of the SDKMESH file is the Mesh declarations. I am not actually making any use of this data, but in the event that you had multiple meshes defined in a single model file, you would want to use this information to marry up the vertex, index buffers, subsets and bones for each mesh in the model scene. I am also not making any use of the bone data for the mesh, as I am only interested in loading a static mesh. At some point in the future, I may circle back and implement this if I find a need to support skinned meshes in the SDKMESH format.

Regardless, we are going to read in this data, for the purposes of advancing our BinaryReader.

[StructLayout(LayoutKind.Sequential)]
internal struct SdkMeshMesh {
    public readonly string Name;
    public readonly byte NumVertexBuffers;
    public readonly List<uint> VertexBuffers;
    public readonly uint IndexBuffer;
    public readonly uint NumSubsets;
    public readonly uint NumFrameInfluences; // bones

    public readonly Vector3 BoundingBoxCenter;
    public readonly Vector3 BoundingBoxExtents;

    public readonly UInt64 SubsetOffset;
    public readonly UInt64 FrameInfluenceOffset; // offset to bone data
    public readonly List<int> SubsetData;
    private const int MaxMeshName = 100;
    private const int MaxVertexStreams = 16;
    public SdkMeshMesh(BinaryReader reader) {

        Name = Encoding.Default.GetString(reader.ReadBytes(MaxMeshName));
        NumVertexBuffers = reader.ReadByte();
        reader.ReadBytes(3);
        VertexBuffers = new List<uint>();
        for (int j = 0; j < MaxVertexStreams; j++) {
            VertexBuffers.Add(reader.ReadUInt32());
        }
        IndexBuffer = reader.ReadUInt32();
        NumSubsets = reader.ReadUInt32();
        NumFrameInfluences = reader.ReadUInt32();
        BoundingBoxCenter = new Vector3(reader.ReadSingle(), reader.ReadSingle(), reader.ReadSingle());
        BoundingBoxExtents = new Vector3(reader.ReadSingle(), reader.ReadSingle(), reader.ReadSingle());
        reader.ReadUInt32();
        SubsetOffset = reader.ReadUInt64();
        FrameInfluenceOffset = reader.ReadUInt64();

        SubsetData = new List<int>();
        if (NumSubsets > 0) {
            ReadSubsets(reader);
        }
        // NOTE: not bothering with bone data now
    }

    private void ReadSubsets(BinaryReader reader) {
        var curPos = reader.BaseStream.Position;
        reader.BaseStream.Seek((long)SubsetOffset, SeekOrigin.Begin);
        for (int i = 0; i < NumSubsets; i++) {
            var subsetId = reader.ReadInt32();
            SubsetData.Add(subsetId);
        }

        reader.BaseStream.Position = curPos;
    }
}

SdkMeshSubset Struct

The next block of the SDKMESH file defines the subsets of the mesh. A subset is a collection of vertices and indices that are rendered using the same material (color, texture).

[StructLayout(LayoutKind.Sequential)]
internal struct SdkMeshSubset {
    private const int MaxSubsetName = 100;
    public readonly string Name;
    public readonly uint MaterialID;
    public readonly uint PrimitiveType;
    public readonly UInt64 IndexStart;
    public readonly UInt64 IndexCount;
    public readonly UInt64 VertexStart;
    public readonly UInt64 VertexCount;

    public SdkMeshSubset(BinaryReader reader) {

        Name = Encoding.Default.GetString(reader.ReadBytes(MaxSubsetName));
        if (Name[0] == '\0') {
            Name = "";
        }
        MaterialID = reader.ReadUInt32();
        PrimitiveType = reader.ReadUInt32();
        reader.ReadUInt32();
        IndexStart = reader.ReadUInt64();
        IndexCount = reader.ReadUInt64();
        VertexStart = reader.ReadUInt64();
        VertexCount = reader.ReadUInt64();

    }
}

SdkMeshFrame Struct

Following the subset data, comes the frame (bone) data. Again, since we are only concerned with loading static meshes, we won't actually be doing anything with this information, besides reading it to advance our BinaryReader. At some time in the future, I might make use of this in order to load skinned meshes with animations, but all of the meshes I have looked at so far for the HLSL Development Cookbook only use simple static meshes.

[StructLayout(LayoutKind.Sequential)]
internal struct SdkMeshFrame {
    private const int MaxFrameName = 100;
    public readonly string Name;
    public readonly uint Mesh;
    public readonly int ParentFrame;
    public readonly int ChildFrame;
    public readonly int SiblingFrame;
    public readonly Matrix Matrix;
    public readonly int AnimationDataIndex;

   public SdkMeshFrame(BinaryReader reader) {

        Name = Encoding.Default.GetString(reader.ReadBytes(MaxFrameName));
        if (Name[0] == '\0') {
            Name = "";
        }
        Mesh = reader.ReadUInt32();
        ParentFrame = reader.ReadInt32();
        ChildFrame = reader.ReadInt32();
        SiblingFrame = reader.ReadInt32();
        Matrix = new Matrix();
        for (int j = 0; j < 4; j++) {
            for (int k = 0; k < 4; k++) {
                Matrix[k, j] = reader.ReadSingle();
            }
        }
        AnimationDataIndex = reader.ReadInt32();

    }
}

SdkMeshMaterial

Our final structure in the SDKMESH file is the SdkMeshMaterial structure. This structure defines color and texture information for a mesh subset. Conceptually, this is similar to the old D3DXMATERIAL structure from DirectX9, with additional support for normal and specular map textures. This structure ends with six 64-bit pointers, which in DXUT are allocated to hold pointers to the textures and ShaderResourceViews for the diffuse, normal and specular textures, which we do not need, but must read in order to advance the BinaryReader.

[StructLayout(LayoutKind.Sequential)]
internal struct SdkMeshMaterial {
    private const int MaxMaterialName = 100;
    private const int MaxMaterialPath = 260;
    private const int MaxTextureName = 260;

    public readonly string Name;
    public readonly string MaterialInstancePath;
    public readonly string DiffuseTexture;
    public readonly string NormalTexture;
    public readonly string SpecularTexture;

    public readonly Color4 Diffuse;
    public readonly Color4 Ambient;
    public readonly Color4 Specular;
    public readonly Color4 Emissive;
    public readonly float Power;

    public SdkMeshMaterial(BinaryReader reader) {
        Name = Encoding.Default.GetString(reader.ReadBytes(MaxMaterialName));
        if (Name[0] == '\0') {
            Name = "";
        }
        MaterialInstancePath = Encoding.Default.GetString(reader.ReadBytes(MaxMaterialPath)).Trim(new[] { ' ', '\0' });
        DiffuseTexture = Encoding.Default.GetString(reader.ReadBytes(MaxTextureName)).Trim(new[] { ' ', '\0' });
        NormalTexture = Encoding.Default.GetString(reader.ReadBytes(MaxTextureName)).Trim(new[] { ' ', '\0' });
        SpecularTexture = Encoding.Default.GetString(reader.ReadBytes(MaxTextureName)).Trim(new[] { ' ', '\0' });

        Diffuse = new Color4 {
            Red = reader.ReadSingle(),
            Green = reader.ReadSingle(),
            Blue = reader.ReadSingle(),
            Alpha = reader.ReadSingle()
        };
        Ambient = new Color4 {
            Red = reader.ReadSingle(),
            Green = reader.ReadSingle(),
            Blue = reader.ReadSingle(),
            Alpha = reader.ReadSingle()
        };
        Specular = new Color4 {
            Red = reader.ReadSingle(),
            Green = reader.ReadSingle(),
            Blue = reader.ReadSingle(),
            Alpha = reader.ReadSingle()
        };
        Emissive = new Color4 {
            Red = reader.ReadSingle(),
            Green = reader.ReadSingle(),
            Blue = reader.ReadSingle(),
            Alpha = reader.ReadSingle()
        };
        Power = reader.ReadSingle();
        // Padding...
        reader.ReadUInt64();
        reader.ReadUInt64();
        reader.ReadUInt64();
        reader.ReadUInt64();
        reader.ReadUInt64();
        reader.ReadUInt64();
    }
}

Loading the SDKMESH as a BasicModel

Now that we have code to load and parse the model data from an SDKMESH file, we will create a new static method in our BasicModel class which will load the SDKMESH file, parse it into our SdkMesh class, and then extract the information from that object and create an instance of a BasicModel.

In broad strokes, this is a very similar process to the way in which we create a BasicModel from an Assimp.net Scene object. We need to extract the vertex, index, subset, material and texture information from the SdkMesh instance, calculate a bounding box, and initialize the BasicModel members with the extracted data.

public static BasicModel LoadSdkMesh(Device device, TextureManager texMgr, string filename, string texturePath) {
    // NOTE: this assumes that the model file only contains a single mesh
    var sdkMesh = new SdkMesh(filename);
    var ret = new BasicModel();

    var faceStart = 0;
    var vertexStart = 0;
    foreach (var sdkMeshSubset in sdkMesh.Subsets) {
        var subset = new MeshGeometry.Subset {
            FaceCount = (int) (sdkMeshSubset.IndexCount / 3),
            FaceStart = faceStart,
            VertexCount = (int) sdkMeshSubset.VertexCount,
            VertexStart = vertexStart
        };
        // fixup any subset indices that assume that all vertices and indices are not in the same buffers
        faceStart = subset.FaceStart + subset.FaceCount;
        vertexStart = subset.VertexStart + subset.VertexCount;
        ret.Subsets.Add(subset);
    }
    var max = new Vector3(float.MinValue);
    var min = new Vector3(float.MaxValue);
    foreach (var vb in sdkMesh.VertexBuffers) {
        foreach (var vertex in vb.Vertices) {
            max = Vector3.Maximize(max, vertex.Pos);
            min = Vector3.Minimize(min, vertex.Pos);
            ret.Vertices.Add(vertex);
        }
    }
    ret.BoundingBox = new BoundingBox(min, max);

    foreach (var ib in sdkMesh.IndexBuffers) {
        ret.Indices.AddRange(ib.Indices.Select(i=>(short)i));
    }
    foreach (var sdkMeshMaterial in sdkMesh.Materials) {
        var material = new Material {
            Ambient = sdkMeshMaterial.Ambient,
            Diffuse = sdkMeshMaterial.Diffuse,
            Reflect = Color.Black,
            Specular = sdkMeshMaterial.Specular
        };
        material.Specular.Alpha = sdkMeshMaterial.Power;
        ret.Materials.Add(material);
        if (!string.IsNullOrEmpty(sdkMeshMaterial.DiffuseTexture)) {
            ret.DiffuseMapSRV.Add(texMgr.CreateTexture(Path.Combine(texturePath, sdkMeshMaterial.DiffuseTexture)));
        } else {
            ret.DiffuseMapSRV.Add(texMgr["default"]);
        }
        if (!string.IsNullOrEmpty(sdkMeshMaterial.NormalTexture)) {
            ret.NormalMapSRV.Add(texMgr.CreateTexture(Path.Combine(texturePath, sdkMeshMaterial.NormalTexture)));
        } else {
            ret.NormalMapSRV.Add(texMgr["defaultNorm"]);
        }
    }
    ret.ModelMesh.SetSubsetTable(ret.Subsets);
    ret.ModelMesh.SetVertices(device, ret.Vertices);
    ret.ModelMesh.SetIndices(device, ret.Indices);

    return ret;
}

After all this work, we can now easily load an SDKMESH file, and use it in the same way as any of our other BasicModel objects.

_bunnyModel = BasicModel.LoadSdkMesh(Device, _texMgr, "Models/bunny.sdkmesh", "Textures");

Next Time...

Gah, I hope I manage to find some time to work on this stuff more regularly. Last month I was on vacation in northern Quebec, with no electricity, let alone internet, for two weeks, and most of the rest of the month was devoted to getting stuff tied up at work before I left, and getting back up to speed after I came back.

I have about six months of on-again-off-again work on computing Voronoi maps that is just about ready to release, once I clean things up and implement some remaining bits of functionality.

Hopefully I'll also find the ambition to dig deeper into the HLSL Development Cookbook, now that I have this stumbling block out of the way.

Thanks for reading, if you've made it this far, haha.