Writing DOOM WAD files

In a prior post, I described id's WAD format used by classic games such as DOOM and how to read them. This post covers how to write them. As with my first post, this only covers the original WAD format, not the enhanced ones which followed.

The Format

A brief recap on the format. There is a 12 byte header which details the wad type, the number of lumps of data it contains, and an offset where the directory index is located.

The directory index is comprised of (16 * number of lumps) bytes which describe the lumps. Each 16 byte header details the size, the position in the data and the lump name.

All integer values are in little-endian format.

Considerations

The nature of the WAD file means that in theory that you should be able to make changes to it without having to rewrite the entire file. For example, adding a new lump of data could simply be added to the end of the existing data, overwriting the existing directory index, and then a new index appended. Replacing a lump with data the same size or small could overwrite the existing lump, and adjusting the directory index entry. Even when removing a lump you could opt to leave the data behind (or zero it out!) and simply remove the meta data from the directory index.

However, the simplest means (albeit most inefficient) is to rewrite the whole WAD. At this point I am simply exploring the format (and its subsequent iterations) so therefore I'm not going to go out of my way to complicate things and so this demonstration program will recreate the WAD each time it is changed.

Creating a WAD From Scratch

The previous post introduced the WadReader class, a way of quickly enumerating a WAD file. Here, I introduce the WadOutputStream class as a counterpart. This class can be used to easy create a WAD file by calling PutNextLump with the name of the lump to add, then write the lump data via the usual Stream methods. Once done, flushing or closing the stream will automatically write the directory entry. I borrowed this pattern from DotNetZip as I find it a convenient way of creating Zip files. In fact, I'll likely refactor WadReader at some point to act more like ZipInputStream as well.

using (Stream output = File.Create("test.wad"))
{
  using (WadOutputStream target = new WadOutputStream(output))
  {
    using (BinaryWriter writer = new BinaryWriter(target, Encoding.UTF8, true))
    {
      target.PutNextLump("PHOTO1");
      writer.Write(File.ReadAllBytes("photo1.jpg"));
      target.PutNextLump("PHOTO2");
      writer.Write(File.ReadAllBytes("photo2.jpg"));
      target.PutNextLump("PHOTO5");
      writer.Write(File.ReadAllBytes("photo5.jpg"));
    }
  }
}

When creating an instance of WadOutputStream, it will immediately write a placeholder header to the stream. This will have the correct WAD type, but the count and directory position will all be defaults until filled in at the end.

As with the previous article, error handling, parameter validation and non-essential code has been elided from these snippets.

public class WadOutputStream : Stream
{
  private readonly List<WadLump> _lumps;
  private readonly Stream _output;
  private readonly long _start;
  private bool _writtenDirectory;

  public WadOutputStream(Stream output, WadType type)
  {
    _output = output;
    _start = output.Position;
    _lumps = new List<WadLump>();

    this.WriteWadHeader(type);
  }

  private void WriteWadHeader(WadType type)
  {
    byte[] buffer;

    buffer = new byte[WadConstants.WadHeaderLength];

    buffer[0] = type == WadType.Internal ? (byte)'I' : (byte)'P';
    buffer[1] = (byte)'W';
    buffer[2] = (byte)'A';
    buffer[3] = (byte)'D';
    // positions 4 - 11 left at zero for now

    _output.Write(buffer, 0, WadConstants.WadHeaderLength);
  }

Although the WadOutputStream inherits from Stream, you can't just randomly write to it. Prior to writing a lump, you need to call the PutNextLump method. This method both finalises the previous lump, if applicable, and prepares the new lump. This is required so that the class can keep track of the lumps in order to write the directory entry at the end.

  public void PutNextLump(string name)
  {
    this.FinaliseLump();

    _lumps.Add(new WadLump
    {
      Name = name,
      Offset = (int)_output.Position
    });
  }

  private void FinaliseLump()
  {
    if (_lumps.Count > 0)
    {
      WadLump lump;

      lump = _lumps[_lumps.Count - 1];
      lump.Size = (int)_output.Position - lump.Offset;
    }
  }

Finally, when we're done writing, we need to finish off the WAD by writing the directory index and updating the header. We'll do this by overriding both Flush and Dispose.

  protected override void Dispose(bool disposing)
  {
    if (disposing && !_writtenDirectory)
    {
      this.Flush();
    }

    base.Dispose(disposing);
  }

  public override void Flush()
  {
    if (!_writtenDirectory)
    {
      this.FinaliseLump();
      this.WriteDirectory();
    }

    _output.Flush();
  }

When preparing for writing the directory, we again check to see if there are any lumps and finalise the last one as we did when adding a new lump.

Now we can finalise the file header. We do this by creating a new 8 byte array and place the lump count in the first four bytes, the current stream position into the latter four, representing where the directory index will be written. We then write these 8 bytes at the start of the stream, overwriting the zero block written earlier.

To write an integer into the byte array I'm using a custom PutInt32Le method. While the BitConverter class has a GetBytes method, firstly this will result in repeated allocations from byte array creation, and secondly I'd then have to copy the contents in the destination array. Finally, BitConverter will return the results based on the endian-ness of the system and we need to ensure that little-endian is used.

Once the WAD header is updated we enumerate each of our lumps and build the 16 byte directory header containing the lump offset, size and the padded name and then write those at the end of the file.

  public static void PutInt32Le(int value, byte[] buffer, int offset)
  {
    buffer[offset + 3] = (byte)((value & 0xFF000000) >> 24);
    buffer[offset + 2] = (byte)((value & 0x00FF0000) >> 16);
    buffer[offset + 1] = (byte)((value & 0x0000FF00) >> 8);
    buffer[offset] = (byte)((value & 0x000000FF) >> 0);
  }

  private void WriteDirectory()
  {
    byte[] buffer;
    long position;

    buffer = new byte[WadConstants.DirectoryHeaderLength];
    position = _output.Position;

    // first update the header
    WordHelpers.PutInt32Le(_lumps.Count, buffer, 0);
    WordHelpers.PutInt32Le((int)position, buffer, 4);

    _output.Position = _start + 4;
    _output.Write(buffer, 0, 8);

    _output.Position = position;

    // now the directory entries
    for (int i = 0; i < _lumps.Count; i++)
    {
      WadLump lump;

      lump = _lumps[i];

      WordHelpers.PutInt32Le(lump.Offset, buffer, WadConstants.LumpStartOffset);
      WordHelpers.PutInt32Le(lump.Size, buffer, WadConstants.LumpSizeOffset);

      for (int j = 0; j < lump.Name.Length; j++)
      {
        buffer[WadConstants.LumpNameOffset + j] = (byte)lump.Name[j];
      }

      for (int j = lump.Name.Length; j < WadConstants.LumpNameLength; j++)
      {
        buffer[WadConstants.LumpNameOffset + j] = 0;
      }

      _output.Write(buffer, 0, buffer.Length);
    }

    _writtenDirectory = true;
  }
}

Rewriting an existing WAD

This example, taken from the WadFile class, enumerates all existing lumps and then writes them into a new stream. Although not demonstrated here, it assumes the GetInputStream for a given WadLump will return either the original data for existing lumps, the modified data for existing lumps that have been altered, or the data for new lumps.

As it can't write to the source stream whilst also reading from it, it does all this to a temporary stream, and then, when done, copies the contents of the temporary stream over the original stream.

This isn't exactly the most efficient approach, but does avoid all of the complexity of determining which parts of the file to update, which parts to clear, keeping a list of changed items for batch saving, etc. This is most likely something I will investigate further in a future topic.

public void Save(Stream stream)
{
  using (Stream temp = this.GetTemporaryStream())
  {
    using (WadOutputStream output = new WadOutputStream(temp, _type))
    {
      for (int i = 0; i < _lumps.Count; i++)
      {
        WadLump lump;

        lump = _lumps[i];
        output.PutNextLump(lump.Name);

        using (Stream input = lump.GetInputStream())
        {
          input.CopyTo(output);
        }
      }

      output.Flush();
    }

    stream.Position = 0;
    stream.SetLength(0);

    temp.Position = 0;
    temp.CopyTo(stream);
  }
}

To Pad, Or Not To Pad

After I discovered that the data in the DOOM picture lumps were padded, I was curious if the WAD file itself was. I copied the hex viewer project from that solution and used it to highlight the different ranges in a WAD. To my surprise, it seemed that in even though picture lumps already had their own padding, the lumps themselves were also padded to always have an even number of bytes. Interestingly, sometimes if a lump started on an even number two padding bytes were still included. I suppose there is a reason but I didn't dig further info it and so didn't build padding support into the writer classes.

Also possibly worthy of note, I checked the DARKWAR.WAD file and this didn't use padding at all.

Does it work?

In a word, yes. I tested dumping DOOM.WAD into separate data files using the waddemo program, then repacking them into a brand new WAD. I then ran DOOM using the new wad and played through the first level. Everything seemed to be running fine.

Getting the source code

As noted in the first article in this series, there isn't a single download available per post as I've done a larger-than-usual demonstration solution. The full project is available from our GitHub page.

Like what you're reading? Perhaps you like to buy us a coffee?