Saga Musix' Demoszene-Blog.

S3M Format Shenanigans 21. August 2021

Most people would probably expect that there's nothing left to discover in the most widespread legacy module formats in 2021.
And yet, sometimes little details are discovered that were apparently overlooked by everyone else so far, or simply noone cared about them. This is exactly what happened not too long ago when I dug around in some S3M files, and while it is of course very well possible that someone else has made the same discoveries as I have made here, I am not aware of anyone documenting them before, so here we go because this was quite an epiphany worth talking about!

For many years I have been aware that there are some small but sometimes significant differences between Gravis Ultrasound (GUS) and SoundBlaster (SB) playback in Scream Tracker 3. For example, sample swapping works with the SB driver but not the GUS driver.
This is annoying, because there's not one objectively correct way to implement an S3M player, and a few S3Ms rely on either GUS or SB playback style. For a long while I have thought how one could heuristically detect how an S3M file should be played (something that OpenMPT already does with MOD files), but I couldn't come up with any convincing heuristics. Scanning sample texts? Too fragile. Looking for panning commands? Nice idea in theory, but there are many songs written for external players which supported panning regardless of which sound card was used.

Interlude: Many oldskool trackers simply dumped their internal data structures into their file formats. This is the reason why sometimes seemingly unnecessary data fields can be found in those files. Often these can be used to fingerprint whether a file was made with the original tracker that introduced the format or a clone, because the latter would typically zero out those fields, while the original tracker might keep some random data from runtime in them (e.g. EMS handles or whatever).
For most formats this kind of fingerprinting is not necessary, but for once, this turned out to be useful.

So, a while ago in June 2021, 8bitbubsy and I were talking about Skaven's "maxtestsong", a Scream Tracker module using 32 sample channels. Scream Tracker 3 itself could only use 16 sample channels simultaneously, so the file must have been created with a custom internal Scream Tracker build that could handle more sample channels.
This lead us to talking about "Call Me An Angel (Iri Mix)", another S3M module using 32 sample channels, but using a different channel allocation table - in "maxtestsong", the extra 16 channels were turned off according to the file header, while "Call Me An Angel" essentially instructs the player to re-use the same physical sample channels that were already used for the first 16 channels, which may produce interesting results when played back in Scream Tracker. OpenMPT detected this file as being written in Scream Tracker 3 rather than a known third-party tracker. So just to be sure, I re-saved the file in Scream Tracker 3 - if the re-saved file was 100% identical to the original file, it was unlikely that another tool was used to create this track.

But... the new file wasn't identical. There were some tiny differences in the sample headers, which turned out to be in the "Int:Gp" field - the internal address of the sample in GUS memory, which is of course useless to store in the file, but it turned out to be a goldmine.

I quickly figured out that Scream Tracker 3 cleanly initializes the "Int:Gp" values to 0 for empty sample slots and 1 for normal samples when using the SB driver, while the values would differ between samples when using the GUS driver. Yes, this was it! It was possible to reliably tell if the file was last saved with the GUS or SB driver loaded, and it should be a fairly safe assumption that the last driver used while saving the file is also the driver that is intended to be used for playing the file. If Scream Tracker didn't clean this field when re-saving a previously GUS-saved file with the SB driver, this discovery might have been useless.

After figuring this out, there were some remaining questions to answer - would this work for all Scream Tracker versions? Early Scream Tracker 3.00 versions only supported SoundBlaster output, and those versions always wrote 0 into the "Int:Gp" field. But at some point in late 1992, GUS support was added, and all Scream Tracker versions since then populate the "Int:Gp" field in a way that could be easily fingerprinted.

I guess I'm incredibly lucky that I use the GUS driver by default in my Scream Tracker DOSBox setup (a relic from more than ten years ago when I was running DOSBox on a much slower CPU), because otherwise I might have never turned a closer look at the "Int:Gp field" - "Call Me An Angel" was saved with the SB driver, and that's the only reason I saw those differences in the file.

Soon after, I made another discovery that might explain a long-standing mystery that I have been wondering about for many years. In "Who is the real Timelord?", I mentioned that there were some S3M files with IRC chat logs at the end of the sample data. There are also many S3Ms with other seemingly random crap past the sample loop end. I always wondered how this could happen - did Scream Tracker not read those sample files correctly from disk and accidentally read some cluster from an unrelated file? It must have been a Scream Tracker thing, because this phenomenon is incredibly common in S3M files while being practically non-existent in other module formats.

It turned out that the answer to this is also in the sample header fields. For the SB driver, there is some sample loop unrolling code in Scream Tracker that doesn't quite do the right thing. Supposedly Scream Tracker tried to put the first 512 samples of the sample loop past its loop end, and for some reason the offset to write those samples to is stored in the "Int:512" sample header field.
However, what Scream Tracker ends up doing is moving a block of sample data 512 bytes closer to the sample loop end (which is stored in "Int:512"), but this block extends 512 bytes past the actual sample end - and this means that whatever 512 bytes were following the sample end would now become part of the sample. Often this would turn out to be some other sample data - or just random bytes left over in memory from some other application that was previously running (remember, no memory protection in the DOS days!). So if you re-save a file in Scream Tracker where the "Int:512" field of some sample is set, you will trash that sample's last 512 bytes with some more or less random memory. With this knowledge it is now also obvious why so many samples in S3M files seem to have their loop end perfectly matched up against some follow-up garbage - the garbage wasn't there when the loop was initially created.

Alright, that's it for now. I haven't written a blog post in a long while, but I thought that this topic was worth taking about in more than just some dry developer documentation.