April 2025 - News archive

I extracted the GRPs from WarCraft I: Orcs and Humans from its data archive files. The format is nearly identical to Uncompressed GRPs, except that the header is four bytes instead of six (in WarCraft I there is first a two byte frame count, and then a byte with the maximum width followed by a byte of the maximum height - In StarCraft and WarCraft II the max width and max height are both two bytes each).

This caused a surprising amount of code changes in IronGRP, but nothing too difficult. There is still work to be done in automatically detecting whether the header is four or six bytes, though. I implemented a primitive check, but it fails to differentiate between WarCraft I GRPs and Extended Uncompressed GRPs. Some more work is needed to make it correct and more robust.

After spending the Easter holidays being sick, I looked a bit more at the last three GRPs that I mentioned last time.

The art/orc.grp and art/human.grp turned out to be what I call Extended Uncompressed GRPs for lack of a better name. A frame in the GRP files only have room for sprites of up to 255x255 pixels. However, for these two files, Blizzard has done a little trick. Each frame contain an offset to where the image data starts. By setting the high bit of the offset, it indicates that the frame is wider than 255 pixels. The actual width then, is what the frame reports, plus 256, which for these two files becomes 300 pixels wide. IronGRP now supports this format hack, and converting from GRP to PNGs to GRP renders identical result as the input.

As for the art/unit/orc/black.grp — the Orc Blacksmith from WarCraft II — it turns out that Blizzard for some reason tweaked a parameter when encoding it. GRPs use RLE (Run-Length Encoding) compression, which on a high level works like this: a control byte is used to communicate one of three command types: one being that the next x pixels are transparent; one being that the next x pixels are identical; one being to copy the next x pixels, which can be different. So if there were eight identical pixels in a row, the command byte would signal identicality, and the next byte would be the pixel value. This thus saves 8 - 2 = 6 bytes of space.

Normally, four pixels must be identical for the identicality instruction to be used. However, for the Orc Blacksmith GRP — and that GRP alone — a different more efficient threshold of three pixels is used. As this saves more space than the normal threshold of four pixels, it is unclear why this is not the standard.

IronGRP supports three different compression types: the Normal RLE compression, Uncompressed, and a more Optimised compression type. The Optimised compression type would previously do three things: use this more efficient threshold of three identical pixels; attempt to reuse the encoded ending pixels of a row if that matched the first pixels of the next row; as well as continuing to read identical pixels rather than finishing because too many pixels had been read.

These latter two optimisations do result in smaller file sizes, but only by very little, and they add complexity to the code. I have thus decided to remove them, and instead just use the more efficient threshold of three pixels for the identicality instruction. Not only do these two optimisations not yield a significant size reduction, but by not doing them, the Optimised compression type now creates an identical output of the Blacksmith.

A user then has the option to use the Normal RLE compression type, or the Optimised compression type. The games should have no problem using either, so it is up to the user whether they want the Blizzard standard way, or a slightly more efficient way.

With this, all GRPs of StarCraft, StarCraft: Brood Wars and WarCraft II: Battle.net edition can now be perfectly converted (apart from the WarCraft II portrait files I mentioned previously, which can be handled but with a difference, which I regard as an error on Blizzard's part).

With release 0.3 of IronGRP, it now supports converting to and from uncompressed GRPs. That means that every last GRP from StarCraft and StarCraft: Brood War can be converted to and from PNGs. The program is available on GitHub.

I now turn my attention to WarCraft II: Battle.net Edition. The GRP format is the same as in StarCraft, but a few files have differences when going from GRP to PNGs to GRP again, and some files fail due to format errors. For WarCraft II: BNE, we have:

The seven outliers are:

1. War2Dat/Art/Unit/Orc/black.grp
2. War2Dat/Art/unit/Portrait/portrait.grp
3. War2Dat/Art/unit/Portrait/l_port.grp
4. War2Dat/Art/unit/Portrait/s_port.grp
5. War2Dat/Art/unit/Portrait/x_port.grp
6. War2Dat/art/orc.grp
7. War2Dat/art/human.grp

The first one, the Orc Blacksmith, is a normal GRP. There are significant differences in the input and output GRPs. I need to investigate further what is happening there.

The four portrait files all have the same underlying issue. Identical frames can be reused to save space, which is frequently utilized in the GRPs. The frames 182 and 183 are identical and the image data is reused. Ditto with frames 184 and 185. However, frame 158 and 159 are also identical, but their image data is not reused but instead duplicated. For all these three pairs of frames, IronGRP detects that they are identical and reuse the image data. I will write this off as an error on Blizzard's part, as there is no reason to reuse some frames and not others.

The last two files are probably Uncompressed GRPs or a variant thereof. Either my handling of Uncompressed GRPs is not solid enough, or they are a different variant compared to the Uncompressed GRPs of StarCraft.

IronGRP now correctly handles the last eight normal GRPs that previously were encoded differently than their input! When frames had identical image contents, it would reuse the same frame data, including the vertical and horizontal offsets. It turned out that identical image data does not necessarily mean identical offsets. For instance, the red blinking "nuke dot" jumps around a little bit, which is accomplished by setting different offsets, although the image data is the same.

This means that all normal GRPs are now handled correctly! Remaining are ten uncompressed GRPs, that I will turn my attention to next.

The ten uncompressed GRPs are:

1. Patch_rt.mpq: dlgs/protoss.grp
2. Patch_rt.mpq: dlgs/terran.grp
3. Patch_rt.mpq: dlgs/zerg.grp
4. Patch_rt.mpq: glue/PalNl/Dlg.grp
5. BrooDat.mpq: glue/ScorePd/iScore.grp
6. BrooDat.mpq: glue/ScorePv/iScore.grp
7. BrooDat.mpq: glue/ScoreTd/iScore.grp
8. BrooDat.mpq: glue/ScoreTv/iScore.grp
9. BrooDat.mpq: glue/ScoreZd/iScore.grp
10. BrooDat.mpq: glue/ScoreZv/iScore.grp

So I've been playing around with IronGRP for a few hours now, and I just realized what the cause of (almost all) the remaining differences between the input GRPs and the output GRPs are.

The algorithm I use to encode PNG files into GRPs is based on reading the colour values of each pixel in the PNG, and finding the index of the corresponding colour in the palette. This is done by searching through the palette for the matching colour (or if the user has deviated from the palette, the closest colour match), and then using that index in the GRP encoding.

However, I just realized that the palette file that I have been using has several entries with duplicated colours. I have thus been trying to figure out why on earth the program would swap a single colour for another (I thought it was some integer overflow situation again), when what happened is that it looked for a colour matching the pixel value and correctly returned the first match it found, when the input GRP referred to a different index with an identical colour value.

When I switched to a palette file that has no duplicated colours, the results were better. When running on all GRP files in StarDat.mpq, BrooDat.mpq and patch_rt.mpq, the results are now as follows:

We're getting there! The offending GRPs are:

1. StarDat.mpq: unit/neutral/nddsha2.grp
2. StarDat.mpq: unit/thingy/dbl_exp.grp
3. StarDat.mpq: unit/thingy/nukebeam.grp
4. StarDat.mpq: unit/thingy/sbalarge.grp
5. Patch_rt.mpq: dlgs/protoss.grp *
6. Patch_rt.mpq: dlgs/terran.grp *
7. Patch_rt.mpq: dlgs/zerg.grp *
8. Patch_rt.mpq: glue/PalNl/Dlg.grp *
9. BrooDat.mpq: glue/ScorePd/iScore.grp *
10. BrooDat.mpq: glue/ScorePv/iScore.grp *
11. BrooDat.mpq: glue/ScoreTd/iScore.grp *
12. BrooDat.mpq: glue/ScoreTv/iScore.grp *
13. BrooDat.mpq: glue/ScoreZd/iScore.grp *
14. BrooDat.mpq: glue/ScoreZv/iScore.grp *
15. BrooDat.mpq: unit/neutral/ncicShad.grp
16. BrooDat.mpq: unit/neutral/nckShad.grp
17. BrooDat.mpq: unit/neutral/nddSha2.grp
18. BrooDat.mpq: unit/wirefram/wirefram.grp

The ones marked with * are uncompressed I believe, for which there is no support yet. As for the rest, I did a quick analysis. None have unused "residue" bytes (which is a good thing), but they all have several sets of identical GRP frames. My hunch is that it is related to that, or to some corner case regarding vertical and horisontal offsets.

I have continued to look at how the GRP converter, IronGRP, decodes and re-encodes the GRP files of StarCraft. I have examined the output of the Terran Battlecruiser, the Protoss Dragoon and the Protoss Nexus GRPs, since those were three random GRPs out of the bunch where my tool failed to perfectly convert the GRP to PNGs and then back to GRP again.

I have found the root causes of why it would differ: for the Battlecruiser, it was a corner case where my algorithm was somewhat more efficient than Blizzard's. For the Dragoon, I mistakenly printed out all image data of duplicated frames, which not only resulted in the GRPs to differ, but for mine to be nearly eight times as large as the original. Ooops. For the Nexus, it was a type conversion error that resulted in integer overflow errors. These things are now fixed so that it matches Blizzard's way.

So like last time, I converted all GRPs to PNGs, and then back to GRPs again, for all GRPs in unit/protoss, unit/thingy, unit/terran and unit/zerg from StarDat.mpq. The results are now as follows:

Much better than last time, but still some work to be done.

I started working on a TBL converter. TBL files are used in WarCraft II and StarCraft for holding text data, such as unit names, menu text, and other game strings. They also act as mappings to sprite and sound files. Since I've written the tool in Rust, I went with the same naming as the previous tool, and called it IronTBL.

It can convert TBL files to plain text files and back. As a little extra, I also added yazi preview integration. There's some minor things I still want to fix for it, such as adding a Readme file, and provide a way to autogenerate shell completions (because why not).

Work continues on the GRP converter, IronGRP, when time permits. It can now detect duplicated frames and reuse them, instead of copying identical frames multiple times.

The GRP format has no concept of transparency, whereas PNGs do, in their alpha channel. IronGRP can now handle PNGs with alpha channels, in the sense that completely transparent pixels are treated as belonging to palette index 0, which is the transparent colour in the GRP format. Pixels with complete opacity are treated as normal pixels. Any other value in the alpha channel gives rise to a warning, but is otherwise treated as a normal pixel.

In the previous news item, I claimed that IronGRP converts flawlessly, but this turned out to be a premature claim. I converted all GRPs to PNGs, and then back to GRPs again, for all GRPs in unit/protoss, unit/thingy, unit/terran and unit/zerg from StarDat.mpq. The results are as follows:

I was hoping for more identical GRPs, and I'll need some time to investigate what these differences are. I doubt they are really visible; this is probably more encoding differences making individual bytes different, or perhaps "residue" bytes that are not actually read by the game.

Still remaining is also to handle Uncompressed GRPs.

The GRP converter, IronGRP, that I've been working on for the past few days, now seems to work flawlessly (at least on the Valkyrie GRP; I need to test on more GRPs). Previously, it would sometimes fail to consume enough pixels when reading a row of pixels, which caused slight deviations when converting from GRP to PNG and back to GRP. That was the cause of the slight difference in output that I referred to in the last post. Now, GRP to PNG to GRP conversion is byte-to-byte identical (for the Valkyrie).

Since it now produces identical output to its input, I've promoted the code to version 0.1. The code is available on GitHub.

As I wrote in last post, there are some things I still want to add:

• Handling of Uncompressed GRPs.
• Handling PNGs with alpha channels (transparency).
• Detecting duplicated frames and reusing them rather than writing them anew.

Once that is in place, it seems fitting to call it version 1.0.

The unit and building sprites of WarCraft I, WarCraft II and StarCraft I are found in GRP files, which need to be converted into other image formats in order to be edited. For that, I'm working on a little GRP to PNG converter tool, called IronGRP.

Back in the day, RetroGRP was used by everyone, but it is a closed source Windows only utility. ShadowFlare made grpapi which is no doubt very competent, but not too straight-forward to compile on Linux (probably speaks more about my own short-comings than anything else). There is also PyGRP as part of PyMS by poiuyqwert, but it seems to be stuck in Python 2 land. Worth mentioning is also libgrp by Bradley Clemetson.

Instead of fighting to compile 20 year old source code, I decided to take a shot at creating a utility from scratch. IronGRP is the fruit of that labour. As of this writing, it can convert GRPs to PNGs and vice versa, and it can create tiled images, where all frames are in one tiled image:

The decoding and encoding algorithm appears to create near-perfect results compared to Blizzard's own file, but there are some instances where the output differs ever so slightly.

I'm trying to see if these differences are fixable. I also want to handle uncompressed GRPs, and more gracefully handle PNGs with alpha channels (transparency). Support for detecting duplicated frames and re-using them should be added as well. But for now, I'm happy to say that what I have feels like a pretty solid start.