News

In December, I wrote an MPQDraft plugin, MpqFileLister that lists all the filenames that a games accesses in its MPQ archives — or at least that was the intention. A peculiarity of MPQs are that they don't contain the file names of the content of the archives. But when accessing them, the game will call functions in Storm.dll to open those files, and by hooking into those calls, we can intercept the file it is attempting to open, thus construct a list of all files that are accessed.

I had thought that hooking into the SFileOpenFile and SFileOpenFileEx calls would be enough, but when luauser32167 pointed out on GitHub that the resulting list files were not containing all assets that the game loaded, I realized those two calls were not enough.

I have just made a new release of MpqFileLister that in addition to SFileOpenFile and SFileOpenFileEx and also hook into SVidPlayBegin, SFileLoadFile, SFileLoadFileEx, SBmpLoadImage and SBmpAllocLoadImage. I thus believe all Storm.dll functions that load files by their file names are covered.

In addition, this new release allows the user to freely specify the log format and timestamps, some odd bugs have been fixed and the code has been refactored.

You can find version 1.2.0 of MpqFileLister on GitHub.

By the way, you might be wondering what the need then is for namebreak-cuda, which I wrote about in the previous entry. The answer is simple: not all files are accessed, but finding and naming them is still of interest.

For many years, Blizzard used MPQ files as their archive files for many of their games. MPQ files are basically like ZIP files; they store many other files in a compressed format. One quirk of MPQ files is that they don't contain the filenames of their own content. Some MPQ files contain a listfile consisting of the filenames of their content, but other MPQ files do not. For these, an external listfile is needed.

The War2Dat.mpq archive of WarCraft II: Battle.net Edition has had a 99% complete listfile for years, but there has been one single file that the community has not known the name of.

Around a year ago, I made a tool, namebreak-cuda, which bruteforces filenames on Nvidia GPUs. While I put the heavy lifting in place, I never quite finished it then. So now, after tweaking it for a bit, I ran it for less than two hours, and it returned the missing filename:

ART\CHAT_BNE_PLAIN.PCX

The War2Dat.mpq now has complete listfiles after all these years. I was happily surprised that the search was so quick.

There are other archives that also don't have complete listfiles, to which I turn my attention next. I plan to make the tool more polished, create a server which can coordinate the efforts of finding more file names, and publish the code on GitHub.

As I wrote about back in May, I've been working on scraping forums and retrieving long gone content from the Internet Archives. It has turned out to be way more fiddly than I expected, mostly because I decided to turn the HTML forum posts into valid Markdown format, and people have used and abused HTML in all sorts of ways. I hope to host the scraped content of dead sites here in not too long.

The scraper is available here for those who want to make their own backups of sites, but please remember that it takes a long time to run, and every page of every thread puts load on the webservers, so please be considerate. We don't want to push any sites further towards unmaintainability.

Some might have noticed that Campaign Creations went down recently. It was once a hugely popular site, but has for a long time been reduced to a forum (with no other content available) that was neither active nor possible to register accounts at. They went dark a few months ago, which I figured was the deathknell for the site. However, I am happy to see that they once again are alive and kicking, and at least some of the content that they used to host seems to be accessible again. And once more, you are now able to register at the forum!

Over the past few days, there's been two releases of mpqcli, and StormLib has also had a release. mpqcli is a command-line tool to create, add and remove files from MPQ archives, and StormLib is the underlying library powering it. MPQ archives were used for many years by Blizzard games such as StarCraft I and Warcraft II, and contain the game assets that game modifications change.

Changes in mpqcli 0.10.1:

Added

• Command completion for common shells (bash, zsh, fish and powershell)

Fixed

• Memory leak in read subcommand
• Inconsistencies in documentation

Updated

• Code style to conform to Google CPP style with slight variations
• StormLib dependency for a variety of fixes

Changes in mpqcli 0.10.2:

Fixed

• verify incorrectly reporting success for a tampered archive
• verify failure messages now clarify whether a signature is missing or invalid
• add overwrite flag not always working as expected
• Archive size incorrectly reported for large archives
• File sizes shown as negative for large files

Updated

• Simplified how strong archive signatures are read internally

Changes in StormLib 9.40:

The changelog for StormLib is brief and only states that it "Added functions SFileOpenFileArchive and SFilegetFileArchive", but the release is actually the accumulation of changes since November 2024.

Download:

Find version 0.10.2 of mpqcli here, and version 9.40 of StormLib here.

For StarCraft I modders, one of the best set of tools has for a long time been PyMS, a graphical tool suite that allows several file types to be modded.

The author, poiuy_qwert, has been hard at work converting the source from Python 2 to Python 3, and writes:

Hey all, I've been putting a lot of work into the python3 version of PyMS recently, but again I only have time to work on the tools not to actually use them, so I need to lean on others to use/test the programs and report any issues. If anyone is able to play with them please let me know!

See here for how to raise any issues.

Also, while the source of PyMS has been available for a long time, it has so far been without any explicit license, but poiuy_qwert just now released it under the MIT license.

Find the Python 3 branch of PyMS on GitHub.

StarCraft I and Warcraft II use TBL files to store game strings for things such as unit names, dialog strings, game tips, mission dialogs etc. Roughly a year ago, I wrote a little utility, IronTbl to convert these TBL files to TXT files and back.

I recently decided to give IronTBL some love and fixed some minor bugs and issues, and spent some time hardening it. So with that, version 2.0.0 is now out.

The previous --mode flag is now replaced by proper subcommands — since this is a breaking change for the interface, I bumped the version to 2.0.0 even though the release doesn't offer any new features.

IronTBL 2.0.0 is available on GitHub.

Thomas Laurenson seems to have figured that yesterday's 0.9.10 release of mpqcli merited a version bump, since it is a rather big release.

He also included a contribution by me to add the compact subcommand which defragments an MPQ archive, thus shrinking it in size if it contains any gaps from previous adds or removes.

So, version 0.9.10 is not available anymore, but instead version 0.10.0 of mpqcli is out!

Thomas Laurenson just released version 0.9.10 of mpqcli two hours ago. As previously covered here, mpqcli is a tool for creating and modding MPQ files, used for loading custom data into games.

Both Thomas Laurenson and myself have made several contributions to the release, which is rather large:

Added

• Support passing multiple files to the add and remove subcommands
• Support reading file paths from stdin in the add and remove subcommands
• Support adding entire directories with the add subcommand
• Add an --update flag to the add subcommand. When given, files whose archived size matches the on-disk size won't be added to save time.

Fixed

• extract subcommand now reports an error when the output directory cannot be created
• Path traversal guard in extract uses fully resolved paths, closing a potential bypass
• Crash when reading strong signatures from malformed or truncated archives
• Docker glibc image updated to ubuntu:24.04
• Adding files is now ordered and operating system agnostic
• add subcommand now returns non-zero exit codes on failure
• Error messages now include StormLib error codes
• Duplicate error messages removed
• Makefile build targets corrected

Changed

• The --path flag on add now acts as an archive path prefix when a directory is given, and the full name when a file is given.

Removed

• --filename-in-archive and --directory-in-archive. They are now replaced by a unified --path flag

Find version 0.9.10 of mpqcli here.

Yesterday, TheNitesWhoSay released version 2.12 of the StarCraft I map editor Chkdraft.

It is a small bugfix with a single change:

• Fix a trigger action display-text issue where the move unit to/from locations were swapped

Version 2.12 is available here.

Today, May 16, marks the 30 year anniversary of the release of Warcraft II: Beyond the Dark Portal. The game is an expansion set for Warcraft II: Tides of Darkness, which was released December 1995.

Both Warcraft: Orcs from 1994, and Humans and Warcraft II: Tides of Darkness had been major hits for Blizzard Entertainment, the company that created the games. Their press-release proudly stated that "We have been overwhelmed by the tremendous success of Warcraft II, and since the game’s release, we have been flooded with requests for an add-on disk," said Allen Adham, president and founder of Blizzard Entertainment. "With the expansion disk, players extend their Warcraft experience as they delve into the Orcs' homeland for the first time to experience a entirely new set of challenges."

While Warcraft II: Beyond the Dark Portal was only in development for a few months, and during that time was passed from one studio to another, it still came to be very popular among gamers. Its raw feature set is maybe not so much to boost with, but it arguably marked a turning point in Blizzard Entertainment's game design philosophy, and RTS campaign design as a whole.

Development

During the mid 1990s, Blizzard Entertainment was still a small studio. The development team for Warcraft II: Tides of Darkness, had only been six core programmers and two support programmers.1

Blizzard Entertainment saw economic success with both Warcraft: Orcs and Humans, and with Warcraft II: Tides of Darkness, but while it was a rising star in the industry, it was far from an economic behemoth. They thus needed to follow up with more releases. Patrick Wyatt, the Producer of Warcraft: Orcs and Humans, writes this in one of his very interesting articles about his experiences:

Blizzard’s business strategy was driven by Allen Adham, the company’s president. Allen was a student of both gaming and business, and under the tutelage of Bob Davidson (CEO of Davidson and Associates, the educational software company that first acquired Blizzard), they planned the company’s development pipeline with a keen eye towards maximizing the revenue and profit of our studio, as would any corporate leader. [...] Allen endeavored to build a pipeline with predictable game releases[...]

There is surprisingly little information available on how Warcraft II: Beyond the Dark Portal came to be. After Warcraft II: Tides of Darkness was released in December 1995, the company started working on StarCraft, grinding to release it by the end of 1996 (it was eventually released in May 1998).2 To make sure they could capitalize on the success of Tides of Darkness, they wanted to make a follow up game.3

In 1995, they had contracted Cyberlore Studios to develop an expansion for WarCraft II. Letting another developer handle the work had seemed like a win-win: All Cyberlore had to do was whip up multiplayer maps and a single-player campaign that continued the base game’s story while Blizzard’s internal team devoted its full attention to StarCraft, their next real-time strategy game. During a milestone check-in with Cyberlore, however, Blizzard’s managers had deemed the work subpar. They cancelled the contract, finished the add-on pack, Beyond the Dark Portal, themselves, and resolved that only an in-house team could nurture a Blizzard property to an acceptable level of quality.

In retrospect, the game might be something of a turningpoint, where instead of simply pushing out the game, Blizzard dedicated people and effort into raising it to their standards, even as these people were direly needed on other projects. One might here see the seed of Blizzard's mentality of "it'll be ready when it's ready" and obsessing over quality. They did not again reach out to other studios to contract away their work, and they also went against conventional wisdom and decided to miss their target of releasing Diablo for Christmas 1996, instead pushing it off until January 1997, so that they could make sure it held their standards.4 And StarCraft famously was "two months from release" for fourteen months.1

New features

The game brings a new tileset, representing the Orc homeworld of Draenor. Online play was in its infancy and widely inaccessible in 1996, so the new single player campaign of 12 Human and 12 Orc missions, in addition to 50 custom maps, were very welcome by players.

The game most memorably introduces several heroes. They don't have abilities beyond what normal units have, but they are much tougher and more dangerous, although most missions that feature them also require them to stay alive through the game. They can be seen as "proto-heroes" and paved the way for characters in later games — and of course Warcraft III, released in 2002, would heavily expand on the concept.

The campaign itself is rather varied for the time and given the game engine constraints. The basics of most missions are to build up your base, create an army and conquer the AI opponents, but a few levels offer variation. Some missions include rescuing units and moving them to safety on the map; there are no-build missions where you make your way with a small force through hostile territory; and one mission where an Orc clan betrays their kin and joins the Humans, letting you play as the opposite race.

Blizzard created a little bonus song, I'm a Medieval Man, as a homage to Comand and Conquer's Mechanical Man. One can listen to the song by inserting the Warcraft CD into a audio CD player, or by typing the "cheat code" disco in the game. The lyrics to the song are from Warcraft: Orcs and Humans, funnily enough.

Personally, I find it a bit interesting that no new units, buildings or abilities were introduced. Especially as the game is symmetrical in the sense that the two races are nearly perfectly mirrored (apart from the spells, which while they differ still show quite the symmetry). It seems like it would have been an easy task to add in pairs of new Human and Orc units without needing to worry too much about game balance. I'm speculating, but probably the development time was too limited to allow for this.

Today's view by players

While Warcraft II: Tides of Darkness started with plenty of easy levels that gradually grew more demanding, Warcraft II: Beyond the Dark Portal is known for being unforgiving right from the start. Many on Reddit, such as MDoc84, share their frustration about the campaign difficulty: "I've NEVER been able to beat Beyond the Dark Portal. The difficulty curve comparted to Tides of Darkness is insane". From the same reddit-thread:

The difficulty is pretty absurd, I never made it anywhere without cheats. It's an issue that game devs and modders run into. I call it skill inflation:

The testers/team, when making the expansion pack, have likely sank hundreds of hours into the game by that point. They tuned it to what felt like a reasonable challenge to them, but back then you couldn't get public feedback as easily before the game got released. Could be too, that the members of the public they did talk to were diehards as well.

There are indeed also plenty of these die-hard fans who showcase the opposite thoughts. One of my all-time favourite videos on YouTube is this absolutely insane lossless speedrun by texture3D, where they play Orc mission 12 without losing a single unit, and doing so at absolutely incredible speed. It is pure art to watch how they pulverize the AI.

I also highly recommend watching Admiral Iddy play through the entire Warcraft II: Beyond the Dark Portal campaign lossless. He makes it clear just how grueling it is to attempt such a feat.

Modding

The modding community at the time had already been dissecting the games files and data archives. One of the major milestones was when Mordraug (Peter Hatch) mapped out parts of the memory of the game executable, enabling Merlin (Cameron Buschardt) to write his famous tool WarHack, which allowed users to change the commands and tech trees of units.

For those still interested, I can highly recommend visiting the Warcraft II Forums, a community of players, modders and mappers who are still going strong.

Later versions

Blizzard would continue to capitalize on the game and release it in several bundles and versions:

• WarCraft: Battle Chest, released in 1996, was a bundle which included WarCraft: Orcs and Humans, WarCraft II: Tides of Darkness, and WarCraft II: Beyond the Dark Portal.
• WarCraft II: The Dark Saga, released in 1997, was a port for the Sega Saturn and PlayStation consoles by Electronic Arts, including the campaigns from both Tides of Darkness and Beyond the Dark Portal.
• WarCraft II: Battle.net Edition, released in 1999, ported the game's code to Microsoft Windows, fixed some minor bugs, and enabled multiplayer support via Blizzard's online service, Battle.net.
• WarCraft II: Remastered, released in November 2024, is modern remaster of Tides of Darkness and Beyond the Dark Portal, with improved graphics and updated controls.