News

Two weeks ago, I wrote about namebreak-bombe, the tool for cracking the last unknown filenames in Blizzard's MPQ archives on GPUs. A lot has happened since then! At the time it was just me and Heinermann running it; now people have contributed to the coordinated search, and together we have hashed more than 8 quintillion ($8\cdot {10}^{18}$) candidate names. A big thanks to everyone who has joined, in particular Heinermann, lucifirius, Johnny C, Jesse and mika (and myself!), who have all found names.

Names found since last time

StarCraft: Brood War French edition, BroodDat.mpq:

SOUND\PROTOSS\DARCHON\PDAWHT011.WAV
SOUND\PROTOSS\DARCHON\PDAWHT021.WAV
SOUND\PROTOSS\DARCHON\PDAWHT031.WAV

StarCraft: Brood War French edition, Install.exe:

NEWMUSIC\PDEFEAT.WAV
NEWMUSIC\PRDYROOM.WAV
NEWMUSIC\PROTOSS1.WAV

StarCraft German edition, StarDat.mpq:

SOUND\TERRAN\RAYNORM\TMARDY00.WAV

StarCraft: Brood War German edition, Install.exe:

CAMPAIGN\EXPTERRAN\TERRAN07\STAREDIT\WAV\T7M06STU.WAV
FILES\LOCALOLD.DLL

StarCraft Brazil edition, StarDat.mpq:

SCRIPTS\CRDT_LST.TXT

StarCraft: Brood War Brazil edition, Install.exe:

SETUPDAT\INST_F~1.$$$
SETUPDAT\STRINGS.$$$

setupdat_working_sp\bwunin.exe
setupdat_working_sp\bwunin.pif
setupdat_working_sp\cleanup.ins
setupdat_working_sp\debug.ins
setupdat_working_sp\defaults.vis
setupdat_working_sp\directx.ins
setupdat_working_sp\gendefs.ins
setupdat_working_sp\inst.ins
setupdat_working_sp\inst.vis
setupdat_working_sp\installed.ins
setupdat_working_sp\instgen.ins
setupdat_working_sp\inst_files.ins
setupdat_working_sp\inst_gen.ins
setupdat_working_sp\inst_regs.ins
setupdat_working_sp\inst_sys.ins
setupdat_working_sp\iswinnt.ins
setupdat_working_sp\license.txt
setupdat_working_sp\mainplay.vis
setupdat_working_sp\noboot.dat
setupdat_working_sp\normal.ins
setupdat_working_sp\prepbrood.ins
setupdat_working_sp\savetemp.ins
setupdat_working_sp\scr_blizzard.vis
setupdat_working_sp\scr_isp.vis
setupdat_working_sp\scr_main.vis
setupdat_working_sp\scr_options.vis
setupdat_working_sp\scunin.exe
setupdat_working_sp\scunin.pif
setupdat_working_sp\setup.vis
setupdat_working_sp\single.ins
setupdat_working_sp\spawn.ins
setupdat_working_sp\spawn.vis
setupdat_working_sp\strings.ins
setupdat_working_sp\INST_F~1.$$$
setupdat_working_sp\upgrade.ins
setupdat_working_sp\upgrade.vis
setupdat_working_sp\95\comctl32.dll
setupdat_working_sp\audio\battlenetclick.wav
setupdat_working_sp\audio\installermusic.wav
setupdat_working_sp\audio\mousedown.wav
setupdat_working_sp\audio\mouseover.wav
setupdat_working_sp\font\font.ccd
setupdat_working_sp\font\font16.fnt
setupdat_working_sp\font\font16x.fnt
setupdat_working_sp\font\font32.fnt
setupdat_working_sp\gen\allied.lst
setupdat_working_sp\gen\broodwar.lst
setupdat_working_sp\gen\bwladder.lst
setupdat_working_sp\gen\ladder.lst
setupdat_working_sp\gen\oldladder.lst
setupdat_working_sp\gen\webmaps.lst
setupdat_working_sp\images\install.pcx
setupdat_working_sp\nt\comctl32.dll
setupdat_working_sp\STRINGS.$$$
setupdat_working_sp\instcc.exe
setupdat_working_sp\riched20.dll
setupdat_working_sp\vssver.scc

The last one is particularly nice. The Brazilian Brood War installer contains 57 files, one after another, that turned out to be an older, Spanish working copy of the installer's SETUPDAT directory, but under a different name. Most of their basenames could be worked out, but the directory itself eluded us. Finding SETUPDAT_WORKING_SP thus gave names to the other 56 files in it as well, including its subdirectories 95, AUDIO, FONT, GEN, IMAGES and NT.

Also of note are SETUPDAT\INST_F~1.$$$, SETUPDAT\STRINGS.$$$ that contain the characters $ and ~ that we normally don't search for. These files themselves seemed to be half-done versions of other files, and some editors in the 90's would save temporary copies of files with .$$$ extension. The ~1 part is the filesystems of the times that would trim longer file names in this way. We were lucky to have discovered these thanks to AI that tried a bunch of more or less likely patterns.

Diablo II: Lord of Destruction Japanese edition, setup.mpq:

SETUPDAT\JPN\VSSVER.SCC
SETUPDAT\LOC_INSTALLED.INS
SETUPDAT\US_INSTALLED.INS

Encryption keys give away the basename

Several of the names above were cracked by first finding only the basename (the part after the last backslash), and then the directory separately. This is possible because many files in MPQs are encrypted, and the key a file is encrypted with is derived from a hash of its basename. Strictly speaking, the key is the basename's hash, adjusted by the file's position and size in the archive, but that adjustment is easily undone. The key itself isn't stored anywhere, but since we know what the first few bytes of an encrypted file must decrypt to, the key can be recovered from the encrypted data.

This is very useful: it means a single file gives us three hashes rather than two, and that one of them only depends on the basename. For example, the French Brood War music files were identified to be identical to PDEFEAT.WAV, PRDYROOM.WAV and PROTOSS1.WAV. Because of the encryption key, we can now verify that the files indeed has these basenames. The remaining search is then for the directory, which turned out to be NEWMUSIC. The key also lets the client discard false matches; with the number of candidates we hash, names matching both of the regular hashes purely by chance do occur, but a chance match whose basename also hashes to the key is far less likely.

I've made a pull request to mpqcli so that it can correctly show the encryption keys of files, which makes this easier for anyone wanting to look into an archive themselves. This is now part of mpqcli version 0.13.0.

New search modes: dictionary and meet-in-the-middle

So far, the client has only been able to bruteforce every combination of characters from an alphabet. That works well for short names, but every additional character multiplies the search space by around 40. I've therefore added two new search modes to the client.

Dictionary search builds the candidates out of words instead of individual characters: one word, or several with separators such as _, - or a space between them. The client comes with an English word list of about 64,000 words, and the coordinator can hand out dedicated word lists, for instance with StarCraft-related words. Long names made of real words, which are far out of reach for plain bruteforcing, can be found this way.

Meet-in-the-middle search is for finding unknown directories, and it is what cracked SETUPDAT_WORKING_SP. The MPQ hash function processes a name one character at a time, and it turns out that each step can be undone if the character is known. When many files are known to be in the same directory (as they were in this copy of SETUPDAT), it is possible to recover what the hash state must be right after that directory. The search can then work from both ends: forward from the start, building a table of every possible beginning of the directory name and which hash state each leads to, and backward from the recovered state, rewinding possible endings of the name and looking up whether they meet any of the beginnings in the table. A 5-character table costs 1.4 GB of memory, but it means that 5 characters of the name come for free, which turns a search that would take a year of GPU time into one that takes minutes. For this to be useful, we must know that several files exist in the same directory, we need to know their basenames, and we need to have enough files to be able to recover the hash state — in other words, it's quite special circumstances needed for it to be a useful search.

Both modes are available in the latest releases, both when running locally and when connected to the coordinator, on CUDA, OpenCL, Metal and the CPU.

What we're struggling with

Some targets are still stubbornly resisting:

• The "Poor Fellar's Dog" cutscene from the Brazilian StarCraft Install.exe, a version of the music without voiceover in the MUSIC directory. Over 3 quintillion names have been tried by brute force, and dictionary searches of up to five words have been exhausted without a hit. The name is probably >= 13 characters long and made of words or abbreviations, or it contains some special character we haven't searched for.
• Two sounds in the REZ directory of the StarCraft installers: copies of swishlock.wav and swishout.wav. Several quintillion candidates have been searched for these too.

If you find this interesting, please join the search by running namebreak-bombe on your machine! The dashboard here shows the current progress on the remaining names.

Yesterday, TheNitesWhoSay released version 2.16 of the StarCraft I map editor Chkdraft.

It is a smaller release with two bug-fixes:

• Fix a bug where zero-sized anims (e.g. start locations) rendered at the wrong positions on HD skins
• Fix a crash where cutting units while not having a selection resulted in a crash

Version 2.16 is available here.

Thomas Laurenson just made a new release of mpqcli. The release makes the subcommands consistent, with the archive now always being the first argument, and lets the create and extract subcommands work on several files at once. Note that this breaks compatibility with scripts written for earlier versions, as some arguments have moved and a few flags have been removed.

Changes in mpqcli 0.13.0:

Added

• Accept several files, or a list read from stdin, in create and extract
• Add a flat option to extract
• Add short flags -e, -j, -q and -x for the locale, flat, property and print options

Changed

• Take the archive as the first argument in every subcommand, including create and read
• Keep folders when extracting named files
• Rename the create version option to format-version
• Reject an unknown locale in extract and read instead of using the default locale

Fixed

• Show the encryption keys of encrypted files listed without their names
• Accept Windows line endings in paths read from stdin
• Stop add and create from storing an archive inside itself
• Leave no archive behind when create cannot add every file

Removed

• Remove create -o, extract -f and -k, and the -p flag from info, list and verify

Download:

Find version 0.13.0 of mpqcli here.

Milestone and iDoodler have released the first alpha of StarDraft 2, a new map editor for StarCraft: Remastered on Windows. It can be used for classic maps, but its main focus is EUD maps, with new units, weapons and game rules. Apparently, modding support is on the way.

The name is a homage to the original StarDraft by Camelot Systems from 1998, which pretty much started the StarCraft modding scene and gave modders what they needed to make their creations in the early 2000s. StarDraft 2 aims to be a spiritual successor for StarCraft modders in 2026 and beyond. Just like the original, StarDraft 2 seems to be aiming to be a one-stop-shop for all modding eventually.

EUD mapping has opened up exciting possibilities for StarCraft maps, but getting started can be challenging. Typically, one needs to set up euddraft alongside one or more editors, learn which tool handles what, and switch between them to edit units, triggers and other game data. The goal of StarDraft 2 is to bring all of this together in one place, making EUD mapping more accessible to newcomers while giving experienced mappers a faster and more integrated workflow.

Some of the features:

• Maps: terrain brushes with symmetry, generated terrain, HD, SD and Carbot graphics, and a town area overlay
• Classic maps: a classic trigger editor, unit, upgrade and tech settings, and a briefing editor with a preview
• EUD projects: editing of every .dat file, as well as strings, requirements and button sets, with previews of things such as ranges, images and the command card
• Triggers: trigger blocks are turned into epScript, a checker finds EUD errors and desyncs, and you can link your own .eps files
• Debugging: a debug mode, and epTrace profiling that shows which script lines take the most time
• Building: Ctrl+B builds the playable map, and StarDraft 2 downloads euddraft for you

More features are planned, such as campaign projects and support for mods. The authors update the editor often based on feedback, and the editor checks for updates when it starts. Feedback can be given by right-clicking almost anything and selecting Report feedback, or on the StarDraft 2 Discord.

As this is an alpha, expect bugs and features that are not fully implemented. Save often and keep backups of your maps!

A video showing the editor can be seen here.

Download:

Find the latest version of StarDraft 2 on GitHub.

jeany55 has been working on scmJS, a new map editor for StarCraft, Brood War and StarCraft: Remastered. It takes inspiration from StarEdit, SCMDraft 2 and StarForge, and runs either directly in the browser or as a desktop app for Windows, macOS and Linux. The editor is still in beta, but it already has an impressive set of features.

scmJS opens and saves .scm and .scx maps as well as bare .chk scenarios. Map sections that the editor does not understand, as well as malformed ones, are written back unchanged when saving, so maps are not damaged by sections the editor does not know about. Backups (.bak) and recovery copies are kept as well.

Some of the features:

• Classic graphics, or StarCraft: Remastered graphics if you point it to a Remastered installation
• Real-time collaborative editing, where several people can edit the same map at once through a shared link
• TrigScript, which lets you write triggers in TypeScript, with a simulator for testing them
• A Repair plugin for damaged or protected maps, and a Section Explorer that shows the contents of the map file byte by byte
• Plugins for graphing trigger dependencies, showing walkability (islands, chokepoints and distances), importing terrain from images and recording timelapses as GIF or WebM
• A stamp library for reusable map pieces, and conversion of doodads to terrain
• Searching for and opening maps from scmscx.com
• English and Korean interfaces

No game data is bundled with the editor. On first use, it offers to download the needed files from Blizzard, or you can use the .mpq files from a classic installation.

Try it out:

The editor can be used directly in the browser at scmjs.dev/editor, where the maps stay on your own disk. Desktop builds are available on GitHub, and there is also a Docker container image:

docker run --rm -p 8080:80 ghcr.io/scm-js/scm-js:latest

The documentation, including a user guide, a TrigScript guide, a trigger reference and a reference for the CHK format, is available at docs.scmjs.dev. The source code is available on GitHub under the MIT license.

For many years, Blizzard used MPQ files as their archive files for many of their games. MPQ files are basically like ZIP files; they store many other files in a compressed format. One quirk of MPQ files is that they don't contain the filenames of their own content. Some MPQ files contain a listfile consisting of the filenames of their content, but other MPQ files do not. For these, an external listfile is needed. By now, we know the names of nearly all files of these games, but there are a few files whos names are yet to be cracked.

I have made a tool, namebreak-bombe (available on GitHub), which bruteforces filenames on GPUs. By default, it will contact a central server which will assign it a range to search in, thus allowing coordinated efforts where many people simultaneously can look for filenames (though it can also be set to run locally only without talking to servers).

I ask everyone who finds this interesting to join in on the search by running namebreak-bombe on their machines - the more machines that run this, the faster we crack this. The dashboard here shows the current progress on the remaining names.

I have over the past month optimised the client significantly, and me and Heinermann have been testing it together on Linux and Windows 10 over the past few days. Already we have uncovered quite a few names that were previously unknown:

WarCraft II: Battle.net Edition, War2Dat.mpq:

ART\CHAT_BNE_PLAIN.PCX

WarCraft II: Battle.net Edition, Install.exe:

MacFiles\DF.ModuleUniverselUSBHID
MACFILES\DF.PATCH.TXT
MacFiles\RF.ModuleUniverselUSBHID
MACFILES\RF.PATCH.TXT

StarCraft Beta, StarDat.mpq:

GLUE\BATTLE.NET\ICONS\RAD_PRESSED.PCX
GLUE\BATTLE.NET\ICONS\RAD_UNPRESSED.PCX

StarCraft Brazil edition, Install.exe:

CAMPAIGN\EXPTERRAN\TERRAN03\STAREDIT\WAV\T3M02GAL.WAV

Diablo II Mac Installer TOME:

103b\DF.BNUpdate
103b\DF.Diablo II
103b\DF.Storm
103b\RF.BNUpdate
103b\RF.Diablo II
103b\RF.Storm

103c\DF.Diablo II
103c\RF.Diablo II
103c\DF.Storm
103c\RF.Storm
103c\DF.BNUpdate
103c\RF.BNUpdate

DF.InputSprocket Extension
RF.InputSprocket Extension

Again, if you find this interesting, please join the search by running namebreak-bombe on your machine! The dashboard here shows the current progress on the remaining names.

IronGRP is a GRP to PNG converter for WarCraft I, WarCraft II and StarCraft.

I just released IronGRP version 0.6. It mainly focuses on bug-fixes, hardening the program and taking care of corner cases. The command-line interface now uses subcommands, which changes the interface compared to before.

Some of the breaking changes:

The command-line interface uses subcommands instead of --mode, and takes the input and output paths as positional arguments:

• --mode grp-to-png --input-path X --output-path Y is now grp-to-png X Y
• --mode png-to-grp --input-path X --output-path Y is now png-to-grp X Y
• --mode analyse-grp --input-path X is now analyse X (analyse-grp is accepted as an alias)
• --generate-shell-completions SHELL is now completions SHELL

Some flags are renamed: --pal-path to --palette, --use-transparency to --transparent, --frame-number to --frame, --compression-type to --compression and --analyse-row-number to --row. The old names are still accepted. New short forms: -c for --compression, -f for --frame and -r for --row.

Some other safety features:

• Existing files are no longer silently overwritten. png-to-grp refuses to overwrite an existing GRP, and grp-to-png refuses to overwrite existing PNGs or to write all frames into a directory that already contains frame PNGs, unless the new --force flag is given.
• png-to-grp writes the GRP to a temporary file and renames it into place, so a failure no longer leaves a partially written GRP.
• png-to-grp reports up front when the input path is not a directory.

These are just a few of the many changes. See the full changelog here.

IronGRP is available on GitHub.

Thomas Laurenson just made a new release of mpqcli. The release mainly refactors the code, overhauls the install scripts and release artifacts, and makes the program work more in accordance with how terminal programs are expected to behave.

Changes in mpqcli 0.12.0:

Added

• Stop cleanly on Ctrl-C, leaving no partial archive behind

Changed

• Ship one static musl Linux binary per architecture, with the release assets renamed by OS and architecture
• Publish install scripts and a signed checksums file with each release
• Run the Docker image in /data so mounted paths need no prefix
• Print progress and warnings on stderr, leaving stdout for results alone
• Mark an overwritten file as changed rather than added when adding

Fixed

• Fail with a non-zero exit code on an unknown subcommand, a missing argument or an internal error
• Fail with a non-zero exit code when closing, signing, listing or printing a signature fails
• Complete every locale, game profile and archive path in the zsh, fish and PowerShell scripts

Removed

• Remove the install scripts from the repository; each release now carries its own

Download:

Find version 0.12.0 of mpqcli here.

Recently, milestone started working on some tools for Warcraft II: PudForge and RuneFoundry.

PudForge is a very competent map editor. The map format is the same in the original DOS versions of Warcraft II as well as the Battle.net Edition and the more recent Remastered version. PudForge thus works for all of them. The author, milestone, has made a video showing the features, and it has some really nice capabilities. I was especially impressed with the terrain editing, as the editor supports copying and pasting terrain, replacing a given terrain type with other types, generating maps, drawing symmetrical shapes and much more.

RuneFoundry is a set of mod tools for Warcraft II Remastered only — not the DOS originals or Battle.net Edition. It has two programs, a mod launcher and an editor. A video showing its features is available here along with a video of an example campaign which plays through a campaign made with RuneFoundry. It can edit things like the campaign, AI-scripts, sprites and unit and weapon properties.

It is fantastic to see such feature rich and capable tools still being put out! Hats off to milestone for the efforts!

Find PudForge here and RuneFoundry here.

I just saw that Thomas Laurenson made a new release of mpqcli earlier this week. Many of the changes are things I contributed a while back, that has now made it into release 0.11.0. mpqcli is a command-line tool to create, add and remove files from MPQ archives. MPQ archives were used for many years by Blizzard games such as StarCraft I and Warcraft II, and contain the game assets that game modifications change.

I'd like to highlight a few of the changes that I contributed to this release:

• A rename subcommand has been added, which allows you to rename a file inside an archive without having to remove and re-add it.

• The logic for updating files in an archive has been improved. When working on archives containing the (attributes) file, mpqcli will now — if the --update flag is given — compare timestamps and checksums in addition to just file size when deciding whether to update a file. This means that it will much more quickly decide whether a file needs to be updated or not, and will avoid unnecessary writes to the archive. This was quite an extensive change, that also involved me making some contributions to StormLib, the MPQ library that mpqcli uses to read and write MPQ archives.

Changes in mpqcli 0.11.0:

Added

• Add rename subcommand to rename a file inside an archive

Changed

• Compare timestamps and checksums when updating files, not size alone
• Accept the --update flag for single files as well as directories
• Reject the --update and --overwrite flags on add when both are given

Fixed

• Report filesystem errors instead of aborting when a path cannot be resolved
• Add to and create archives on volumes that cannot report real paths
• Block path traversal in extract before anything is written to disk

Updated

• Update StormLib for a variety of fixes
• Add hash-library as a submodule dependency
• Build with StormLib's bundled compression libraries instead of system packages
• Require CMake 3.25 or later

Download:

Find version 0.11.0 of mpqcli here.