Replace ZAPDTR and OTRExporter with Torch - #6989
Merged
Merged
Conversation
Plan for replacing ZAPDTR + OTRExporter with Torch as SoH's only asset pipeline, following HarbourMasters/Torch#219 ("oot support"). Co-Authored-By: Claude <noreply@anthropic.com>
- Name the actual repo (briaguya0/soh-asset-yml); it holds yml only, and is edited directly rather than regenerated. zapd_to_torch.py is a one-shot conversion, not an ongoing workflow, since nobody edits the ZAPD XMLs once ZAPD is gone. - Phase 5 is pure deletion. Generator inputs are already preserved by the harness's shipwright@95d8f7e submodule pin and by git history, so nothing needs snapshotting or moving out of Shipwright. - Clarify soh/CMakeLists.txt:107-109 as a configure-soh-directly fallback with nothing left to fall back to. - Downgrade the extraction wall-clock risk; perf work landed in Torch#219. Co-Authored-By: Claude <noreply@anthropic.com>
Adds torch-migration/PHASE0.md with the full gate detail: what each gate's
variable is, the driver source, exact commands, cost, and exit criteria.
Two gates the plan was missing:
- Gate C (Release build) - the harness 14/14 is a Debug measurement; CI
and releases build -O3.
- Gate A2 (two extractions in one process) - free once the driver exists,
retires the re-entrancy risk offline instead of by hand in the game.
Also: gates run full-tree + check.sh rather than test_assets.py (which
hardcodes the torch path and copies a filtered yml subset), 19 ROM dumps
not 17, and libgfxd being USE_STANDALONE-only is called out as the real
hazard in Gate A.
The "filtered subset" was a misread. The log line prints the pre-recursion yml count (1320); setup_scratch's external_files recursion then pulls in the remaining 130, so an unfiltered run copies all 1450 - verified by replaying the logic. So the gates use test_assets.py, the same tool that produced the existing 14/14, with a TORCH_BIN env override as the only harness patch. Gate A2 keeps the check.sh path, since it needs two extractions in one process and test_assets.py execs the binary once per ROM.
Baseline Torch build (USE_STANDALONE=ON, all games, Debug, PORT_VERSION_ ENDIANNESS=ON) reproduces every reference archive: 19 ROM dumps, all 0 failed / 0 not generated / 0 not in reference. Counts match the harness README's 14-target table, and the duplicate dumps agree with each other, which the 14-target matrix never showed. Reference manifests also spot-checked by re-derivation from two o2r files -- the baseline the gates measure against hasn't drifted.
USE_STANDALONE=OFF plus driving Companion by hand produces byte-identical archives for all 19 ROM dumps. The real question was libgfxd, which is fetched only if(USE_STANDALONE) -- _deps/ confirms it is absent, so the Binary export path is genuinely gfxd-free. Reading said so; now it's run. Progress denominator confirmed exact: SetPhaseCallback fires once per yml file, matching the on-disk count for all three classes (1449/1450/1480). Phase 3's bar can be monotonic with no change to OTRGlobals. Also observed for Phase 2: zlib is fetched even with BUILD_STORMLIB=OFF and no zlib.h use in torch/src, confirming risk #1; spdlog is not fetched, so only the global-logger stomping applies there.
Two extractions in one process, vanilla then MQ, both byte-identical to their references. The second run's phases=1450 matches pal_mq's on-disk yml count exactly, which is the direct refutation of the gProcessedFiles risk -- a leak would have skipped files and truncated the archive. A fresh Companion per extraction is sufficient; PLAN.md risk #6 retired. Also records what the gate turned up on the way: torch writes 25 duplicate archive entries, benign because libultraship indexes by CRC64 of the path. Filed as Torch#233 item 5. Corrects the Gate A note that attributed the whole ~343 KB size delta to compression -- 76 KB of it is these duplicates.
OoT-only (eight BUILD_<game> flags OFF) and Release (-O3) each 19/19 identical, one variable moved apiece -- verified against the option() defaults so the other flags genuinely match the baseline cache. Gate C mattered: every parity measurement to date, including the original 14/14, was Debug, while CI and releases build -O3. Also records extraction wall-clock from torch's own timing line: Release averages 11.4s per ROM against Debug's 26.7s, and the OoT-only flags cost nothing. SoH ships Release, so PLAN.md risk #5 looks much smaller than the Debug numbers suggested -- though ZAPD's time on the same machine is still unmeasured.
Static lib + OoT-only + Release together: 19/19 identical plus the vanilla-then-MQ pair. Flags verified from the generated cache rather than assumed, so Phase 2's CMake block and TorchExtract.cpp can be transcriptions of this build and RunOnce respectively. Archive bytes turn out to be configuration-independent -- the pair produced exactly the sizes the Debug static-lib run did, phases=1450 on both runs. Extraction 11.8s mean, so the static-lib wrapper costs nothing over Gate C. Confirms both Phase 2 dependency risks in the configuration that actually ships: zlib fetched with BUILD_STORMLIB=OFF and no zlib.h includes, tinyxml2 fetched with BUILD_NAUDIO=OFF removing its only consumers. Both FetchContent declarations are unconditional.
Unblocks the rest of the migration. soh.o2r is produced by ZAPD via OTRExporter/extract_assets.py, both of which this migration deletes; its replacement is a small in-tree packer that has to reproduce ZAPD's texture quantisation exactly. Rather than block Phases 2/3/5 on that packer, check the archive in and make GenerateSohOtr a copy. The archive doubles as Phase 4's acceptance oracle -- the packer is correct when its output matches this entry-for-entry -- which is strictly more useful than Gate E's planned manifest of hashes of it. Generation turns out to be content-deterministic but NOT byte- deterministic: two runs give identical payloads for all 1,042 entries but different file hashes, because zip stores per-entry timestamps. So the Phase 4 comparison must be entry-wise; whole-file would fail even on a correct archive. Recorded in prebuilt/README.md. Adds a configure-time guard on the baked-in portVersion. OTRGlobals.cpp :283 requires exact major.minor.patch equality and RunExtract exit(1)s with "soh.o2r is outdated", which is an opaque symptom for a stale checked-in file; verified the guard fires by building against 9.2.4. ExtractAssets, CI and copy-existing-otrs.cmake are deliberately untouched -- ExtractAssets still generates a correct archive while ZAPD exists, and the CI job keeps working since it just builds this target.
Adds soh/assets/yml as a submodule (briaguya0/soh-asset-yml, 20,353 asset definitions) and FetchContents torch at 4cae4416, configured exactly as Gate A' measured it: static lib, OoT only, no UI, no StormLib. find_package(ZLIB) has to run before torch is declared. Torch declares zlib with OVERRIDE_FIND_PACKAGE, so it takes over the whole build's find_package(ZLIB), and its copy provides no ZLIB::ZLIB -- which both StormLib and CMake's FindPNG link by name. Filed upstream as Torch#233. CallZapd becomes CallTorch: no chdir, no symlinked assets dir, no 22-entry argv. Everything that needed the working directory is now a parameter. TorchExtract.cpp is the only TU that includes Companion.h, since torch exports its whole lib/ as PUBLIC includes. Progress uses torch's phase callback, which fires once per yml file, with the denominator counted off disk. Gate A confirmed the two match exactly for all three yml counts, so the bar runs 0-100 without the sawtooth the old per-file counter would have produced. GetZapdVerStr becomes GetTorchVersionDir, returning the version directory under the yml tree rather than a ZAPD xml directory. soh.elf links torch and no longer references zapd_report.
Vanilla then MQ extracted in one process, both loaded and played. Closes PLAN.md risk #6 in the game rather than only against the harness driver, and confirms config.yml-driven archive naming end to end.
soh links torch as a static library, which compiles out torch's own CLI, so build-time extraction needs an entry point. soh-torch supplies one around the same SohTorch::Extract the game calls -- one implementation, so the CLI and the in-game path can't drift. ExtractAssets keeps its old contract: SOH_ROM_PATH accepts roms and/or directories of roms and defaults to roms/, so dropping a vanilla and a master quest rom in produces oot.o2r and oot-mq.o2r in one run, and chaining GenerateSohOtr keeps soh.o2r coming out of it too. Extract now returns the archive name torch chose rather than deriving it from IsMasterQuest(), so config.yml is the only thing naming archives. The name comes from Companion::GetOutputPath(): scanning the destination for *.o2r picks up an unrelated soh.o2r when extracting into the build directory, which is only invisible in the game because it extracts into an empty temp dir. ExtractAssetHeaders stays as a target that explains why it can't run -- torch registers no OoT header exporter and emits no #define d<sym>, which soh source references ~24k times. Documented in BUILDING.md alongside it. This removes the last ZAPD dependency from the build.
BUILDING.md points people at roms/, but git can't track an empty directory, so a fresh clone didn't have one. Its .gitignore keeps everything but itself and the readme out.
Nothing drives them any more: ExtractAssets runs soh-torch, the in-game extractor calls Companion directly, GenerateSohOtr copies a prebuilt archive, and soh links torch instead of ZAPDLib. Verified with a build from a fresh tree with both submodules absent from disk. Removes the submodules, the 7,680 ZAPD xmls under soh/assets/xml, the extractor inputs (Config_*.xml, TexturePool.xml, filelists, symbols) and copy-existing-otrs.cmake. The xmls remain in this repo's history and in the test harness's pinned shipwright submodule, which is where the yaml was generated from. Packaging changes shape as well as content: the extractor assets used to install into the 'extractor' component, which Packaging-2.cmake only includes for the AppImage generator, so linux zips shipped without them. torch is linked into soh rather than being a separate binary, so the yml tree installs into 'ship' on every platform and every package can extract a rom.
Torch's fetched copy satisfies libultraship today, so the install steps are redundant -- but that rests on torch declaring tinyxml2 with an unconditional OVERRIDE_FIND_PACKAGE, which is an implementation detail, and one we've asked upstream to reconsider in Torch#233. If the fetch gets gated, find_package needs a system copy again and CI would break because we removed the steps providing it. Also notes that the generate-soh-otr slim-down was written before the prebuilt archive, which makes that job build nothing for now.
soh-o2r-packer builds soh.o2r from soh/assets/custom, reproducing the ZAPD/OTRExporter archive exactly: 1,042 entries, 0 missing, 0 extra, 0 content mismatch, checked entry-wise because zip stores per-entry timestamps and whole-file hashes never match between runs. Most of it is torch's: Companion::Pack walks the directory, zips it and writes portVersion; BaseExporter::WriteHeader writes the resource header; TextureType and CalculateTextureSize come from TextureUtils. The packer stages the assets into the shape the archive should have and hands that over. Encoding a PNG into an N64 texture is the only piece nothing else provides -- torch decodes rom data that is already N64 format, never the reverse -- so PngTexture.cpp is the whole of what had to be written, and it follows ZAPD's quantisation rather than n64graphics', which scales where ZAPD shifts. GenerateSohOtr copies the libultraship shaders into assets/custom again before packing; assets/custom/shaders is gitignored and nothing else populates it, so a fresh clone would otherwise pack three files short. Removes prebuilt/ and its configure-time portVersion guard.
soh-torch and soh-o2r-packer both exist to turn assets into archives, so they sit better beside the assets they read than at the repo root. Nothing tangled: both targets are declared in the root CMakeLists with explicit paths rather than add_subdirectory, and soh/CMakeLists.txt globs only include/, soh/ and src/, so sources under assets/ aren't swept into the soh target.
SOH_TOOLS_ONLY returns from the root CMakeLists once the asset tools are declared, before libultraship and soh are added, so a configure that only needs soh-o2r-packer never reaches LUS's find_package(SDL2 REQUIRED). That lets generate-soh-otr drop building SDL2 from source, SDL2_net, and the deps cache that existed to hold the SDL2 tarball -- eleven steps down to six. Verified a tools-only build produces the archive with all 1,042 entries matching. The tool targets move above the sub-projects so the early return can sit between them; they only ever needed torch. The packer creates its output directory, which previously came for free from soh's build directory.
macos: libultraship declares spdlog with OVERRIDE_FIND_PACKAGE so its own find_package(spdlog REQUIRED) resolves, but torch declares spdlog without it, and FetchContent_Declare is first-wins. Configuring torch first made LUS's declaration a no-op and left find_package with no config to find. It only showed on macos because that's the platform where neither project finds an installed spdlog -- it's kept out of macports.yml because its fmt dependency breaks the universal build -- so both fall through to fetching. Declaring it ahead of both keeps macos on the same source-built v1.16.0 it used before. Torch already uses OVERRIDE_FIND_PACKAGE for tinyxml2 and zlib, so the real fix is upstream. windows: soh builds with /WX, and TorchExtract.cpp is the one translation unit reaching yaml-cpp through torch's headers, which trips the dll-interface warnings. Suppress those two on that file. EXCLUDE_FROM_ALL on soh-torch and soh-o2r-packer: they are build-time tools, so building the game shouldn't compile them. ExtractAssets and GenerateSohOtr still pull them in through DEPENDS. Drops the .gitattributes rule for the prebuilt archive, which is gone.
Left over with the .gitattributes rule when prebuilt/ was removed.
Deleting 7,680 xmls puts the diff past what GitHub will render, and they are 99.8% of it. Restoring them here leaves the ~46 files that actually changed, which reviews in the compare view. Revert this before opening the PR.
Scaffolding for the migration itself rather than anything the repo needs to carry afterwards.
verMap's two master quest rows keyed off the GameCube constants instead of the MQ ones, so with duplicate keys dropped the map never contained OOT_NTSC_US_MQ or OOT_NTSC_JP_MQ and both roms were filtered out before extraction. IsMasterQuest already handles both, and the asset yml covers ntsc_u_mq and ntsc_j_mq, so keying them correctly is all that's needed. Predates this migration; the torch cli was unaffected because it hashes the rom rather than consulting verMap.
…ewable" This reverts commit 28f4113.
Renaming CallZapd to CallTorch made the call sites one character longer and left their continuation lines under-indented. Co-Authored-By: Claude <noreply@anthropic.com>
Member
otherwise happy to consolidate with other ports |
serprex
approved these changes
Jul 25, 2026
Member
|
Considering the amount of manual asset headers I added in the various version support PRs, I don't think ExtractAssetHeaders is a major deal to be missing as even with ZAPD it wouldn't generate headers for all the symbols needed for compiling |
Contributor
Author
|
Extraction wall clock was an open risk on this, so I measured it: Torch is ~2.1x faster. Measured through the in-game extractor rather than the CLI, since that's the path users actually wait on. Two throwaway branches replace the ROM picker's follow-on flow with "pick a ROM, extract it 10 times back to back, append each run's wall clock to a CSV". The patch is identical on both sides apart from the
Same ROM (PAL GC), same machine (32 cores), both Release, fresh temp dir per run.
all ten runs, seconds
Torch also holds steadier: a 0.41s spread, 0.35s of which is run 1 warming up, against ZAPD's steady ~0.4s upward drift from run 2 onward. Ten back-to-back extractions in one process showed no state accumulation on either side. Both produced complete archives — 35,411 entries / 33.2 MB for Torch against 35,386 / 32.8 MB for ZAPD. That 25-entry gap is the duplicate-name issue already filed on HarbourMasters/Torch#233, not missing content. Caveat: one ROM, one machine, 32 cores. Some of the margin is likely parallelism a 4-core machine won't have, so read 2.1x as an upper bound rather than what every user gets. The point is that extraction doesn't regress. Edit: Windows is faster too, once HarbourMasters/Torch#235 is in. Torch's pinned yaml-cpp was constructing a named |
HarbourMasters/Torch#234 fixes three Windows-only defects in the OoT factories: a 64 KiB over-read past the code segment that crashes extraction, backslashes in exported scene resource names that crash at scene load, and external_files being keyed differently from the directory walk, which processed every dependency twice. Without them a Windows build of this branch cannot extract a rom. Goes back to a HarbourMasters sha once that PR merges. Co-Authored-By: Claude <noreply@anthropic.com>
HarbourMasters/Torch#234 landed as 65eb11c, squashing the three Windows fixes this branch was pinned to a fork for. The squashed tree is identical to the fork commit that was tested, and extraction output is unchanged: 35,386 entries, same CRCs. Co-Authored-By: Claude <noreply@anthropic.com>
The submodule existed only so this PR's diff stayed reviewable; squash-merging it would have put a submodule on develop. This replaces it with the 20,353 files themselves, from briaguya0/soh-asset-yml@523be1d. The submodule's own README is dropped rather than vendored, since it describes that repo and was showing up in the assets folder of release downloads. Co-Authored-By: Claude <noreply@anthropic.com>
briaguya0
marked this pull request as ready for review
July 26, 2026 03:22
HarbourMasters/Torch#235 landed as e92c210. The pinned yaml-cpp predated jbeder/yaml-cpp@4fe2fb8, so every scalar conversion constructed a named std::locale; cheap on glibc, expensive on MSVC. Windows extraction of a PAL GC rom drops from 49.0s to 13.9s on the same machine, which also puts it ahead of ZAPDTR's 48.4s rather than fractionally behind. The merged commit is tree-identical to the fork commit those numbers were measured on. Extraction output is unchanged: 35,386 entries, same CRCs. Co-Authored-By: Claude <noreply@anthropic.com>
Reviewer feedback on HarbourMasters#6989: FetchContent makes iterating on torch itself awkward, since testing a change means pushing it somewhere the pin can reach. A submodule is editable in place, which is how libultraship already works. Nothing else changes -- FetchContent_MakeAvailable was already doing add_subdirectory, so the include topology and every build option are the same. The pin moves from GIT_TAG into the gitlink, at the same sha. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
One thing I noticed so far. Way too many YMLs contain |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces ZAPDTR + OTRExporter with Torch as the only asset pipeline, following the OoT support that landed in HarbourMasters/Torch#219.
ZAPDTR and OTRExporter are deleted outright. Asset definitions move from 7,680 ZAPD XMLs to 20,353 Torch YAMLs at
soh/assets/yml. Torch comes in as a submodule alongside libultraship, and is linked intosohas a static library, so dragging a ROM onto the game still works the same way.Windows testing turned up three Windows-only defects in the OoT factories, two of them hard crashes. Those are fixed in HarbourMasters/Torch#234, which has landed, and the submodule is pinned to that upstream sha.
Verification
The YAML wasn't hand-converted and hoped for; it was measured. Torch's output was compared file-by-file against an
oot.o2rproduced by ZAPDTR/OTRExporter at95d8f7eb5:19 of 19 ROM dumps extract byte-for-byte identically — 35,352–39,066 assets each,
0 failed, 0 not generated, 0 not in reference. Confirmed across CLI and static-library Torch builds, all-games and OoT-only, Debug and Release, and with two extractions in one process.soh.o2ris likewise reproduced entry-for-entry against the archive ZAPD produced: 1,042 entries, 0 missing, 0 extra, 0 content mismatch.Full detail, including why each build configuration was checked separately:
Manually verified in-game: extracting a vanilla ROM and then an MQ ROM in one session, both loading and playing.
Known gaps, worth a follow-up
Neither blocks the migration — review settled the first, and nobody has objected to the second. Both are worth doing once this lands.
1. Asset header regeneration is unavailable
ExtractAssetHeadersno longer works. It's kept as a target that fails with an explanation rather than vanishing, anddocs/BUILDING.mdsays the same.Torch can't replace it yet, for two independent reasons: every
REGISTERinsrc/factories/oot/isREGISTER(Binary, ...), so an export would silently omit ~10k assets whose factories have no Header exporter; and Torch emits onlystatic const ALIGN_ASSET(2) char sym[], not the paired#define d<sym>that soh source references ~24,698 times.What still works: the 1,085 checked-in
.hfiles undersoh/assetsare untouched ordinary source, and the build doesn't need regeneration. Only adding or renaming an asset symbol needs it — and as @inspectredc points out below, ZAPD's target didn't emit every symbol needed to compile either, so the version support PRs were already adding headers by hand.The follow-up: ~14 mechanical Header exporters plus the
#defineline, upstream in Torch, then restore the target.2. Torch findings we worked around
Doing this turned up a few things in Torch: duplicate entries in generated archives, zlib declared with
OVERRIDE_FIND_PACKAGEin a way that breaks the consumer'sfind_package(ZLIB), nothing that encodes a PNG into an N64 texture, and spdlog declared withoutOVERRIDE_FIND_PACKAGE— which broke the macOS build until we worked around it.They're written up as comments on Torch#233, each linking the commit here that works around it.
Windows testing since then found three more that couldn't be worked around ship-side: a 64 KiB over-read past the code segment that crashes extraction, backslashes in exported scene resource names that crash at scene load, and
external_fileskeyed differently from the directory walk, which processed every dependency twice — 2025 phase callbacks for 1450 ymls, and 42,119 archive entries against Linux's 35,411. All three are fixed in HarbourMasters/Torch#234, which has landed and is what this branch pins; with it, a Windows extraction produces an archive identical to the Linux one, which it did not before.The follow-up: fix the #233 items upstream and drop the ship-side workarounds.
Reviewing this
7,680 deleted XMLs and 20,353 added YAMLs are all but a rounding error of this diff, and make the PR painful to look at. To see it without either:
develop...1875a0a — the same changes at 44 files.
That's the branch at the last commit before the XMLs were deleted again: 28f4113 puts them back purely so that view exists, and 6ec230c removes them for real.
What's where afterwards
soh-torch— build-time ROM extraction forExtractAssets;USE_STANDALONE=OFFcompiles out Torch's own CLI, and this drives the same code the game runssoh-o2r-packer— buildssoh.o2r; most of it isCompanion::Pack, with PNG→N64 encoding as the only piece Torch has no equivalent forSOH_TOOLS_ONLY— configures just the tools, skipping libultraship and soh, which is all CI needs forsoh.o2rroms/now rather than the deletedOTRExporter/Build Artifacts