oot support - #219
Merged
Merged
Conversation
Recovered from filesystem after data loss. This squashes ~58 commits originally made between 2026-03-23 and 2026-03-28. The full original reflog is preserved in docs/recovered-git-history.md. New OoT-specific factories: - OoTSceneFactory (OOT:SCENE, OOT:ROOM) — scene command parsing and binary export - OoTSkeletonFactory — skeleton, limb, and skin vertex support - OoTAnimationFactory — normal, curve, legacy, and player animations - OoTCollisionFactory — collision mesh with camera data and waterboxes - OoTArrayFactory — Shipwright-compatible VTX and Vec3s arrays Modified upstream: - DisplayListFactory — OoT cross-segment DList handling, VTX consolidation, virtual segment 0x80, G_BRANCH_Z discovery, ZAPD compatibility fixes - Companion — OoT factory registration, BUILD_OOT cmake option - ResourceType — OoT type codes (OSKL, OSLB, OANM, OROM, OCOL, OPTH, OTXT) Tooling (soh/): - zapd_to_torch.py — converts ZAPDTR/OTRExporter XML to Torch YAML - test_assets.sh, check.sh, verify.sh, manifest.sh, lib.sh — test harness - list_assets.py — asset manifest query tool Status at time of loss: 20,432 assets passing, 0 failures. 14,355 scene assets in progress (scene/room factory implemented, iterating on binary format correctness). OoTTextFactory was not recovered and needs recreation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- identify_roms.sh: identifies OoT ROMs by SHA1, renames to standardized format, handles duplicates - extract_dma.py: extracts DMA tables from all 17 ROM versions using Shipwright filelists, outputs JSON keyed by filename - Pre-computed DMA tables for all 17 versions (14 unique) - Manifests directory with gitignore for generated hash files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
config.yml moved from soh/ to soh/assets/yml/ where Torch expects it. Generated per-version YAML dirs are gitignored via local .gitignore rather than the top-level one. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add lib/libyaz0/ with decode support following libmio0/libyay0 pattern - Wire YAZ0 into Decompressor::Decode and AutoDecode - Add missing PendingVtx struct in DeferredVtx namespace - Add missing IS_VIRTUAL_SEGMENT macro in BaseFactory.h - Add libyaz0 to CMake C_FILES glob Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- TranslateAddr now recognizes high segments (>= 0x80) when they exist in the segment map, not just standard segments (0x01-0x1F) - ASSET_PTR extracts segment offset for virtual segments too, preventing raw 0x80XXXXXX addresses from being used as buffer offsets Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OTRExporter writes 0-byte files for LimbTable entries. BlobFactory crashed when trying to Write() a null buffer. Guard the write with an empty check. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Empty blobs (e.g. LimbTable) now write 0 bytes to match OTRExporter reference output instead of writing a header with size 0 - test_assets.sh auto-logs to soh/logs/ with timestamp - New compare_asset.sh tool for hex-diffing individual assets between reference and generated O2R Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
*.o2r are generated archive files. torch.hash.yml is a Torch build cache tracking which YAMLs have been processed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Hash all extracted files in a single sha256sum call instead of one process per file - Redirect torch output to a log file instead of piping through grep - Collapse duplicate jq reduce into one pass with inline fail count Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrites the asset test script in Python to avoid per-file process spawning. YAML collection, O2R extraction, and hashing are all done in-process. Hashes assets directly from the zip without extracting to disk. 107s → 1.6s for 17,516 object assets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add BUILD_OOT option (default ON) following pattern of other games, defines OOT_SUPPORT so OoT factories are registered - Stub OoTTextFactory so it compiles (real impl is task #5) - Expose DeferredVtx::BeginDefer in DisplayListFactory.h so OoTSceneFactory can call it Enables 16,952 additional assets: 12,377 → 29,329 passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Enable GFX auto-discovery for auto-discovered limbs (previously disabled, causing 573 limbs to have empty DList paths) - Fix LOD limb DList suffix: use "FarDL" instead of "DL2" to match OTRExporter/ZAPDTR naming convention - Fix Curve limb DList suffixes: "CurveDL"/"Curve2DL" to match ZAPDTR - Resolve LOD far DList before near, so shared-address limbs use the Far name for both fields (matches OTRExporter behavior) - Rewrite compare_asset.sh as compare_asset.py (takes two O2Rs, no torch run needed) - test_assets.py now saves generated.o2r to soh/o2r/ by default Objects: 17,322 passed, 1 failed (MTX), 193 not generated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OTRExporter writes a 0-byte file for each skeleton's limb array (e.g. gKeeseSkeletonLimbs). Add this to the skeleton factory's parse to match. Objects: 17,515 passed, 1 failed (MTX), 0 not generated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OTRExporter/ZAPDTR reads the N64 Mtx as 16 sequential int32 BE values and writes them back as-is. Our exporter was writing individual uint16 int-part values, which produced byte-swapped output within each 32-bit word. Now reads and stores the raw int32 values in the parser and writes them in the binary exporter, matching the reference format. Objects: 17,516 passed, 0 failed. Code: 11 passed, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When multiple segments map to the same physical ROM address (common for overlays which alias segments 8-13 to their code data), the virtual address patcher was returning a segment 0x0D address instead of segment 0x80. This caused texture lookups to fail because textures are registered under segment 0x80 offsets in the YAML. Now explicitly prefers segment 0x80 when it maps to the same physical address, matching how YAML offsets are declared. Overlays: 325 passed, 0 failed (was 101 failures). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scene/room DLists are auto-discovered by the scene factory with room-prefixed names matching OTRExporter output. Pre-declared DList entries from ZAPDTR XMLs used different naming (gXxxDL_ vs xxx_room_0DL_) causing mismatches. Scenes: 10,729 passed, 0 failed (was 27 failures). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Room mesh DLists are auto-discovered by the scene factory with correct room-prefixed names. Pre-declared DLists from ZAPDTR XMLs (both room-named and scene-named) conflict with auto-discovery. 18 scene-level DLists declared in room files (e.g. gKinsutaDL_0030B0) are now missing — these need to be handled by the scene factory or a separate mechanism. Tracked as part of scene work. 31,156 passed, 1 failed (version), 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scene/room alternate headers (SetAlternateHeaders command) are now recursively processed as sub-assets. Processing is deferred until after the primary header's commands (especially SetMesh) complete, so primary DLists are registered first and alternate headers reuse their names for shared ROM addresses. DeferredVtx state is saved/restored around each alternate header to prevent VTX consolidation corruption. Exposes SaveAndClearPending/RestorePending and PendingVtx struct in DisplayListFactory.h for use by scene factory. 31,436 passed (+280), 128 scene failures (Sets/Cutscenes), 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Alternate headers pass parent's baseName for sub-asset naming (DLists, backgrounds, cutscenes, pathways) so names match OTRExporter which doesn't prefix with Set_ - Fix cutscene suffix: "CutsceneData" instead of "Cs" to match OTRExporter's GetSegmentedPtrName convention 31,501 passed (+345 from session start), 108 failed, 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Cutscenes use entryName (with Set_ prefix) matching OTRExporter - Pathways use baseName (parent name) matching OTRExporter - Fix cutscene suffix: CutsceneData instead of Cs 31,583 passed, 109 failed (84 Set command data, 24 cutscenes, 1 version). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use getNeighborSize to limit pathway entry scanning instead of a hard 256 maximum. This helps some alternate headers with tight boundaries, though pathway count inference remains imperfect without XML metadata. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OTRExporter creates empty placeholder files for actor list data (e.g. Bmori1_room_0ActorEntry_000054). Add these as companion files in the scene factory. 32,151 passed (+568), 109 failed, 0 regressions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OoT alternate headers reference the same DLists as primary headers under Set_-prefixed names. OTRExporter creates both files with identical content. - Add RegisterAssetAlias to Companion for creating duplicate O2R entries with the same binary data under different names - Scene factory uses entryName for DList symbols and ResolveGfxWithAlias to register aliases when an existing DList is found at the same offset - Alias files are written during the export phase using the already-serialized binary data (zero re-parsing overhead) 34,539 passed (+2,388), 109 failed, 738 not generated. Session total: 12,377 → 34,539 (34.9% → 97.6%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace naive 0xFFFFFFFF scan with a command-aware parser that correctly determines cutscene boundaries by parsing the command structure (ID + entry count + entry size per type). Handles camera splines (terminated by continueFlag), scene transitions (0x2D), destinations (0x3E8), and standard commands. Cutscene sizes are now correct, but content still differs from reference because OTRExporter re-serializes with different byte ordering (ROM is BE, O2R is LE with CMD_HH packing). Full re-serialization is the next step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document the BE→LE field re-packing needed for each command type. Raw copy doesn't work because OTRExporter uses CMD_HH/CMD_BBH/CMD_HBB macros to pack fields into uint32 words differently than ROM layout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace raw cutscene copy with proper BE→LE re-serialization using CMD_HH/CMD_BBH/CMD_HBB field packing to match OTRExporter output. Handles camera splines, actor cues, misc/lighting/BGM, textbox, rumble, settime, transition, and destination commands. 33 additional cutscenes now match. 76 failures remain (likely a subtle issue with uint16/uint32 field reading in some entries). 34,572 passed (97.7%), 76 failed, 738 not generated. Session total: 12,377 → 34,572 (34.9% → 97.7%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Actor cue entries have rotY/rotZ as the 3rd word packed with CMD_HH, not a raw uint32. Differentiate actor cues from misc/lighting/BGM commands to apply correct packing. 34,602 passed (97.8%), 46 failed (44 cutscene, 1 pathway, 1 version). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
34,602/35,386 (97.8%) passing. Remaining: 44 cutscene format issues, 598 audio (no factory), 135 scene sub-assets, 4 text. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Keep the original main code at its original indent level inside if (!OoT::DListHelpers::Handle...) blocks to minimize the diff against main. Add comments explaining the intentional non-indentation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace per-opcode wrapper blocks in DisplayListFactory with three method-level replacements (SearchVtx, Export, Parse) that early-return when OoT is active. DisplayListFactory.cpp diff vs main is now just 10 insertions (1 include + 3 early-return blocks), with main's code completely untouched. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strip all OoT work-in-progress documentation and SoH-specific tooling, manifests, DMA tables, VTX data, and test infrastructure to produce a clean Torch-only branch for PR review. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closed
inspectredc
reviewed
Apr 9, 2026
inspectredc
left a comment
inspectredc
left a comment
Member
There was a problem hiding this comment.
First of all this is really cool to see! I haven't gone through and tested the factories yet but left a few questions about the PR overall in the major sections (the main major blocker being the issue in the DisplayListFactory)
The message-text reader assumed single-byte characters and stopped at the first 0x00/0x02/0x07 byte, which truncates Japanese messages (whose two-byte characters routinely contain those byte values) — jpn_message_data_static came out ~10x too small. Add a Japanese reader alongside the English one. The NTSC/PAL split is the table format (inline offset vs language table); English vs Japanese is the text encoding, so ParseMessagesNTSC becomes ParseMessagesEnglishNTSC and a sibling ParseMessagesJapaneseNTSC handles two-byte text. Japanese characters are big-endian and stored byte-swapped; control codes take trailing shorts and 0x8170 ends the message (mirrors the English IsEndOfMessageCode / GetTrailingBytes helper structure, ported from ZAPD's ZText). parse() routes to it when the YAML declares language: Japanese. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The OoT display-list exporter's ExportMoveMem was ported from Torch's generic DisplayListFactory path, which resolves a G_MOVEMEM (e.g. a gsSPLight static light-load) to a G_MOVEMEM_OTR_HASH reference or, when the target isn't a declared asset, leaves the raw segmented pointer. OTRExporter has no G_MOVEMEM handler at all: its DisplayListExporter switch falls through to the default (undefined-opcode) case, emitting just the opcode byte with all operands zeroed (w0 = G_MOVEMEM << 24, w1 = 0). For byte-parity with the reference O2R, emit that single 8-byte command in the OoT path. This surfaced on N64 1.0's nintendo_rogo_static/gNintendo64LogoDL, whose logo DL contains two G_MV_LIGHT movemem loads (the revised 1.1 DL has none). The generic DisplayListFactory and PM64 paths are unchanged and keep the hash form. Verified against the zapd-to-torch test harness: ntsc_1-0 now 38390/0/0/0 (was 1 failed); no regressions on pal_gc, ntsc_u_gc, pal_1-0. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#4) Three fixes that let debug ROMs (pal_gc_dbg, pal_mq_dbg) extract without crashing: 1. Decompressor::AutoDecode (uncompressed branch): if the resolved file offset is past the end of the buffer, `buffer.size() - fileOffset` underflows size_t and the function hands back an out-of-bounds pointer. Guard the offset and return an empty segment instead. 2. SceneCommandWriter::WriteSetAlternateHeaders: an uninitialized alt-header pointer with a bogus segment resolves out of bounds. Skip the alternate headers (emit none) in that case, matching OTRExporter, rather than reading from an invalid address. 3. OoTTextFactory: the code segment was always decoded as YAZ0, but some ROMs store it uncompressed. Detect the compression type at code_phys_start and read it directly when uncompressed. Verified against the zapd-to-torch test harness: pal_gc_dbg now completes (35645 passed; remaining diffs are the syotes debug stub scene) and pal_mq_dbg completes as well; no regressions on the retail versions. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
syotes_room_0 is a debug test room whose data at offset 0 is a raw mesh header, not a command list. Parsing it as commands produced a bogus 407 KB room plus spurious actor/cutscene/pathway assets from garbage pointers. ZAPD special-cases this room by name (ZRoom::ParseXML sets the syotes_room hack, ZRoom::SyotesRoomFix emits no commands). Match that: when the room symbol is syotes_room_0, skip command parsing and emit an empty room. Its mesh DLs/textures/vtx are declared separately, so they are unaffected. Verified against the zapd-to-torch test harness: pal_gc_dbg is now fully clean (35646/0/0/0, was 1 failed + 8 extras); pal_mq_dbg's syotes extras are likewise cleared. No effect on retail ROMs (they have no syotes room). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#6) ZAPD zeroes the vertex flag field for display-list-discovered vertices (DisplayListExporter re-parses them from a VTX() text representation that drops the flag and writes 0), but preserves it for XML-declared <Array><Vtx/> arrays (ArrayExporter writes the parsed flag). OoTArrayFactory previously always wrote the ROM flag, so a discovered vertex array whose source bytes carry a non-zero flag (e.g. pal_mq code/z_fbdemo_circle/codeVtx_0FAFA8) diverged from the reference, which zeroes it. Conversely, XML-declared arrays with genuine non-zero flags (e.g. overlays/ovl_file_choose/D_80811F30_NTSC on NTSC ROMs) must keep them. Honor a zero_flag node field: write 0 for the flag when set, else the parsed value. zapd_to_torch sets it on supplemental-injected VTX arrays (the discovered ones) and leaves XML-declared arrays untagged, matching ZAPD's split exactly. Verified: pal_mq now 35352/0/0/0; ntsc_u_gc stays 39066/0/0/0 with D_80811F30's real flags preserved. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
ExportVtx previously nulled ANY cross-file VTX reference (directory mismatch between the DL and the target vtx). But OTRExporter only nulls cross-file references to display-list-discovered vertices (its VTX() text path emits no cross-file-resolvable symbol); it resolves cross-file references to XML-declared vertices via the owning file's declaration. Flip the default to resolve, and null only when the target vtx node carries the null_cross_file directive. zapd_to_torch sets it on supplemental (discovered) VTX arrays alongside zero_flag, and leaves XML-declared arrays untagged. This fixes the debug ROMs' object_fish/object_zo ripple DLs, which reference a gameplay_keep vtx cross-file: on debug that vtx is XML-declared (gGameplayKeepVtx_01A120) -> resolve; on retail it is supplemental (gameplay_keepVtx_01A120, null_cross_file) -> null. Both match the reference. Verified: pal_mq_dbg now 35612/0/0/0; all other versions unchanged. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
DisplayListFactory.cpp is a shared factory compiled regardless of
BUILD_OOT, but it referenced OoT::DListHelpers unconditionally, so a
BUILD_OOT=OFF build failed to compile ('OoT' has not been declared at
the SearchVtx / Export / parse call sites).
Guard the OoTDListHelpers include and the three helper call sites with
#ifdef OOT_SUPPORT, matching how Companion.cpp and DisplayListFactory.h
already gate their OoT includes and factory registrations. When
OOT_SUPPORT is defined the behavior is unchanged; when it's off the
factory falls through to its generic path.
Verified: BUILD_OOT=OFF now configures, compiles, and links; BUILD_OOT=ON
still produces the PAL GC O2R byte-for-byte (35386 passed, 0 failed).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Every source file under src/factories/oot/ wrapped its entire contents
in #ifdef OOT_SUPPORT ... #endif. That guard is redundant: CMakeLists
already excludes the whole src/factories/oot/* directory from the build
when BUILD_OOT is off (list(FILTER SRC_DIR EXCLUDE REGEX .../oot/*)), so
these translation units never compile without OOT_SUPPORT anyway, and
every include of an OoT header from a shared file (Companion.cpp,
DisplayListFactory.{h,cpp}) is itself already guarded.
Remove the file-level guard from all 45 OoT factory sources/headers.
Headers keep their #pragma once.
Verified: BUILD_OOT=ON still produces the PAL GC O2R byte-for-byte
(35386 passed, 0 failed); BUILD_OOT=OFF still configures, compiles, and
links (the oot dir remains excluded).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The AddAsset "undeclared asset" throw was hardcoded to fire only when gbi.subversion == OoT. That behavior (every asset must be pre-declared in the YAML; error instead of dynamically adding on a gap) is a project preference, not something intrinsic to OoT — other projects may want it too, and an OoT project mid-enrichment may not. Add a `strict_declarations` config setting (TorchConfig::strictDeclarations, parsed from the per-ROM config block, default false) and gate the throw on it instead of the GBI subversion. Projects opt in from config. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 22, 2026
Merged
The ROM CRC is read at header offset 0x10 with a big-endian reader, so ReadUInt32 already yields the correct native value; that value is written into the version file. Ports/decomps that were built against Torch's earlier output compared against the byte-swapped CRC, so flipping it can break their version checks. Add a ROM_CRC_BSWAP CMake option (default OFF) gating a BSWAP32 on the CRC, so those consumers can opt back into the legacy byte order without a source change. Default (OFF) keeps the corrected order. Verified via the test harness (PAL GC): default build is unchanged (35386 passed, 0 failed); a -DROM_CRC_BSWAP=ON build differs in exactly one asset — the version file. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Brings in upstream's 31 commits since the clang-format merge-base (75f6416): the parse/export phase split, BK64 asset extraction, the BUILD_UI interactive viewer, compressed/BKZIP segment handling, the GetNodesByType -> GetNodesByTypeRef API, and assorted audio/PM64 fixes. Conflicts (6 files) resolved as follows: - Companion.cpp (parse/export split): upstream independently converged on our per-file flow (segment parse -> gAddrMap population -> reset -> ParseCurrentFileConfig) and added the phase split. Dropped our now- redundant PreparseConfig/PopulateAddrMap/ResetTemporalState helpers and the duplicate ProcessFile, adopting upstream's ProcessParseFile / ProcessExportFile / ProcessFile. Re-added two OoT bits upstream's orchestrator lacked: gCurrentDirectory (default + :config directory override) set before the gAddrMap loop, and per-file gManualSegments clearing. - Companion.cpp (ProcessExportFile): took upstream's mShouldProcess/try + no_export structure; preserved AliasManager::WriteAliases. - Companion.cpp (GetNodeByAddr): kept our ResolveVirtualAddr-based OoT overlay resolution; added upstream's GetCompressedSegmentOffset (BKZIP) fallback (a no-op for OoT). - Companion.cpp (ResolveVirtualAddr / PatchVirtualAddr): kept ours; pulled upstream's GetFileOffsetFromCompressedSegmentedAddr out as a standalone. - Decompressor.cpp: kept YAZ0 in the decode group; took upstream's BKZIP case and upstream's out-of-bounds clamp guard. - DisplayListFactory.cpp: kept OoT SearchVtx; adopted GetNodesByTypeRef. - ResourceType.h / Companion.h / CMakeLists.txt: additive keep-both. - Removed a duplicate RegisterFactory declaration (kept upstream's private one; only called within Companion.cpp). Verified byte-for-byte against the reference O2Rs: all 14 OoT ROM targets pass (N passed, 0 failed, 0 not generated, 0 not in reference), including the debug ROMs (OOB guard) and :config-directory scenes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
briaguya0
marked this pull request as ready for review
July 23, 2026 01:41
KiritoDv
approved these changes
Jul 24, 2026
This was referenced Jul 25, 2026
serprex
pushed a commit
to HarbourMasters/Shipwright
that referenced
this pull request
Aug 1, 2026
* Add Torch migration plan Plan for replacing ZAPDTR + OTRExporter with Torch as SoH's only asset pipeline, following HarbourMasters/Torch#219 ("oot support"). Co-Authored-By: Claude <noreply@anthropic.com> * Plan: yml repo is source of truth, drop the move-to-assets-repo steps - Name the actual repo (briaguya0/soh-asset-yml); it holds yml only, and is edited directly rather than regenerated. zapd_to_torch.py is a one-shot conversion, not an ongoing workflow, since nobody edits the ZAPD XMLs once ZAPD is gone. - Phase 5 is pure deletion. Generator inputs are already preserved by the harness's shipwright@95d8f7e submodule pin and by git history, so nothing needs snapshotting or moving out of Shipwright. - Clarify soh/CMakeLists.txt:107-109 as a configure-soh-directly fallback with nothing left to fall back to. - Downgrade the extraction wall-clock risk; perf work landed in Torch#219. Co-Authored-By: Claude <noreply@anthropic.com> * Phase 0: flesh out the verification gates Adds torch-migration/PHASE0.md with the full gate detail: what each gate's variable is, the driver source, exact commands, cost, and exit criteria. Two gates the plan was missing: - Gate C (Release build) - the harness 14/14 is a Debug measurement; CI and releases build -O3. - Gate A2 (two extractions in one process) - free once the driver exists, retires the re-entrancy risk offline instead of by hand in the game. Also: gates run full-tree + check.sh rather than test_assets.py (which hardcodes the torch path and copies a filtered yml subset), 19 ROM dumps not 17, and libgfxd being USE_STANDALONE-only is called out as the real hazard in Gate A. * Phase 0: correct the test_assets.py claim The "filtered subset" was a misread. The log line prints the pre-recursion yml count (1320); setup_scratch's external_files recursion then pulls in the remaining 130, so an unfiltered run copies all 1450 - verified by replaying the logic. So the gates use test_assets.py, the same tool that produced the existing 14/14, with a TORCH_BIN env override as the only harness patch. Gate A2 keeps the check.sh path, since it needs two extractions in one process and test_assets.py execs the binary once per ROM. * Phase 0: preflight green, 19/19 Baseline Torch build (USE_STANDALONE=ON, all games, Debug, PORT_VERSION_ ENDIANNESS=ON) reproduces every reference archive: 19 ROM dumps, all 0 failed / 0 not generated / 0 not in reference. Counts match the harness README's 14-target table, and the duplicate dumps agree with each other, which the 14-target matrix never showed. Reference manifests also spot-checked by re-derivation from two o2r files -- the baseline the gates measure against hasn't drifted. * Phase 0: Gate A green, 19/19 USE_STANDALONE=OFF plus driving Companion by hand produces byte-identical archives for all 19 ROM dumps. The real question was libgfxd, which is fetched only if(USE_STANDALONE) -- _deps/ confirms it is absent, so the Binary export path is genuinely gfxd-free. Reading said so; now it's run. Progress denominator confirmed exact: SetPhaseCallback fires once per yml file, matching the on-disk count for all three classes (1449/1450/1480). Phase 3's bar can be monotonic with no change to OTRGlobals. Also observed for Phase 2: zlib is fetched even with BUILD_STORMLIB=OFF and no zlib.h use in torch/src, confirming risk #1; spdlog is not fetched, so only the global-logger stomping applies there. * Phase 0: Gate A2 green, and the duplicate-entry finding Two extractions in one process, vanilla then MQ, both byte-identical to their references. The second run's phases=1450 matches pal_mq's on-disk yml count exactly, which is the direct refutation of the gProcessedFiles risk -- a leak would have skipped files and truncated the archive. A fresh Companion per extraction is sufficient; PLAN.md risk #6 retired. Also records what the gate turned up on the way: torch writes 25 duplicate archive entries, benign because libultraship indexes by CRC64 of the path. Filed as Torch#233 item 5. Corrects the Gate A note that attributed the whole ~343 KB size delta to compression -- 76 KB of it is these duplicates. * Phase 0: Gates B and C green OoT-only (eight BUILD_<game> flags OFF) and Release (-O3) each 19/19 identical, one variable moved apiece -- verified against the option() defaults so the other flags genuinely match the baseline cache. Gate C mattered: every parity measurement to date, including the original 14/14, was Debug, while CI and releases build -O3. Also records extraction wall-clock from torch's own timing line: Release averages 11.4s per ROM against Debug's 26.7s, and the OoT-only flags cost nothing. SoH ships Release, so PLAN.md risk #5 looks much smaller than the Debug numbers suggested -- though ZAPD's time on the same machine is still unmeasured. * Phase 0: Gate A' green -- the shipping configuration is the spec Static lib + OoT-only + Release together: 19/19 identical plus the vanilla-then-MQ pair. Flags verified from the generated cache rather than assumed, so Phase 2's CMake block and TorchExtract.cpp can be transcriptions of this build and RunOnce respectively. Archive bytes turn out to be configuration-independent -- the pair produced exactly the sizes the Debug static-lib run did, phases=1450 on both runs. Extraction 11.8s mean, so the static-lib wrapper costs nothing over Gate C. Confirms both Phase 2 dependency risks in the configuration that actually ships: zlib fetched with BUILD_STORMLIB=OFF and no zlib.h includes, tinyxml2 fetched with BUILD_NAUDIO=OFF removing its only consumers. Both FetchContent declarations are unconditional. * Copy soh.o2r from a checked-in prebuilt archive (TEMPORARY) Unblocks the rest of the migration. soh.o2r is produced by ZAPD via OTRExporter/extract_assets.py, both of which this migration deletes; its replacement is a small in-tree packer that has to reproduce ZAPD's texture quantisation exactly. Rather than block Phases 2/3/5 on that packer, check the archive in and make GenerateSohOtr a copy. The archive doubles as Phase 4's acceptance oracle -- the packer is correct when its output matches this entry-for-entry -- which is strictly more useful than Gate E's planned manifest of hashes of it. Generation turns out to be content-deterministic but NOT byte- deterministic: two runs give identical payloads for all 1,042 entries but different file hashes, because zip stores per-entry timestamps. So the Phase 4 comparison must be entry-wise; whole-file would fail even on a correct archive. Recorded in prebuilt/README.md. Adds a configure-time guard on the baked-in portVersion. OTRGlobals.cpp :283 requires exact major.minor.patch equality and RunExtract exit(1)s with "soh.o2r is outdated", which is an opaque symptom for a stale checked-in file; verified the guard fires by building against 9.2.4. ExtractAssets, CI and copy-existing-otrs.cmake are deliberately untouched -- ExtractAssets still generates a correct archive while ZAPD exists, and the CI job keeps working since it just builds this target. * Extract ROMs with Torch instead of ZAPD Adds soh/assets/yml as a submodule (briaguya0/soh-asset-yml, 20,353 asset definitions) and FetchContents torch at 4cae4416, configured exactly as Gate A' measured it: static lib, OoT only, no UI, no StormLib. find_package(ZLIB) has to run before torch is declared. Torch declares zlib with OVERRIDE_FIND_PACKAGE, so it takes over the whole build's find_package(ZLIB), and its copy provides no ZLIB::ZLIB -- which both StormLib and CMake's FindPNG link by name. Filed upstream as Torch#233. CallZapd becomes CallTorch: no chdir, no symlinked assets dir, no 22-entry argv. Everything that needed the working directory is now a parameter. TorchExtract.cpp is the only TU that includes Companion.h, since torch exports its whole lib/ as PUBLIC includes. Progress uses torch's phase callback, which fires once per yml file, with the denominator counted off disk. Gate A confirmed the two match exactly for all three yml counts, so the bar runs 0-100 without the sawtooth the old per-file counter would have produced. GetZapdVerStr becomes GetTorchVersionDir, returning the version directory under the yml tree rather than a ZAPD xml directory. soh.elf links torch and no longer references zapd_report. * Record the two-ROM session passing in-game Vanilla then MQ extracted in one process, both loaded and played. Closes PLAN.md risk #6 in the game rather than only against the harness driver, and confirms config.yml-driven archive naming end to end. * Add soh-torch, rebuild ExtractAssets on it soh links torch as a static library, which compiles out torch's own CLI, so build-time extraction needs an entry point. soh-torch supplies one around the same SohTorch::Extract the game calls -- one implementation, so the CLI and the in-game path can't drift. ExtractAssets keeps its old contract: SOH_ROM_PATH accepts roms and/or directories of roms and defaults to roms/, so dropping a vanilla and a master quest rom in produces oot.o2r and oot-mq.o2r in one run, and chaining GenerateSohOtr keeps soh.o2r coming out of it too. Extract now returns the archive name torch chose rather than deriving it from IsMasterQuest(), so config.yml is the only thing naming archives. The name comes from Companion::GetOutputPath(): scanning the destination for *.o2r picks up an unrelated soh.o2r when extracting into the build directory, which is only invisible in the game because it extracts into an empty temp dir. ExtractAssetHeaders stays as a target that explains why it can't run -- torch registers no OoT header exporter and emits no #define d<sym>, which soh source references ~24k times. Documented in BUILDING.md alongside it. This removes the last ZAPD dependency from the build. * Add the roms directory ExtractAssets defaults to BUILDING.md points people at roms/, but git can't track an empty directory, so a fresh clone didn't have one. Its .gitignore keeps everything but itself and the readme out. * Delete ZAPDTR and OTRExporter Nothing drives them any more: ExtractAssets runs soh-torch, the in-game extractor calls Companion directly, GenerateSohOtr copies a prebuilt archive, and soh links torch instead of ZAPDLib. Verified with a build from a fresh tree with both submodules absent from disk. Removes the submodules, the 7,680 ZAPD xmls under soh/assets/xml, the extractor inputs (Config_*.xml, TexturePool.xml, filelists, symbols) and copy-existing-otrs.cmake. The xmls remain in this repo's history and in the test harness's pinned shipwright submodule, which is where the yaml was generated from. Packaging changes shape as well as content: the extractor assets used to install into the 'extractor' component, which Packaging-2.cmake only includes for the AppImage generator, so linux zips shipped without them. torch is linked into soh rather than being a separate binary, so the yml tree installs into 'ship' on every platform and every package can extract a rom. * Keep CI providing tinyxml2 itself Torch's fetched copy satisfies libultraship today, so the install steps are redundant -- but that rests on torch declaring tinyxml2 with an unconditional OVERRIDE_FIND_PACKAGE, which is an implementation detail, and one we've asked upstream to reconsider in Torch#233. If the fetch gets gated, find_package needs a system copy again and CI would break because we removed the steps providing it. Also notes that the generate-soh-otr slim-down was written before the prebuilt archive, which makes that job build nothing for now. * Generate soh.o2r again, replacing the prebuilt archive soh-o2r-packer builds soh.o2r from soh/assets/custom, reproducing the ZAPD/OTRExporter archive exactly: 1,042 entries, 0 missing, 0 extra, 0 content mismatch, checked entry-wise because zip stores per-entry timestamps and whole-file hashes never match between runs. Most of it is torch's: Companion::Pack walks the directory, zips it and writes portVersion; BaseExporter::WriteHeader writes the resource header; TextureType and CalculateTextureSize come from TextureUtils. The packer stages the assets into the shape the archive should have and hands that over. Encoding a PNG into an N64 texture is the only piece nothing else provides -- torch decodes rom data that is already N64 format, never the reverse -- so PngTexture.cpp is the whole of what had to be written, and it follows ZAPD's quantisation rather than n64graphics', which scales where ZAPD shifts. GenerateSohOtr copies the libultraship shaders into assets/custom again before packing; assets/custom/shaders is gitignored and nothing else populates it, so a fresh clone would otherwise pack three files short. Removes prebuilt/ and its configure-time portVersion guard. * Move the asset tools under soh/assets soh-torch and soh-o2r-packer both exist to turn assets into archives, so they sit better beside the assets they read than at the repo root. Nothing tangled: both targets are declared in the root CMakeLists with explicit paths rather than add_subdirectory, and soh/CMakeLists.txt globs only include/, soh/ and src/, so sources under assets/ aren't swept into the soh target. * Build soh.o2r in CI without libultraship or soh SOH_TOOLS_ONLY returns from the root CMakeLists once the asset tools are declared, before libultraship and soh are added, so a configure that only needs soh-o2r-packer never reaches LUS's find_package(SDL2 REQUIRED). That lets generate-soh-otr drop building SDL2 from source, SDL2_net, and the deps cache that existed to hold the SDL2 tarball -- eleven steps down to six. Verified a tools-only build produces the archive with all 1,042 entries matching. The tool targets move above the sub-projects so the early return can sit between them; they only ever needed torch. The packer creates its output directory, which previously came for free from soh's build directory. * Fix macos and windows builds, keep the tools out of the game build macos: libultraship declares spdlog with OVERRIDE_FIND_PACKAGE so its own find_package(spdlog REQUIRED) resolves, but torch declares spdlog without it, and FetchContent_Declare is first-wins. Configuring torch first made LUS's declaration a no-op and left find_package with no config to find. It only showed on macos because that's the platform where neither project finds an installed spdlog -- it's kept out of macports.yml because its fmt dependency breaks the universal build -- so both fall through to fetching. Declaring it ahead of both keeps macos on the same source-built v1.16.0 it used before. Torch already uses OVERRIDE_FIND_PACKAGE for tinyxml2 and zlib, so the real fix is upstream. windows: soh builds with /WX, and TorchExtract.cpp is the one translation unit reaching yaml-cpp through torch's headers, which trips the dll-interface warnings. Suppress those two on that file. EXCLUDE_FROM_ALL on soh-torch and soh-o2r-packer: they are build-time tools, so building the game shouldn't compile them. ExtractAssets and GenerateSohOtr still pull them in through DEPENDS. Drops the .gitattributes rule for the prebuilt archive, which is gone. * Drop the gitignore negation for the prebuilt archive Left over with the .gitattributes rule when prebuilt/ was removed. * TEMPORARY: put soh/assets/xml back so the branch diff is reviewable Deleting 7,680 xmls puts the diff past what GitHub will render, and they are 99.8% of it. Restoring them here leaves the ~46 files that actually changed, which reviews in the compare view. Revert this before opening the PR. * Remove the migration planning docs Scaffolding for the migration itself rather than anything the repo needs to carry afterwards. * Fix NTSC master quest roms being rejected as unsupported verMap's two master quest rows keyed off the GameCube constants instead of the MQ ones, so with duplicate keys dropped the map never contained OOT_NTSC_US_MQ or OOT_NTSC_JP_MQ and both roms were filtered out before extraction. IsMasterQuest already handles both, and the asset yml covers ntsc_u_mq and ntsc_j_mq, so keying them correctly is all that's needed. Predates this migration; the torch cli was unaffected because it hashes the rom rather than consulting verMap. * Revert "TEMPORARY: put soh/assets/xml back so the branch diff is reviewable" This reverts commit 28f4113. * Run clang-format Renaming CallZapd to CallTorch made the call sites one character longer and left their continuation lines under-indented. Co-Authored-By: Claude <noreply@anthropic.com> * Temporarily pin torch to the fork carrying the Windows fixes HarbourMasters/Torch#234 fixes three Windows-only defects in the OoT factories: a 64 KiB over-read past the code segment that crashes extraction, backslashes in exported scene resource names that crash at scene load, and external_files being keyed differently from the directory walk, which processed every dependency twice. Without them a Windows build of this branch cannot extract a rom. Goes back to a HarbourMasters sha once that PR merges. Co-Authored-By: Claude <noreply@anthropic.com> * Point torch back at upstream HarbourMasters/Torch#234 landed as 65eb11c, squashing the three Windows fixes this branch was pinned to a fork for. The squashed tree is identical to the fork commit that was tested, and extraction output is unchanged: 35,386 entries, same CRCs. Co-Authored-By: Claude <noreply@anthropic.com> * Vendor the asset yml instead of submoduling it The submodule existed only so this PR's diff stayed reviewable; squash-merging it would have put a submodule on develop. This replaces it with the 20,353 files themselves, from briaguya0/soh-asset-yml@523be1d. The submodule's own README is dropped rather than vendored, since it describes that repo and was showing up in the assets folder of release downloads. Co-Authored-By: Claude <noreply@anthropic.com> * Bump torch to pick up the yaml-cpp 0.9.0 bump HarbourMasters/Torch#235 landed as e92c210. The pinned yaml-cpp predated jbeder/yaml-cpp@4fe2fb8, so every scalar conversion constructed a named std::locale; cheap on glibc, expensive on MSVC. Windows extraction of a PAL GC rom drops from 49.0s to 13.9s on the same machine, which also puts it ahead of ZAPDTR's 48.4s rather than fractionally behind. The merged commit is tree-identical to the fork commit those numbers were measured on. Extraction output is unchanged: 35,386 entries, same CRCs. Co-Authored-By: Claude <noreply@anthropic.com> * Consume torch as a submodule instead of FetchContent Reviewer feedback on #6989: FetchContent makes iterating on torch itself awkward, since testing a change means pushing it somewhere the pin can reach. A submodule is editable in place, which is how libultraship already works. Nothing else changes -- FetchContent_MakeAvailable was already doing add_subdirectory, so the include topology and every build option are the same. The pin moves from GIT_TAG into the gitlink, at the same sha. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
louist103
pushed a commit
to louist103/Shipwright-1
that referenced
this pull request
Aug 7, 2026
Plan for replacing ZAPDTR + OTRExporter with Torch as SoH's only asset pipeline, following HarbourMasters/Torch#219 ("oot support"). Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
i've built a dedicated test harness that checks this branch against a Shipwright/OTRExporter reference O2R, file by file:
https://github.com/briaguya0/zapd-to-torch-test-harness
it converts Shipwright XML → Torch YAML, runs this branch's
torchto produce an O2R, and compares every asset's sha256 against a manifest generated from the OTRExporter reference (Torch and Shipwright are both vendored as submodules so it's reproducible independent of this repo). "clean" = byte-for-byte identical to the reference: 0 failed, 0 not generated, 0 extra.current status: all 14 OoT ROM targets clean — every retail/debug/MQ variant across N64 1.0/1.1/1.2 and GC (PAL / NTSC-U / NTSC-J / MQ / debug) extracts byte-for-byte identical to the reference. full table: https://github.com/briaguya0/zapd-to-torch-test-harness#status
follow-up issues to create once this lands:
from oot support #219 (comment)
from oot support #219 (comment)
move from OOT ARRAY of type VTX to standard shared VTX format
from oot support #219 (comment)
see if we need the alias system or if having different bhash values for the same data in different paths is something we can work with instead
from oot support #219 (comment)
see what it'd take to fully drop the swap the CRC logic (what ports/decomps currently rely on the swapping)