diff --git a/Crusader.rep/idata/00/~0000000e.db/db.3.gbf b/Crusader.rep/idata/00/~0000000e.db/db.5.gbf similarity index 99% rename from Crusader.rep/idata/00/~0000000e.db/db.3.gbf rename to Crusader.rep/idata/00/~0000000e.db/db.5.gbf index de97c90..d0160ae 100644 Binary files a/Crusader.rep/idata/00/~0000000e.db/db.3.gbf and b/Crusader.rep/idata/00/~0000000e.db/db.5.gbf differ diff --git a/Crusader.rep/idata/01/~00000010.db/db.1.gbf b/Crusader.rep/idata/01/~00000010.db/db.2.gbf similarity index 78% rename from Crusader.rep/idata/01/~00000010.db/db.1.gbf rename to Crusader.rep/idata/01/~00000010.db/db.2.gbf index f437992..b2e845c 100644 Binary files a/Crusader.rep/idata/01/~00000010.db/db.1.gbf and b/Crusader.rep/idata/01/~00000010.db/db.2.gbf differ diff --git a/Crusader.rep/idata/01/~00000010.db/db.3.gbf b/Crusader.rep/idata/01/~00000010.db/db.3.gbf new file mode 100644 index 0000000..1ab1333 Binary files /dev/null and b/Crusader.rep/idata/01/~00000010.db/db.3.gbf differ diff --git a/Crusader.rep/projectState b/Crusader.rep/projectState index 6244408..f55e322 100644 --- a/Crusader.rep/projectState +++ b/Crusader.rep/projectState @@ -3,7 +3,11 @@ + + + + diff --git a/Crusader.rep/user/00/0000000f.prp b/Crusader.rep/user/00/0000000f.prp new file mode 100644 index 0000000..b8bc179 --- /dev/null +++ b/Crusader.rep/user/00/0000000f.prp @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/Crusader.rep/user/00/~00000008.db/db.27.gbf b/Crusader.rep/user/00/~00000008.db/db.28.gbf similarity index 99% rename from Crusader.rep/user/00/~00000008.db/db.27.gbf rename to Crusader.rep/user/00/~00000008.db/db.28.gbf index b8bb439..00f7e2e 100644 Binary files a/Crusader.rep/user/00/~00000008.db/db.27.gbf and b/Crusader.rep/user/00/~00000008.db/db.28.gbf differ diff --git a/Crusader.rep/user/00/~00000008.db/db.26.gbf b/Crusader.rep/user/00/~00000008.db/db.29.gbf similarity index 99% rename from Crusader.rep/user/00/~00000008.db/db.26.gbf rename to Crusader.rep/user/00/~00000008.db/db.29.gbf index 0883758..8e9070a 100644 Binary files a/Crusader.rep/user/00/~00000008.db/db.26.gbf and b/Crusader.rep/user/00/~00000008.db/db.29.gbf differ diff --git a/Crusader.rep/user/00/~0000000c.db/db.1.gbf b/Crusader.rep/user/00/~0000000c.db/db.4.gbf similarity index 99% rename from Crusader.rep/user/00/~0000000c.db/db.1.gbf rename to Crusader.rep/user/00/~0000000c.db/db.4.gbf index ee48367..942fe65 100644 Binary files a/Crusader.rep/user/00/~0000000c.db/db.1.gbf and b/Crusader.rep/user/00/~0000000c.db/db.4.gbf differ diff --git a/Crusader.rep/user/00/~0000000c.db/db.5.gbf b/Crusader.rep/user/00/~0000000c.db/db.5.gbf new file mode 100644 index 0000000..20ee39a Binary files /dev/null and b/Crusader.rep/user/00/~0000000c.db/db.5.gbf differ diff --git a/Crusader.rep/user/00/~0000000f.db/db.1.gbf b/Crusader.rep/user/00/~0000000f.db/db.1.gbf new file mode 100644 index 0000000..a4637ca Binary files /dev/null and b/Crusader.rep/user/00/~0000000f.db/db.1.gbf differ diff --git a/Crusader.rep/user/00/~0000000f.db/db.2.gbf b/Crusader.rep/user/00/~0000000f.db/db.2.gbf new file mode 100644 index 0000000..f65eaf3 Binary files /dev/null and b/Crusader.rep/user/00/~0000000f.db/db.2.gbf differ diff --git a/Crusader.rep/user/~index.dat b/Crusader.rep/user/~index.dat index 35d5aa2..a78168c 100644 --- a/Crusader.rep/user/~index.dat +++ b/Crusader.rep/user/~index.dat @@ -2,6 +2,7 @@ VERSION=1 / 0000000a:udf_c0a86451c281202637836837200:c0a86451c652220429919955700 0000000c:udf_c0a86451c285202638031072100:c0a86451fcd2547353220145000 + 0000000f:udf_c0a86451c287202638146478500:c0a86451fe5683084012760000 00000008:udf_c0a86451c28c202638381579400:c0a86451f608205075819887000 0000000b:udf_c0a86451c28e202638509414500:ac18b01ab332438409229485800 0000000d:udf_c0a86451c52a19721794868600:c0a86451c52b19721868788800 @@ -15,5 +16,5 @@ VERSION=1 00000000:udf_c0a8647bf0178892741854800:c0a8647bd36236342207469100 00000001:udf_c0a8647bf4b212984786819600:c0a8647bd36336342224113900 00000003:udf_c0a8647bfe7615910786193500:c0a8647bd36536342248279100 -NEXT-ID:f +NEXT-ID:10 MD5:d41d8cd98f00b204e9800998ecf8427e diff --git a/docs/usecode-roundtrip-ir.md b/docs/usecode-roundtrip-ir.md index f72b5a3..4a08db3 100644 --- a/docs/usecode-roundtrip-ir.md +++ b/docs/usecode-roundtrip-ir.md @@ -805,4 +805,12 @@ That gets to a reversible editor sooner than waiting for a full semantic VM reco - **Next Steps:**: (1) Implement compare-direction fix in the expression builder and add small semantic regression tests, (2) re-run unit tests and a corpus-wide render+validate sweep, (3) regenerate affected pseudocode files for inspection. - **Files of Interest:**: [tools/poc_crusader_usecode_parser.py](tools/poc_crusader_usecode_parser.py), [tools/tests/test_usecode_structuring.py](tools/tests/test_usecode_structuring.py), [USECODE/EUSECODE_extracted/pseudocode/BART/slot_0F_enterFastArea.txt](USECODE/EUSECODE_extracted/pseudocode/BART/slot_0F_enterFastArea.txt). +## **Recent Renderer Work (2026-03-31)** + +- **Opcode Status:**: The map renderer was already loading the recovered JP opcode table from [usecode_opcodes.txt](k:/ghidra/crusader-disasm/usecode_opcodes.txt); no additional opcode-name integration was required in this pass. +- **VM Semantics Fix:**: The JS renderer in [src/lib/usecode-decompiler.js](k:/ghidra/crusader_map_viewer/map_renderer/src/lib/usecode-decompiler.js) now follows the Pentagram/ScummVM VM for two core cases: opcode `0x24 cmp` is equality, not inequality, and opcode `0x51 IF` is a relative branch on false, not on true. +- **Readability Impact:**: False branches are now emitted with the negated high-level condition, so the existing structurer can recover counted loops as `while (counter <= limit)` instead of the previously inverted `while (counter > limit)` pattern. +- **Regression Coverage:**: Added a focused renderer-side regression script at [scripts/test-usecode-structuring.mjs](k:/ghidra/crusader_map_viewer/map_renderer/scripts/test-usecode-structuring.mjs) to guard one equality-based selector case and one counted-loop case. +- **Next Steps:**: Rebuild a fresh renderer usecode cache and inspect representative families like `BART`, `_BOOT`, and `EVENT` for any remaining cases where other compare producers still leak VM-oriented phrasing. + If you want, I can (a) implement the comparison/operand polarity fix next, (b) run the unit tests and a fresh corpus sweep, and (c) open a PR-ready commit with these doc and code updates. \ No newline at end of file diff --git a/docs/versions/family-export-differences.md b/docs/versions/family-export-differences.md new file mode 100644 index 0000000..e09bf83 --- /dev/null +++ b/docs/versions/family-export-differences.md @@ -0,0 +1,72 @@ +# Family Export Differences + +This note summarizes the first run of the renderer-side family comparison script and points to the generated artifacts for deeper inspection. + +## How to Regenerate + +- Run `npm run compare-family-export-data` from `k:\ghidra\crusader_map_viewer\map_renderer` +- Generated artifacts: + - `k:\ghidra\crusader_map_viewer\map_renderer\generated\version-differences\family-export-differences.json` + - `k:\ghidra\crusader_map_viewer\map_renderer\generated\version-differences\family-export-differences.md` + +## Normalization Rules + +- The script compares the same `scene.json` payload that `export-static` emits for each map. +- It strips build metadata before hashing: + - `build.*` + - `metadata.game` + - `metadata.gameLabel` + - `metadata.usage.note` + - `metadata.usage.tableAddress` +- Result: matching hashes mean the exported scene payload is content-identical after removing packaging-only metadata. Differing hashes mean something in the exported map payload changed, even if the high-level item counts stayed the same. + +## No Remorse Family + +- Map presence: + - retail, `1.01`, and JP all expose the same `63` detected maps + - demo exposes only map `1` +- Shared-map headline: + - no shared Remorse-family map hashed identically across compared versions in this first run + - that does not mean every map has large layout changes; many differences are below the top-line counters +- Retail vs `1.01`: + - this is the clearest structural delta in the family + - many shared maps have small but real count changes in raw fixed rows, rendered items, or sprite totals + - representative examples from the generated report: + - map `1`: raw `7547 -> 7530`, rendered `17584 -> 17567`, sprites `790 -> 791` + - map `15`: raw `7473 -> 7486`, rendered `13667 -> 13680`, sprites `686 -> 687` + - map `24`: raw `2296 -> 2285`, rendered `5028 -> 5020`, sprites `517 -> 520` +- Retail vs JP: + - all `63` shared maps differ at the normalized payload level + - the top-line counts stayed the same in the summary lines sampled from the generated report, which suggests the interesting deltas are likely deeper in the serialized scene payload rather than simple map-size changes + - that makes JP a good candidate for focused scene-structure or sprite-definition diffing rather than another map-count pass +- Retail vs demo: + - only map `1` is shared + - map `1` differs in normalized payload, but the headline counts remained the same in the first-pass summary (`raw 7547`, `render 17584`, `sprites 790` on both sides) + - that again points to deeper scene-serialization or resource differences instead of a broad layout mismatch +- `1.01` vs JP: + - the maps with visible count deltas line up with the retail-vs-`1.01` changes, which is consistent with JP often tracking retail more closely than `1.01` does for top-line counts + +## No Regret Family + +- Map presence: + - retail exposes `38` detected maps + - demo exposes only maps `1` and `2` +- Shared-map headline: + - maps `1` and `2` both differ between retail and demo +- Concrete count deltas from the generated report: + - map `1`: raw `5073 -> 5104`, rendered `8809 -> 8840`, sprites `629 -> 639` + - map `2`: raw `3078 -> 3093`, rendered `5544 -> 5559`, sprites `523 -> 531` +- Category shift pattern: + - both shared demo maps gained fixed-source rows and editor records relative to retail + - map `1` also shifts the base/editor/terrain mix slightly, which makes it a strong candidate for targeted scene-item diff tooling later + +## Practical Hunting Order + +- First: Regret retail vs demo maps `1` and `2`, because the delta set is small and the count changes are obvious +- Second: Remorse retail vs `1.01` on maps with visible count deltas such as `1`, `15`, `24`, `29`, `40`, and `41` +- Third: Remorse retail vs JP on maps where counts stay identical, because those likely hide more subtle structural or resource-order differences + +## Notes + +- The Regret demo export path now depends on per-entry `SHAPES.FLX` fallback to retail Regret art for sparse demo archives. That fallback is intentional and currently required for successful export. +- This report is meant to narrow the search space. When a pair/map looks interesting, the next pass should diff the specific exported `scene.json` entries or inspect the underlying `FIXED.DAT` rows and shape references directly. \ No newline at end of file diff --git a/docs/versions/version-differences.md b/docs/versions/version-differences.md new file mode 100644 index 0000000..7bbacb8 --- /dev/null +++ b/docs/versions/version-differences.md @@ -0,0 +1,77 @@ +# Version Export Differences + +## 2026-03-31 exporter check + +- Renderer config correction: the retail No Remorse and No Regret static folders now both use `EUSECODE.FLX`, not `USECODE.FLX`, and the map renderer config has been updated to match the files now present in `STATIC/` and `STATIC_REGRET/`. +- Mission-table correction: Remorse 1.01 also needed a version-specific mission-table override. Its executable does contain the same 17-entry base-map sequence as retail, but at file offset `0x0e16c6` rather than the retail location. + +### No Remorse JP + +- Status: supported in the map renderer. JP scene export works, and JP mission metadata now generates correctly from a verified live Ghidra table read. +- Verified commands: + - `node src/build-cache.js remorse-jp` + - `node src/export-static.js --game=remorse-jp --output=_tmp_export_remorse_jp` +- Implemented renderer support: + - JP mission table is now supplied from a verified Ghidra live-table read at `0x0047b72c` + - extracted base-map sequence: `0, 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 40` + - consumer path verified in `FUN_00428e00`, with the key table-read site at `0x00429056` +- Current renderer/export fix: if `STATIC_JP/GAMEPAL.PAL` is missing, the exporter now falls back to `STATIC/GAMEPAL.PAL`. +- Important limitation: this is only a temporary compatibility fallback. The Japanese asset set does not currently include a visible `GAMEPAL.PAL`, and it still needs investigation whether the correct palette lives in another file such as `PALETTE.DAT`, one of the other `.PAL` assets, or an executable/resource path. +- Asset/executable mismatch note: the Japanese static folder uses `JUSECODE.FLX`, but a Ghidra string pass over `/ja/CRUSADER.EXE` found `eusecode.flx` at `0x0047be90` and found no `jusecode` string at all. The string is consumed by `startup_apply_u_override_if_present`, which looks like a startup override hook for the usecode path rather than a JP-specific rename path. +- Additional Ghidra note: the JP executable string pass found `static\fixed.dat`, `fixed.dat`, `shapes.flx`, `glob.flx`, `fonts.flx`, `eusecode.flx`, and `dtable.flx`, but no explicit `.pal` filename strings. `GAMEPAL` only surfaced as the source path string `".\\gamepal.c"`, so palette filename handling still needs more direct loader tracing. +- Ghidra documentation added: + - comment at `0x00466ebc` for the JP startup usecode override path + - comment at `0x0047b72c` for the verified warp base-map table + - comment at `0x00429056` for the table consumer path that adds `-mapoff` +- Archive comparison note: `FIXDEMO.DAT` is in `STATIC_JP`, not `STATIC_DEMO`. It is byte-for-byte identical to the prefix of JP `FIXED.DAT`, but it is truncated: only the first two non-empty map payloads are readable, and the remaining 61 non-empty table entries run past EOF. That makes it look more like a cut-down/demo fragment than a complete alternate archive. It may still be worth exposing separately later for comparison, but it is not a full parallel map set. + +### No Remorse Demo + +- Status: supported in the map renderer. The detected demo map exports cleanly, and demo mission metadata now generates correctly. +- Verified commands: + - `node src/build-cache.js remorse-demo 1` + - `node src/export-static.js --game=remorse-demo --output=_tmp_export_remorse_demo` +- Implemented renderer support: + - demo mission table now reads from `STATIC_DEMO/CRUSADER.EXE` at file offset `0x0e3a88` + - extraction is fixed-length (`17` entries) instead of sentinel-bounded + - exported demo mission metadata now resolves map `1` as the base map for mission `1` +- Important naming difference: demo usecode is stored as `EUSECODE.FLX` rather than plain `USECODE.FLX`. +- Investigation note: `STATIC_DEMO` also contains `USECODE.ZIP` plus an extracted `USECODEZIP/` folder with `EUSECODE.FLX`, `OVERLOAD.DAT`, `UNKCOFF.DAT`, and `UNKDS.DAT`. This needs a later provenance/content check to determine whether the zip is archival noise, patch material, or the authoritative demo usecode bundle. +- Root cause and fix: the earlier garbage mission extraction came from using the retail Remorse file offset (`0x0e4088`) against the demo executable. The demo binary does still carry the same 17-entry base-map sequence, but at `0x0e3a88`, and it does not expose the immediate retail-style double-zero terminator after those entries. +- Ghidra string/xref pass on `/demo/CRUSADER.EXE`: + - `eusecode.flx` is present at `1478:07aa` and is consumed by `FUN_1420_0cdf`, a 16-bit startup routine with the same broad shape as the JP/retail `-u` override path. It force-writes the first character to `'e'`, resolves the full path, checks file existence, and only then swaps the usecode source. + - `gamepal.pal` is present explicitly at `1478:07d9`, unlike the JP executable. It is referenced from `FUN_1028_0000`, which looks like an early palette/static resource initialization path. + - `fixed.dat` is present at `1478:075f` and is referenced from `FUN_10b8_0616`, which resolves a path under the `static` directory before passing it into a loader helper. + - `nonfixed.dat` is present at `1478:0769`, but this pass did not recover any direct xrefs to the string yet, so it may be unused, late-bound, or referenced through a table path not exposed by the current MCP queries. + - The demo executable also carries `-demo`, `"Loading demo: ["`, and `"Demo mode.\n\r"` strings, which supports the idea that this build has a materially different startup/game-flow path from retail. + - Added Ghidra comments at `1420:0cdf`, `1028:0000`, and `10b8:0616` to preserve those verified findings in the demo database. + +### Other follow-up + +- Demo renderer/export support is now working end-to-end. +- JP renderer/export support is now working end-to-end. +- No Regret demo renderer/export support is now working end-to-end. +- The viewer now preserves map selection more intelligently across versions: equal-count variants within a family try to keep the same map ID, one-map builds auto-select their only map, and switching between the Remorse and Regret families falls back to the last remembered map for the target family. + +### No Regret Demo + +- Status: supported in the map renderer. Both detected demo maps export cleanly. +- Verified commands: + - `node src/export-static.js --game=regret-demo --output=_tmp_export_regret_demo` + - `node scripts/compare-family-export-data.js` +- Verified asset layout: + - `STATIC_REGRET_DEMO/FIXED.DAT` exposes two non-empty maps: `1` and `2` + - `STATIC_REGRET_DEMO/REGRET.EXE` is a tiny `MZ` stub-like launcher (`11,942` bytes) + - `STATIC_REGRET_DEMO/REGRET.DAT` is the real payload-bearing image (`988,372` bytes) and contains the expected `fixed.dat`, `nonfixed.dat`, `eusecode.flx`, `gamepal.pal`, and `-demo` strings +- Mission-table support: + - the renderer now reads the demo base-map table from `REGRET.DAT`, not `REGRET.EXE` + - verified table location: absolute file offset `0x0e295c` + - extraction remains fixed-length at `17` entries, matching the other supported variants +- Export blocker and fix: + - the first Regret demo export failed because `STATIC_REGRET_DEMO/SHAPES.FLX` contains empty entries that retail Regret still has art for; shape `404` was the first hard failure + - the renderer now supports per-entry `SHAPES.FLX` fallback through `fallbackStaticDirs`, so sparse demo shape archives can borrow only the missing retail shapes instead of replacing the whole archive + - build fingerprinting was updated so these layered archive inputs participate in cache invalidation +- Family comparison note: + - retail Regret and Regret demo share only maps `1` and `2` + - both shared maps differ in the normalized exported scene payload; the demo versions have more fixed-source rows, more editor items, and more sprites than retail for those two maps +- Follow-up report: cross-version export diffs now live in `docs/versions/family-export-differences.md`, generated from `map_renderer/generated/version-differences/family-export-differences.{json,md}`.