Map sorting and usecode
This commit is contained in:
parent
589bfc31ef
commit
af5b77ea13
7 changed files with 1497 additions and 39 deletions
221
docs/map-rendering.md
Normal file
221
docs/map-rendering.md
Normal file
|
|
@ -0,0 +1,221 @@
|
|||
# Crusader Map Rendering Workbench
|
||||
|
||||
## Purpose
|
||||
|
||||
This note starts a dedicated lane for offline Crusader map extraction and PNG rendering from the shipped data files in this workspace.
|
||||
|
||||
Current implementation entry point:
|
||||
|
||||
- `tools/render_crusader_map.py`
|
||||
|
||||
Current supported data roots:
|
||||
|
||||
- `STATIC` for No Remorse
|
||||
- `STATIC_REGRET` for No Regret
|
||||
|
||||
Current asset note:
|
||||
|
||||
- `STATIC_REGRET` in this workspace now includes `FIXED.DAT`
|
||||
- the renderer still accepts `--fixed-dat` so alternate map copies can be tested without changing the rest of the static asset path
|
||||
|
||||
The immediate goal is practical and narrow: load a fixed map, expand glob placements, decode the required shapes from `SHAPES.FLX`, apply `GAMEPAL.PAL`, and render a deterministic PNG.
|
||||
|
||||
## Source Cross-Checks Used
|
||||
|
||||
The first renderer is grounded in the overlapping parts of three sources rather than in ad hoc guesses.
|
||||
|
||||
1. Pentagram Crusader shape/map loaders
|
||||
- `convert/crusader/ConvertShapeCrusader.cpp`
|
||||
- `graphics/Shape.cpp`
|
||||
- `graphics/ShapeFrame.cpp`
|
||||
- `world/Map.cpp`
|
||||
- `world/MapGlob.cpp`
|
||||
- `graphics/Palette.cpp`
|
||||
- `graphics/TypeFlags.cpp`
|
||||
|
||||
2. ScummVM Ultima8 Crusader paths
|
||||
- `gfx/shape_archive.cpp`
|
||||
- `gfx/type_flags.cpp`
|
||||
- `world/map.cpp`
|
||||
- `world/glob_egg.cpp`
|
||||
- `world/coord_utils.h`
|
||||
- `world/item_sorter.cpp`
|
||||
- `world/sort_item.cpp`
|
||||
|
||||
3. Local workspace evidence
|
||||
- `docs/scummvm-crusader-reference.md`
|
||||
- `docs/pentagram-crusader-reference.md`
|
||||
- `docs/raw-0007-rendering.md`
|
||||
- `crusader-disasm/shapedata.txt`
|
||||
- `crusader-disasm/mapdump/mapdump.py`
|
||||
|
||||
## File Formats Used By The First Tool
|
||||
|
||||
### `FIXED.DAT`
|
||||
|
||||
The map container is treated as a header plus a map table:
|
||||
|
||||
- map count at file offset `0x54`
|
||||
- map table at file offset `0x80`
|
||||
- each table row is `<u32 offset, u32 size>`
|
||||
|
||||
Each map payload is read as packed 16-byte item records:
|
||||
|
||||
- `x: u16`
|
||||
- `y: u16`
|
||||
- `z: u8`
|
||||
- `shape: u16`
|
||||
- `frame: u8`
|
||||
- `flags: u16`
|
||||
- `quality: u16`
|
||||
- `npc_num: u8`
|
||||
- `map_num: u8`
|
||||
- `next: u16`
|
||||
|
||||
Crusader-specific coordinate adjustment matches the Pentagram and ScummVM runtime loaders:
|
||||
|
||||
- world `x = disk_x * 2`
|
||||
- world `y = disk_y * 2`
|
||||
|
||||
### `GLOB.FLX`
|
||||
|
||||
`GLOB.FLX` is handled as a normal FLEX archive, not as a one-off format.
|
||||
|
||||
Each non-empty glob object contains:
|
||||
|
||||
- object count: `u16`
|
||||
- repeated entries of `x:u8 y:u8 z:u8 shape:u16 frame:u8`
|
||||
|
||||
Glob expansion matches the Crusader `GlobEgg::enterFastArea()` rule in ScummVM/Pentagram:
|
||||
|
||||
- `coordmask = ~0x3ff`
|
||||
- `coordshift = 2`
|
||||
- `offset = 2`
|
||||
- `itemx = (parent_x & coordmask) + (glob_x << 2) + 2`
|
||||
- `itemy = (parent_y & coordmask) + (glob_y << 2) + 2`
|
||||
- `itemz = parent_z + glob_z`
|
||||
|
||||
The first renderer expands glob contents and skips drawing the source glob egg itself.
|
||||
|
||||
### `SHAPES.FLX`
|
||||
|
||||
World shapes use the Crusader shape layout documented by Pentagram/ScummVM:
|
||||
|
||||
- shape header: 6 bytes
|
||||
- 4 bytes unknown
|
||||
- 2-byte frame count
|
||||
- frame header table: 8 bytes per frame
|
||||
- 3-byte frame offset
|
||||
- 1 unknown byte
|
||||
- 4-byte frame length
|
||||
- frame body header: 28 bytes
|
||||
- 8 unknown bytes
|
||||
- 4-byte compression flag
|
||||
- 4-byte width
|
||||
- 4-byte height
|
||||
- 4-byte x offset
|
||||
- 4-byte y offset
|
||||
- then `height` 4-byte line offsets
|
||||
- then per-line RLE data
|
||||
|
||||
The current decoder follows the runtime line walker used in `ShapeFrame::getPixelAtPoint()`:
|
||||
|
||||
- each line is a series of skip/run pairs
|
||||
- compressed runs use the low bit to choose literal versus repeated-color mode
|
||||
- pixels absent from the RLE stream are treated as transparent
|
||||
|
||||
### `GAMEPAL.PAL`
|
||||
|
||||
`GAMEPAL.PAL` is read as 768 bytes of VGA-style palette data.
|
||||
|
||||
Each component is promoted from `0..63` to `0..255` using the same scaling used by Pentagram:
|
||||
|
||||
- `rgb8 = (rgb6 * 255) / 63`
|
||||
|
||||
### `TYPEFLAG.DAT`
|
||||
|
||||
The renderer currently uses Crusader's 9-byte records to extract:
|
||||
|
||||
- family id
|
||||
- shape footpad dimensions (`x`, `y`, `z`)
|
||||
- editor flag
|
||||
|
||||
This is enough for:
|
||||
|
||||
- skipping known egg families in the first pass
|
||||
- expanding `SF_GLOBEGG`
|
||||
- documenting future work toward a better sorter
|
||||
|
||||
The current tool does not yet use the footpad values for full ItemSorter-equivalent overlap resolution.
|
||||
|
||||
## Current Projection And Painting Rules
|
||||
|
||||
The renderer anchors each shape at the same world-to-screen bottom point used by the runtime shape painter:
|
||||
|
||||
$$
|
||||
screen_x = \frac{x - y}{4}
|
||||
$$
|
||||
|
||||
$$
|
||||
screen_y = \frac{x + y}{8} - z
|
||||
$$
|
||||
|
||||
Frame placement then follows the shape-frame offsets used by the runtime sorter:
|
||||
|
||||
- unflipped: `left = screen_x - xoff`
|
||||
- flipped: `left = screen_x + xoff - width`
|
||||
- `top = screen_y - yoff`
|
||||
|
||||
The renderer now uses a ScummVM/Pentagram-style dependency graph sorter rather than a plain scalar key.
|
||||
|
||||
The current implementation ports the crucial parts of `SortItem` and `ItemSorter`:
|
||||
|
||||
- footpad-derived world boxes from `TYPEFLAG.DAT`
|
||||
- screen-diamond overlap and containment checks
|
||||
- `below()` ordering rules for flat pieces, tall pieces, roofs, translucent items, and Crusader inventory-item families
|
||||
- dependency expansion so overlapping items are painted only after everything behind them
|
||||
|
||||
This is materially better than the initial `z / x+y` heuristic and is the main path for reducing wall and prop overdraw artifacts, though it still omits some of the engine's more specialized runtime-only cases.
|
||||
|
||||
## Command Examples
|
||||
|
||||
Render No Remorse map `0`:
|
||||
|
||||
```powershell
|
||||
c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 0 --output out/map0-remorse.png
|
||||
```
|
||||
|
||||
Render No Regret map `0` and emit metadata:
|
||||
|
||||
```powershell
|
||||
c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game regret --fixed-dat K:/path/to/REGRET/FIXED.DAT --map 0 --output out/map0-regret.png --metadata out/map0-regret.json
|
||||
```
|
||||
|
||||
Render a bounded world-space region only:
|
||||
|
||||
```powershell
|
||||
c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 0 --world-rect 0 0 4096 4096 --output out/map0-quarter.png
|
||||
```
|
||||
|
||||
## Current Deliberate Limits
|
||||
|
||||
This tool is a start, not a complete engine clone.
|
||||
|
||||
Current gaps:
|
||||
|
||||
1. It renders `FIXED.DAT` only. It does not yet merge save-state or `NONFIXED.DAT` style movable items.
|
||||
2. It expands globs, but it does not yet emulate broader fast-area/runtime-driven materialization behavior.
|
||||
3. It skips several egg-family placements instead of trying to visualize their hidden runtime helpers.
|
||||
4. It now implements the core dependency graph sorter, but it still omits experimental occlusion grouping and some runtime-only sprite/highlight cases.
|
||||
5. It does not yet consume `ANIM.DAT`, `DAMAGE.FLX`, `DTABLE.FLX`, `WPNOVLAY.DAT`, or palette transforms such as `XFORMPAL.DAT`.
|
||||
6. It uses `GAMEPAL.PAL` directly and does not yet model alternate or transformed palettes.
|
||||
7. It writes a plain RGBA PNG using only the standard library; there is no zoomed viewer, tile atlas exporter, or sprite manifest yet.
|
||||
|
||||
## Immediate Follow-Ups
|
||||
|
||||
1. Validate and tune the dependency sorter against representative Remorse and Regret rooms, especially tall wall seams and dense prop clusters.
|
||||
2. Add optional atlas export for all shapes touched by a chosen map.
|
||||
3. Add a second path for movable/dynamic content once the relevant Crusader save/runtime files are pinned down for both games.
|
||||
4. Compare a few rendered regions against known in-game screenshots to tighten projection and ordering errors.
|
||||
5. Add optional per-item manifest output with `(shape, frame, x, y, z, source)` rows for debugging bad composites.
|
||||
6. Revisit raw `0007` rendering notes and the live executable only if the current Pentagram/ScummVM overlap model proves insufficient for specific remaining errors.
|
||||
|
|
@ -790,4 +790,19 @@ The strongest present path to a usable compiler/decompiler is:
|
|||
5. Attach ScummVM event and intrinsic names as hints, not as truth.
|
||||
6. Recompile by rebuilding the original class header and event table layout first, then re-emitting decoded and opaque ops together.
|
||||
|
||||
That gets to a reversible editor sooner than waiting for a full semantic VM recovery.
|
||||
That gets to a reversible editor sooner than waiting for a full semantic VM recovery.
|
||||
|
||||
## **Recent Research (2026-03-26)**
|
||||
|
||||
- **Root Cause:**: The structuring pass left forward/back-edge loops and counted-loop headers detached in fallback output, which produced unstructured pseudocode for some bodies (notably BART slot 0x0F).
|
||||
- **Renderer Fixes:**: Added a conservative loop-lifting helper and a restricted infinite-loop lift in the partial fallback renderer to fold loops into structured blocks where safe. See the modified renderer at [tools/poc_crusader_usecode_parser.py](tools/poc_crusader_usecode_parser.py).
|
||||
- **Validator Added:**: A lightweight pseudocode syntax/label validator was added to detect brace mismatches and missing goto/label targets before exporting pseudocode.
|
||||
- **Tests:**: Added and adjusted unit tests in [tools/tests/test_usecode_structuring.py](tools/tests/test_usecode_structuring.py) to guard loop-lifting behavior and fallback conservatism.
|
||||
- **Corpus Validation:**: Ran a corpus-wide render+validator pass over 977 decoded bodies; result: `TOTAL_BODIES=977, FAILURES=0` (no syntax/label failures).
|
||||
- **Real-World Output:**: Regenerated the BART pseudocode file — [USECODE/EUSECODE_extracted/pseudocode/BART/slot_0F_enterFastArea.txt](USECODE/EUSECODE_extracted/pseudocode/BART/slot_0F_enterFastArea.txt) now shows an outer `while(true)` with nested structured branches and counted loops instead of detached labels.
|
||||
- **Scope & Safety:**: Fully-structured renderer remains conservative; the loop-lifting helper is reused where safe. The outer infinite-loop lift was narrowed to partial fallback after tests revealed regressions when it was too broad.
|
||||
- **Remaining Semantic Gap:**: Expression/comparison operand polarity still needs correction (some counted-loop conditions show inverted comparisons). Next work: fix operand ordering in the expression builder so loop headers reflect correct comparison direction.
|
||||
- **Next Steps:**: (1) Implement compare-direction fix in the expression builder and add small semantic regression tests, (2) re-run unit tests and a corpus-wide render+validate sweep, (3) regenerate affected pseudocode files for inspection.
|
||||
- **Files of Interest:**: [tools/poc_crusader_usecode_parser.py](tools/poc_crusader_usecode_parser.py), [tools/tests/test_usecode_structuring.py](tools/tests/test_usecode_structuring.py), [USECODE/EUSECODE_extracted/pseudocode/BART/slot_0F_enterFastArea.txt](USECODE/EUSECODE_extracted/pseudocode/BART/slot_0F_enterFastArea.txt).
|
||||
|
||||
If you want, I can (a) implement the comparison/operand polarity fix next, (b) run the unit tests and a fresh corpus sweep, and (c) open a PR-ready commit with these doc and code updates.
|
||||
Loading…
Add table
Add a link
Reference in a new issue