Crusader_Decomp/docs/map-rendering.md

# Crusader Map Rendering Workbench

## Purpose

This note starts a dedicated lane for offline Crusader map extraction and PNG rendering from the shipped data files in this workspace.

Current implementation entry point:

- `tools/render_crusader_map.py`
- `render_maps.bat` for whole-game batch runs into `out/remorse` and `out/regret`

Current renderer diagnostics:

- large maps now emit progress checkpoints during item collection, dependency sorting, paint-order resolution, and blitting
- `collect_render_items()` only expands `FIXED.DAT` glob eggs once, instead of recursively re-expanding glob-emitted glob eggs
- metadata now records sampled invalid shape/frame references plus a conservative map-usage hint block
- roofs/exploration obscurers are now optional and disabled by default
- editor/debug/marker-style map content is now enabled by default instead of being silently discarded

Internal package layout:

- `tools/crusader_map/formats.py` for Crusader archive and record parsing
- `tools/crusader_map/sorting.py` for the dependency-graph overlap sorter
- `tools/crusader_map/png.py` for PNG buffer/blit helpers
- `tools/crusader_map/cli.py` for command-line orchestration

Current supported data roots:

- `STATIC` for the default No Remorse retail set when populated
- `STATIC_1.01` for No Remorse 1.01
- `STATIC_DEMO` for a future No Remorse demo set
- `STATIC_JP` for a future No Remorse Japanese set
- `STATIC_REGRET` for No Regret

Current asset note:

- `STATIC_REGRET` in this workspace now includes `FIXED.DAT`
- `map_renderer/STATIC` now contains a populated No Remorse retail-style asset set including `CRUSADER.EXE`, `FIXED.DAT`, and `EUSECODE.FLX`
- `map_renderer/STATIC_1.01` currently contains `CRUSADER.EXE`, `FIXED.DAT`, `EUSECODE.FLX`, `DTABLE.FLX`, and the rest of the shipped renderer inputs
- the renderer still accepts `--fixed-dat` so alternate map copies can be tested without changing the rest of the static asset path

## Versioned Asset Audit

Current map-renderer workspace findings for No Remorse version support:

- `map_renderer/STATIC` and `map_renderer/STATIC_1.01` are both populated and can now be compared directly
- verified SHA-1 and size differences exist between the retail-style `STATIC` set and `STATIC_1.01` for `CRUSADER.EXE`, `DAMAGE.FLX`, `EUSECODE.FLX`, `FIXED.DAT`, `GLOB.FLX`, `OVERLOAD.DAT`, `TYPEFLAG.DAT`, `UNKCOFF.DAT`, and `UNKDS.DAT`
- `CRUSADER.EXE` differs materially: retail-style `STATIC/CRUSADER.EXE` is `991878` bytes with SHA-1 `5F3EB75202FA54CDC6703B875BFC505EB1EE320D`, while `STATIC_1.01/CRUSADER.EXE` is `979382` bytes with SHA-1 `3BE9A196E5504E7ACB4E84FE3DF16503C16616FD`
- `EUSECODE.FLX` differs materially: retail-style `STATIC/EUSECODE.FLX` is `556613` bytes with SHA-1 `D7912724673F27C5D2E5B4FB9C41B22D67AE828A`, while `STATIC_1.01/EUSECODE.FLX` is `418556` bytes with SHA-1 `B2176648C9A795156AF764E4CEB45E4CF1973010`
- `FIXED.DAT` differs materially: retail-style `STATIC/FIXED.DAT` is `3735392` bytes with SHA-1 `4A8CF1ED99996B8B37A3A0FD33ACF09C1BB642E5`, while `STATIC_1.01/FIXED.DAT` is `3734592` bytes with SHA-1 `685757B0AC8871C0247F7B5963C4EB5ED68DC89C`
- `GLOB.FLX` also differs: retail-style `STATIC/GLOB.FLX` is `264974` bytes with SHA-1 `9E26B13F81E425C33CFB1D5A6CF202AAEB8C0F00`, while `STATIC_1.01/GLOB.FLX` is `264884` bytes with SHA-1 `01A31DAF4806946B8AEA0D3483DD5C3BA1D149ED`
- many shipped art/palette/data files are unchanged between these two roots, including `ANIM.DAT`, `DTABLE.FLX`, `SHAPES.FLX`, `GAMEPAL.PAL`, `FONTS.FLX`, `GUMPS.FLX`, `TRIG.DAT`, `WPNOVLAY.DAT`, and `XFORMPAL.DAT`
- the web renderer should therefore treat available versions as detected data roots rather than assuming one remorse asset set stands in for all releases

Renderer version-support changes:

- the UI now exposes a version selector above the map selector and only lists variants whose static roots actually contain usable map data
- No Remorse variants are modeled as separate cache/export identities so `STATIC_1.01`, future `STATIC_DEMO`, and future `STATIC_JP` builds do not collide in `.cache` or `site/data/maps`
- shared annotation sources such as the remorse shape catalog CSV and DTable name dump remain reused across No Remorse variants until version-specific tables are added
- the build path now resolves `REMORSE_XFORMPAL_PATH` for every No Remorse variant, not just the legacy `remorse` id

Follow-up comparison plan once more data roots are added:

- compare `CRUSADER.EXE` hashes between `STATIC`, `STATIC_1.01`, `STATIC_DEMO`, and `STATIC_JP`
- compare `FIXED.DAT` hashes and map counts between versions
- compare `EUSECODE.FLX` hashes between versions
- record any map-count or mission-table differences in this note once the missing folders are populated

The immediate goal is practical and narrow: load a fixed map, expand glob placements, decode the required shapes from `SHAPES.FLX`, apply `GAMEPAL.PAL`, and render a deterministic PNG.

## Source Cross-Checks Used

The first renderer is grounded in the overlapping parts of three sources rather than in ad hoc guesses.

1. Pentagram Crusader shape/map loaders
   - `convert/crusader/ConvertShapeCrusader.cpp`
   - `graphics/Shape.cpp`
   - `graphics/ShapeFrame.cpp`
   - `world/Map.cpp`
   - `world/MapGlob.cpp`
   - `graphics/Palette.cpp`
   - `graphics/TypeFlags.cpp`

2. ScummVM Ultima8 Crusader paths
   - `gfx/shape_archive.cpp`
   - `gfx/type_flags.cpp`
   - `world/map.cpp`
   - `world/glob_egg.cpp`
   - `world/coord_utils.h`
   - `world/item_sorter.cpp`
   - `world/sort_item.cpp`

3. Local workspace evidence
   - `docs/scummvm-crusader-reference.md`
   - `docs/pentagram-crusader-reference.md`
   - `docs/raw-0007-rendering.md`
   - `crusader-disasm/shapedata.txt`
   - `crusader-disasm/mapdump/mapdump.py`

## File Formats Used By The First Tool

### `FIXED.DAT`

The map container is treated as a header plus a map table:

- map count at file offset `0x54`
- map table at file offset `0x80`
- each table row is `<u32 offset, u32 size>`

Each map payload is read as packed 16-byte item records:

- `x: u16`
- `y: u16`
- `z: u8`
- `shape: u16`
- `frame: u8`
- `flags: u16`
- `quality: u16`
- `npc_num: u8`
- `map_num: u8`
- `next: u16`

Crusader-specific coordinate adjustment matches the Pentagram and ScummVM runtime loaders:

- world `x = disk_x * 2`
- world `y = disk_y * 2`

### `GLOB.FLX`

`GLOB.FLX` is handled as a normal FLEX archive, not as a one-off format.

Each non-empty glob object contains:

- object count: `u16`
- repeated entries of `x:u8 y:u8 z:u8 shape:u16 frame:u8`

Glob expansion matches the Crusader `GlobEgg::enterFastArea()` rule in ScummVM/Pentagram:

- `coordmask = ~0x3ff`
- `coordshift = 2`
- `offset = 2`
- `itemx = (parent_x & coordmask) + (glob_x << 2) + 2`
- `itemy = (parent_y & coordmask) + (glob_y << 2) + 2`
- `itemz = parent_z + glob_z`

The first renderer expands glob contents and skips drawing the source glob egg itself.

### `SHAPES.FLX`

World shapes use the Crusader shape layout documented by Pentagram/ScummVM:

- shape header: 6 bytes
  - 4 bytes unknown
  - 2-byte frame count
- frame header table: 8 bytes per frame
  - 3-byte frame offset
  - 1 unknown byte
  - 4-byte frame length
- frame body header: 28 bytes
  - 8 unknown bytes
  - 4-byte compression flag
  - 4-byte width
  - 4-byte height
  - 4-byte x offset
  - 4-byte y offset
- then `height` 4-byte line offsets
- then per-line RLE data

The current decoder follows the runtime line walker used in `ShapeFrame::getPixelAtPoint()`:

- each line is a series of skip/run pairs
- compressed runs use the low bit to choose literal versus repeated-color mode
- pixels absent from the RLE stream are treated as transparent

### `GAMEPAL.PAL`

`GAMEPAL.PAL` is read as 768 bytes of VGA-style palette data.

Each component is promoted from `0..63` to `0..255` using the same scaling used by Pentagram:

- `rgb8 = (rgb6 * 255) / 63`

### `TYPEFLAG.DAT`

The renderer currently uses Crusader's 9-byte records to extract:

- family id
- shape footpad dimensions (`x`, `y`, `z`)
- editor flag

This is enough for:

- skipping known egg families in the first pass
- expanding `SF_GLOBEGG`
- documenting future work toward a better sorter

Current offline rendering policy differs from the live game intentionally:

- `SI_ROOF` shapes are hidden by default because they commonly act as exploration obscurers or roof covers that gameplay later removes or pops
- editor/debug/marker-style content is shown by default so offline renders expose more of what the shipped data actually contains

The current tool does not yet use the footpad values for full ItemSorter-equivalent overlap resolution.

## Current Projection And Painting Rules

The renderer anchors each shape at the same world-to-screen bottom point used by the runtime shape painter:

$$
screen_x = \frac{x - y}{4}
$$

$$
screen_y = \frac{x + y}{8} - z
$$

Frame placement then follows the shape-frame offsets used by the runtime sorter:

- unflipped: `left = screen_x - xoff`
- flipped: `left = screen_x + xoff - width`
- `top = screen_y - yoff`

The renderer now uses a ScummVM/Pentagram-style dependency graph sorter rather than a plain scalar key.

The current implementation ports the crucial parts of `SortItem` and `ItemSorter`:

- footpad-derived world boxes from `TYPEFLAG.DAT`
- screen-diamond overlap and containment checks
- `below()` ordering rules for flat pieces, tall pieces, roofs, translucent items, and Crusader inventory-item families
- dependency expansion so overlapping items are painted only after everything behind them

This is materially better than the initial `z / x+y` heuristic and is the main path for reducing wall and prop overdraw artifacts, though it still omits some of the engine's more specialized runtime-only cases.

## Command Examples

Render No Remorse map `0`:

```powershell
c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 0 --output out/map0-remorse.png
```

Render No Regret map `0` and emit metadata:

```powershell
c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game regret --fixed-dat K:/path/to/REGRET/FIXED.DAT --map 0 --output out/map0-regret.png --metadata out/map0-regret.json
```

Render a bounded world-space region only:

```powershell
c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 0 --world-rect 0 0 4096 4096 --output out/map0-quarter.png
```

Render with roofs restored:

```powershell
c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 1 --include-roofs --output out/map1-with-roofs.png
```

Render without the extra hidden/editor marker content:

```powershell
c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 1 --no-include-hidden-markers --no-include-editor --output out/map1-minimal.png
```

Render every No Remorse map to `out/remorse`:

```cmd
render_maps.bat remorse
```

Render every No Regret map to `out/regret`:

```cmd
render_maps.bat regret
```

The batch runner also accepts an optional `start_map end_map` range for partial runs while validating changes:

```cmd
render_maps.bat remorse 1 3
```

You can also forward extra renderer arguments through the `RENDER_ARGS` environment variable, for example a bounded validation run:

```cmd
set RENDER_ARGS=--world-rect 0 0 16384 16384
render_maps.bat regret 5 5
```

Batch behavior notes:

- empty maps are skipped in batch mode and do not produce PNG or JSON outputs
- there is no default max-pixel cap anymore; full-map renders are attempted unless you pass `--max-pixels`
- batch item-count skipping is now opt-in only; set `BATCH_MAX_ITEMS` to a positive value if you want the batch runner to skip very large full maps
- the renderer emits progress by default every 2000 items; pass `--progress-every 0` through `RENDER_ARGS` to silence it
- batch runs now default to `include_editor=true`, `include_hidden_markers=true`, and `include_roofs=false`

Metadata notes:

- `invalid_items` contains a capped sample of bad `(shape, frame, x, y, z, source, reason)` records so broken `FIXED.DAT` references can be inspected without rerunning a scan
- `usage` is conservative: it reports known reference-backed map hints when available and otherwise stays `unknown`; it does not yet prove orphan status
- `base_item_summary` reports how many roof, editor, egg-family, invisible, and NPC-linked records were present in the raw map payload
- `filters` records whether the render included roofs, editor shapes, and hidden marker content

## Current Deliberate Limits

This tool is a start, not a complete engine clone.

Current gaps:

1. It renders `FIXED.DAT` only. It does not yet merge save-state or `NONFIXED.DAT` style movable items.
2. It expands globs, but it does not yet emulate broader fast-area/runtime-driven materialization behavior.
3. It skips several egg-family placements instead of trying to visualize their hidden runtime helpers.
4. It now implements the core dependency graph sorter, but it still omits experimental occlusion grouping and some runtime-only sprite/highlight cases.
5. It does not yet consume `ANIM.DAT`, `DAMAGE.FLX`, `DTABLE.FLX`, `WPNOVLAY.DAT`, or palette transforms such as `XFORMPAL.DAT`.
6. It uses `GAMEPAL.PAL` directly and does not yet model alternate or transformed palettes.
7. It writes a plain RGBA PNG using only the standard library; there is no zoomed viewer, tile atlas exporter, or sprite manifest yet.
8. Some maps still contain invalid shape/frame references in `FIXED.DAT`; the renderer now skips those items instead of aborting the whole map, but that means some broken placements remain missing until the source of those references is understood.

## Immediate Follow-Ups

1. Validate and tune the dependency sorter against representative Remorse and Regret rooms, especially tall wall seams and dense prop clusters.
2. Add optional atlas export for all shapes touched by a chosen map.
3. Add a second path for movable/dynamic content once the relevant Crusader save/runtime files are pinned down for both games.
4. Compare a few rendered regions against known in-game screenshots to tighten projection and ordering errors.
5. Add optional per-item manifest output with `(shape, frame, x, y, z, source)` rows for debugging bad composites.
6. Revisit raw `0007` rendering notes and the live executable only if the current Pentagram/ScummVM overlap model proves insufficient for specific remaining errors.