Crusader_Decomp/docs/map-rendering.md

Crusader Map Rendering Workbench

Purpose

This note starts a dedicated lane for offline Crusader map extraction and PNG rendering from the shipped data files in this workspace.

Current implementation entry point:

• tools/render_crusader_map.py
• render_maps.bat for whole-game batch runs into out/remorse and out/regret

Current renderer diagnostics:

• large maps now emit progress checkpoints during item collection, dependency sorting, paint-order resolution, and blitting
• collect_render_items() only expands FIXED.DAT glob eggs once, instead of recursively re-expanding glob-emitted glob eggs
• metadata now records sampled invalid shape/frame references plus a conservative map-usage hint block
• roofs/exploration obscurers are now optional and disabled by default
• editor/debug/marker-style map content is now enabled by default instead of being silently discarded

Internal package layout:

• tools/crusader_map/formats.py for Crusader archive and record parsing
• tools/crusader_map/sorting.py for the dependency-graph overlap sorter
• tools/crusader_map/png.py for PNG buffer/blit helpers
• tools/crusader_map/cli.py for command-line orchestration

Current supported data roots:

• STATIC for the default No Remorse retail set when populated
• STATIC_1.01 for No Remorse 1.01
• STATIC_DEMO for a future No Remorse demo set
• STATIC_JP for a future No Remorse Japanese set
• STATIC_REGRET for No Regret

Current asset note:

• STATIC_REGRET in this workspace now includes FIXED.DAT
• map_renderer/STATIC now contains a populated No Remorse retail-style asset set including CRUSADER.EXE, FIXED.DAT, and EUSECODE.FLX
• map_renderer/STATIC_1.01 currently contains CRUSADER.EXE, FIXED.DAT, EUSECODE.FLX, DTABLE.FLX, and the rest of the shipped renderer inputs
• the renderer still accepts --fixed-dat so alternate map copies can be tested without changing the rest of the static asset path

Versioned Asset Audit

Current map-renderer workspace findings for No Remorse version support:

• map_renderer/STATIC and map_renderer/STATIC_1.01 are both populated and can now be compared directly
• verified SHA-1 and size differences exist between the retail-style STATIC set and STATIC_1.01 for CRUSADER.EXE, DAMAGE.FLX, EUSECODE.FLX, FIXED.DAT, GLOB.FLX, OVERLOAD.DAT, TYPEFLAG.DAT, UNKCOFF.DAT, and UNKDS.DAT
• CRUSADER.EXE differs materially: retail-style STATIC/CRUSADER.EXE is 991878 bytes with SHA-1 5F3EB75202FA54CDC6703B875BFC505EB1EE320D, while STATIC_1.01/CRUSADER.EXE is 979382 bytes with SHA-1 3BE9A196E5504E7ACB4E84FE3DF16503C16616FD
• EUSECODE.FLX differs materially: retail-style STATIC/EUSECODE.FLX is 556613 bytes with SHA-1 D7912724673F27C5D2E5B4FB9C41B22D67AE828A, while STATIC_1.01/EUSECODE.FLX is 418556 bytes with SHA-1 B2176648C9A795156AF764E4CEB45E4CF1973010
• FIXED.DAT differs materially: retail-style STATIC/FIXED.DAT is 3735392 bytes with SHA-1 4A8CF1ED99996B8B37A3A0FD33ACF09C1BB642E5, while STATIC_1.01/FIXED.DAT is 3734592 bytes with SHA-1 685757B0AC8871C0247F7B5963C4EB5ED68DC89C
• GLOB.FLX also differs: retail-style STATIC/GLOB.FLX is 264974 bytes with SHA-1 9E26B13F81E425C33CFB1D5A6CF202AAEB8C0F00, while STATIC_1.01/GLOB.FLX is 264884 bytes with SHA-1 01A31DAF4806946B8AEA0D3483DD5C3BA1D149ED
• many shipped art/palette/data files are unchanged between these two roots, including ANIM.DAT, DTABLE.FLX, SHAPES.FLX, GAMEPAL.PAL, FONTS.FLX, GUMPS.FLX, TRIG.DAT, WPNOVLAY.DAT, and XFORMPAL.DAT
• the web renderer should therefore treat available versions as detected data roots rather than assuming one remorse asset set stands in for all releases

Renderer version-support changes:

• the UI now exposes a version selector above the map selector and only lists variants whose static roots actually contain usable map data
• No Remorse variants are modeled as separate cache/export identities so STATIC_1.01, future STATIC_DEMO, and future STATIC_JP builds do not collide in .cache or site/data/maps
• shared annotation sources such as the remorse shape catalog CSV and DTable name dump remain reused across No Remorse variants until version-specific tables are added
• the build path now resolves REMORSE_XFORMPAL_PATH for every No Remorse variant, not just the legacy remorse id

Follow-up comparison plan once more data roots are added:

• compare CRUSADER.EXE hashes between STATIC, STATIC_1.01, STATIC_DEMO, and STATIC_JP
• compare FIXED.DAT hashes and map counts between versions
• compare EUSECODE.FLX hashes between versions
• record any map-count or mission-table differences in this note once the missing folders are populated

The immediate goal is practical and narrow: load a fixed map, expand glob placements, decode the required shapes from SHAPES.FLX, apply GAMEPAL.PAL, and render a deterministic PNG.

Source Cross-Checks Used

The first renderer is grounded in the overlapping parts of three sources rather than in ad hoc guesses.

1. Pentagram Crusader shape/map loaders

   • convert/crusader/ConvertShapeCrusader.cpp
   • graphics/Shape.cpp
   • graphics/ShapeFrame.cpp
   • world/Map.cpp
   • world/MapGlob.cpp
   • graphics/Palette.cpp
   • graphics/TypeFlags.cpp

2. ScummVM Ultima8 Crusader paths

   • gfx/shape_archive.cpp
   • gfx/type_flags.cpp
   • world/map.cpp
   • world/glob_egg.cpp
   • world/coord_utils.h
   • world/item_sorter.cpp
   • world/sort_item.cpp

3. Local workspace evidence

   • docs/scummvm-crusader-reference.md
   • docs/pentagram-crusader-reference.md
   • docs/raw-0007-rendering.md
   • crusader-disasm/shapedata.txt
   • crusader-disasm/mapdump/mapdump.py

File Formats Used By The First Tool

FIXED.DAT

The map container is treated as a header plus a map table:

• map count at file offset 0x54
• map table at file offset 0x80
• each table row is <u32 offset, u32 size>

Each map payload is read as packed 16-byte item records:

• x: u16
• y: u16
• z: u8
• shape: u16
• frame: u8
• flags: u16
• quality: u16
• npc_num: u8
• map_num: u8
• next: u16

Crusader-specific coordinate adjustment matches the Pentagram and ScummVM runtime loaders:

• world x = disk_x * 2
• world y = disk_y * 2

GLOB.FLX

GLOB.FLX is handled as a normal FLEX archive, not as a one-off format.

Each non-empty glob object contains:

• object count: u16
• repeated entries of x:u8 y:u8 z:u8 shape:u16 frame:u8

Glob expansion matches the Crusader GlobEgg::enterFastArea() rule in ScummVM/Pentagram:

• coordmask = ~0x3ff
• coordshift = 2
• offset = 2
• itemx = (parent_x & coordmask) + (glob_x << 2) + 2
• itemy = (parent_y & coordmask) + (glob_y << 2) + 2
• itemz = parent_z + glob_z

The first renderer expands glob contents and skips drawing the source glob egg itself.

SHAPES.FLX

World shapes use the Crusader shape layout documented by Pentagram/ScummVM:

• shape header: 6 bytes
  • 4 bytes unknown
  • 2-byte frame count
• frame header table: 8 bytes per frame
  • 3-byte frame offset
  • 1 unknown byte
  • 4-byte frame length
• frame body header: 28 bytes
  • 8 unknown bytes
  • 4-byte compression flag
  • 4-byte width
  • 4-byte height
  • 4-byte x offset
  • 4-byte y offset
• then height 4-byte line offsets
• then per-line RLE data

The current decoder follows the runtime line walker used in ShapeFrame::getPixelAtPoint():

• each line is a series of skip/run pairs
• compressed runs use the low bit to choose literal versus repeated-color mode
• pixels absent from the RLE stream are treated as transparent

GAMEPAL.PAL

GAMEPAL.PAL is read as 768 bytes of VGA-style palette data.

Each component is promoted from 0..63 to 0..255 using the same scaling used by Pentagram:

• rgb8 = (rgb6 * 255) / 63

TYPEFLAG.DAT

The renderer currently uses Crusader's 9-byte records to extract:

• family id
• shape footpad dimensions (x, y, z)
• editor flag

This is enough for:

• skipping known egg families in the first pass
• expanding SF_GLOBEGG
• documenting future work toward a better sorter

Current offline rendering policy differs from the live game intentionally:

• SI_ROOF shapes are hidden by default because they commonly act as exploration obscurers or roof covers that gameplay later removes or pops
• editor/debug/marker-style content is shown by default so offline renders expose more of what the shipped data actually contains

The current tool does not yet use the footpad values for full ItemSorter-equivalent overlap resolution.

Current Projection And Painting Rules

The renderer anchors each shape at the same world-to-screen bottom point used by the runtime shape painter:

screen_x = \frac{x - y}{4}

screen_y = \frac{x + y}{8} - z

Frame placement then follows the shape-frame offsets used by the runtime sorter:

• unflipped: left = screen_x - xoff
• flipped: left = screen_x + xoff - width
• top = screen_y - yoff

The renderer now uses a ScummVM/Pentagram-style dependency graph sorter rather than a plain scalar key.

The current implementation ports the crucial parts of SortItem and ItemSorter:

• footpad-derived world boxes from TYPEFLAG.DAT
• screen-diamond overlap and containment checks
• below() ordering rules for flat pieces, tall pieces, roofs, translucent items, and Crusader inventory-item families
• dependency expansion so overlapping items are painted only after everything behind them

This is materially better than the initial z / x+y heuristic and is the main path for reducing wall and prop overdraw artifacts, though it still omits some of the engine's more specialized runtime-only cases.

Command Examples

Render No Remorse map 0:

c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 0 --output out/map0-remorse.png

Render No Regret map 0 and emit metadata:

c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game regret --fixed-dat K:/path/to/REGRET/FIXED.DAT --map 0 --output out/map0-regret.png --metadata out/map0-regret.json

Render a bounded world-space region only:

c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 0 --world-rect 0 0 4096 4096 --output out/map0-quarter.png

Render with roofs restored:

c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 1 --include-roofs --output out/map1-with-roofs.png

Render without the extra hidden/editor marker content:

c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 1 --no-include-hidden-markers --no-include-editor --output out/map1-minimal.png

Render every No Remorse map to out/remorse:

render_maps.bat remorse

Render every No Regret map to out/regret:

render_maps.bat regret

The batch runner also accepts an optional start_map end_map range for partial runs while validating changes:

render_maps.bat remorse 1 3

You can also forward extra renderer arguments through the RENDER_ARGS environment variable, for example a bounded validation run:

set RENDER_ARGS=--world-rect 0 0 16384 16384
render_maps.bat regret 5 5

Batch behavior notes:

• empty maps are skipped in batch mode and do not produce PNG or JSON outputs
• there is no default max-pixel cap anymore; full-map renders are attempted unless you pass --max-pixels
• batch item-count skipping is now opt-in only; set BATCH_MAX_ITEMS to a positive value if you want the batch runner to skip very large full maps
• the renderer emits progress by default every 2000 items; pass --progress-every 0 through RENDER_ARGS to silence it
• batch runs now default to include_editor=true, include_hidden_markers=true, and include_roofs=false

Metadata notes:

• invalid_items contains a capped sample of bad (shape, frame, x, y, z, source, reason) records so broken FIXED.DAT references can be inspected without rerunning a scan
• usage is conservative: it reports known reference-backed map hints when available and otherwise stays unknown; it does not yet prove orphan status
• base_item_summary reports how many roof, editor, egg-family, invisible, and NPC-linked records were present in the raw map payload
• filters records whether the render included roofs, editor shapes, and hidden marker content

Current Deliberate Limits

This tool is a start, not a complete engine clone.

Current gaps:

1. It renders FIXED.DAT only. It does not yet merge save-state or NONFIXED.DAT style movable items.
2. It expands globs, but it does not yet emulate broader fast-area/runtime-driven materialization behavior.
3. It skips several egg-family placements instead of trying to visualize their hidden runtime helpers.
4. It now implements the core dependency graph sorter, but it still omits experimental occlusion grouping and some runtime-only sprite/highlight cases.
5. It does not yet consume ANIM.DAT, DAMAGE.FLX, DTABLE.FLX, WPNOVLAY.DAT, or palette transforms such as XFORMPAL.DAT.
6. It uses GAMEPAL.PAL directly and does not yet model alternate or transformed palettes.
7. It writes a plain RGBA PNG using only the standard library; there is no zoomed viewer, tile atlas exporter, or sprite manifest yet.
8. Some maps still contain invalid shape/frame references in FIXED.DAT; the renderer now skips those items instead of aborting the whole map, but that means some broken placements remain missing until the source of those references is understood.

Immediate Follow-Ups

1. Validate and tune the dependency sorter against representative Remorse and Regret rooms, especially tall wall seams and dense prop clusters.
2. Add optional atlas export for all shapes touched by a chosen map.
3. Add a second path for movable/dynamic content once the relevant Crusader save/runtime files are pinned down for both games.
4. Compare a few rendered regions against known in-game screenshots to tighten projection and ordering errors.
5. Add optional per-item manifest output with (shape, frame, x, y, z, source) rows for debugging bad composites.
6. Revisit raw 0007 rendering notes and the live executable only if the current Pentagram/ScummVM overlap model proves insufficient for specific remaining errors.