Crusader_Decomp/docs/map_renderer/translucency-xformpal.md

Translucency and XFORMPAL Work Log

This note tracks the renderer-side work on Crusader translucent shapes, especially the recurring wrong yellow/white failure on editor walls and related helper surfaces.

Current Goal

Make translucent shapes in the public map renderer use colors closer to the original Crusader appearance instead of the old bright yellow/white fallback.

Verified Test Anchors

Work in this batch used the local renderer project at K:/ghidra/crusader_map_viewer/map_renderer with the retail executables now present in STATIC/CRUSADER.EXE and STATIC_REGRET/REGRET.EXE.

Representative translucent shapes used as direct probes:

• 0x005A, 0x005B, 0x005C, 0x005D, 0x0067, 0x0068, 0x0069: invisible/editor wall family
• 0x00E9: additional invisible wall helper
• 0x02F8: unnamed translucent base shape present in Remorse map 10
• 0x044D, 0x044E: Zappy_Surface_SW_044D / Zappy_Surface_SE_044E

Verified scene/cache anchors from this batch:

• old remap-only build: fingerprint 5795f5362a7450fa (v11-atlas-scene-xformpal-opaque-remap)
• ScummVM-slot-only build: fingerprint 34b9a03b15bdf9da (v12-atlas-scene-crusader-xform-blend)
• current hybrid build: fingerprint 9bc2bb02d545279d (v13-atlas-scene-crusader-xform-remap-rgb)

Verified fresh scene file for the current implementation:

• K:/ghidra/crusader_map_viewer/map_renderer/.cache/scene-cache/remorse/map-10/9bc2bb02d545279d/scene.json

What The Pixel Data Proved

Direct frame inspection for the bad editor-wall and helper shapes showed that they are dominated by palette indices 8, 9, 10, 11, 13, and sometimes 14.

That matters because ScummVM and Pentagram treat those low slot values as special Crusader xform slots rather than ordinary color indices.

Example direct probe results from this batch:

• shape 0x005A / 90: top source indices 11, 10, 13, 9
• shape 0x00E9 / 233: top source indices 11, 9, 10, 13
• shape 0x044D / 1101: almost entirely slot 8, with a small amount of 10
• shape 0x044E / 1102: almost entirely slot 10, with a small amount of 8

This falsified the earlier local assumption that XFORMPAL.DAT should be consumed as a simple 256 -> 256 color remap for translucent world shapes.

Attempt 1: Simple XFORMPAL Remap

Implementation shape:

• parse the first 256 bytes of XFORMPAL.DAT
• remap every translucent source pixel to a different palette index
• bake the result into the atlas PNG

Result:

• failed visually
• translucent shapes still landed in the same obviously wrong bright yellow/white family
• this also did not model the destination-aware blend behavior used by ScummVM/Pentagram

Useful negative result:

• XFORMPAL.DAT object 0 is not a single 256-byte table; it is 2304 bytes
• the current files in both Remorse and Regret are 2480 bytes total with:
  • entry 0: offset 144, length 2304
  • entry 1: offset 2448, length 32

Attempt 2: ScummVM Crusader Slot Blend

Implementation shape:

• use the Crusader slot alpha/color behavior modeled in ScummVM CruXFormPal
• treat slots 8..14 as special translucent blend slots
• keep those pixels per-pixel translucent instead of globally fading the entire item

What this fixed:

• removed the earlier double fade behavior where translucent items were being globally faded on top of a pre-faded atlas
• direct probe on shape 0x005A produced the expected slot alpha set 64, 80, 128, 140

What still failed:

• editor walls still looked like bright white/yellow slabs
• this was better structurally than attempt 1, but still wrong in hue

Status:

• partial success only

Attempt 3: Compare All On-Disk XFORMPAL Tables

The 2304-byte payload was also split into 9 separate 256-byte remap tables and compared visually against the bad translucent shapes.

Observed result:

• tables are not equivalent
• several tables still produced extreme orange/red/black results
• some later tables produced substantially more muted grey/blue/red translucent helper colors than the ScummVM hardcoded white/yellow slot colors

Most useful empirical result from this comparison:

• remap table index 5 (zero-based) was the most consistently plausible across the tested shape families in this batch
• it improved the editor wall family and also gave better-looking muted tones for the tested Zappy_Surface_* probes

This table choice is still empirical, not format-closed.

Current Implementation: Hybrid Alpha + XFORMPAL RGB

Current code path in the renderer:

• keep the Crusader slot alpha behavior from the ScummVM-style 8..14 xform slots
• source the translucent RGB from XFORMPAL.DAT remap table 5 instead of the hardcoded white/yellow slot RGB
• do not apply the old item-wide 0.7 opacity to translucent items anymore
• invalidate scene caches with renderer cache version v13-atlas-scene-crusader-xform-remap-rgb

Files touched in the renderer repo for this batch:

• src/lib/formats.js
• src/lib/png.js
• src/lib/build-manager.js

Verified result:

• fresh isolated build probe succeeded with scene fingerprint 9bc2bb02d545279d
• fresh dark-background debug render no longer showed the old bright yellow/white slabs for the tested editor-wall and helper-family probes
• the current debug artifact from the live code path is:
  • K:/ghidra/crusader_map_viewer/map_renderer/.cache/translucency-debug-sheet-v13.png

Best short summary of the current state:

• alpha behavior: materially improved and now matches Crusader xform-slot expectations much better than the old remap-only path
• hue behavior: materially improved for the tested shapes, but still not proven closed for every translucent family

What Is Still Unresolved

• the exact semantic structure of XFORMPAL.DAT is still not closed
• entry 1 (32 bytes) remains unexplained in this batch
• the choice of remap table 5 is evidence-backed for the tested shapes, but still empirical rather than formally decoded
• other translucent families may need a different XFORMPAL table than the current default
• this batch validated via direct shape probes and a fresh scene build, but not yet via a broad multi-map manual browser pass

Recommended Next Steps

1. Run a manual browser-side check on a few representative maps and note which translucent families now look correct versus still wrong.
2. Identify at least one real window family shape from live cached scenes and add it to the comparison set, instead of relying mostly on editor-wall and helper probes.
3. Compare remap table 5 versus nearby candidates 6 and 7 for the remaining suspect families if any still look off.
4. Reverse-engineer the meaning of the 32-byte second XFORMPAL entry and look for any runtime selector or table-id field that explains why multiple remap tables exist.
5. If one-family-per-table evidence appears, move from a single global translucent RGB table to a family- or shape-driven selector.