Crusader_Decomp/psx-map-exporter/docs/spec.md

PSX Map Exporter Spec

Goal

psx-map-exporter is a standalone Node.js probe for Crusader PSX map extraction.

It exists to prove a fresh end-to-end path from raw LSET*.WDL input to:

• extracted intermediate sprite assets under .cache
• a rendered map PNG under .output

This project does not reuse Crusader-Map-Viewer code, scene caches, donor mappings, or sidecar summaries as binding inputs. It only consumes raw PSX assets plus the documented executable-backed findings from docs/psx and the live Ghidra session.

Scope

Version v0 is intentionally narrow.

It will:

• read one PSX LSET*.WDL file
• parse the documented 0x38-byte top-level header
• carve the post-audio map/art regions from header-derived boundaries
• parse the loader-sized post-audio sections as a second, higher-value view of the file layout
• extract the dense constructor-placement family from post_audio_section_00
• keep the smaller root-dispatch family available as a comparison probe
• render a layered authored probe that can combine constructor placements with the smaller root-dispatch lane
• scan post_audio_region_04 for type-4/type-5 sprite bundles
• decode bundle frames directly from the raw WDL
• write extracted frame PNGs to .cache
• compose a probe map PNG to .output

It will not claim full runtime parity yet.

Known non-goals for v0:

• exact CLUT reproduction
• full stage-1 dependency-graph ordering
• full post_audio_region_01 / post_audio_region_02 semantic decode

Landed in the current pass (was previously a non-goal):

• loader-faithful DAT_800758d8 active-header bank binding via explicit parses of the artInstall and override blocks in both SPEC_A.WDL and the map-local LSET*.WDL. See Loader Layout and Art Binding Rule below.

Evidence Constraints

The implementation is grounded in these current facts from the docs and Ghidra:

• LSET*.WDL begins with 14 little-endian u32 size fields (56 bytes total) describing the sequence of post-header blocks. The loader (wdl_resource_bundle_load_by_index @ 0x80039444) reads each size and carves the blocks in order: packPreamble, dispatchRootsSize, ctorPlacementsSize, packTailRewindSize, ctorPlacementSection, sectionPackBaseSize, policyTableSize, table8006754cSize, opcodeStreamsSize, detachedBlobSize, artInstallSize, stateBankSize, overrideSize, stateBank2Size.
• SPEC_A.WDL (global bundle A) begins with a fixed 0x3520-byte VRAM preload, followed by the same 14-field size header. Bundle A skips the sectionPack and detachedBlob blocks entirely; the remaining blocks are still present in the same order.
• The old "audio blob at word[1]" model was incorrect for this stream; the first 14 u32 words are block-size descriptors, not a single audio size.
• The post-audio four-region carve is kept as a fallback diagnostic view but is no longer the primary input for record or art extraction.
• The small count-prefixed section-0 root-dispatch rows are real, but they are not the whole map object set.
• The dense constructor-placement records recovered from loader-sized post_audio_section_00 are the live-object seed source for rendering.
• post_audio_region_04 is retained only as a fallback bundle source; real art now flows through the artInstall and override blocks parsed out of the 14-u32 layout.
• Type-4/type-5 drawable bundles expose width, height, palette mode/index, frame count, frame table offset, and data offset in the raw 0x58-byte bundle header.
• Bundle frame entries use a 20-byte row with size, relative data offset, width, height, origin x/y, and flags.
• sprite_rle_decode_rows uses row-local control bytes:
  • positive: repeat next byte N times
  • negative: copy next abs(N) literal bytes
  • zero: end row
• The executable projection basis (per psx_project_object_main_visible @ 0x80040d44) is, in pixel units, with no extra scale factor:

screen_x = y - x

screen_y = 2z - \frac{x + y}{2}

• Record X/Y values are already in screen-pixel units. The live view-cull box is camera +/- 0x140 = +/- 320 pixels, matching PSX screen width. The exporter therefore uses PSX_SCREEN_SCALE = 1; earlier builds multiplied by 2, producing over-spaced maps.

Loader Layout

Both SPEC_A.WDL and LSET*.WDL are fed to the same loader body, once per WDL pass. Each pass runs two art installs, two state-bank installs, and one override install. The loader reads the 14-u32 size header starting at offset 0 (LSET) or 0x3520 (SPEC_A) and lays out blocks sequentially.

• artInstall block (at 0x800396a0 for bundle A, 0x80039988 for bundle B): directory and payloads live at block + 0x2718. The first 0x2710 bytes of the block are a scratch header cache used while resources are built. The directory format is { u32 count; u32 directoryOffset; } at the start of the block, then count entries of { u32 size; u32 typeId; } at block + 0x2718 + directoryOffset. For each non-zero entry the loader installs a built-resource pair { u16 kind; u16 _; u32 resource_ptr } into DAT_800758d8[typeId] (0x18-byte stride).
• override block (at 0x80039730 for bundle A, 0x80039a18 for bundle B): same directory format, but the payload cursor starts at block + 8 (directly after the 8-byte prefix). Each non-zero entry payload is a raw 0x58-byte drawable header whose pointer is written straight into DAT_800758d8[typeId] at 0x8003977c / 0x80039a64, overwriting whatever the earlier artInstall pass installed. Zero-size entries clear the bank slot.
• Apply order per loader call: SPEC_A artInstall → SPEC_A override → LSET artInstall → LSET override. Later writes win, so the final DAT_800758d8 state is a mix of built-resource pointers and raw override headers.

Evidence retained for reference

• The direct typeWord -> bundle slot scan-order binding is disproven as a final art rule and is retained only as a diagnostic bundle-family probe.

Input Model

The exporter accepts either:

• a direct --wdl path
• or a --source path relative to a PSX disc root

Default disc root for local workspace runs:

• d:/Ghidra/Crusader-Map-Viewer/map_renderer/STATIC_PSX

Expected source examples:

• LSET1/L0.WDL
• LSET4/L37.WDL

Output Layout

.cache

Per-run cache path:

• .cache/<map-stem>/

Contents:

• wdl-summary.json
• records.json
• bundles.json
• frame-manifest.json
• active-header-overrides.json
• sprites/<bundle-offset>/frame_<n>.png

The cache is disposable. It exists to preserve intermediate evidence and make re-runs inspectable.

records.json now also records constructor-stream detection metadata when available: stream header offset, record start offset, reported count, and the initial structured-prefix run.

The cache also records candidate late DAT_800758d8 header-only override blobs as a standalone diagnostic. Those candidates are not used as final art binding yet.

wdl-summary.json now also emits sceneInterpretation, which is an explicit warning-bearing classification of what the current export most likely represents. For constructor-placement exports this should currently read as a constructor-fed live-object seed lane rather than a final visible-world reconstruction.

.output

Per-run final outputs:

• .output/<map-stem>.png
• .output/<map-stem>.json
• .output/<map-stem>_<layer>.png for each rendered authored layer when layered mode is active

The JSON stores the final probe scene manifest used to draw the PNG.

The .output folder is reset at the start of each export so evaluation only sees artifacts from the current run.

The .output/<map-stem>.json manifest inherits sceneInterpretation from wdl-summary.json so consumers do not need to infer that warning from prose docs alone.

Record Extraction Rules

v0 pulls scene records from two loader-faithful lanes inside the section pack, matching the executable's two dispatch iterators. Both lanes are indexed through packSubranges from the 14-u32 loader layout.

Constructor placements (12-byte stride)

• Source: ctorPlacements pack subrange (word 2).
• Dispatcher: psx_dispatch_section0_constructor_placements @ 0x800258cc.
• Layout: [u32 count][count * { u16 typeWord; u16 X; u16 Y; u16 Z; u16 selector; u16 flags }].
• The dispatcher passes each record directly to descriptor_table[typeWord].slot0(record, 0) and downstream spawners (e.g. psx_object_create_compound_record) read exactly the six u16 fields.
• Older heuristic region-01 / section-0 scans are retained as compatibility fallbacks when the loader block is absent or empty.

Dispatch roots (24-byte stride)

• Source: dispatchRoots pack subrange (word 1).
• Dispatcher: psx_dispatch_section0_dispatch_roots @ 0x800256b0.
• Layout per record: [u32 count] followed by 24-byte entries whose dispatcher-visible fields are:
  • +0x04 u16 typeId indexes psx_type_descriptor_table
  • +0x08 u16 screenX used directly by the +/- 0x140 view-cull
  • +0x0A u16 screenY same
  • +0x10 u16 flags bit 3 skips the record
• Remaining fields are forwarded to descriptor slot 0. The exporter empirically projects +0x06 as z, +0x0C as selector, +0x0E as lane, with relaxed plausibility because the live dispatcher only requires the fields above.

Selection modes

• auto / combined / layered merges both lanes into one layered probe.
• constructors / region01 returns only the 12-byte constructor placement records (preferring the loader block; falling back to the region-01 heuristic stream).
• roots / region00 returns only the 24-byte dispatch-root records (preferring the loader block; falling back to the region-00 paired-record scan).

Renderable-record counts for the current validation set (auto mode):

• LSET1/L0.WDL: 2334 total (1182 constructor placements + 1152 dispatch roots).
• LSET4/L37.WDL: 1463 total.

This is now a loader-faithful schema for the two main visible-object lanes. The older count-prefixed region heuristics are kept only as compatibility fallbacks.

Art Binding Rule

v0 now binds art via a loader-faithful DAT_800758d8 parse. For each scene record with typeWord = T:

1. First preference: the bundle installed at DAT_800758d8[T] by the LSET override pass (bundleSource = override-bank-lset).
2. Then: SPEC_A override pass (bundleSource = override-bank-spec-a).
3. Then: LSET artInstall pass (bundleSource = art-install-lset).
4. Then: SPEC_A artInstall pass (bundleSource = art-install-spec-a).
5. Fallback only when no loader block covers the type: raw post_audio_region_04 scan slot (bundleSource = raw-scan).

Mapping sources are recorded per item so failures stay auditable. For the current L0 / L9 / L37 validation runs there are no raw-scan fallbacks; every rendered type resolves through artInstall or override.

The opt-in runtime-map0-masked-proxy mode is retained as a secondary override for research against the runtime map-0 RAM snapshot. It no longer supplies the primary binding.

The older typeWord -> bundle slot scan-order rule is retained only as a named binding mode (raw) for negative-evidence experiments. It is not claimed as executable truth.

When debug labels are enabled for a map render, labels identify unique rendered resources rather than per-instance placements. The stable label key is bundle offset + clamped frame + resolved palette.

Rendering Rule

For each record:

• compute screenX and screenY from the documented projection basis
• select frame index from selectorWord, clamped to available frames
• place sprite top-left at:
  • screenX - originX
  • screenY - originY

Projection sign convention

psx_project_object_main_visible @ 0x80040d44 writes proj_x = Y - X and proj_y = 2*Z - (X + Y) / 2 to obj+0x78/0x7c. The engine's draw step at 0x80040e3c/0x80040e5c then computes screen = (cam - proj) - origin, which flips the sign of the projection relative to canvas Y-down space. For a camera-less full-map export the exporter bakes that flip into projectCtorPlacement, so higher world-Z and smaller (X+Y) sit visibly higher on the output PNG. The same projection is applied to both constructor placements and dispatch-root records; dispatch-root X/Y fields are in world coordinates, not pre-projected screen coordinates, despite the runtime camera +/- 0x140 cull comparing against them directly.

Authored layer semantics

The two authored lanes carry different responsibilities:

• constructors: static level geometry — walls, floors, architecture placed by psx_dispatch_section0_constructor_placements.
• roots: interactive / dynamic objects — crates, terminals, doors, pickups placed by psx_dispatch_section0_dispatch_roots.

Despite the dispatcher name, the roots lane is not the map background; it is the live-object seed list. For the exporter, "constructors" is the geometry layer and "roots" is the object layer.

Painter's order

The exporter sorts items before blitting using the following keys, in order:

1. stage ascending. stage = 1 when typeWord === 4 or laneWord & 0x0400 is set; those overlays draw last.
2. Authored-layer priority: constructors (0) before roots (1). Static geometry draws first so interactive props in the same room do not get hidden behind the floor or wall pieces that occupy the same cell.
3. Isometric depth ascending: back-to-front by world X + Y (isometric ground-depth axis). Falls back to projected screenY when world coordinates are unavailable.
4. World Z ascending within the same ground cell so lower elevations draw before taller objects sharing the same footprint.
5. screenX ascending as a stable tie-breaker.

This is still an approximation of the engine's stage-1 graph order but is closer to what an isometric painter's algorithm would produce than the earlier screenY-only sort.

The rendered PNG uses a neutral opaque background by default so probe silhouettes are legible without relying on transparency.

Color Rule

The exporter resolves palettes entirely from the WDL contents. It does not require any RAM or VRAM dump; those paths are now optional research overrides.

The map-local palette blob lives at headerWords[2] .. headerWords[2] + 0x1000 (4096 bytes = 2048 colors = 128 × 16-entry CLUTs). The blob is what the engine uploads to VRAM rows 0xf0 .. 0xf7 on map load; each VRAM row is 16 CLUTs wide so the 128 CLUTs tile exactly 8 rows of 16 CLUTs.

Resolution rules by bundle mode:

• Mode 2 (4bpp): the bundle header's paletteIndex at +0x14 is the 16-entry CLUT index into the WDL palettes16 bank. When that index points at a sparse/empty CLUT the exporter falls back to a per-bundle palette sweep that picks a CLUT covering the pixel-index set used by the bundle frames.
• Mode 1 (8bpp): the 256-color CLUT is the concatenation of 16 consecutive 16-entry CLUTs from the WDL bank. The bundle's paletteIndex is treated as the starting CLUT index. For the current L0 dataset every mode-1 bundle stores paletteIndex = 0, which is the top-left 256-color bank. Mode-1 color fidelity is therefore approximate until the level-specific 256-CLUT source (suspected to live in the stateBank block) is decoded — tracked as a follow-up.

Transparent pixel index 0 stays transparent during blit regardless of the color value stored at CLUT index 0.

CLI

Primary command:

node src/cli.js --source LSET1/L0.WDL

Supported options:

• --source <relative-path>
• --wdl <absolute-or-relative-file>
• --disc-root <path>
• --binding-mode <raw|runtime-map0-masked-proxy>
• --map-source <auto|combined|layered|constructors|roots|region01|region00>
• --out-name <stem>

Success Criteria

v0 is successful if it can:

• parse a raw LSET*.WDL
• recover the loader-sized section view alongside the region carve
• scan bundles directly from post_audio_region_04
• decode at least one frame from raw data
• extract a stable constructor-placement record set from post_audio_section_00
• write extracted sprite PNGs into .cache
• write a readable diagnostic probe PNG into .output

Planned Follow-Ups

• decode the stateBank and stateBank2 blocks to recover the level-specific 256-color CLUT used by mode-1 sprites. Current mode-1 palettes default to CLUT-bank start 0, which produces plausible colors for some sprites but renders many indoor floor tiles as solid green plates.
• extend sceneInterpretation so it reflects the landed loader-faithful binding instead of the older repeated-wrong-art warning.
• recover the engine's stage-1 graph ordering instead of approximating with isometric (X + Y, Z, screenX) sort keys.
• compare the probe scene against fixed live samples such as map 104 without reintroducing viewer-side donor assumptions.