261 lines
13 KiB
Markdown
261 lines
13 KiB
Markdown
# PSX Map Exporter Spec
|
|
|
|
## Goal
|
|
|
|
`psx-map-exporter` is a standalone Node.js probe for Crusader PSX map extraction.
|
|
|
|
It exists to prove a fresh end-to-end path from raw `LSET*.WDL` input to:
|
|
|
|
- extracted intermediate sprite assets under `.cache`
|
|
- a rendered map PNG under `.output`
|
|
|
|
This project does not reuse `Crusader-Map-Viewer` code, scene caches, donor mappings, or sidecar summaries as binding inputs. It only consumes raw PSX assets plus the documented executable-backed findings from `docs/psx` and the live Ghidra session.
|
|
|
|
## Scope
|
|
|
|
Version `v0` is intentionally narrow.
|
|
|
|
It will:
|
|
|
|
- read one PSX `LSET*.WDL` file
|
|
- parse the documented `0x38`-byte top-level header
|
|
- carve the post-audio map/art regions from header-derived boundaries
|
|
- parse the loader-sized post-audio sections as a second, higher-value view of the file layout
|
|
- extract the dense constructor-placement family from `post_audio_section_00`
|
|
- keep the smaller root-dispatch family available as a comparison probe
|
|
- render a layered authored probe that can combine constructor placements with the smaller root-dispatch lane
|
|
- scan `post_audio_region_04` for type-4/type-5 sprite bundles
|
|
- decode bundle frames directly from the raw WDL
|
|
- write extracted frame PNGs to `.cache`
|
|
- compose a probe map PNG to `.output`
|
|
|
|
It will not claim full runtime parity yet.
|
|
|
|
Known non-goals for `v0`:
|
|
|
|
- exact CLUT reproduction
|
|
- full stage-1 dependency-graph ordering
|
|
- full `post_audio_region_01` / `post_audio_region_02` semantic decode
|
|
|
|
Landed in the current pass (was previously a non-goal):
|
|
|
|
- loader-faithful `DAT_800758d8` active-header bank binding via explicit parses of the `artInstall` and `override` blocks in both `SPEC_A.WDL` and the map-local `LSET*.WDL`. See `Loader Layout` and `Art Binding Rule` below.
|
|
|
|
## Evidence Constraints
|
|
|
|
The implementation is grounded in these current facts from the docs and Ghidra:
|
|
|
|
- `LSET*.WDL` begins with 14 little-endian `u32` size fields (56 bytes total) describing the sequence of post-header blocks. The loader (`wdl_resource_bundle_load_by_index @ 0x80039444`) reads each size and carves the blocks in order: `packPreamble`, `dispatchRootsSize`, `ctorPlacementsSize`, `packTailRewindSize`, `ctorPlacementSection`, `sectionPackBaseSize`, `policyTableSize`, `table8006754cSize`, `opcodeStreamsSize`, `detachedBlobSize`, `artInstallSize`, `stateBankSize`, `overrideSize`, `stateBank2Size`.
|
|
- `SPEC_A.WDL` (global bundle A) begins with a fixed `0x3520`-byte VRAM preload, followed by the same 14-field size header. Bundle A skips the `sectionPack` and `detachedBlob` blocks entirely; the remaining blocks are still present in the same order.
|
|
- The old "audio blob at `word[1]`" model was incorrect for this stream; the first 14 u32 words are block-size descriptors, not a single audio size.
|
|
- The post-audio four-region carve is kept as a fallback diagnostic view but is no longer the primary input for record or art extraction.
|
|
- The small count-prefixed section-0 root-dispatch rows are real, but they are not the whole map object set.
|
|
- The dense constructor-placement records recovered from loader-sized `post_audio_section_00` are the live-object seed source for rendering.
|
|
- `post_audio_region_04` is retained only as a fallback bundle source; real art now flows through the `artInstall` and `override` blocks parsed out of the 14-u32 layout.
|
|
- Type-4/type-5 drawable bundles expose width, height, palette mode/index, frame count, frame table offset, and data offset in the raw `0x58`-byte bundle header.
|
|
- Bundle frame entries use a `20`-byte row with size, relative data offset, width, height, origin x/y, and flags.
|
|
- `sprite_rle_decode_rows` uses row-local control bytes:
|
|
- positive: repeat next byte N times
|
|
- negative: copy next `abs(N)` literal bytes
|
|
- zero: end row
|
|
- The executable projection basis (per `psx_project_object_main_visible @ 0x80040d44`) is, in pixel units, with no extra scale factor:
|
|
|
|
$$
|
|
screen_x = y - x
|
|
$$
|
|
|
|
$$
|
|
screen_y = 2z - \frac{x + y}{2}
|
|
$$
|
|
|
|
- Record X/Y values are already in screen-pixel units. The live view-cull box is `camera +/- 0x140` = `+/- 320` pixels, matching PSX screen width. The exporter therefore uses `PSX_SCREEN_SCALE = 1`; earlier builds multiplied by 2, producing over-spaced maps.
|
|
|
|
## Loader Layout
|
|
|
|
Both `SPEC_A.WDL` and `LSET*.WDL` are fed to the same loader body, once per WDL pass. Each pass runs two art installs, two state-bank installs, and one override install. The loader reads the 14-u32 size header starting at offset `0` (LSET) or `0x3520` (SPEC_A) and lays out blocks sequentially.
|
|
|
|
- `artInstall` block (at `0x800396a0` for bundle A, `0x80039988` for bundle B): directory and payloads live at `block + 0x2718`. The first `0x2710` bytes of the block are a scratch header cache used while resources are built. The directory format is `{ u32 count; u32 directoryOffset; }` at the start of the block, then `count` entries of `{ u32 size; u32 typeId; }` at `block + 0x2718 + directoryOffset`. For each non-zero entry the loader installs a built-resource pair `{ u16 kind; u16 _; u32 resource_ptr }` into `DAT_800758d8[typeId]` (`0x18`-byte stride).
|
|
- `override` block (at `0x80039730` for bundle A, `0x80039a18` for bundle B): same directory format, but the payload cursor starts at `block + 8` (directly after the 8-byte prefix). Each non-zero entry payload is a raw `0x58`-byte drawable header whose pointer is written straight into `DAT_800758d8[typeId]` at `0x8003977c` / `0x80039a64`, overwriting whatever the earlier `artInstall` pass installed. Zero-size entries clear the bank slot.
|
|
- Apply order per loader call: SPEC_A `artInstall` → SPEC_A `override` → LSET `artInstall` → LSET `override`. Later writes win, so the final `DAT_800758d8` state is a mix of built-resource pointers and raw override headers.
|
|
|
|
## Evidence retained for reference
|
|
|
|
- The direct `typeWord -> bundle slot` scan-order binding is disproven as a final art rule and is retained only as a diagnostic bundle-family probe.
|
|
|
|
## Input Model
|
|
|
|
The exporter accepts either:
|
|
|
|
- a direct `--wdl` path
|
|
- or a `--source` path relative to a PSX disc root
|
|
|
|
Default disc root for local workspace runs:
|
|
|
|
- `d:/Ghidra/Crusader-Map-Viewer/map_renderer/STATIC_PSX`
|
|
|
|
Expected source examples:
|
|
|
|
- `LSET1/L0.WDL`
|
|
- `LSET4/L37.WDL`
|
|
|
|
## Output Layout
|
|
|
|
### `.cache`
|
|
|
|
Per-run cache path:
|
|
|
|
- `.cache/<map-stem>/`
|
|
|
|
Contents:
|
|
|
|
- `wdl-summary.json`
|
|
- `records.json`
|
|
- `bundles.json`
|
|
- `frame-manifest.json`
|
|
- `active-header-overrides.json`
|
|
- `sprites/<bundle-offset>/frame_<n>.png`
|
|
|
|
The cache is disposable. It exists to preserve intermediate evidence and make re-runs inspectable.
|
|
|
|
`records.json` now also records constructor-stream detection metadata when available: stream header offset, record start offset, reported count, and the initial structured-prefix run.
|
|
|
|
The cache also records candidate late `DAT_800758d8` header-only override blobs as a standalone diagnostic. Those candidates are not used as final art binding yet.
|
|
|
|
`wdl-summary.json` now also emits `sceneInterpretation`, which is an explicit warning-bearing classification of what the current export most likely represents. For constructor-placement exports this should currently read as a constructor-fed live-object seed lane rather than a final visible-world reconstruction.
|
|
|
|
### `.output`
|
|
|
|
Per-run final outputs:
|
|
|
|
- `.output/<map-stem>.png`
|
|
- `.output/<map-stem>.json`
|
|
- `.output/<map-stem>_<layer>.png` for each rendered authored layer when layered mode is active
|
|
|
|
The JSON stores the final probe scene manifest used to draw the PNG.
|
|
|
|
The `.output` folder is reset at the start of each export so evaluation only sees artifacts from the current run.
|
|
|
|
The `.output/<map-stem>.json` manifest inherits `sceneInterpretation` from `wdl-summary.json` so consumers do not need to infer that warning from prose docs alone.
|
|
|
|
## Record Extraction Rules
|
|
|
|
`v0` pulls scene records from two loader-faithful lanes inside the section pack, matching the executable's two dispatch iterators. Both lanes are indexed through `packSubranges` from the 14-u32 loader layout.
|
|
|
|
### Constructor placements (12-byte stride)
|
|
|
|
- Source: `ctorPlacements` pack subrange (word 2).
|
|
- Dispatcher: `psx_dispatch_section0_constructor_placements @ 0x800258cc`.
|
|
- Layout: `[u32 count][count * { u16 typeWord; u16 X; u16 Y; u16 Z; u16 selector; u16 flags }]`.
|
|
- The dispatcher passes each record directly to `descriptor_table[typeWord].slot0(record, 0)` and downstream spawners (e.g. `psx_object_create_compound_record`) read exactly the six u16 fields.
|
|
- Older heuristic region-01 / section-0 scans are retained as compatibility fallbacks when the loader block is absent or empty.
|
|
|
|
### Dispatch roots (24-byte stride)
|
|
|
|
- Source: `dispatchRoots` pack subrange (word 1).
|
|
- Dispatcher: `psx_dispatch_section0_dispatch_roots @ 0x800256b0`.
|
|
- Layout per record: `[u32 count]` followed by 24-byte entries whose dispatcher-visible fields are:
|
|
- `+0x04 u16 typeId` indexes `psx_type_descriptor_table`
|
|
- `+0x08 u16 screenX` used directly by the `+/- 0x140` view-cull
|
|
- `+0x0A u16 screenY` same
|
|
- `+0x10 u16 flags` bit 3 skips the record
|
|
- Remaining fields are forwarded to descriptor slot 0. The exporter empirically projects `+0x06` as z, `+0x0C` as selector, `+0x0E` as lane, with relaxed plausibility because the live dispatcher only requires the fields above.
|
|
|
|
### Selection modes
|
|
|
|
- `auto` / `combined` / `layered` merges both lanes into one layered probe.
|
|
- `constructors` / `region01` returns only the 12-byte constructor placement records (preferring the loader block; falling back to the region-01 heuristic stream).
|
|
- `roots` / `region00` returns only the 24-byte dispatch-root records (preferring the loader block; falling back to the region-00 paired-record scan).
|
|
|
|
Renderable-record counts for the current validation set (auto mode):
|
|
|
|
- `LSET1/L0.WDL`: 2334 total (1182 constructor placements + 1152 dispatch roots).
|
|
- `LSET4/L37.WDL`: 1463 total.
|
|
|
|
This is now a loader-faithful schema for the two main visible-object lanes. The older count-prefixed region heuristics are kept only as compatibility fallbacks.
|
|
|
|
## Art Binding Rule
|
|
|
|
`v0` now binds art via a loader-faithful `DAT_800758d8` parse. For each scene record with `typeWord = T`:
|
|
|
|
1. First preference: the bundle installed at `DAT_800758d8[T]` by the LSET `override` pass (`bundleSource = override-bank-lset`).
|
|
2. Then: SPEC_A `override` pass (`bundleSource = override-bank-spec-a`).
|
|
3. Then: LSET `artInstall` pass (`bundleSource = art-install-lset`).
|
|
4. Then: SPEC_A `artInstall` pass (`bundleSource = art-install-spec-a`).
|
|
5. Fallback only when no loader block covers the type: raw `post_audio_region_04` scan slot (`bundleSource = raw-scan`).
|
|
|
|
Mapping sources are recorded per item so failures stay auditable. For the current L0 / L9 / L37 validation runs there are no `raw-scan` fallbacks; every rendered type resolves through `artInstall` or `override`.
|
|
|
|
The opt-in `runtime-map0-masked-proxy` mode is retained as a secondary override for research against the runtime map-0 RAM snapshot. It no longer supplies the primary binding.
|
|
|
|
The older `typeWord -> bundle slot` scan-order rule is retained only as a named binding mode (`raw`) for negative-evidence experiments. It is not claimed as executable truth.
|
|
|
|
When debug labels are enabled for a map render, labels identify unique rendered resources rather than per-instance placements. The stable label key is `bundle offset + clamped frame + resolved palette`.
|
|
|
|
## Rendering Rule
|
|
|
|
For each record:
|
|
|
|
- compute `screenX` and `screenY` from the documented projection basis
|
|
- select frame index from `selectorWord`, clamped to available frames
|
|
- place sprite top-left at:
|
|
- `screenX - originX`
|
|
- `screenY - originY`
|
|
|
|
Current draw order is conservative:
|
|
|
|
- main-visible before special-visible
|
|
- then ascending `screenY`
|
|
- then ascending `screenX`
|
|
|
|
This is a probe approximation. The later graph-based stage-1 ordering still belongs to a future pass.
|
|
|
|
The rendered PNG uses a neutral opaque background by default so probe silhouettes are legible without relying on transparency.
|
|
|
|
## Color Rule
|
|
|
|
`v0` emits grayscale art from raw pixel indices.
|
|
|
|
Reason:
|
|
|
|
- bundle frame decode is already well constrained
|
|
- full CLUT parity is not
|
|
- grayscale preserves shape/variant evidence without pretending the palette problem is solved
|
|
|
|
Transparent index `0` stays transparent.
|
|
|
|
## CLI
|
|
|
|
Primary command:
|
|
|
|
```powershell
|
|
node src/cli.js --source LSET1/L0.WDL
|
|
```
|
|
|
|
Supported options:
|
|
|
|
- `--source <relative-path>`
|
|
- `--wdl <absolute-or-relative-file>`
|
|
- `--disc-root <path>`
|
|
- `--binding-mode <raw|runtime-map0-masked-proxy>`
|
|
- `--map-source <auto|combined|layered|constructors|roots|region01|region00>`
|
|
- `--out-name <stem>`
|
|
|
|
## Success Criteria
|
|
|
|
`v0` is successful if it can:
|
|
|
|
- parse a raw `LSET*.WDL`
|
|
- recover the loader-sized section view alongside the region carve
|
|
- scan bundles directly from `post_audio_region_04`
|
|
- decode at least one frame from raw data
|
|
- extract a stable constructor-placement record set from `post_audio_section_00`
|
|
- write extracted sprite PNGs into `.cache`
|
|
- write a readable diagnostic probe PNG into `.output`
|
|
|
|
## Planned Follow-Ups
|
|
|
|
- extend `sceneInterpretation` so it reflects the landed loader-faithful binding instead of the older repeated-wrong-art warning
|
|
- identify and parse the separate static-world or subordinate level substrate that complements the constructor-fed live-object lane, instead of treating section-0 constructor placements as the whole map
|
|
- add palette/CLUT reconstruction
|
|
- add stage-1 graph ordering recovery
|
|
- compare the probe scene against fixed live samples such as `map 104` without reintroducing viewer-side donor assumptions
|