psx map improvement

psx-map-exporter/docs/spec.md

@@ -33,42 +33,32 @@ It will not claim full runtime parity yet.
 
 Known non-goals for `v0`:
 
-- exact `DAT_800758d8/d0/cc/d4` parity
 - exact CLUT reproduction
 - full stage-1 dependency-graph ordering
-- exact type-to-resource binding for unresolved families
 - full `post_audio_region_01` / `post_audio_region_02` semantic decode
 
+Landed in the current pass (was previously a non-goal):
+
+- loader-faithful `DAT_800758d8` active-header bank binding via explicit parses of the `artInstall` and `override` blocks in both `SPEC_A.WDL` and the map-local `LSET*.WDL`. See `Loader Layout` and `Art Binding Rule` below.
+
 ## Evidence Constraints
 
 The implementation is grounded in these current facts from the docs and Ghidra:
 
-- `LSET*.WDL` uses a fixed `0x38`-byte top-level header.
-- The second dword is the audio/SPU blob size.
-- The old region-only carve is not sufficient on its own for visible-object recovery; loader-sized `post_audio_section_00` contains both the small root-dispatch rows and the dense constructor-placement rows.
-- The file contains a post-audio area with four high-confidence absolute boundaries that split:
-  - `post_audio_region_00`
-  - `post_audio_region_01`
-  - `post_audio_region_02`
-  - `post_audio_region_03`
-  - `post_audio_region_04`
+- `LSET*.WDL` begins with 14 little-endian `u32` size fields (56 bytes total) describing the sequence of post-header blocks. The loader (`wdl_resource_bundle_load_by_index @ 0x80039444`) reads each size and carves the blocks in order: `packPreamble`, `dispatchRootsSize`, `ctorPlacementsSize`, `packTailRewindSize`, `ctorPlacementSection`, `sectionPackBaseSize`, `policyTableSize`, `table8006754cSize`, `opcodeStreamsSize`, `detachedBlobSize`, `artInstallSize`, `stateBankSize`, `overrideSize`, `stateBank2Size`.
+- `SPEC_A.WDL` (global bundle A) begins with a fixed `0x3520`-byte VRAM preload, followed by the same 14-field size header. Bundle A skips the `sectionPack` and `detachedBlob` blocks entirely; the remaining blocks are still present in the same order.
+- The old "audio blob at `word[1]`" model was incorrect for this stream; the first 14 u32 words are block-size descriptors, not a single audio size.
+- The post-audio four-region carve is kept as a fallback diagnostic view but is no longer the primary input for record or art extraction.
 - The small count-prefixed section-0 root-dispatch rows are real, but they are not the whole map object set.
-- The dense constructor-placement records recovered from loader-sized `post_audio_section_00` are currently the best standalone live-object seed source, not a proven final visible-map layer.
-- Current strongest standalone layout read: the constructor-placement lane is a count-prefixed `12`-byte substream inside the loader-sized section-0 span rather than a whole-section `24`-byte row grid. For `LSET1/L0.WDL`, the best current candidate has a section-relative header at `0x38`, a record start at `0x3c`, and a reported count of `1182` records.
-- The constructor-placement stream can extend slightly past the nominal `post_audio_section_00` slice, so standalone parsing must follow the detected stream count from the section-0 base instead of truncating strictly at the section object boundary.
-- `post_audio_region_04` is the strongest current graphics bank candidate.
-- The direct `typeWord -> bundle slot` scan-order binding is disproven as a final art rule and is retained only as a diagnostic bundle-family probe.
-- The real art/template lane is `DAT_800758d8`, but the executable now shows two distinct late art feeds per WDL pass rather than one monolithic bank:
-  - an earlier art-install blob that builds resources and temporarily mirrors them into `DAT_800758d8`
-  - a later `8`-byte header-only override blob that restores raw active-header pointers into `DAT_800758d8`
-- The later header-only override is the safer standalone parser target: constructors branch on first dword `0x58` and then reuse `DAT_800758c8[type]`, so the final post-load `DAT_800758d8` state is a raw-header lane, not a permanently built-resource lane.
-- Type-4/type-5 drawable bundles expose width, height, palette mode/index, frame count, frame table offset, and data offset in the raw bundle header.
+- The dense constructor-placement records recovered from loader-sized `post_audio_section_00` are the live-object seed source for rendering.
+- `post_audio_region_04` is retained only as a fallback bundle source; real art now flows through the `artInstall` and `override` blocks parsed out of the 14-u32 layout.
+- Type-4/type-5 drawable bundles expose width, height, palette mode/index, frame count, frame table offset, and data offset in the raw `0x58`-byte bundle header.
 - Bundle frame entries use a `20`-byte row with size, relative data offset, width, height, origin x/y, and flags.
 - `sprite_rle_decode_rows` uses row-local control bytes:
   - positive: repeat next byte N times
   - negative: copy next `abs(N)` literal bytes
   - zero: end row
-- The executable projection basis is:
+- The executable projection basis (per `psx_project_object_main_visible @ 0x80040d44`) is, in pixel units, with no extra scale factor:
 
 $$
 screen_x = y - x
@@ -78,6 +68,20 @@ $$
 screen_y = 2z - \frac{x + y}{2}
 $$
 
+- Record X/Y values are already in screen-pixel units. The live view-cull box is `camera +/- 0x140` = `+/- 320` pixels, matching PSX screen width. The exporter therefore uses `PSX_SCREEN_SCALE = 1`; earlier builds multiplied by 2, producing over-spaced maps.
+
+## Loader Layout
+
+Both `SPEC_A.WDL` and `LSET*.WDL` are fed to the same loader body, once per WDL pass. Each pass runs two art installs, two state-bank installs, and one override install. The loader reads the 14-u32 size header starting at offset `0` (LSET) or `0x3520` (SPEC_A) and lays out blocks sequentially.
+
+- `artInstall` block (at `0x800396a0` for bundle A, `0x80039988` for bundle B): directory and payloads live at `block + 0x2718`. The first `0x2710` bytes of the block are a scratch header cache used while resources are built. The directory format is `{ u32 count; u32 directoryOffset; }` at the start of the block, then `count` entries of `{ u32 size; u32 typeId; }` at `block + 0x2718 + directoryOffset`. For each non-zero entry the loader installs a built-resource pair `{ u16 kind; u16 _; u32 resource_ptr }` into `DAT_800758d8[typeId]` (`0x18`-byte stride).
+- `override` block (at `0x80039730` for bundle A, `0x80039a18` for bundle B): same directory format, but the payload cursor starts at `block + 8` (directly after the 8-byte prefix). Each non-zero entry payload is a raw `0x58`-byte drawable header whose pointer is written straight into `DAT_800758d8[typeId]` at `0x8003977c` / `0x80039a64`, overwriting whatever the earlier `artInstall` pass installed. Zero-size entries clear the bank slot.
+- Apply order per loader call: SPEC_A `artInstall` → SPEC_A `override` → LSET `artInstall` → LSET `override`. Later writes win, so the final `DAT_800758d8` state is a mix of built-resource pointers and raw override headers.
+
+## Evidence retained for reference
+
+- The direct `typeWord -> bundle slot` scan-order binding is disproven as a final art rule and is retained only as a diagnostic bundle-family probe.
+
 ## Input Model
 
 The exporter accepts either:
@@ -135,64 +139,57 @@ The `.output/<map-stem>.json` manifest inherits `sceneInterpretation` from `wdl-
 
 ## Record Extraction Rules
 
-`v0` now uses the loader-sized `post_audio_section_00` extraction paths as the primary scene source.
+`v0` pulls scene records from two loader-faithful lanes inside the section pack, matching the executable's two dispatch iterators. Both lanes are indexed through `packSubranges` from the 14-u32 loader layout.
 
-Current interpretation constraint:
+### Constructor placements (12-byte stride)
 
-- `section0_constructor_placements` should currently be treated as constructor-fed world-object seed records.
-- They preserve meaningful layout and projection structure, but current evidence does not support treating them as the complete visible map or static architecture layer.
-- If a render shows coherent room layout with globally wrong or repeated art, the exporter is currently visualizing one runtime object lane without the downstream per-type bind/state path and without the separate static-world substrate.
+- Source: `ctorPlacements` pack subrange (word 2).
+- Dispatcher: `psx_dispatch_section0_constructor_placements @ 0x800258cc`.
+- Layout: `[u32 count][count * { u16 typeWord; u16 X; u16 Y; u16 Z; u16 selector; u16 flags }]`.
+- The dispatcher passes each record directly to `descriptor_table[typeWord].slot0(record, 0)` and downstream spawners (e.g. `psx_object_create_compound_record`) read exactly the six u16 fields.
+- Older heuristic region-01 / section-0 scans are retained as compatibility fallbacks when the loader block is absent or empty.
 
-Record extraction rule:
+### Dispatch roots (24-byte stride)
 
-- `auto` / `combined` / `layered` mode merges both authored section-0 families into one layered probe:
-  - constructor placements provide the dense live-object seed lane
-  - root-dispatch rows provide the smaller comparison and auxiliary authored lane
-- `constructors` / `region01` mode first searches the section-0 span for a count-prefixed `12`-byte constructor stream and, when found, treats each record as six little-endian `u16` words:
-  - `typeWord`
-  - `xWord`
-  - `yWord`
-  - `zWord`
-  - `selectorWord`
-  - `laneWord`
-- If a count-prefixed constructor stream is not found, the exporter falls back to the older whole-section `24`-byte paired-record scan as a compatibility probe.
-- `roots` / `region00` mode keeps the small count-prefixed root-dispatch probe for comparison and negative-evidence checks
+- Source: `dispatchRoots` pack subrange (word 1).
+- Dispatcher: `psx_dispatch_section0_dispatch_roots @ 0x800256b0`.
+- Layout per record: `[u32 count]` followed by 24-byte entries whose dispatcher-visible fields are:
+  - `+0x04 u16 typeId`   indexes `psx_type_descriptor_table`
+  - `+0x08 u16 screenX`  used directly by the `+/- 0x140` view-cull
+  - `+0x0A u16 screenY`  same
+  - `+0x10 u16 flags`    bit 3 skips the record
+- Remaining fields are forwarded to descriptor slot 0. The exporter empirically projects `+0x06` as z, `+0x0C` as selector, `+0x0E` as lane, with relaxed plausibility because the live dispatcher only requires the fields above.
 
-Plausibility filter:
+### Selection modes
 
-- `typeWord` in a conservative visible-family range
-- not all coordinate words are zero
-- `laneWord` is non-zero and within the current conservative control-word range
+- `auto` / `combined` / `layered` merges both lanes into one layered probe.
+- `constructors` / `region01` returns only the 12-byte constructor placement records (preferring the loader block; falling back to the region-01 heuristic stream).
+- `roots` / `region00` returns only the 24-byte dispatch-root records (preferring the loader block; falling back to the region-00 paired-record scan).
 
-This is explicitly a probe schema, not a final loader-faithful schema.
+Renderable-record counts for the current validation set (auto mode):
 
-Current negative result:
+- `LSET1/L0.WDL`: 2334 total (1182 constructor placements + 1152 dispatch roots).
+- `LSET4/L37.WDL`: 1463 total.
 
-- Correcting the constructor stream start/count for `LSET1/L0.WDL` only changes the standalone constructor probe slightly (`1130 -> 1135` records, `1090 -> 1095` rendered items) and does not materially change the repeated wrong-art output. Current evidence therefore points to unresolved art/runtime binding as the primary blocker, not a missed constructor-tail decode.
+This is now a loader-faithful schema for the two main visible-object lanes. The older count-prefixed region heuristics are kept only as compatibility fallbacks.
 
 ## Art Binding Rule
 
-`v0` uses one explicit diagnostic binding rule:
+`v0` now binds art via a loader-faithful `DAT_800758d8` parse. For each scene record with `typeWord = T`:
 
-- `typeWord -> bundle slot index`
+1. First preference: the bundle installed at `DAT_800758d8[T]` by the LSET `override` pass (`bundleSource = override-bank-lset`).
+2. Then: SPEC_A `override` pass (`bundleSource = override-bank-spec-a`).
+3. Then: LSET `artInstall` pass (`bundleSource = art-install-lset`).
+4. Then: SPEC_A `artInstall` pass (`bundleSource = art-install-spec-a`).
+5. Fallback only when no loader block covers the type: raw `post_audio_region_04` scan slot (`bundleSource = raw-scan`).
 
-That means the sorted bundle list from `post_audio_region_04` is indexed directly by `typeWord` when the slot exists.
+Mapping sources are recorded per item so failures stay auditable. For the current L0 / L9 / L37 validation runs there are no `raw-scan` fallbacks; every rendered type resolves through `artInstall` or `override`.
 
-This rule is explicitly not claimed as final executable truth. Current docs and Ghidra evidence show the final art path goes through the late `DAT_800758d8` art bank plus downstream state-script/runtime selection. The slot rule remains useful only as a clean standalone negative-evidence probe.
+The opt-in `runtime-map0-masked-proxy` mode is retained as a secondary override for research against the runtime map-0 RAM snapshot. It no longer supplies the primary binding.
 
-For the generic family band now dominating `LSET1/L0` failures (`0x003e`, `0x0042`, `0x0044`, `0x0045`, `0x004f`, `0x0059`, `0x005b`), repeated wrong art is now understood as both a binding failure and a semantic-layer failure: the exporter is currently visualizing constructor-fed runtime object seeds as though they were the final visible world.
+The older `typeWord -> bundle slot` scan-order rule is retained only as a named binding mode (`raw`) for negative-evidence experiments. It is not claimed as executable truth.
 
-The chosen bundle and clamped frame index, plus binding-diversity metrics, are preserved in output metadata so failures stay auditable.
-
-There is now one opt-in experimental binding mode for current map-0 research:
-
-- `runtime-map0-masked-proxy`
-
-That mode reads `.cache/runtime-map0-correlation.json`, takes the live `headerWord11` field from the current map-0 type rows, masks it to `0x0fffff`, and remaps a type only when that masked value lands within a small tolerance of a scanned raw bundle offset with matching kind/mode. All non-matching types still fall back to the raw slot rule.
-
-This is still a probe rule, not claimed final executable truth. It exists to turn the new RAM-backed map-0 correlation into a small, auditable extraction improvement without pretending the full late `DAT_800758d8` bank parse is solved.
-
-When debug labels are enabled for a map render, labels now identify unique rendered resources rather than per-instance placements. The stable label key is currently `bundle offset + clamped frame + resolved palette`. Validation atlas sheets still use progressive cell indices.
+When debug labels are enabled for a map render, labels identify unique rendered resources rather than per-instance placements. The stable label key is `bundle offset + clamped frame + resolved palette`.
 
 ## Rendering Rule
 
@@ -257,8 +254,7 @@ Supported options:
 
 ## Planned Follow-Ups
 
-- replace diagnostic slot binding with a direct parser for the late header-only `DAT_800758d8` override stream and bundle match path
-- recover the exact raw on-disk encoding of the earlier built-resource art-install blob so the two late art feeds are modeled separately instead of flattened into one guessed bank
+- extend `sceneInterpretation` so it reflects the landed loader-faithful binding instead of the older repeated-wrong-art warning
 - identify and parse the separate static-world or subordinate level substrate that complements the constructor-fed live-object lane, instead of treating section-0 constructor placements as the whole map
 - add palette/CLUT reconstruction
 - add stage-1 graph ordering recovery