diff --git a/Crusader.rep/idata/01/~0000001b.db/change.data.gbf b/Crusader.rep/idata/01/~0000001b.db/change.data.gbf index 5daeace..dc09744 100644 Binary files a/Crusader.rep/idata/01/~0000001b.db/change.data.gbf and b/Crusader.rep/idata/01/~0000001b.db/change.data.gbf differ diff --git a/Crusader.rep/idata/01/~0000001b.db/change.map.gbf b/Crusader.rep/idata/01/~0000001b.db/change.map.gbf index ce48377..8acd9ec 100644 Binary files a/Crusader.rep/idata/01/~0000001b.db/change.map.gbf and b/Crusader.rep/idata/01/~0000001b.db/change.map.gbf differ diff --git a/Crusader.rep/idata/01/~0000001b.db/db.65.gbf b/Crusader.rep/idata/01/~0000001b.db/db.67.gbf similarity index 98% rename from Crusader.rep/idata/01/~0000001b.db/db.65.gbf rename to Crusader.rep/idata/01/~0000001b.db/db.67.gbf index 39526f3..156e23a 100644 Binary files a/Crusader.rep/idata/01/~0000001b.db/db.65.gbf and b/Crusader.rep/idata/01/~0000001b.db/db.67.gbf differ diff --git a/Crusader.rep/projectState b/Crusader.rep/projectState index 6244408..877ece6 100644 --- a/Crusader.rep/projectState +++ b/Crusader.rep/projectState @@ -3,6 +3,8 @@ + + diff --git a/Crusader.rep/user/00/~0000000d.db/db.9.gbf b/Crusader.rep/user/00/~0000000d.db/db.11.gbf similarity index 99% rename from Crusader.rep/user/00/~0000000d.db/db.9.gbf rename to Crusader.rep/user/00/~0000000d.db/db.11.gbf index a86e3c0..dca17b2 100644 Binary files a/Crusader.rep/user/00/~0000000d.db/db.9.gbf and b/Crusader.rep/user/00/~0000000d.db/db.11.gbf differ diff --git a/psx-map-exporter/docs/spec.md b/psx-map-exporter/docs/spec.md index 2dfef44..dbccc85 100644 --- a/psx-map-exporter/docs/spec.md +++ b/psx-map-exporter/docs/spec.md @@ -33,42 +33,32 @@ It will not claim full runtime parity yet. Known non-goals for `v0`: -- exact `DAT_800758d8/d0/cc/d4` parity - exact CLUT reproduction - full stage-1 dependency-graph ordering -- exact type-to-resource binding for unresolved families - full `post_audio_region_01` / `post_audio_region_02` semantic decode +Landed in the current pass (was previously a non-goal): + +- loader-faithful `DAT_800758d8` active-header bank binding via explicit parses of the `artInstall` and `override` blocks in both `SPEC_A.WDL` and the map-local `LSET*.WDL`. See `Loader Layout` and `Art Binding Rule` below. + ## Evidence Constraints The implementation is grounded in these current facts from the docs and Ghidra: -- `LSET*.WDL` uses a fixed `0x38`-byte top-level header. -- The second dword is the audio/SPU blob size. -- The old region-only carve is not sufficient on its own for visible-object recovery; loader-sized `post_audio_section_00` contains both the small root-dispatch rows and the dense constructor-placement rows. -- The file contains a post-audio area with four high-confidence absolute boundaries that split: - - `post_audio_region_00` - - `post_audio_region_01` - - `post_audio_region_02` - - `post_audio_region_03` - - `post_audio_region_04` +- `LSET*.WDL` begins with 14 little-endian `u32` size fields (56 bytes total) describing the sequence of post-header blocks. The loader (`wdl_resource_bundle_load_by_index @ 0x80039444`) reads each size and carves the blocks in order: `packPreamble`, `dispatchRootsSize`, `ctorPlacementsSize`, `packTailRewindSize`, `ctorPlacementSection`, `sectionPackBaseSize`, `policyTableSize`, `table8006754cSize`, `opcodeStreamsSize`, `detachedBlobSize`, `artInstallSize`, `stateBankSize`, `overrideSize`, `stateBank2Size`. +- `SPEC_A.WDL` (global bundle A) begins with a fixed `0x3520`-byte VRAM preload, followed by the same 14-field size header. Bundle A skips the `sectionPack` and `detachedBlob` blocks entirely; the remaining blocks are still present in the same order. +- The old "audio blob at `word[1]`" model was incorrect for this stream; the first 14 u32 words are block-size descriptors, not a single audio size. +- The post-audio four-region carve is kept as a fallback diagnostic view but is no longer the primary input for record or art extraction. - The small count-prefixed section-0 root-dispatch rows are real, but they are not the whole map object set. -- The dense constructor-placement records recovered from loader-sized `post_audio_section_00` are currently the best standalone live-object seed source, not a proven final visible-map layer. -- Current strongest standalone layout read: the constructor-placement lane is a count-prefixed `12`-byte substream inside the loader-sized section-0 span rather than a whole-section `24`-byte row grid. For `LSET1/L0.WDL`, the best current candidate has a section-relative header at `0x38`, a record start at `0x3c`, and a reported count of `1182` records. -- The constructor-placement stream can extend slightly past the nominal `post_audio_section_00` slice, so standalone parsing must follow the detected stream count from the section-0 base instead of truncating strictly at the section object boundary. -- `post_audio_region_04` is the strongest current graphics bank candidate. -- The direct `typeWord -> bundle slot` scan-order binding is disproven as a final art rule and is retained only as a diagnostic bundle-family probe. -- The real art/template lane is `DAT_800758d8`, but the executable now shows two distinct late art feeds per WDL pass rather than one monolithic bank: - - an earlier art-install blob that builds resources and temporarily mirrors them into `DAT_800758d8` - - a later `8`-byte header-only override blob that restores raw active-header pointers into `DAT_800758d8` -- The later header-only override is the safer standalone parser target: constructors branch on first dword `0x58` and then reuse `DAT_800758c8[type]`, so the final post-load `DAT_800758d8` state is a raw-header lane, not a permanently built-resource lane. -- Type-4/type-5 drawable bundles expose width, height, palette mode/index, frame count, frame table offset, and data offset in the raw bundle header. +- The dense constructor-placement records recovered from loader-sized `post_audio_section_00` are the live-object seed source for rendering. +- `post_audio_region_04` is retained only as a fallback bundle source; real art now flows through the `artInstall` and `override` blocks parsed out of the 14-u32 layout. +- Type-4/type-5 drawable bundles expose width, height, palette mode/index, frame count, frame table offset, and data offset in the raw `0x58`-byte bundle header. - Bundle frame entries use a `20`-byte row with size, relative data offset, width, height, origin x/y, and flags. - `sprite_rle_decode_rows` uses row-local control bytes: - positive: repeat next byte N times - negative: copy next `abs(N)` literal bytes - zero: end row -- The executable projection basis is: +- The executable projection basis (per `psx_project_object_main_visible @ 0x80040d44`) is, in pixel units, with no extra scale factor: $$ screen_x = y - x @@ -78,6 +68,20 @@ $$ screen_y = 2z - \frac{x + y}{2} $$ +- Record X/Y values are already in screen-pixel units. The live view-cull box is `camera +/- 0x140` = `+/- 320` pixels, matching PSX screen width. The exporter therefore uses `PSX_SCREEN_SCALE = 1`; earlier builds multiplied by 2, producing over-spaced maps. + +## Loader Layout + +Both `SPEC_A.WDL` and `LSET*.WDL` are fed to the same loader body, once per WDL pass. Each pass runs two art installs, two state-bank installs, and one override install. The loader reads the 14-u32 size header starting at offset `0` (LSET) or `0x3520` (SPEC_A) and lays out blocks sequentially. + +- `artInstall` block (at `0x800396a0` for bundle A, `0x80039988` for bundle B): directory and payloads live at `block + 0x2718`. The first `0x2710` bytes of the block are a scratch header cache used while resources are built. The directory format is `{ u32 count; u32 directoryOffset; }` at the start of the block, then `count` entries of `{ u32 size; u32 typeId; }` at `block + 0x2718 + directoryOffset`. For each non-zero entry the loader installs a built-resource pair `{ u16 kind; u16 _; u32 resource_ptr }` into `DAT_800758d8[typeId]` (`0x18`-byte stride). +- `override` block (at `0x80039730` for bundle A, `0x80039a18` for bundle B): same directory format, but the payload cursor starts at `block + 8` (directly after the 8-byte prefix). Each non-zero entry payload is a raw `0x58`-byte drawable header whose pointer is written straight into `DAT_800758d8[typeId]` at `0x8003977c` / `0x80039a64`, overwriting whatever the earlier `artInstall` pass installed. Zero-size entries clear the bank slot. +- Apply order per loader call: SPEC_A `artInstall` → SPEC_A `override` → LSET `artInstall` → LSET `override`. Later writes win, so the final `DAT_800758d8` state is a mix of built-resource pointers and raw override headers. + +## Evidence retained for reference + +- The direct `typeWord -> bundle slot` scan-order binding is disproven as a final art rule and is retained only as a diagnostic bundle-family probe. + ## Input Model The exporter accepts either: @@ -135,64 +139,57 @@ The `.output/.json` manifest inherits `sceneInterpretation` from `wdl- ## Record Extraction Rules -`v0` now uses the loader-sized `post_audio_section_00` extraction paths as the primary scene source. +`v0` pulls scene records from two loader-faithful lanes inside the section pack, matching the executable's two dispatch iterators. Both lanes are indexed through `packSubranges` from the 14-u32 loader layout. -Current interpretation constraint: +### Constructor placements (12-byte stride) -- `section0_constructor_placements` should currently be treated as constructor-fed world-object seed records. -- They preserve meaningful layout and projection structure, but current evidence does not support treating them as the complete visible map or static architecture layer. -- If a render shows coherent room layout with globally wrong or repeated art, the exporter is currently visualizing one runtime object lane without the downstream per-type bind/state path and without the separate static-world substrate. +- Source: `ctorPlacements` pack subrange (word 2). +- Dispatcher: `psx_dispatch_section0_constructor_placements @ 0x800258cc`. +- Layout: `[u32 count][count * { u16 typeWord; u16 X; u16 Y; u16 Z; u16 selector; u16 flags }]`. +- The dispatcher passes each record directly to `descriptor_table[typeWord].slot0(record, 0)` and downstream spawners (e.g. `psx_object_create_compound_record`) read exactly the six u16 fields. +- Older heuristic region-01 / section-0 scans are retained as compatibility fallbacks when the loader block is absent or empty. -Record extraction rule: +### Dispatch roots (24-byte stride) -- `auto` / `combined` / `layered` mode merges both authored section-0 families into one layered probe: - - constructor placements provide the dense live-object seed lane - - root-dispatch rows provide the smaller comparison and auxiliary authored lane -- `constructors` / `region01` mode first searches the section-0 span for a count-prefixed `12`-byte constructor stream and, when found, treats each record as six little-endian `u16` words: - - `typeWord` - - `xWord` - - `yWord` - - `zWord` - - `selectorWord` - - `laneWord` -- If a count-prefixed constructor stream is not found, the exporter falls back to the older whole-section `24`-byte paired-record scan as a compatibility probe. -- `roots` / `region00` mode keeps the small count-prefixed root-dispatch probe for comparison and negative-evidence checks +- Source: `dispatchRoots` pack subrange (word 1). +- Dispatcher: `psx_dispatch_section0_dispatch_roots @ 0x800256b0`. +- Layout per record: `[u32 count]` followed by 24-byte entries whose dispatcher-visible fields are: + - `+0x04 u16 typeId` indexes `psx_type_descriptor_table` + - `+0x08 u16 screenX` used directly by the `+/- 0x140` view-cull + - `+0x0A u16 screenY` same + - `+0x10 u16 flags` bit 3 skips the record +- Remaining fields are forwarded to descriptor slot 0. The exporter empirically projects `+0x06` as z, `+0x0C` as selector, `+0x0E` as lane, with relaxed plausibility because the live dispatcher only requires the fields above. -Plausibility filter: +### Selection modes -- `typeWord` in a conservative visible-family range -- not all coordinate words are zero -- `laneWord` is non-zero and within the current conservative control-word range +- `auto` / `combined` / `layered` merges both lanes into one layered probe. +- `constructors` / `region01` returns only the 12-byte constructor placement records (preferring the loader block; falling back to the region-01 heuristic stream). +- `roots` / `region00` returns only the 24-byte dispatch-root records (preferring the loader block; falling back to the region-00 paired-record scan). -This is explicitly a probe schema, not a final loader-faithful schema. +Renderable-record counts for the current validation set (auto mode): -Current negative result: +- `LSET1/L0.WDL`: 2334 total (1182 constructor placements + 1152 dispatch roots). +- `LSET4/L37.WDL`: 1463 total. -- Correcting the constructor stream start/count for `LSET1/L0.WDL` only changes the standalone constructor probe slightly (`1130 -> 1135` records, `1090 -> 1095` rendered items) and does not materially change the repeated wrong-art output. Current evidence therefore points to unresolved art/runtime binding as the primary blocker, not a missed constructor-tail decode. +This is now a loader-faithful schema for the two main visible-object lanes. The older count-prefixed region heuristics are kept only as compatibility fallbacks. ## Art Binding Rule -`v0` uses one explicit diagnostic binding rule: +`v0` now binds art via a loader-faithful `DAT_800758d8` parse. For each scene record with `typeWord = T`: -- `typeWord -> bundle slot index` +1. First preference: the bundle installed at `DAT_800758d8[T]` by the LSET `override` pass (`bundleSource = override-bank-lset`). +2. Then: SPEC_A `override` pass (`bundleSource = override-bank-spec-a`). +3. Then: LSET `artInstall` pass (`bundleSource = art-install-lset`). +4. Then: SPEC_A `artInstall` pass (`bundleSource = art-install-spec-a`). +5. Fallback only when no loader block covers the type: raw `post_audio_region_04` scan slot (`bundleSource = raw-scan`). -That means the sorted bundle list from `post_audio_region_04` is indexed directly by `typeWord` when the slot exists. +Mapping sources are recorded per item so failures stay auditable. For the current L0 / L9 / L37 validation runs there are no `raw-scan` fallbacks; every rendered type resolves through `artInstall` or `override`. -This rule is explicitly not claimed as final executable truth. Current docs and Ghidra evidence show the final art path goes through the late `DAT_800758d8` art bank plus downstream state-script/runtime selection. The slot rule remains useful only as a clean standalone negative-evidence probe. +The opt-in `runtime-map0-masked-proxy` mode is retained as a secondary override for research against the runtime map-0 RAM snapshot. It no longer supplies the primary binding. -For the generic family band now dominating `LSET1/L0` failures (`0x003e`, `0x0042`, `0x0044`, `0x0045`, `0x004f`, `0x0059`, `0x005b`), repeated wrong art is now understood as both a binding failure and a semantic-layer failure: the exporter is currently visualizing constructor-fed runtime object seeds as though they were the final visible world. +The older `typeWord -> bundle slot` scan-order rule is retained only as a named binding mode (`raw`) for negative-evidence experiments. It is not claimed as executable truth. -The chosen bundle and clamped frame index, plus binding-diversity metrics, are preserved in output metadata so failures stay auditable. - -There is now one opt-in experimental binding mode for current map-0 research: - -- `runtime-map0-masked-proxy` - -That mode reads `.cache/runtime-map0-correlation.json`, takes the live `headerWord11` field from the current map-0 type rows, masks it to `0x0fffff`, and remaps a type only when that masked value lands within a small tolerance of a scanned raw bundle offset with matching kind/mode. All non-matching types still fall back to the raw slot rule. - -This is still a probe rule, not claimed final executable truth. It exists to turn the new RAM-backed map-0 correlation into a small, auditable extraction improvement without pretending the full late `DAT_800758d8` bank parse is solved. - -When debug labels are enabled for a map render, labels now identify unique rendered resources rather than per-instance placements. The stable label key is currently `bundle offset + clamped frame + resolved palette`. Validation atlas sheets still use progressive cell indices. +When debug labels are enabled for a map render, labels identify unique rendered resources rather than per-instance placements. The stable label key is `bundle offset + clamped frame + resolved palette`. ## Rendering Rule @@ -257,8 +254,7 @@ Supported options: ## Planned Follow-Ups -- replace diagnostic slot binding with a direct parser for the late header-only `DAT_800758d8` override stream and bundle match path -- recover the exact raw on-disk encoding of the earlier built-resource art-install blob so the two late art feeds are modeled separately instead of flattened into one guessed bank +- extend `sceneInterpretation` so it reflects the landed loader-faithful binding instead of the older repeated-wrong-art warning - identify and parse the separate static-world or subordinate level substrate that complements the constructor-fed live-object lane, instead of treating section-0 constructor placements as the whole map - add palette/CLUT reconstruction - add stage-1 graph ordering recovery diff --git a/psx-map-exporter/src/bundles.js b/psx-map-exporter/src/bundles.js index fe47551..7c8bb16 100644 --- a/psx-map-exporter/src/bundles.js +++ b/psx-map-exporter/src/bundles.js @@ -231,96 +231,119 @@ export function scanSpriteBundles(region) { continue; } - const kind = readU32LE(region.buffer, offset + 0x00); - const width = readU32LE(region.buffer, offset + 0x08); - const height = readU32LE(region.buffer, offset + 0x0c); - const mode = readU32LE(region.buffer, offset + 0x10); - const paletteIndex = readU32LE(region.buffer, offset + 0x14); - const dataOffset = readU32LE(region.buffer, offset + 0x1c); - const frameCount = readU32LE(region.buffer, offset + 0x20); - const frameTableOffset = 0x34; - - if (paletteIndex > 127) { + const payloadBuffer = region.buffer.subarray(offset); + const bundle = extractBundleFromHeader(payloadBuffer, region.offset + offset); + if (!bundle) { continue; } - - const frames = []; - let valid = true; - - for (let index = 0; index < frameCount; index += 1) { - const entryOffset = offset + frameTableOffset + (index * 20); - const flags = readU32LE(region.buffer, entryOffset + 0x00); - const relativeDataOffset = readU32LE(region.buffer, entryOffset + 0x08); - const frameWidth = readU16LE(region.buffer, entryOffset + 0x0c); - const frameHeight = readU16LE(region.buffer, entryOffset + 0x0e); - const originX = readU16LE(region.buffer, entryOffset + 0x10); - const originY = readU16LE(region.buffer, entryOffset + 0x12); - - const dataStart = offset + dataOffset + (((flags & 1) === 1) ? relativeDataOffset * 4 : relativeDataOffset); - const rawSize = rowByteWidth(frameWidth, mode) * frameHeight; - if ( - frameWidth === 0 || - frameHeight === 0 || - frameWidth > 512 || - frameHeight > 512 || - dataStart >= region.buffer.length || - (((flags & 1) === 0) && (dataStart + rawSize > region.buffer.length)) - ) { - valid = false; - break; - } - - let consumed; - if ((flags & 1) === 1) { - const decoded = decodeRleRows(region.buffer, dataStart, frameWidth, frameHeight, mode); - if (!decoded) { - valid = false; - break; - } - consumed = decoded.consumed; - } else { - consumed = rawSize; - } - - frames.push({ - index, - consumed, - relativeDataOffset, - width: frameWidth, - height: frameHeight, - originX, - originY, - flags, - dataStart, - absoluteDataStart: region.offset + dataStart, - }); - } - - if (!valid) { - continue; - } - - seenRanges.push([offset, offset + dataOffset]); - - bundles.push({ - slot: bundles.length, - offsetInRegion: offset, - absoluteOffset: region.offset + offset, - kind, - width, - height, - mode, - paletteIndex, - dataOffset, - frameCount, - frameTableOffset, - frames, - }); + bundle.slot = bundles.length; + bundle.offsetInRegion = offset; + bundle.sourceBuffer = payloadBuffer; + bundle.bindingSource = bundle.bindingSource ?? 'raw-scan'; + seenRanges.push([offset, offset + bundle.dataOffset]); + bundles.push(bundle); } return bundles; } +// Parses a single kind-4/kind-5 drawable-header bundle starting at offset 0 of +// `payloadBuffer`. Returns null if the header is not a plausible bundle. +// +// `absoluteHeaderOffset` is the file-level offset of the header, used to tag +// the returned bundle and its frames for cache metadata / diagnostics. Frame +// decode later needs the same `payloadBuffer` (or a region whose `buffer` +// slice starts at the header) so that `frame.dataStart` lands correctly. +export function extractBundleFromHeader(payloadBuffer, absoluteHeaderOffset) { + if (!payloadBuffer || payloadBuffer.length < 0x34) { + return null; + } + if (!isValidBundleHeader(payloadBuffer, 0)) { + return null; + } + + const kind = readU32LE(payloadBuffer, 0x00); + const width = readU32LE(payloadBuffer, 0x08); + const height = readU32LE(payloadBuffer, 0x0c); + const mode = readU32LE(payloadBuffer, 0x10); + const paletteIndex = readU32LE(payloadBuffer, 0x14); + const dataOffset = readU32LE(payloadBuffer, 0x1c); + const frameCount = readU32LE(payloadBuffer, 0x20); + const frameTableOffset = 0x34; + + if (paletteIndex > 127) { + return null; + } + + const frames = []; + for (let index = 0; index < frameCount; index += 1) { + const entryOffset = frameTableOffset + (index * 20); + if (entryOffset + 20 > payloadBuffer.length) { + return null; + } + + const flags = readU32LE(payloadBuffer, entryOffset + 0x00); + const relativeDataOffset = readU32LE(payloadBuffer, entryOffset + 0x08); + const frameWidth = readU16LE(payloadBuffer, entryOffset + 0x0c); + const frameHeight = readU16LE(payloadBuffer, entryOffset + 0x0e); + const originX = readU16LE(payloadBuffer, entryOffset + 0x10); + const originY = readU16LE(payloadBuffer, entryOffset + 0x12); + + const dataStart = dataOffset + (((flags & 1) === 1) ? relativeDataOffset * 4 : relativeDataOffset); + const rawSize = rowByteWidth(frameWidth, mode) * frameHeight; + + if ( + frameWidth === 0 + || frameHeight === 0 + || frameWidth > 512 + || frameHeight > 512 + || dataStart >= payloadBuffer.length + || (((flags & 1) === 0) && (dataStart + rawSize > payloadBuffer.length)) + ) { + return null; + } + + let consumed; + if ((flags & 1) === 1) { + const decoded = decodeRleRows(payloadBuffer, dataStart, frameWidth, frameHeight, mode); + if (!decoded) { + return null; + } + consumed = decoded.consumed; + } else { + consumed = rawSize; + } + + frames.push({ + index, + consumed, + relativeDataOffset, + width: frameWidth, + height: frameHeight, + originX, + originY, + flags, + dataStart, + absoluteDataStart: absoluteHeaderOffset + dataStart, + }); + } + + return { + slot: 0, + offsetInRegion: 0, + absoluteOffset: absoluteHeaderOffset, + kind, + width, + height, + mode, + paletteIndex, + dataOffset, + frameCount, + frameTableOffset, + frames, + }; +} + function decodeRleRows(buffer, start, width, height, mode) { const expectedSize = rowByteWidth(width, mode) * height; const output = []; @@ -434,21 +457,28 @@ function indexedToColorRgba(pixels, palette) { export function decodeBundleFrame(region, bundle, frameIndex, palette = null) { const frame = bundle.frames[Math.max(0, Math.min(frameIndex, bundle.frames.length - 1))]; const rawSize = rowByteWidth(frame.width, bundle.mode) * frame.height; + // Bundle-local source buffer when present (override-bank bundles carry their + // own payload slice); fall back to the supplied region buffer for bundles + // recovered via raw scan of post_audio_region_04. + const sourceBuffer = bundle.sourceBuffer ?? region?.buffer; + if (!sourceBuffer) { + throw new Error('decodeBundleFrame requires a bundle.sourceBuffer or region.buffer'); + } let rawPixels; let consumed; if ((frame.flags & 1) === 1) { - const decoded = decodeRleRows(region.buffer, frame.dataStart, frame.width, frame.height, bundle.mode); + const decoded = decodeRleRows(sourceBuffer, frame.dataStart, frame.width, frame.height, bundle.mode); if (!decoded) { throw new Error(`Failed to decode RLE frame at 0x${frame.absoluteDataStart.toString(16)}`); } rawPixels = decoded.rawPixels; consumed = decoded.consumed; } else { - if (frame.dataStart + rawSize > region.buffer.length) { - throw new Error(`Frame overruns bundle region at 0x${frame.absoluteDataStart.toString(16)}`); + if (frame.dataStart + rawSize > sourceBuffer.length) { + throw new Error(`Frame overruns source buffer at 0x${frame.absoluteDataStart.toString(16)}`); } - rawPixels = region.buffer.subarray(frame.dataStart, frame.dataStart + rawSize); + rawPixels = sourceBuffer.subarray(frame.dataStart, frame.dataStart + rawSize); consumed = rawSize; } diff --git a/psx-map-exporter/src/cli.js b/psx-map-exporter/src/cli.js index fb1f502..101a870 100644 --- a/psx-map-exporter/src/cli.js +++ b/psx-map-exporter/src/cli.js @@ -139,6 +139,7 @@ async function main() { projectRoot, wdlPath, sourceRelPath: options.source, + discRoot: options.discRoot, mapSource: options.mapSource, bindingMode: options.bindingMode, sceneScope: options.sceneScope, diff --git a/psx-map-exporter/src/export-map.js b/psx-map-exporter/src/export-map.js index 9497bc1..49a7455 100644 --- a/psx-map-exporter/src/export-map.js +++ b/psx-map-exporter/src/export-map.js @@ -9,8 +9,9 @@ import { extractPaletteSets, } from './bundles.js'; import { decodeBundleFrame, encodePng, scanSpriteBundles } from './bundles.js'; +import { buildOverrideBundleBindings, parseOverrideBank } from './override-bank.js'; import { renderMap } from './render.js'; -import { parseLsetWdl, parseRegion00Records, parseRegion01Records, summarizeRegion02 } from './wdl.js'; +import { parseCtorPlacementsBlock, parseDispatchRootsBlock, parseLoaderLayout, parseLsetWdl, parseRegion00Records, parseRegion01Records, summarizeRegion02 } from './wdl.js'; const DEFAULT_BACKGROUND = { red: 18, green: 18, blue: 18, alpha: 255 }; @@ -85,17 +86,38 @@ function chooseRecordSet(wdl, mapSource) { ? parseRegion01Records(region01Source) : { source: 'region01', recordStartOffset: 0, records: [] }; + // Loader-faithful paths: pull ctorPlacements and dispatchRoots directly from + // the packSubranges in the 14-u32 loader header. When present these override + // the heuristic region scans because they match the executable's dispatch + // iterators 1:1. + const packSubranges = wdl.loaderLayout?.packSubranges ?? []; + const ctorBlock = packSubranges.find((block) => block.name === 'ctorPlacements'); + const dispatchBlock = packSubranges.find((block) => block.name === 'dispatchRoots'); + const ctorPlacementsLoader = ctorBlock + ? parseCtorPlacementsBlock(ctorBlock) + : { source: 'ctorPlacements', records: [] }; + const dispatchRootsLoader = dispatchBlock + ? parseDispatchRootsBlock(dispatchBlock) + : { source: 'dispatchRoots', records: [] }; + + const constructorRecordSet = ctorPlacementsLoader.records.length > 0 + ? ctorPlacementsLoader + : region01Records; + const rootRecordSet = dispatchRootsLoader.records.length > 0 + ? dispatchRootsLoader + : region00Records; + if (mapSource === 'region00' || mapSource === 'roots') { - return region00Records; + return rootRecordSet; } if (mapSource === 'region01' || mapSource === 'constructors') { - return region01Records; + return constructorRecordSet; } if (mapSource === 'combined' || mapSource === 'layered' || mapSource === 'auto') { const records = [ - ...region01Records.records.map((record) => ({ ...record, authoredLayer: 'constructors' })), - ...region00Records.records.map((record) => ({ ...record, authoredLayer: 'roots' })), + ...constructorRecordSet.records.map((record) => ({ ...record, authoredLayer: record.authoredLayer ?? 'constructors' })), + ...rootRecordSet.records.map((record) => ({ ...record, authoredLayer: record.authoredLayer ?? 'roots' })), ]; return { @@ -103,13 +125,25 @@ function chooseRecordSet(wdl, mapSource) { recordStartOffset: 0, records, layers: { - constructors: region01Records.records.length, - roots: region00Records.records.length, + constructors: constructorRecordSet.records.length, + roots: rootRecordSet.records.length, + }, + loaderSources: { + ctorPlacements: { + used: ctorPlacementsLoader.records.length > 0, + count: ctorPlacementsLoader.records.length, + reportedCount: ctorPlacementsLoader.reportedCount ?? null, + }, + dispatchRoots: { + used: dispatchRootsLoader.records.length > 0, + count: dispatchRootsLoader.records.length, + reportedCount: dispatchRootsLoader.reportedCount ?? null, + }, }, }; } - return region01Records.records.length >= 8 ? region01Records : region00Records; + return constructorRecordSet.records.length >= 8 ? constructorRecordSet : rootRecordSet; } function chooseBundleForType(bundles, typeWord) { @@ -127,6 +161,146 @@ async function loadJsonIfExists(filePath) { } } +async function loadOverrideBankBindingsForBuffer(buffer, layout, variant, paletteSets, options = {}) { + if (!layout) { + return { + bundles: [], + byType: new Map(), + overrideBank: null, + artInstallBank: null, + failedEntries: [], + }; + } + const overrideBlock = layout.blocksByName.get('override'); + const artInstallBlock = layout.blocksByName.get('artInstall'); + + const result = { + bundles: [], + byType: new Map(), + overrideBank: null, + artInstallBank: null, + failedEntries: [], + }; + + // Parse art-install first: it provides the initial per-type drawable bindings + // for both kind-4 and kind-5 resources. The later override pass then + // replaces some slots with raw-header pointers. + if (artInstallBlock && artInstallBlock.buffer) { + const bank = parseOverrideBank(artInstallBlock, { + variant, + payloadBase: 0x2718, + }); + result.artInstallBank = bank; + if (bank.valid) { + const { bundles: raw, failedEntries } = buildOverrideBundleBindings(bank, { buffer }); + const tagged = raw.map((bundle) => ({ + ...bundle, + bundleSource: variant === 'spec_a' ? 'art-install-spec-a' : 'art-install-lset', + })); + const resolved = resolveBundlePalettes(tagged, paletteSets, { + mode1RuntimePalette: options.mode1RuntimePalette, + }); + for (const bundle of resolved) { + if (!bundle.sourceBuffer && Number.isInteger(bundle.absoluteOffset)) { + bundle.sourceBuffer = buffer.subarray(bundle.absoluteOffset); + } + result.byType.set(bundle.typeId, bundle); + result.bundles.push(bundle); + } + result.failedEntries.push(...failedEntries); + } + } + + if (overrideBlock && overrideBlock.buffer) { + const bank = parseOverrideBank(overrideBlock, { variant, payloadBase: 8 }); + result.overrideBank = bank; + if (bank.valid) { + const { bundles: raw, failedEntries } = buildOverrideBundleBindings(bank, { buffer }); + const tagged = raw.map((bundle) => ({ + ...bundle, + bundleSource: variant === 'spec_a' ? 'override-bank-spec-a' : 'override-bank-lset', + })); + const resolved = resolveBundlePalettes(tagged, paletteSets, { + mode1RuntimePalette: options.mode1RuntimePalette, + }); + for (const bundle of resolved) { + if (!bundle.sourceBuffer && Number.isInteger(bundle.absoluteOffset)) { + bundle.sourceBuffer = buffer.subarray(bundle.absoluteOffset); + } + // Late override wins over earlier art-install for the same type, + // matching the loader's bank-slot overwrite behaviour. + result.byType.set(bundle.typeId, bundle); + result.bundles.push(bundle); + } + result.failedEntries.push(...failedEntries); + } + } + + return result; +} + +async function loadSpecAOverrideBank(wdlPath, discRoot, paletteSets, options = {}) { + const candidatePaths = []; + if (discRoot) { + candidatePaths.push(path.join(discRoot, 'SPEC_A.WDL')); + } + const wdlDir = path.dirname(wdlPath); + candidatePaths.push(path.resolve(wdlDir, '..', 'SPEC_A.WDL')); + candidatePaths.push(path.resolve(wdlDir, 'SPEC_A.WDL')); + + for (const candidate of candidatePaths) { + try { + const buffer = await fs.readFile(candidate); + const layout = parseLoaderLayout(buffer, { variant: 'spec_a' }); + const result = await loadOverrideBankBindingsForBuffer( + buffer, + layout, + 'spec_a', + paletteSets, + options, + ); + result.sourcePath = candidate; + return result; + } catch { + // try next candidate + } + } + + return { bundles: [], byType: new Map(), overrideBank: null, failedEntries: [], sourcePath: null }; +} + +async function loadOverrideBankBindings(lsetWdl, wdlPath, discRoot, paletteSets, options = {}) { + // SPEC_A.WDL provides the base (bundle A) override table; the map-local + // LSET*.WDL provides the override B table. The loader runs SPEC_A first + // then LSET, and for matching typeIds LSET wins because it overwrites the + // bank slot. We merge in the same order here. + const specAResult = await loadSpecAOverrideBank(wdlPath, discRoot, paletteSets, options); + const lsetLayout = lsetWdl.loaderLayout; + const lsetResult = await loadOverrideBankBindingsForBuffer( + lsetWdl.buffer, + lsetLayout, + 'lset', + paletteSets, + options, + ); + + const bundles = [...specAResult.bundles, ...lsetResult.bundles]; + const byType = new Map(); + for (const bundle of specAResult.bundles) { + byType.set(bundle.typeId, bundle); + } + // LSET wins over SPEC_A for identical typeIds (matches loader behaviour). + for (const bundle of lsetResult.bundles) { + byType.set(bundle.typeId, bundle); + } + return { + bundles, + byType, + specA: specAResult, + lset: lsetResult, + }; +} + function buildRuntimeMap0MaskedBindings(correlation, bundles, tolerance = 0x200) { const byType = new Map(); const diagnostics = []; @@ -226,6 +400,23 @@ function buildRuntimeMap0MaskedBindings(correlation, bundles, tolerance = 0x200) } function chooseBundleBinding(record, bundles, options = {}) { + // Primary path: per-type override-bank binding. The late header-only override + // block of each WDL pass (SPEC_A.WDL followed by the map-local LSET*.WDL) + // writes raw 0x58-byte active-header pointers into + // `psx_type_art_active_header_bank[type]`. Constructors and the render + // submitters read that bank at runtime, so an override-bank bundle IS the + // drawable resource bound to that type for this level. + const overrideBundle = options.overrideBundlesByType?.get(record.typeWord) ?? null; + if (overrideBundle) { + return { + bundle: overrideBundle, + mappingSource: overrideBundle.overrideVariant === 'spec_a' + ? 'override-bank-spec-a' + : 'override-bank-lset', + runtimeBinding: null, + }; + } + if (options.bindingMode === 'runtime-map0-masked-proxy') { const runtimeBinding = options.runtimeMap0Bindings?.get(record.typeWord) ?? null; if (runtimeBinding?.bundle) { @@ -233,6 +424,13 @@ function chooseBundleBinding(record, bundles, options = {}) { } } + if (options.bindingMode === 'override-bank') { + // Strict mode: refuse to fall back to the diagnostic slot rule. Types with + // no override binding get dropped from the scene so the render reflects + // only executable-backed art. + return null; + } + const rawTypeBundle = chooseBundleForType(bundles, record.typeWord); if (!rawTypeBundle) { return null; @@ -1033,6 +1231,20 @@ export async function exportMap(options) { { mode1RuntimePalette } ); + // Override-bank bindings: per-type drawable-header pointers installed by the + // late override pass in wdl_resource_bundle_load_by_index. This is the same + // runtime state constructors and render submitters read when resolving art, + // so it gives executable-backed typeWord -> bundle truth for the bundle A + // (SPEC_A.WDL) and bundle B (map-local LSET WDL) passes combined. + const overrideBankResult = await loadOverrideBankBindings( + wdl, + options.wdlPath, + options.discRoot ?? null, + paletteSets, + { mode1RuntimePalette }, + ); + bundles = [...bundles, ...overrideBankResult.bundles]; + const runtimeMap0Correlation = options.bindingMode === 'runtime-map0-masked-proxy' ? await loadJsonIfExists(path.join(cacheBaseRoot, 'runtime-map0-correlation.json')) : null; @@ -1044,6 +1256,7 @@ export async function exportMap(options) { mode1RuntimePalette, mode1PaletteBank, runtimeMap0Bindings: runtimeMap0BindingResult.byType, + overrideBundlesByType: overrideBankResult.byType, }); const bindingDiversity = summarizeBindingDiversity(sceneItems); const authoredLayerSummary = summarizeAuthoredLayers(recordSet.records); diff --git a/psx-map-exporter/src/override-bank.js b/psx-map-exporter/src/override-bank.js new file mode 100644 index 0000000..39cfc30 --- /dev/null +++ b/psx-map-exporter/src/override-bank.js @@ -0,0 +1,164 @@ +// Parser for the late-override per-type active-header block. +// +// This block is emitted twice per loader pass (bundle A from SPEC_A.WDL and +// bundle B from the map-local LSET*.WDL). After the earlier art-install pass +// builds kind-4/kind-5 built resources and mirrors them into +// `psx_type_art_active_header_bank`, this block REPLACES the active-header +// slot for each type with a raw pointer to a self-describing 0x58-byte +// drawable descriptor that lives inside the block itself. +// +// Constructors, object state-script and render submitters all read +// `psx_type_art_active_header_bank[type]` at runtime, so parsing this block +// gives us the definitive typeWord -> drawable-header binding. +// +// Block layout: +// u32 count +// u32 directoryOffset (from payload base = block+8 to directory) +// byte payload[ directoryOffset ] (concatenated 0x58-byte headers, variable sz) +// { u32 size; u32 typeId; } directory[count] +// +// A directory entry with size == 0 clears the slot (writes 0 to the bank). +// Non-zero entries emit `payloadCursor` as the active-header pointer for +// `typeId` and advance the cursor by `size`. + +import { extractBundleFromHeader } from './bundles.js'; + +function readU32LE(buffer, offset) { + return buffer.readUInt32LE(offset); +} + +export function parseOverrideBank(overrideBlock, options = {}) { + if (!overrideBlock || !overrideBlock.buffer || overrideBlock.size < 8) { + return { + valid: false, + reason: 'missing-or-undersized-block', + entries: [], + }; + } + + const buffer = overrideBlock.buffer; + const blockOffset = overrideBlock.offset; + const count = readU32LE(buffer, 0); + const directoryOffset = readU32LE(buffer, 4); + + if (count === 0 || count > 0x400) { + return { valid: false, reason: 'implausible-count', count, entries: [] }; + } + + // Override block layout uses payloadBase = 8 (directly after the 8-byte + // prefix). Art-install blocks use payloadBase = 0x2718 because the loader + // reserves a fixed 0x2710-byte header-cache area before payloads start. + // Both variants store the directory at `payloadBase + directoryOffset` and + // describe payloads as { u32 size; u32 typeId; } pairs. + const payloadBase = options.payloadBase ?? 8; + const directoryBase = payloadBase + directoryOffset; + if (directoryBase + count * 8 > buffer.length) { + return { valid: false, reason: 'directory-overrun', count, directoryOffset, payloadBase, entries: [] }; + } + + const entries = []; + let payloadCursor = payloadBase; + let nonZeroCount = 0; + let clearCount = 0; + + for (let index = 0; index < count; index += 1) { + const size = readU32LE(buffer, directoryBase + index * 8 + 0); + const typeId = readU32LE(buffer, directoryBase + index * 8 + 4); + + if (size === 0) { + entries.push({ index, typeId, size: 0, payloadOffset: null, kind: 'clear' }); + clearCount += 1; + continue; + } + + if (payloadCursor + size > directoryBase) { + return { + valid: false, + reason: 'payload-overruns-directory', + count, + directoryOffset, + entries, + truncatedAt: index, + }; + } + + entries.push({ + index, + typeId, + size, + payloadOffset: payloadCursor, + absolutePayloadOffset: blockOffset + payloadCursor, + kind: 'install', + }); + payloadCursor += size; + nonZeroCount += 1; + } + + return { + valid: true, + count, + directoryOffset, + directoryBase, + payloadBase, + directoryAbsoluteOffset: blockOffset + directoryBase, + nonZeroCount, + clearCount, + payloadEnd: payloadCursor, + entries, + overrideBlockOffset: blockOffset, + overrideBlockSize: overrideBlock.size, + variant: options.variant ?? 'unknown', + }; +} + +// For each installed entry, treat its payload as a single drawable-header +// bundle (the 0x58-byte descriptor + embedded frame table + embedded data). +// Returns a map from typeId to a fully-hydrated bundle with `frames[]`. +export function buildOverrideBundleBindings(overrideBank, options = {}) { + if (!overrideBank || !overrideBank.valid) { + return { bundles: [], byType: new Map(), failedEntries: [] }; + } + + const buffer = options.buffer; + if (!buffer) { + throw new Error('buildOverrideBundleBindings requires { buffer } of the full WDL file'); + } + + const bundles = []; + const byType = new Map(); + const failedEntries = []; + + for (const entry of overrideBank.entries) { + if (entry.kind !== 'install') { + continue; + } + + const absoluteHeaderOffset = entry.absolutePayloadOffset; + const payloadLimit = Math.min(entry.size, buffer.length - absoluteHeaderOffset); + if (payloadLimit < 0x34) { + failedEntries.push({ ...entry, reason: 'payload-too-small-for-header' }); + continue; + } + + const payloadBuffer = buffer.subarray(absoluteHeaderOffset, absoluteHeaderOffset + payloadLimit); + const bundle = extractBundleFromHeader(payloadBuffer, absoluteHeaderOffset); + if (!bundle) { + failedEntries.push({ ...entry, reason: 'invalid-bundle-header' }); + continue; + } + + bundle.bindingSource = 'override-bank'; + bundle.overrideVariant = overrideBank.variant; + bundle.typeId = entry.typeId; + bundle.overrideEntryIndex = entry.index; + bundle.slot = entry.typeId; + bundle.sourceBuffer = payloadBuffer; + bundles.push(bundle); + + // Later bundle pass (override B) wins over earlier (override A) for the + // same type; the loader mirrors this exactly by overwriting the bank slot. + byType.set(entry.typeId, bundle); + } + + return { bundles, byType, failedEntries }; +} diff --git a/psx-map-exporter/src/wdl.js b/psx-map-exporter/src/wdl.js index 8df3350..25b6ecb 100644 --- a/psx-map-exporter/src/wdl.js +++ b/psx-map-exporter/src/wdl.js @@ -9,12 +9,150 @@ function readU16LE(buffer, offset) { } const ALLOWED_LANE_WORDS = new Set([0x20, 0x22, 0x30]); -const PSX_SCREEN_SCALE = 2; +// Loader-faithful projection: per `psx_project_object_main_visible` the +// executable computes `screenX = Y - X` and `screenY = 2*Z - (X+Y)/2` in pixel +// units with no extra scale (the view-cull box is +/-0x140 = +/-320 pixels, +// matching PSX screen width). Keeping an explicit constant at 1 documents the +// 1:1 world-to-screen mapping and leaves room for experimentation. +const PSX_SCREEN_SCALE = 1; function uniqueSorted(values) { return [...new Set(values)].sort((left, right) => left - right); } +// Loader-faithful layout per wdl_resource_bundle_load_by_index @ 0x80039444. +// The 0x38-byte header at the start of an LSET*.WDL file contains 14 u32 size +// words that parameterize every downstream read. Names follow the decompiled +// local variable order (local_c8 is read first into file offset 0x00). +const LOADER_SIZE_FIELDS = [ + 'packPreamble', // +0x00 local_c8 - bytes consumed before dispatch roots inside the section pack + 'dispatchRootsSize', // +0x04 local_c4 - section-0 root dispatch rows + 'ctorPlacementsSize', // +0x08 local_c0 - section-0 constructor placement records (12-byte stride) + 'packTailRewindSize', // +0x0c local_bc - trailing bytes in pack that are heap-rewound after load + 'ctorPlacementSection',// +0x10 local_b8 - psx_ctor_placement_section_ptr block + 'sectionPackBaseSize', // +0x14 local_b4 - psx_level_section_pack_base to psx_type_policy_table_ptr + 'policyTableSize', // +0x18 local_b0 - psx_type_policy_table_ptr block (per-type policy bits) + 'table8006754cSize', // +0x1c local_ac - auxiliary table between policy and opcode streams + 'opcodeStreamsSize', // +0x20 local_a8 - psx_control_opcode_stream_table to psx_level_clut_table_ptr + 'detachedBlobSize', // +0x24 local_a4 - psx_level_detached_blob (audio runtime stream) + 'artInstallSize', // +0x28 local_a0 - bundle B: per-type active-header + built-resource install + 'stateBankSize', // +0x2c local_9c - state-script/component/extents bank install + 'overrideSize', // +0x30 local_98 - late header-only per-type active-header override stream + 'stateBank2Size', // +0x34 local_94 - second state-script/component/extents bank install +]; + +// Layout variant for SPEC_A.WDL: the first 0x3520 bytes are a VRAM image pair +// uploaded on first boot; the 0x38-byte loader header starts at offset 0x3520. +// Only the last five size words (a4/a0/9c/98/94) matter — the c8..a8 slots are +// leftover/undefined bytes in the SPEC_A header because it skips the section +// pack entirely and goes straight to bundle A. +const SPEC_A_VRAM_PRELOAD_SIZE = 0x3520; + +export function parseLoaderLayout(buffer, options = {}) { + const variant = options.variant ?? 'lset'; + const headerOffset = variant === 'spec_a' ? SPEC_A_VRAM_PRELOAD_SIZE : 0; + if (buffer.length < headerOffset + 0x38) { + throw new Error(`Buffer too small for loader header at 0x${headerOffset.toString(16)}`); + } + + const sizes = {}; + LOADER_SIZE_FIELDS.forEach((name, index) => { + sizes[name] = readU32LE(buffer, headerOffset + index * 4); + }); + + const blocks = []; + let cursor = headerOffset + 0x38; + const addBlock = (name, size) => { + const entry = { + name, + offset: cursor, + size, + end: cursor + size, + buffer: size > 0 && cursor + size <= buffer.length + ? buffer.subarray(cursor, cursor + size) + : null, + }; + blocks.push(entry); + cursor += size; + return entry; + }; + + if (variant !== 'spec_a') { + // Section pack = sum of c8..a8 read as a single contiguous CD read, then the + // loader subdivides it using the same size words. + const packSize = sizes.packPreamble + + sizes.dispatchRootsSize + + sizes.ctorPlacementsSize + + sizes.packTailRewindSize + + sizes.ctorPlacementSection + + sizes.sectionPackBaseSize + + sizes.policyTableSize + + sizes.table8006754cSize + + sizes.opcodeStreamsSize; + addBlock('sectionPack', packSize); + addBlock('detachedBlob', sizes.detachedBlobSize); + } + + // SPEC_A skips both the section pack and the detached blob: after the + // 0x38-byte header it jumps straight to bundle A. LSET reads the section + // pack first, then the detached blob, then bundle B. Past this point both + // variants share the same art-install / state / override / state2 sequence. + addBlock('artInstall', sizes.artInstallSize); + addBlock('stateBank', sizes.stateBankSize); + addBlock('override', sizes.overrideSize); + addBlock('stateBank2', sizes.stateBank2Size); + + const blocksByName = new Map(blocks.map((block) => [block.name, block])); + + // Subranges inside the section pack follow the loader pointer chain: + // dispatch_roots = pack + packPreamble + // ctor_placements = dispatch_roots + dispatchRootsSize + // ctor_placement_section = ctor_placements + ctorPlacementsSize + // section_pack_base = ctor_placement_section + ctorPlacementSection + // policy_table = section_pack_base + sectionPackBaseSize + // table_8006754c = policy_table + policyTableSize + // opcode_streams = table_8006754c + table8006754cSize + // (pack tail of packTailRewindSize bytes is heap-rewound) + let packSubranges = []; + const pack = blocksByName.get('sectionPack'); + if (pack) { + let sub = pack.offset; + const push = (name, size) => { + packSubranges.push({ + name, + offset: sub, + size, + end: sub + size, + buffer: size > 0 && sub + size <= buffer.length + ? buffer.subarray(sub, sub + size) + : null, + }); + sub += size; + }; + push('packPreamble', sizes.packPreamble); + push('dispatchRoots', sizes.dispatchRootsSize); + push('ctorPlacements', sizes.ctorPlacementsSize); + push('ctorPlacementSection', sizes.ctorPlacementSection); + push('sectionPackBase', sizes.sectionPackBaseSize); + push('policyTable', sizes.policyTableSize); + push('table_8006754c', sizes.table8006754cSize); + push('opcodeStreams', sizes.opcodeStreamsSize); + push('packTailRewind', sizes.packTailRewindSize); + } + + const remaining = buffer.length - cursor; + return { + variant, + headerOffset, + sizes, + blocks, + blocksByName, + packSubranges, + trailingBytes: remaining, + totalConsumed: cursor, + }; +} + export function parseLsetWdl(buffer, filePath) { if (buffer.length < 0x34) { throw new Error(`File too small for LSET header: ${filePath}`); @@ -101,6 +239,13 @@ export function parseLsetWdl(buffer, filePath) { sections, boundaryCandidates, regions, + loaderLayout: (() => { + try { + return parseLoaderLayout(buffer, { variant: 'lset' }); + } catch { + return null; + } + })(), }; } @@ -447,3 +592,157 @@ export function parseRegion01Records(region) { records, }; } + +// Loader-faithful 12-byte constructor-placement records straight out of the +// `ctorPlacements` subrange of the section pack. Layout per +// `psx_dispatch_section0_constructor_placements @ 0x800258cc`: +// [u32 count][count * { u16 typeWord; u16 X; u16 Y; u16 Z; u16 selector; +// u16 flags }] +// Each record is called as `descriptor_table[typeWord].slot0(record, 0)` and +// `psx_object_create_compound_record` then reads exactly those 6 u16 fields. +export function parseCtorPlacementsBlock(block, variant = 'lset') { + if (!block || !block.buffer || block.size < 4) { + return { source: 'ctorPlacements', records: [], count: 0 }; + } + const count = readU32LE(block.buffer, 0); + const records = []; + if (count === 0 || count > 0x2000) { + return { source: 'ctorPlacements', records, count }; + } + for (let index = 0; index < count; index += 1) { + const recordOffset = 4 + index * 12; + if (recordOffset + 12 > block.buffer.length) { + break; + } + const words = []; + for (let cursor = 0; cursor < 12; cursor += 2) { + words.push(readU16LE(block.buffer, recordOffset + cursor)); + } + const record = buildRecord(words, 'ctorPlacements', recordOffset, words); + if (!record) { + continue; + } + record.sourceFamily = 'section0_constructor_placements'; + record.sourceRole = 'constructor-placement'; + record.rowIndex = index; + record.authoredLayer = 'constructors'; + record.authoredVariant = variant; + records.push(record); + } + records.forEach((record, index) => { + record.index = index; + record.absoluteOffset = block.offset + record.offset; + }); + return { + source: 'ctorPlacements', + recordStartOffset: 4, + reportedCount: count, + records, + }; +} + +// Loader-faithful 24-byte dispatch-root records from the `dispatchRoots` +// subrange of the section pack. Layout per +// `psx_dispatch_section0_dispatch_roots @ 0x800256b0`: +// [u32 count][count * 24 bytes] +// Within each record the dispatcher reads: +// +4 u16 typeId (argument to descriptor_table[typeId]) +// +8 u16 screenX (for +/-0x140 cull) +// +10 u16 screenY (for +/-0x140 cull) +// +16 u16 flags (bit 3 = skip this record) +// Z, selector and lane are not universally used by the dispatcher. The empirical +// best mapping that matches constructor-placement convention is: +// +6 u16 zeta (often 0) +// +12 u16 selector/rotation +// +14 u16 lane/flags-lo +// We keep them in rawWords so downstream consumers can probe further, but +// buildRecord uses the cull-verified X/Y/typeId for positioning. +export function parseDispatchRootsBlock(block, variant = 'lset') { + if (!block || !block.buffer || block.size < 4) { + return { source: 'dispatchRoots', records: [], count: 0 }; + } + const count = readU32LE(block.buffer, 0); + const records = []; + if (count === 0 || count > 0x2000) { + return { source: 'dispatchRoots', records, count }; + } + for (let index = 0; index < count; index += 1) { + const recordOffset = 4 + index * 24; + if (recordOffset + 24 > block.buffer.length) { + break; + } + const rawWords = []; + for (let cursor = 0; cursor < 24; cursor += 2) { + rawWords.push(readU16LE(block.buffer, recordOffset + cursor)); + } + // Project into buildRecord's 6-word ctor-placement shape using the fields + // the live dispatcher reads. rawWords indices: + // [0..1] (+0..+3) prefix + // [2] (+4) typeId (dispatch) + // [3] (+6) zeta / z-ish + // [4] (+8) X (cull) + // [5] (+10) Y (cull) + // [6] (+12) selector-ish + // [7] (+14) lane-ish + // [8] (+16) flags (bit 3 skip) + // [9..11] trailing + const flags = rawWords[8]; + if ((flags & 0x8) !== 0) { + continue; + } + const typeWord = rawWords[2]; + const xWord = rawWords[4]; + const yWord = rawWords[5]; + const zWord = rawWords[3] & 0xff; + const selectorWord = rawWords[6]; + const laneWord = rawWords[7]; + // Relaxed plausibility: dispatch-root records can have lane==0 or large + // selectors (e.g. behavior opcodes) because the dispatcher only reads + // typeId, X, Y, and the +0x10 skip flag. We only require typeId in the + // scene-relevant range and X/Y not both zero. This is looser than + // `isPlausibleRecord` on purpose. + if (typeWord < 0x20 || typeWord > 0x1ff) { + continue; + } + if ((xWord | yWord) === 0) { + continue; + } + const words = [typeWord, xWord, yWord, zWord, selectorWord, laneWord]; + const screenX = (yWord - xWord) * PSX_SCREEN_SCALE; + const screenY = ((2 * zWord) - Math.floor((xWord + yWord) / 2)) * PSX_SCREEN_SCALE; + const record = { + index: -1, + source: 'dispatchRoots', + offset: recordOffset, + words, + rawWords: words, + typeWord, + xWord, + yWord, + zWord, + selectorWord, + laneWord, + screenX, + screenY, + sourceFamily: 'section0_dispatch_roots', + sourceRole: 'root-dispatch', + recordSide: null, + rowIndex: index, + authoredLayer: 'roots', + authoredVariant: variant, + dispatchRootRawWords: rawWords, + dispatchRootFlags: flags, + }; + records.push(record); + } + records.forEach((record, index) => { + record.index = index; + record.absoluteOffset = block.offset + record.offset; + }); + return { + source: 'dispatchRoots', + recordStartOffset: 4, + reportedCount: count, + records, + }; +} diff --git a/psx-map-exporter/tmp_correlate_runtime_map0.mjs b/psx-map-exporter/tmp_correlate_runtime_map0.mjs deleted file mode 100644 index 8ec6234..0000000 --- a/psx-map-exporter/tmp_correlate_runtime_map0.mjs +++ /dev/null @@ -1,79 +0,0 @@ -import fs from 'node:fs'; -import path from 'node:path'; - -const projectRoot = path.resolve('..'); -const runtimePath = path.resolve(projectRoot, 'psx-map-exporter', '.cache', 'runtime-snapshot-report.json'); -const recordsPath = path.resolve(projectRoot, 'psx-map-exporter', '.cache', 'ctor_stream_probe', 'records.json'); - -const runtime = JSON.parse(fs.readFileSync(runtimePath, 'utf8')); -const records = JSON.parse(fs.readFileSync(recordsPath, 'utf8')); - -const visibleObjects = runtime.sampleVisibleObjects ?? []; -const visibleTypeMap = new Map(); -for (const object of visibleObjects) { - const key = object.typeId; - const entry = visibleTypeMap.get(key) ?? { - typeId: key, - count: 0, - routeWords: new Set(), - selectorIndexes: new Set(), - latchedTokens: new Set(), - artResourcePtrs: new Set(), - authoredRecordPtrs: [], - }; - entry.count += 1; - entry.routeWords.add(object.routeWord); - entry.selectorIndexes.add(object.selectorIndex); - entry.latchedTokens.add(object.latchedToken); - entry.artResourcePtrs.add(object.artResourcePtr); - if (entry.authoredRecordPtrs.length < 8) { - entry.authoredRecordPtrs.push(object.authoredRecordPtr); - } - visibleTypeMap.set(key, entry); -} - -const typeRows = new Map((runtime.sampleTypeRows ?? []).map((row) => [row.typeId, row])); -const recordTypeCounts = new Map(); -const recordLaneByType = new Map(); -for (const record of records.records ?? []) { - recordTypeCounts.set(record.typeWord, (recordTypeCounts.get(record.typeWord) ?? 0) + 1); - const laneSet = recordLaneByType.get(record.typeWord) ?? new Set(); - laneSet.add(record.laneWord); - recordLaneByType.set(record.typeWord, laneSet); -} - -const combined = [...new Set([...recordTypeCounts.keys(), ...visibleTypeMap.keys()])] - .sort((left, right) => left - right) - .map((typeId) => { - const visible = visibleTypeMap.get(typeId); - const row = typeRows.get(typeId) ?? null; - return { - typeId, - rawRecordCount: recordTypeCounts.get(typeId) ?? 0, - rawLaneWords: [...(recordLaneByType.get(typeId) ?? new Set())].sort((a, b) => a - b), - visibleCount: visible?.count ?? 0, - visibleRouteWords: visible ? [...visible.routeWords].sort((a, b) => a - b) : [], - visibleSelectorIndexes: visible ? [...visible.selectorIndexes].sort((a, b) => a - b) : [], - visibleLatchedTokens: visible ? [...visible.latchedTokens].sort((a, b) => a - b) : [], - visibleArtResourcePtrs: visible ? [...visible.artResourcePtrs] : [], - sampleAuthoredRecordPtrs: visible?.authoredRecordPtrs ?? [], - activeHeader: row?.activeHeader ?? 0, - builtResource: row?.builtResource ?? 0, - headerKind: row?.headerWords?.[0] ?? null, - headerWord3: row?.headerWords?.[3] ?? null, - headerWord4: row?.headerWords?.[4] ?? null, - headerWord8: row?.headerWords?.[8] ?? null, - headerWord10: row?.headerWords?.[10] ?? null, - headerWord11: row?.headerWords?.[11] ?? null, - }; - }) - .filter((entry) => entry.rawRecordCount !== 0 || entry.visibleCount !== 0); - -const outputPath = path.resolve(projectRoot, 'psx-map-exporter', '.cache', 'runtime-map0-correlation.json'); -fs.writeFileSync(outputPath, JSON.stringify({ - currentMapId: runtime.currentMapId, - visibleObjectCount: runtime.visibleObjectCount, - combined, -}, null, 2)); - -console.log(JSON.stringify({ outputPath }, null, 2)); \ No newline at end of file diff --git a/psx-map-exporter/tmp_dump_runtime_snapshot.mjs b/psx-map-exporter/tmp_dump_runtime_snapshot.mjs deleted file mode 100644 index 4841bdd..0000000 --- a/psx-map-exporter/tmp_dump_runtime_snapshot.mjs +++ /dev/null @@ -1,159 +0,0 @@ -import fs from 'node:fs'; -import path from 'node:path'; - -const RAM_BASE = 0x80000000; -const TYPE_TABLE_BASE = 0x800758c4; -const TYPE_ROW_STRIDE = 0x18; -const MAIN_VISIBLE_LIST = 0x8006ad5c; -const MAIN_VISIBLE_READ_INDEX = 0x80067690; -const MAIN_VISIBLE_WRITE_INDEX = 0x800676bc; -const CURRENT_MAP_ID = 0x80067728; -const NEXT_MAP_ID = 0x800678d0; - -const projectRoot = path.resolve('..'); -const ramDumpPath = path.resolve(projectRoot, 'binary', 'Crusader - No Remorse (USA) Main Memory Dump.bin'); -const ram = fs.readFileSync(ramDumpPath); -const outputPath = path.resolve(projectRoot, 'psx-map-exporter', '.cache', 'runtime-snapshot-report.json'); - -function vaToOffset(address) { - const offset = address - RAM_BASE; - if (offset < 0 || offset >= ram.length) { - throw new RangeError(`Address out of dump range: 0x${address.toString(16)}`); - } - return offset; -} - -function canRead(address, size = 4) { - const offset = address - RAM_BASE; - return offset >= 0 && offset + size <= ram.length; -} - -function readU32(address) { - return ram.readUInt32LE(vaToOffset(address)); -} - -function readS32(address) { - return ram.readInt32LE(vaToOffset(address)); -} - -function readU16(address) { - return ram.readUInt16LE(vaToOffset(address)); -} - -function readS16(address) { - return ram.readInt16LE(vaToOffset(address)); -} - -function readHeaderWords(address, wordCount = 10) { - if (!canRead(address, wordCount * 4)) { - return null; - } - return Array.from({ length: wordCount }, (_, index) => readU32(address + index * 4)); -} - -function parseTypeRows(limit = 0x100) { - const rows = []; - for (let typeId = 0; typeId < limit; typeId += 1) { - const rowBase = TYPE_TABLE_BASE + typeId * TYPE_ROW_STRIDE; - const installCount = readU32(rowBase + 0x00); - const builtResource = readU32(rowBase + 0x04); - const stateScript = readU32(rowBase + 0x08); - const simpleComponent = readU32(rowBase + 0x0c); - const extents = readU32(rowBase + 0x10); - const activeHeader = readU32(rowBase + 0x14); - - if ((installCount | builtResource | stateScript | simpleComponent | extents | activeHeader) === 0) { - continue; - } - - const headerWords = activeHeader !== 0 ? readHeaderWords(activeHeader, 12) : null; - rows.push({ - typeId, - rowBase, - installCount, - builtResource, - stateScript, - simpleComponent, - extents, - activeHeader, - headerWords, - }); - } - return rows; -} - -function parseVisibleObjects(limit = 256) { - const readIndex = readU32(MAIN_VISIBLE_READ_INDEX); - const writeIndex = readU32(MAIN_VISIBLE_WRITE_INDEX); - const count = Math.max(0, Math.min(limit, writeIndex)); - const objects = []; - - for (let index = 0; index < count; index += 1) { - const objectPtr = readU32(MAIN_VISIBLE_LIST + index * 4); - if (!canRead(objectPtr, 0xa4)) { - continue; - } - - objects.push({ - index, - objectPtr, - typeId: readU16(objectPtr + 0x18), - routeWord: readU16(objectPtr + 0x1c), - stateFlags: readU16(objectPtr + 0x1e), - screenLeft: readS16(objectPtr + 0x20), - screenTop: readS16(objectPtr + 0x22), - screenRight: readS16(objectPtr + 0x24), - screenBottom: readS16(objectPtr + 0x26), - worldX: readS32(objectPtr + 0x3c), - worldY: readS32(objectPtr + 0x40), - worldZ: readS32(objectPtr + 0x44), - velocityX: readS32(objectPtr + 0x60), - velocityY: readS32(objectPtr + 0x64), - velocityZ: readS32(objectPtr + 0x68), - programPtr: readU32(objectPtr + 0x08), - artResourcePtr: readU32(objectPtr + 0x10), - companionExtentsPtr: readU32(objectPtr + 0x84), - stateScriptPtr: readU32(objectPtr + 0x88), - scriptBasePtr: readU32(objectPtr + 0x8c), - scriptReadPtr: readU32(objectPtr + 0x90), - latchedToken: readU16(objectPtr + 0x94), - scriptCountdown: readU16(objectPtr + 0x96), - selectorIndex: readU16(objectPtr + 0x9e), - authoredRecordPtr: readU32(objectPtr + 0xa0), - }); - } - - return { - readIndex, - writeIndex, - count, - objects, - }; -} - -const typeRows = parseTypeRows(); -const visible = parseVisibleObjects(); -const byType = new Map(); -for (const object of visible.objects) { - byType.set(object.typeId, (byType.get(object.typeId) ?? 0) + 1); -} - -const summary = { - ramDumpPath, - ramSize: ram.length, - currentMapId: readU32(CURRENT_MAP_ID), - nextMapId: readU32(NEXT_MAP_ID), - mainVisibleReadIndex: visible.readIndex, - mainVisibleWriteIndex: visible.writeIndex, - visibleObjectCount: visible.objects.length, - nonZeroTypeRowCount: typeRows.length, - visibleTypeCounts: [...byType.entries()] - .map(([typeId, count]) => ({ typeId, count })) - .sort((left, right) => right.count - left.count || left.typeId - right.typeId), - sampleTypeRows: typeRows.slice(0, 64), - sampleVisibleObjects: visible.objects.slice(0, 128), -}; - -fs.mkdirSync(path.dirname(outputPath), { recursive: true }); -fs.writeFileSync(outputPath, JSON.stringify(summary, null, 2)); -console.log(JSON.stringify({ outputPath }, null, 2)); \ No newline at end of file diff --git a/psx-map-exporter/tmp_inspect_region00.mjs b/psx-map-exporter/tmp_inspect_region00.mjs deleted file mode 100644 index d0269d5..0000000 --- a/psx-map-exporter/tmp_inspect_region00.mjs +++ /dev/null @@ -1,91 +0,0 @@ -import fs from 'node:fs'; -import path from 'node:path'; -import { parseLsetWdl } from './src/wdl.js'; - -const wdlPath = path.resolve('..', '..', 'Crusader-Map-Viewer', 'map_renderer', 'STATIC_PSX', 'LSET1', 'L0.WDL'); -const buffer = fs.readFileSync(wdlPath); -const wdl = parseLsetWdl(buffer, wdlPath); -const section = wdl.sections.find((entry) => entry.name === 'post_audio_section_00'); -const region = wdl.regions.find((entry) => entry.name === 'post_audio_region_00'); - -function readWords(sourceBuffer, offset, wordCount = 6) { - return Array.from({ length: wordCount }, (_, index) => sourceBuffer.readUInt16LE(offset + index * 2)); -} - -function isStructuredCandidate(words) { - const [typeWord, xWord, yWord, zWord, selectorWord, laneWord] = words; - if (typeWord >= 0x200) { - return false; - } - if (xWord === 0 && yWord === 0) { - return false; - } - if (xWord >= 0x4000 || yWord >= 0x4000) { - return false; - } - if (zWord > 0x20 || selectorWord > 0x04) { - return false; - } - return laneWord === 0x20 || laneWord === 0x22 || laneWord === 0x30; -} - -function inspectCountPrefixed12ByteStreams(source) { - const hits = []; - - for (let offset = 0; offset + 4 + 12 <= source.buffer.length; offset += 4) { - const count = source.buffer.readUInt32LE(offset); - if (count === 0 || count > 0x2000) { - continue; - } - - let good = 0; - const preview = []; - for (let index = 0; index < count && offset + 4 + (index + 1) * 12 <= source.buffer.length; index += 1) { - const recordOffset = offset + 4 + index * 12; - const words = readWords(source.buffer, recordOffset); - const structured = isStructuredCandidate(words); - if (index < 6) { - preview.push({ index, recordOffset, words, structured }); - } - if (!structured) { - break; - } - good += 1; - } - - if (good >= 16) { - hits.push({ - offset, - absoluteOffset: source.offset + offset, - count, - good, - preview, - }); - } - } - - return hits; -} - -const sectionHits = inspectCountPrefixed12ByteStreams(section); -const regionHits = inspectCountPrefixed12ByteStreams(region); -const preview = []; -for (let offset = 0; offset < 0x90; offset += 12) { - const words = readWords(section.buffer, offset); - preview.push({ - offset, - absoluteOffset: section.offset + offset, - words, - structured: isStructuredCandidate(words), - }); -} - -console.log(JSON.stringify({ - sectionOffset: section.offset, - sectionSize: section.size, - regionOffset: region.offset, - regionSize: region.size, - sectionHits, - regionHits, - preview, -}, null, 2));