diff --git a/Crusader.rep/idata/00/~00000006.db/db.43.gbf b/Crusader.rep/idata/00/~00000006.db/db.72.gbf similarity index 67% rename from Crusader.rep/idata/00/~00000006.db/db.43.gbf rename to Crusader.rep/idata/00/~00000006.db/db.72.gbf index 8c8c5d1..789de26 100644 Binary files a/Crusader.rep/idata/00/~00000006.db/db.43.gbf and b/Crusader.rep/idata/00/~00000006.db/db.72.gbf differ diff --git a/Crusader.rep/idata/00/~00000006.db/db.42.gbf b/Crusader.rep/idata/00/~00000006.db/db.73.gbf similarity index 67% rename from Crusader.rep/idata/00/~00000006.db/db.42.gbf rename to Crusader.rep/idata/00/~00000006.db/db.73.gbf index 8983d8b..e4c133a 100644 Binary files a/Crusader.rep/idata/00/~00000006.db/db.42.gbf and b/Crusader.rep/idata/00/~00000006.db/db.73.gbf differ diff --git a/Crusader.rep/user/00/~00000005.db/db.7.gbf b/Crusader.rep/user/00/~00000005.db/db.11.gbf similarity index 99% rename from Crusader.rep/user/00/~00000005.db/db.7.gbf rename to Crusader.rep/user/00/~00000005.db/db.11.gbf index a8e917e..021d8e2 100644 Binary files a/Crusader.rep/user/00/~00000005.db/db.7.gbf and b/Crusader.rep/user/00/~00000005.db/db.11.gbf differ diff --git a/Crusader.rep/user/00/~00000005.db/db.8.gbf b/Crusader.rep/user/00/~00000005.db/db.12.gbf similarity index 99% rename from Crusader.rep/user/00/~00000005.db/db.8.gbf rename to Crusader.rep/user/00/~00000005.db/db.12.gbf index 1778358..3130bda 100644 Binary files a/Crusader.rep/user/00/~00000005.db/db.8.gbf and b/Crusader.rep/user/00/~00000005.db/db.12.gbf differ diff --git a/crusader_decompilation_notes.md b/crusader_decompilation_notes.md index 805fa51..4b994e4 100644 --- a/crusader_decompilation_notes.md +++ b/crusader_decompilation_notes.md @@ -50,7 +50,15 @@ Functions at Ghidra `0003:XXXX` / `0004:XXXX` are **Phar Lap extender code** (fl ### `0000:ffff` — NE Fixup Placeholder (not a dispatcher) -`unresolved_far_thunk_dispatch` at `0000:ffff` is NOT a runtime function. Every `CALLF 0x0000:ffff` in the binary is a **different** external or inter-segment call patched by the NE loader at runtime. The decompiler body is garbled (it reads NE fixup-chain sentinel data). Decompiler comment added in Ghidra. See individual call sites for per-site behavioral annotations. +`unresolved_far_thunk_dispatch` at `0000:ffff` is NOT a runtime function. Every `CALLF 0x0000:ffff` in the original NE image is a **different** external or inter-segment call patched by the NE loader at runtime. The body at `0000:ffff` is just fixup placeholder data, so decompiling it as a function is meaningless. + +Repair status in `CRUSADER-RAW.EXE`: +- A PyGhidra repair pass now applies the verified NE relocation table directly to the raw-program bytes for literal internal `CALLF 9A ptr16:16` sites, then re-disassembles each patched instruction. +- Current verified batch results: + - `8851` internal literal `CALLF` sites patched to their real segment:offset targets. + - `2841` far-pointer relocation entries skipped because they were not literal `CALLF` instructions (data or other non-call uses). + - `119` import callsites annotated as `NE IMPORT -> module.symbol`. +- Remaining xrefs to `0000:ffff` should now mostly be import callsites or non-literal far-pointer cases rather than unresolved intra-game far calls. Known call-site classifications (by argument pattern): - `PUSH DS; PUSH imm_ordinal; CALLF` — Phar Lap extender calling a runtime-imported procedure by ordinal @@ -149,9 +157,9 @@ Known call-site classifications (by argument pattern): #### Current limitation in raw import -- The common thunk endpoint is still imported as `unresolved_far_thunk_dispatch` at `0000:ffff`. -- In this raw database, that body decompiles as overlapped/bad instruction data, so exact arithmetic internals of the final coordinate projection cannot yet be recovered from this symbol alone. -- Despite that, caller context + table shape + argument flow make the gameplay role of this helper clear enough for naming and control-flow analysis. +- The placeholder at `0000:ffff` still exists as a symbol, but the relevant internal literal `CALLF` sites are no longer the best source of truth: they have been patched in-place to their real NE targets. +- For `snap_entity_to_ground`, the formerly unresolved call at `0007:2261` now disassembles to `CALLF 0004:e7bd`, i.e. `world_to_screen_coords`. +- Remaining `0000:ffff` sightings in the raw import are now primarily import calls or non-literal far-pointer cases, not evidence that this gameplay helper still dispatches through a single shared runtime function. #### Working pseudocode (behavioral) @@ -189,7 +197,7 @@ Decompiler comment added to `0000:ffff` in Ghidra documenting this. #### Next RE targets for `snap_entity_to_ground` -- The `0007:224b` thunk call is an intra-NE inter-segment call (calling into a different game segment with ground-aligned coordinate math). Identifying it requires the NE relocation table or matching the disassembly in the standalone extracts. +- The repaired call at `0007:2261` now lands at `world_to_screen_coords` (`0004:e7bd`), so the next step is to reinterpret the helper with the real callee in view rather than through the old `0000:ffff` placeholder model. ### Raw 0007 Gameplay Helper Follow-up: AI sweep + checked spawn path @@ -213,6 +221,14 @@ Decompiler comment added to `0000:ffff` in Ghidra documenting this. - Reads player entity FAR pointer from global `0x2de4`. - Copies player world position fields (`+0x40`, `+0x42`) into globals `0x27e7` / `0x27e9` (AI focus position cache used by downstream logic). - Iterates entity IDs from `2` through `255` and dispatches per-entity processing through two sequential thunked calls per entity. +- After the NE far-call repair pass, the first call at `0007:101c` now disassembles and decompiles directly as `entity_resolve_slot_ptr` (`0005:0466`) instead of `CALLF 0000:ffff`. +- The repaired call chain now exposes several concrete helpers used by the sweep: + - `0005:42c8` = `entity_projected_bbox_overlaps_viewport` — projects the entity slot via `world_to_screen_coords`, subtracts entity height from `0x7df5[slot]`, derives sprite/flag context, and returns `bbox_overlap_test` against the active viewport rectangle referenced from global `0x4014`. + - `0005:3cf5` = `entity_class_has_flag2000` — class-word flag test over `entity_get_class_word(slot) & 0x2000`. + - `0005:ff2d` = `entity_class_get_flag8` — returns bit `0x08` from entity-class detail byte `0x7e1e[type*0x79 + 0x59]`. + - `0006:1305` = `entity_class_get_word_02` — raw accessor for word `+0x02` in the `0x7e1e` class-detail record. + - `0006:0ca4` = `entity_class_get_word_0a` — raw accessor for word `+0x0a` in the same class-detail record. + - `0006:11a1` = `entity_class_clear_flag8_and_dispatch` — clears bit `0x08` in class-detail byte `+0x59`, then performs follow-up entity/type checks and callback dispatch. Name intentionally stays flag-centric until the downstream side effects are fully mapped. - New disassembly comments added at both dispatch call sites: - `0007:101c`: `entity_slot_fetch(SS:&entity_id)` — first call; resolves entity slot/pointer from loop ID - `0007:1093`: `entity_tick_dispatch(SS:&entity_id, g_0x27c8)` — second call; per-entity AI tick with global `0x27c8` mode/context word @@ -710,6 +726,7 @@ All 35+ identified functions renamed and annotated in Ghidra. ### Entity Data Table at 0x7e1e - Stride: `0x79` bytes (121 bytes per entry) - Indexed by entity type (integer) or entity slot +- `+0x59` offset = class-detail flags byte (`entity_class_get_flag8` returns bit `0x08`; other callers also clear bit `0x10` here during at-target facing updates) - `+0x5a` offset = flags byte (bit4 = special entity flag, bit3 = armor/shield flag) - `+0x68` = targeting flag @@ -1000,6 +1017,7 @@ Small conservative rename batch from direct field-write behavior in the `0008:ba | `0008:bbb6` | `entity_set_source_type` | Writes entry word field `+0x04` from incoming parameter, then dispatches through FAR thunk path | | `0008:bc27` | `entity_set_event_type_checked` | Writes entry word field `+0x06`; when source field `+0x04` is non-zero, validates old/new event transition, including special checks for `0xF0-0xF7` and upper bound `<= 0x0FFF` | | `0008:bca8` | `entity_set_group_id` | Validates group id range `1..31`, writes low 5-bit group in byte `+0x08`, decrements old per-group counter and increments new one via counter table pointed to by `0x39c5` | +| `0008:bd53` | `entity_dispatch_entry_unlink` | Clears bit `0x1000` in flags2 at `+0x18` and zeroes the four link/state words at `+0x0a..+0x10`; used as the common unlink/reset tail in the local dispatch-entry pruning path | | `0008:be05` | `entity_increment_group_id` | Computes `((entry+0x08)&0x1F)+1`, validates against active-layer assumptions (`0x39c9`), then applies through `entity_set_group_id` | ### Verified call/xref notes @@ -1015,6 +1033,8 @@ Small conservative rename batch from direct field-write behavior in the `0008:ba - This cluster appears to manage core dispatch-entry metadata (`source_type`, `event_type`, group/layer byte and counters) that feeds the seg021 scheduler/event system previously documented. - The field offsets match the current seg021 entity/dispatch layout notes (`+0x04`, `+0x06`, `+0x08`), strengthening confidence in cross-function struct consistency. +- Follow-up on the same cluster: + - `0008:bd79` remains positional, but current decompiler evidence shows it compares an entry extent/position tuple against the player world position (`g_player_entity_farptr + 0x40/+0x42`), optionally fires the vtable callback at `+0x28` when flag `0x100` is armed, then calls `entity_dispatch_entry_unlink`. ## Raw 0008 Pair-Sync Helper Batch (new) @@ -1085,11 +1105,12 @@ Additional conservative renames from the `0008:d1a4-0008:d27d` cluster. |---------|------|---------| | `0008:d1a4` | `entity_set_flag100_in_flags2` | Gate-checked setter: ORs bit `0x100` into entry word at `+0x18` | | `0008:d1dc` | `entity_clear_flag100_in_flags2` | Gate-checked clearer: ANDs entry word at `+0x18` with `0xFEFF` (clears bit `0x100`) | +| `0008:cefb` | `entity_dispatch_entry_ctor_vtbl_3ad2` | Constructor variant: allocates if null, reinitializes via `entity_dispatch_entry_init`, sets vtable `0x3ad2`, sets flag `0x100` at `+0x16`, and zeroes the extension words at `+0x32/+0x34` | | `0008:d214` | `entity_dispatch_entry_ctor_vtbl_3aa6` | Constructor variant: allocates `0x40` bytes if null, reinitializes via `0008:cefb`, sets vtable to `0x3aa6`, sets flag `0x200` at `+0x16`, zeroes fields `+0x38..+0x3e` | Notes: - The `entity_set_flag100_in_flags2` / `entity_clear_flag100_in_flags2` pair is a verified complementary toggle with identical gate logic (`0x39a8/0x39f9/0x3991` check path). -- Constructor naming is intentionally vtable-centric (`0x3aa6`) until more direct gameplay semantics are recovered from its callback dispatch paths. +- Constructor naming is intentionally vtable-centric (`0x3ad2`, `0x3aa6`) until more direct gameplay semantics are recovered from their callback dispatch paths. ## Raw 0008 Periodic/Counter Helpers (new) @@ -1573,6 +1594,190 @@ Compares two 5-byte `map_position` structs: `{ x:word, y:word, layer:byte }`. Re | 98 | `0003:e4d3` | `dos_file_open_wrapper` | 26 | Zeros output byte, delegates to file open impl at 0003:bb92 | | 99 | `0005:033e` | `entity_resolve_base_parent` | 25 | Same hierarchy walk as `entity_resolve_base_type` but returns parent from [0x7ded] | | 100 | `000a:87fd` | `render_clip_rect_to_viewport` | 25 | Clips 4 rect params to viewport bounds at [0x4014], sets dirty flag at 0x8a16, increments draw counter at 0x4716 | +| 101 | `0005:3cf5` | `entity_class_has_flag2000` | 25 | Returns `(entity_get_class_word(slot) & 0x2000) != 0` | +| 102 | `0009:80db` | `bbox_overlap_test` | 25 | Boolean rectangle overlap test for two 4-word bbox structs; unlike `bbox_intersect`, it does not write back an intersection | +| 103 | `000d:cc00` | `entity_compute_proximity_or_visibility_bucket` | 25 | Returns `0x40` if entity is null or projected bbox overlaps viewport; otherwise buckets world-distance from current reference entity (`0x7e22`) into `0x32/0x20/0x10/0x08` | +| 104 | `000d:d413` | `entity_refresh_recent_proximity_or_visibility_buckets` | 24 | Recomputes bucket values for the last four active entries in `0x69ac` and notifies backing handles via `000a:6343` when a bucket changes | + +## Raw 000d Proximity/Visibility Bucket Cluster (new) + +Small conservative rename batch from the `000d:cc00-d413` region after the far-call repair exposed the viewport helper in live decompilation. + +| Address | Name | Evidence | +|---------|------|---------| +| `000d:cc00` | `entity_compute_proximity_or_visibility_bucket` | Returns bucket `0x40` for null or on-screen entities (`entity_projected_bbox_overlaps_viewport`), else computes a distance bucket from the current reference entity at `0x7e22` with thresholds `0x17d`, `0x281`, `0x3c1` mapping to `0x32`, `0x20`, `0x10`, `0x08` | +| `000d:d413` | `entity_refresh_recent_proximity_or_visibility_buckets` | Walks the last four active records in the `0x69ac` array, recomputes the same bucket, stores it back to each entry, and calls `000a:6343` when the bucket changes | +| `000d:cdd0` | `tracked_entity_bucket_prune_invalid_entries` | Walks the `0x69ac` array, validates backing handles through `000a:637a`, and clears entry handles to `0xffff` when the backing object is gone | +| `000d:cd62` | `tracked_entity_bucket_find_free_main_slot` | Finds the first free entry in the main portion of the `0x69ac` array (`0 .. count-4`) | +| `000d:cd9a` | `tracked_entity_bucket_find_free_aux_slot` | Finds the first free entry in the auxiliary tail portion of the `0x69ac` array (`count-4 .. count-1`) | + +Supporting caller notes: +- `000d:ce1e` populates one `0x69ac` entry by reserving a free slot, computing the initial bucket through `entity_compute_proximity_or_visibility_bucket`, storing both current and previous bucket fields, then allocating/linking the backing handle through `000a:5f36`. +- `000d:d409` is a thin wrapper that only calls `entity_refresh_recent_proximity_or_visibility_buckets`. +- `000d:cfad` is an update-or-allocate helper for `(param_1,param_2)` pairs: it tries to update an existing tracked entry through `000a:606a`, clears dead entries, and falls back to `000d:ce1e` allocation when no live match remains. +- `000d:cec5` is the auxiliary-slot allocator: it prunes invalid entries, uses `tracked_entity_bucket_find_free_aux_slot`, tags the new entry with byte `+0x0a = 1`, and seeds its handle via `000a:5f36(..., flag=1)`. +- `000a:606a` = `tracked_entity_bucket_handle_update_or_alloc` — updates the backing handle for an existing tracked bucket entry when possible, or falls back to allocation via `000a:5f36` if the handle has gone stale. +- `000d:d350` = `tracked_entity_bucket_set_value` — finds a tracked `(entity_id, entity_ref)` entry in `0x69ac` and pushes a new bucket value into its backing handle through `000a:6343`. +- `000d:d10b` = `tracked_entity_bucket_clear_ref_field` — clears only the `+0x02` reference field for all matching entries. +- `000d:d151` = `tracked_entity_bucket_remove_by_ref` — marks matching entries' backing handles for removal and clears the local entry handle/reference fields. +- `000d:d1b1` = `tracked_entity_bucket_remove_tagged_by_ref` — same removal path, but only for entries whose byte `+0x0a` tag is set. + +## Raw 000a Tracked-Handle Table (new) + +The `0x4673` table now reads as the backing handle registry for the `0x69ac` tracked-entry bucket subsystem. +That client layer sits on top of a separate generic cache manager rooted at `0x4688..0x46b7`, so the current `tracked_entity_*` names should be read as client-side structure names rather than names for the cache internals themselves. + +| Address | Name | Evidence | +|---------|------|---------| +| `000a:5f02` | `tracked_entity_handle_find_slot` | Linear scan over 12 entries in the `0x4673` table for a matching 32-bit handle id | +| `000a:602b` | `tracked_entity_handle_is_live` | Returns true only when a handle exists in `0x4673` and its flag word at `+0x0a` does not have bit `0x0002` set | +| `000a:60eb` | `tracked_entity_handle_mark_remove` | Sets bit `0x0002` in the handle-table flag word and dispatches through the unresolved cleanup path | +| `000a:612e` | `tracked_entity_handle_mark_remove_all` | Iterates all 12 handle-table entries and marks each live handle for removal | +| `000a:6167` | `tracked_entity_handle_alloc_slot` | Allocates a slot in one of two ranges (`0..7` or `8..11`) depending on the aux flag; when full, wraps in a ring and evicts via `tracked_entity_handle_mark_remove` before reusing the slot | +| `000a:6228` | `tracked_entity_handle_prune_removed` | Reaps entries previously marked with bit `0x0002`, clears dead slots, and refreshes high-index entries through `000a:6b2d` | +| `000a:63bc` | `tracked_entity_handle_find_by_entity` | Finds the first live handle-table entry whose key/entity word at `+0x04` matches the requested entity id | + +Current structural read of one `0x4673` entry (stride `0x0c`): +- `+0x00` = 32-bit handle id +- `+0x04` = key/entity id +- `+0x06` = class/group/source-style selector passed in from tracked-entry allocation +- `+0x08` = current bucket/value +- `+0x0a` = flags (`bit0` set by aux-slot allocation, `bit1` = pending removal) + +Thin public wrappers on top of the tracked-handle client layer: +- `000a:5276` = `entity_bucket_track_default_main` — gated by `0x45aa`; creates or refreshes a main-slot tracked handle with bucket `0x40` and selector `0xff`. +- `000a:5294` = `entity_bucket_track_main` — same path, but takes the bucket value as an argument for the main-slot range. +- `000a:52d0` = `entity_bucket_track_default_aux` — aux-slot variant with default bucket `0x40`. +- `000a:52ee` = `entity_bucket_track_aux` — aux-slot variant with explicit bucket argument. + +## Raw 000a Generic Cache Manager (new) + +Follow-up analysis of `000a:6b2d` and the `0x4688..0x46b7` globals shows that this region is a generic cache manager used by the tracked-handle layer, not part of the tracked-entity subsystem itself. + +| Address | Name | Evidence | +|---------|------|---------| +| `000a:6b2d` | `cache_lookup_or_load_entry_by_id` | Fast-paths the last id via `0x46af/0x46b1`, otherwise searches `0x469d`, evicts older cache slots until there is room under byte budget `0x46a5`, allocates a block from the free-list, clears/initializes the payload, records the id, and dispatches through the loader interface at `0x468c` | +| `000a:6a95` | `cache_release_entry_by_slot` | Releases a cached slot by index, clears any client references through `000a:62d8`, frees its backing block through `cache_free_block_by_slot`, and marks the slot id in `0x469d` as unused (`0xffff`) | +| `000a:6d07` | `cache_alloc_block_for_slot` | Allocates or splits a block from the free-list anchored at `0x4688`, tags it with the owning cache slot index, and updates the in-use byte count at `0x46a9` | +| `000a:6f4d` | `cache_free_block_by_slot` | Finds the free-list node for a cache slot, marks it free, subtracts its size from `0x46a9`, and coalesces adjacent free blocks | +| `000a:67d9` | `cache_shutdown` | Tears down the generic cache manager: flushes/reset state, frees slot arrays at `0x4699/0x469d/0x46b3`, frees the free-list container at `0x4688`, and closes backing state at `0x4691` | +| `000a:6898` | `cache_set_loader_interface` | Installs the backend loader/callback interface pointer at `0x468c` | + +Current structural read of the cache globals: +- `0x4688` = intrusive free-list / block-list head used by `cache_alloc_block_for_slot` and `cache_free_block_by_slot` +- `0x468c` = backend loader interface / callback table; `+0x34` returns the payload size for an id and `+0x0c` loads or binds a block after allocation +- `0x4695` = base pointer for the raw cache payload arena +- `0x4699` = per-slot payload-pointer table +- `0x469d` = per-slot cached id table (`0xffff` = unused) +- `0x46a5` = cache byte budget / arena capacity +- `0x46a9` = current bytes in use +- `0x46af` / `0x46b1` = one-entry fast-path cache of the last requested id and slot index +- `0x46b3` = per-slot block metadata pointer mirror used when releasing or refreshing slots + +### Follow-up: caller resolution for the public bucket wrappers + +- `0005:3b34` = `tracked_entity_bucket_alloc_main_if_enabled` and `0005:3b53` = `tracked_entity_bucket_alloc_aux_if_enabled`; these are the two thin gate wrappers that feed the `0x69ac` tracked-entry layer. +- `0005:3b72` = `tracked_entity_bucket_remove_by_entity_and_ref_if_enabled`, which forwards into `000d:d086 = tracked_entity_bucket_remove_by_entity_and_ref` when `0x45aa` is set. +- `0006:d404` is not a standalone function entry; it is the tail call inside the unnamed destructor block immediately before `0006:d414`. That block tears down a `0x2774` dispatch-entry object, pushes `(entity_id = 0xdb, entity_ref = *0x7e22)`, and removes the matching aux tracked bucket entry. +- `0006:d5ae` is the analogous tail call immediately before `0006:d5be`. It tears down the sibling `0x2750` dispatch-entry object, pushes `(entity_id = 0xa4, entity_ref = *0x7e22)`, and removes the matching aux tracked bucket entry. +- The matching constructor sides are `0006:d370` and `0006:d51a`: both allocate/init a dispatch entry, stamp source type `8`, seed a per-object field from the current reference entity at `0x7e22`, and then call `tracked_entity_bucket_alloc_aux_if_enabled`. +- `0007:cce8` is the tail call at the end of `scroll_camera_set_state_params`. After the camera scroll state is updated and the new screen-space origin is committed to `0x2bb7/0x2bb9`, it refreshes the recent proximity/visibility buckets through `000d:d409` when `0x45aa` is enabled. + +### Follow-up: `0x45aa` gate and cache loader installation + +- `0x45aa` now reads as `tracked_entity_bucket_system_enabled`, not a one-off debug/test flag. It gates all three public wrapper helpers above and the camera-side refresh in `scroll_camera_set_state_params`. +- The enable bit is set only by the unlabeled init block around `000a:51d0..5222`: that code stores the incoming backend/interface pointer into `0x45ab/0x45ad`, installs it into `0x468c` via `cache_set_loader_interface`, allocates the tracked-handle table (`000a:5e00`) and the 32-entry bucket array (`000d:cca3(0x20)`), then sets `0x45aa = 1`. +- The matching unlabeled shutdown block starts at `000a:5223`: it checks `0x45aa`, tears down the tracked-handle table through `000a:5e59`, frees the bucket array through `000d:ccec`, and only then clears/uses the backend pointer state at `0x45ab`. +- `0x468c` remains best named as a generic `cache_loader_interface`: the current verified evidence is a cache backend callback table (`+0x34` = size query, `+0x0c` = load/bind callback) shared by the tracked-entry service. The newly traced gameplay callers prove this service participates in camera/entity-interest updates, but they are not yet strong enough to justify an audio-specific or resource-specific subsystem rename. + +### Follow-up: init/shutdown entry points around `000a:51xx` + +- `000a:5186` = `tracked_entity_bucket_system_init`. +- `000a:538e` = `tracked_entity_bucket_system_init_if_configured`; it only calls the init routine when config/feature gate `0x89f4` is non-zero. +- `000a:5223` = `tracked_entity_bucket_system_shutdown`. +- `0x45ab/0x45ad` now read as `tracked_entity_bucket_backend_object`, a cached backend/interface object pointer used by init/shutdown in addition to the lower-level `cache_loader_interface` callback table at `0x468c`. +- `tracked_entity_bucket_system_init` first allocates a rotating buffer via `0009:3600`, lazily creates `tracked_entity_bucket_backend_object` through `0009:5600` when absent, installs that object into `cache_loader_interface`, allocates the tracked handle table via the missing function entry at `000a:5e00`, allocates the 32-entry `0x69ac` bucket array via `000d:cca3(0x20)`, then sets `tracked_entity_bucket_system_enabled`. +- `tracked_entity_bucket_system_shutdown` is called from the wider engine teardown routine at `0004:621b`; it tears down the tracked handle table, frees the `0x69ac` bucket array, calls backend-object vtable slot `+0x38` with `(3, backend_object)`, and clears `tracked_entity_bucket_backend_object`. + +### Follow-up: backend object constructor at `0009:5600` + +- The missing raw-import function entry at `0009:5600` has now been recovered in-place as `cache_backend_object_init` with body `0009:5600-0009:57b9`. +- Current verified behavior is still structural, but stronger than before: + - Allocates a `0x20`-byte object when the caller passes null. + - Initializes embedded DOS file-handle state via `dos_file_handle_init` (`0009:1c00`). + - Seeds internal method-table / state fields at object offsets `+0x08`, `+0x0c`, `+0x10`, `+0x14`, `+0x16`, `+0x18`, and `+0x1c`. + - Dispatches through the object method table during construction and returns the object pointer later cached at `0x45ab/0x45ad`. +- This is enough to justify the structural `cache_backend_object_init` name, but not yet enough to promote the backend object to a file-, audio-, or resource-specific subsystem name. +- Verified callback roles inside `cache_lookup_or_load_entry_by_id`: + - backend vtable `+0x34` = size query callback for a cache entry id (used before allocation/eviction). + - backend vtable `+0x0c` = load/bind callback that populates the newly allocated slot buffer for the requested id. + +### Follow-up: missing function entry at `000a:5e00` + +- The missing raw-import function entry at `000a:5e00` has now been recovered in-place as `tracked_entity_handle_table_init` with body `000a:5e00-000a:5e58`. +- Verified behavior: if `0x4672` is clear, it allocates `0x90` bytes at `0x4673/0x4675`, aborts through `runtime_init_or_abort` on allocation failure, calls `000a:577d` and local helper `000a:5e95`, then sets `0x4672 = 1`. +- Matching teardown helper promoted alongside it: `000a:5e59` = `tracked_entity_handle_table_shutdown`. +- Current structural names for that local state: + - `0x4672` = `tracked_entity_handle_table_active` + - `0x4673/0x4675` = `tracked_entity_handle_table` +- This matches the existing client-layer interpretation: `0x4673` holds 12 handle entries (`12 * 0x0c = 0x90` bytes), and `000a:5e00` is the table allocator/initializer used by the tracked bucket subsystem startup path. + +### Follow-up: missing function entry at `000a:6600` + +- The missing raw-import function entry at `000a:6600` has now been recovered in-place as `cache_init` with body `000a:6600-000a:67d8`. +- Verified behavior from the repaired body: + - Stores the requested slot count in `0x46ad`. + - Allocates the per-slot payload-pointer table at `0x4699` (`count * 4`) and aborts on failure. + - Seeds each slot with allocator-returned pointers / zero low words before running local pointer normalization helpers (`0009:c496`, `0009:c400`, `0009:c6ae`). + - Queries/derives the cache arena size, subtracts `0x1000`, and stores the byte budget in `0x46a5`. + - Allocates the arena backing object at `0x4691`, derives the payload base pointer `0x4695`, and aborts through `seg091_func_00fd` on failure. + - Allocates the per-slot block-metadata mirror at `0x46b3` (`count * 4`) and per-slot cached-id table at `0x469d` (`count * 2`). + - Allocates and initializes the free-list head object at `0x4688`, then calls local helper `000a:68aa` before returning. + +### Follow-up: cache reset / handle-table helpers + +- `000a:68aa` = `cache_reset_runtime_state`. +- Verified role: shared cache reset/bootstrap helper called from `cache_init`, `cache_shutdown`, and one wider external reset path. It allocates per-slot arena-header / metadata nodes, rebinds slot pointers to the arena base, clears the cached-id table, seeds the free-list head, and resets `0x46a9` (bytes in use) plus `0x46af` (last-id fast path). +- `000a:703e` = `cache_compact_arena_blocks`. +- Verified role: compacts live cache arena blocks into earlier free holes when `cache_alloc_block_for_slot` cannot find a large-enough free block, updates per-slot payload pointers, and merges adjacent free-list headers afterward. +- `000a:5e95` = `tracked_entity_handle_table_clear_and_dispatch`. +- Verified role: when `tracked_entity_handle_table_active` is set, it zeroes the full `0x90`-byte handle table at `0x4673`, resets adjacent local state at `0x4677/0x4679/0x467b`, then dispatches through the remaining thunked follow-up path. +- `000a:5339` = `tracked_entity_handle_mark_remove_all_if_enabled`. +- Verified role: thin gate wrapper that only forwards to `tracked_entity_handle_mark_remove_all` when `tracked_entity_bucket_system_enabled` is set. + +### Follow-up: external reset paths using the cache/tracked-handle layer + +- The unlabeled path around `0004:25a9` now has enough local evidence to classify as an external reset sequence: it calls `cache_reset_runtime_state`, then `tracked_entity_handle_table_clear_and_dispatch`, then continues through additional tracked-entry/cache-side refresh helpers (`000d:cd22`, `000d:44b3`, `0006:ae66`, `0006:ae00`, etc.). +- The unlabeled path around `0004:eb80` is a conditional tracked-bucket reset/update sequence: when `tracked_entity_bucket_system_enabled` is set, it calls `tracked_entity_handle_mark_remove_all_if_enabled`, then `tracked_entity_handle_table_clear_and_dispatch`, then `cache_compact_arena_blocks`, before resuming its outer flow. +- These caller sites strengthen the current interpretation that the `0x45aa` / `0x4673` / `0x4688..46b7` layer is a shared runtime cache service used by gameplay/system reset flows, but they still do not expose a resource-specific subsystem name by themselves. + +### Follow-up: repaired seg004 reset-path function objects + +- `0004:2592` had been mis-modeled as a one-instruction thunk body. It has now been repaired to the full body `0004:2592-25de` and renamed `runtime_cache_reset_sequence`. +- Current verified behavior for `runtime_cache_reset_sequence`: + - Calls `0008:7bfe`. + - Calls `game_mode_init(*(0x27c4))`. + - Executes one still-unresolved follow-up callsite at `0004:25a4`. + - Then resets cache runtime state, clears tracked handles, refreshes tracked-entry/cache helpers, and resumes the wider runtime reset flow. +- Known caller so far: `0004:262d` inside the tiny wrapper at `0004:2620`, which sets byte `+0x40` on the object at `0x6828` before invoking the reset sequence. +- `0004:eb1f` had also been truncated. It has now been repaired to the full body `0004:eb1f-eb9b` and renamed `entity_dispatch_entry_ctor_0f3a_with_cache_reset`. +- Verified behavior for `entity_dispatch_entry_ctor_0f3a_with_cache_reset`: + - Allocates/initializes an entity dispatch entry when needed. + - Stamps entry type `0x0f3a`. + - Stores its two word payload fields from the incoming args. + - Runs local setup through `0004:ebf4`. + - When `tracked_entity_bucket_system_enabled` is set, performs the tracked-handle removal / clear / cache-compaction sequence before finalizing through `0009:b1c3`. +- The sibling at `0004:eb9c` remains separate and valid; it builds the same `0x0f3a` entry type without the extra cache-reset tail, so the repaired `0004:eb1f` boundary stops cleanly at `0004:eb9b`. + +### Repair note: overlapping bad function body + +- Recovery of `cache_init` required a conservative boundary repair: a stray function object `FUN_000a_eee3` had incorrectly claimed body range `000a:6710-000a:fe79`, blocking creation of the real `cache_init` body. +- The bad overlap was removed, `cache_init` was created at `000a:6600-67d8`, and `FUN_000a_eee3` was recreated conservatively as the contiguous visible body `000a:eee3-f00b`. + +Current interpretation of the `0x69ac` / `0x4673` client layer: +- It is an entity-linked consumer of the generic cache manager. +- Its bucket values (`0x40`, `0x32`, `0x20`, `0x10`, `0x08`) still look attenuation- or priority-like rather than purely visibility-like. +- That is enough to keep the structural names, but not enough yet to safely promote the subsystem to a concrete audio/effect name. **Entity Table Pointers (DS-relative, discovered in tier 5):** diff --git a/pyghidra_plans/apply_ne_far_call_fixups.py b/pyghidra_plans/apply_ne_far_call_fixups.py new file mode 100644 index 0000000..2cc8f6f --- /dev/null +++ b/pyghidra_plans/apply_ne_far_call_fixups.py @@ -0,0 +1,160 @@ +from __future__ import annotations + +import csv +from pathlib import Path + +from ghidra.app.cmd.disassemble import DisassembleCommand +from ghidra.util.task import ConsoleTaskMonitor + + +REPO_ROOT = Path(__file__).resolve().parents[1] +INTERNAL_FIXUPS_TSV = REPO_ROOT / "ne_reloc_far_calls.tsv" +IMPORT_FIXUPS_TSV = REPO_ROOT / "ne_reloc_far_imports.tsv" + + +def parse_address_text(address_text: str) -> int: + segment_text, offset_text = address_text.split(":", 1) + return (int(segment_text, 16) << 16) + int(offset_text, 16) + + +def to_address(address_text: str): + address_space = program.getAddressFactory().getDefaultAddressSpace() + return address_space.getAddress(parse_address_text(address_text)) + + +def parse_seg_off(address_text: str) -> tuple[int, int]: + segment_text, offset_text = address_text.split(":", 1) + return int(segment_text, 16), int(offset_text, 16) + + +def signed_byte(value: int) -> int: + return value if value < 0x80 else value - 0x100 + + +def load_internal_fixups() -> dict[str, tuple[str, str]]: + rows: dict[str, tuple[str, str]] = {} + with INTERNAL_FIXUPS_TSV.open("r", encoding="utf-8", newline="") as handle: + reader = csv.DictReader(handle, delimiter="\t") + for row in reader: + source = row["source_ghidra"].strip() + target = row["target_ghidra"].strip() + label = row["target_label"].strip() + previous = rows.get(source) + if previous is not None and previous != (target, label): + raise RuntimeError( + f"conflicting internal fixup mapping for {source}: {previous} vs {(target, label)}" + ) + rows[source] = (target, label) + return rows + + +def load_import_fixups() -> dict[str, str]: + rows: dict[str, str] = {} + with IMPORT_FIXUPS_TSV.open("r", encoding="utf-8", newline="") as handle: + reader = csv.DictReader(handle, delimiter="\t") + for row in reader: + source = row["source_ghidra"].strip() + target = row["target"].strip() + previous = rows.get(source) + if previous is not None and previous != target: + raise RuntimeError( + f"conflicting import fixup mapping for {source}: {previous} vs {target}" + ) + rows[source] = target + return rows + + +def patch_internal_far_call(operand_address_text: str, target_address_text: str, label: str) -> str | None: + memory = program.getMemory() + listing = program.getListing() + + operand_address = to_address(operand_address_text) + instruction_address = operand_address.previous() + instruction_end = instruction_address.add(4) + + opcode = int(memory.getByte(instruction_address)) & 0xFF + if opcode != 0x9A: + return None + + listing.clearCodeUnits(instruction_address, instruction_end, False) + + target_segment, target_offset = parse_seg_off(target_address_text) + operand_bytes = [ + target_offset & 0xFF, + (target_offset >> 8) & 0xFF, + target_segment & 0xFF, + (target_segment >> 8) & 0xFF, + ] + + for index, byte_value in enumerate(operand_bytes): + memory.setByte(operand_address.add(index), signed_byte(byte_value)) + + command = DisassembleCommand(instruction_address, None, True) + if not command.applyTo(program, ConsoleTaskMonitor()): + raise RuntimeError(f"failed to re-disassemble patched CALLF at {instruction_address}") + + comment = f"NE FIXUP APPLIED -> {target_address_text} ({label})" + helpers["set_comment"](program, str(instruction_address), comment, "eol") + return str(instruction_address) + + +def annotate_import_far_call(operand_address_text: str, import_target: str) -> str: + instruction_address = to_address(operand_address_text).previous() + helpers["set_comment"](program, str(instruction_address), f"NE IMPORT -> {import_target}", "eol") + return str(instruction_address) + + +def main() -> None: + internal_fixups = load_internal_fixups() + import_fixups = load_import_fixups() + + applied_internal = 0 + skipped_non_call_internal = 0 + annotated_imports = 0 + internal_examples: list[str] = [] + skipped_examples: list[str] = [] + import_examples: list[str] = [] + + transaction_id = program.startTransaction("Apply NE far-call fixups") + commit = False + try: + for source, (target, label) in sorted(internal_fixups.items()): + instruction_address = patch_internal_far_call(source, target, label) + if instruction_address is None: + skipped_non_call_internal += 1 + if len(skipped_examples) < 10: + skipped_examples.append(f"{source} -> {target} ({label})") + continue + applied_internal += 1 + if len(internal_examples) < 10: + internal_examples.append(f"{instruction_address} -> {target} ({label})") + + for source, target in sorted(import_fixups.items()): + instruction_address = annotate_import_far_call(source, target) + annotated_imports += 1 + if len(import_examples) < 10: + import_examples.append(f"{instruction_address} -> {target}") + + helpers["set_comment"]( + program, + "0000:ffff", + "NE fixup placeholder only. Internal far calls have been patched from the verified relocation table; remaining xrefs, if any, are import callsites or unresolved non-internal cases.", + "pre", + ) + + commit = True + finally: + program.endTransaction(transaction_id, commit) + + print(f"Applied internal far-call patches: {applied_internal}") + for line in internal_examples: + print(f" {line}") + print(f"Skipped internal far-ptr fixups that were not literal CALLF instructions: {skipped_non_call_internal}") + for line in skipped_examples: + print(f" {line}") + print(f"Annotated import far calls: {annotated_imports}") + for line in import_examples: + print(f" {line}") + + +main() \ No newline at end of file diff --git a/tools/pyghidra_crusader/__pycache__/cli.cpython-311.pyc b/tools/pyghidra_crusader/__pycache__/cli.cpython-311.pyc index 1bc6291..ccc61f3 100644 Binary files a/tools/pyghidra_crusader/__pycache__/cli.cpython-311.pyc and b/tools/pyghidra_crusader/__pycache__/cli.cpython-311.pyc differ diff --git a/tools/pyghidra_crusader/__pycache__/cli.cpython-314.pyc b/tools/pyghidra_crusader/__pycache__/cli.cpython-314.pyc index 8a88405..9512d19 100644 Binary files a/tools/pyghidra_crusader/__pycache__/cli.cpython-314.pyc and b/tools/pyghidra_crusader/__pycache__/cli.cpython-314.pyc differ diff --git a/tools/pyghidra_crusader/__pycache__/common.cpython-314.pyc b/tools/pyghidra_crusader/__pycache__/common.cpython-314.pyc index d0f6565..62037bf 100644 Binary files a/tools/pyghidra_crusader/__pycache__/common.cpython-314.pyc and b/tools/pyghidra_crusader/__pycache__/common.cpython-314.pyc differ