Crusader_Decomp/docs/usecode-roundtrip-ir.md

# USECODE Round-Trip IR Plan

## Purpose

This note records the current evidence-backed path from Crusader USECODE bytes to a human-readable, editable, and recompilable script form.

It is intentionally conservative. ScummVM gives strong external anchors for the container layout, class/event numbering, and intrinsic naming, but it is not a symbol map for the DOS binary and it is not a ready-made round-trip compiler.

## Externally Anchored Pieces

### Container and class layout

ScummVM now gives a concrete second implementation for the Crusader USECODE class layout:

- `usecode/usecode_flex.cpp` treats each class body as archive object `classid + 2`.
- Class names come from archive object `1` at `name_object + 4 + 13 * classid`.
- For Crusader, the class base offset is read from class bytes `8..11` and then decremented by `1`.
- Crusader event count is computed as `(base_offset + 19) / 6`.
- `usecode/usecode.cpp` resolves event `N` from class data at `20 + 6 * N`, with the code offset stored in bytes `+2..+5` of each 6-byte event record.

Combined with the already validated FLEX container notes, the current externally anchored container model is:

- FLEX entry count at `0x54`
- FLEX table at `0x80`
- USECODE class object index = `classid + 2`
- Crusader class header contains a four-byte base-offset field at bytes `8..11`
- Crusader event table entries are 6 bytes each, with a known dword code offset and an still-unknown leading word

ScummVM also makes one implementation choice explicit that matters for the current mismatch: `uc_machine.cpp` uses `get_class_base_offset()` as the execution-stream base for Crusader class code, not only as metadata for event counting. That means the `obj[8..11] - 1` value is part of the live code-addressing model in ScummVM, not just a comment-level interpretation.

### Binary-side validation against owner-loaded classes

The first direct local validation pass against sampled owner-loaded EUSECODE class records now splits the ScummVM model into two parts: one part is confirmed, and one part still needs reconciliation.

Confirmed on sampled records (`EVENT`, `NPCTRIG`, `SURCAMNS`, `JELYHACK`, `REE_BOOT`, `SURCAMEW`, `SFXTRIG`):

- The extracted chunk at table offset `0x88` behaves like object `1` for class names.
- For each sampled class body, deriving `object_index = (table_offset - 0x80) / 8`, then `class_id = object_index - 2`, and then reading 13 bytes from object `1` at `4 + 13 * class_id` yields the expected class name.
- The class bodies do have a stable 4-byte header field at bytes `8..11`.
- The region at `class + 20` is a real 6-byte event-slot table with `u16 unknown_word + u32 code_or_payload_field` layout.

Broader family spot-checks now keep the same local structure on the owner-loaded side. In addition to the first validated set, the nearby `_BOOT` and environmental event families (`AND_BOOT`, `BRO_BOOT`, `COR_BOOT`, `VAR_BOOT`, `FLAMEBOX`, `NOSTRIL`, `STEAMBOX`) continue to fit the same `table_offset -> object_index -> class_id` progression with a stable bytes-`8..11` dword and a 6-byte table at `+20`. No contradictory sample has appeared in the local EUSECODE set.

Not yet reconciled with ScummVM's current formula note:

- In the sampled owner-loaded records, the raw dword at bytes `8..11` is `0x00d4`, `0x00da`, or `0x00e6`.
- Treating that dword directly as the first post-event-table offset makes the layout line up cleanly: `(dword_at_8 - 20) / 6` gives 32, 33, or 35 valid slots in the samples.
- Scanning instead with the previously noted ScummVM-style `(base_offset + 19) / 6` interpretation overruns into inline payload and class-name bytes in the same samples.

Current best explanation:

- The mismatch is now best explained as a ScummVM interpretation/detail issue, not as a proven loader-side rewrite.
- The same ScummVM code path that decrements bytes `8..11` by `1` also uses that decremented value as the code-stream base. On the local owner-loaded records, this fits naturally if the raw dword is the first code-byte offset and event-table dword offsets are 1-based relative to `code_base_minus_one`.
- Under that reading, the sampled event-count rule becomes `(code_base_minus_one - 19) / 6`, which is exactly equivalent to `(raw_u32_at_8_11 - 20) / 6` and matches the validated `32/33/35` slot counts.
- The `000d` loader/runtime path (`000d:44df -> 000d:4c99 -> 000d:7000 -> 000d:46ec`) currently shows indexed file loading and slot-table materialization, but no verified per-class header rewrite before the VM consumes owner-backed records.

Current safe conclusion:

- The owner-loaded class records are compatible with `object 1` names, `classid + 2` body lookup, a header field at bytes `8..11`, and 6-byte event records at `+20`.
- The exact meaning of the bytes-`8..11` field is now narrower: on the local owner-loaded records it is best read as the first code-byte offset, with ScummVM's decremented `base_offset` acting as a `code_base_minus_one` anchor for 1-based event code offsets.
- The leading word of each 6-byte event entry remains unresolved.

### VM/runtime model

ScummVM also anchors several VM behaviors that line up with the current raw-binary work:

- `usecode/uc_machine.cpp` uses `ByteSet(0x1000)` for Crusader globals rather than the U8 bitset path.
- Remorse initializes global `0x003c` to avatar number `1`; Regret initializes `0x001e`.
- Opcode `0x11` is class/event dispatch in Crusader: the bytecode operand is an event number that is translated through `get_class_event()` before execution.

That makes the current local reading stronger: the `000d` runtime lane looks like a Crusader-specific object/event VM that should be interpreted against Crusader event ordinals, not against U8 assumptions.

### Event names

`convert/crusader/convert_usecode_crusader.h` gives a named event table for ids `0x00..0x1f`:

- Strongly usable names: `look`, `use`, `anim`, `setActivity`, `cachein`, `hit`, `gotHit`, `hatch`, `schedule`, `release`, `equip`, `unequip`, `combine`, `calledFromAnim`, `enterFastArea`, `leaveFastArea`, `cast`, `justMoved`, `avatarStoleSomething`, `animGetHit`, `unhatch`
- Weak placeholders remain for `0x0d` and `0x16..0x1f` (`func0D`, `func16`..`func1F`)

This is enough to annotate event ordinals safely, but not enough to rename raw binary handlers unless local behavior matches.

### Intrinsic tables

ScummVM provides two distinct kinds of intrinsic evidence:

- `convert/crusader/convert_usecode_crusader.h` and `convert_usecode_regret.h` provide ordinal-to-signature/name tables used for readable conversion.
- `usecode/remorse_intrinsics.h` and `usecode/regret_intrinsics.h` provide the live runtime dispatch tables.

The safe reading is:

- Remorse and Regret share the Crusader event-name table.
- Remorse and Regret do not share a single intrinsic numbering/signature map.
- Intrinsic names are strong hints for arity and broad subsystem identity, but they are still not direct rename authority for the DOS binary.

## Safe Reuse Rules

### Safe to import now

- Event names as labels for event ids `0x00..0x1f` in parsers, reports, and note files.
- Intrinsic ordinal names as `name_hint` or `signature_hint` metadata when the ordinal and argument-byte pattern match.
- High-level subsystem labels such as palette fade, camera, movie, audio, item/actor accessors, and weapon fire when they match existing binary evidence.
- Slot numbers from sampled owner-loaded classes even when the event name is still only a hint.

### Not safe to claim yet

- Direct raw-function renames based only on ScummVM event or intrinsic names.
- Remorse intrinsic numbering from Regret tables, or vice versa.
- Specific descriptor-family to slot-mask mappings that are not yet proven on the binary side.
- Meanings for the unknown leading word in the 6-byte Crusader event table entries.
- That the ScummVM `get_class_event_count()` formula applies unchanged to the sampled owner-loaded EUSECODE records.

## IR Requirements For Round-Tripping

The first script IR should preserve exact recompilation inputs before it tries to look pretty.

### Unit of decompilation

The IR should be organized as:

1. USECODE archive
2. class
3. event slot
4. instruction stream

That matches the externally anchored class/event layout and avoids baking in any still-unproven descriptor-to-runtime assumptions.

### Required top-level records

Each class record should preserve:

- `class_id`
- `class_object_index` (`classid + 2`)
- `name_slot_offset` (`4 + 13 * classid` within object `1`)
- `class_name`
- `raw_header_prefix`
- `raw_code_base_u32`
- `code_base_minus_one`
- `event_count`
- `raw_event_table_bytes`

Each event record should preserve:

- `event_id`
- `event_name_hint`
- `raw_event_entry_word`
- `code_offset`
- `raw_body_bytes`
- `decoded_ops`

## IR v0 Shape

The IR should separate authoritative fields from friendly hints.

```yaml
class:
  class_id: 0x00be
  class_name: EVENT
  class_object_index: 0x00c0
  raw_code_base_u32: 0x0138
  code_base_minus_one: 0x0137
  raw_header_prefix: <bytes>
  events:
    - event_id: 0x04
      event_name_hint: cachein
      raw_event_entry_word: 0x????
      code_offset: 0x00001234
      ops:
        - op: intrinsic_call
          intrinsic_ordinal: 0x001e
          name_hint: Item::I_fireWeapon
          signature_hint: Item::I_fireWeapon(Item *, x, y, z, byte, int, byte)
          arg_bytes: 0x10
        - op: vm_chain_mutation
          vm_ir: APPEND_UNIQUE_INDIRECT
          opcode_hint: 0x19
        - op: unknown_raw
          bytes: <exact original bytes>
```

### Why this shape

- `event_name_hint` is useful for humans but does not replace the event id.
- `name_hint` and `signature_hint` are useful for intrinsics but do not replace the ordinal.
- `unknown_raw` gives a lossless fallback for still-unmapped opcodes or operand forms.
- `raw_event_entry_word` keeps the compiler from losing bytes whose meaning is not yet settled.

## Operation Families Worth Lifting First

The current binary-side evidence supports lifting a small reversible operator set first:

- `intrinsic_call`
- `class_event_call`
- `append_unique_inline`
- `append_unique_indirect`
- `remove_matching_inline`
- `remove_matching_indirect`
- `materialize_or_forward_value`
- `prepend_inline_payload`
- `build_entity_link_matrix`
- `emit_or_pushback_result`
- `push_frame_word_literal`
- `compare_stream_dword_and_push_bool`
- `unknown_raw`

This is enough to represent the verified `000d:0988`, `000d:177c`, `000d:1acb`, `000d:208b`, `000d:21ed`, and `000d:22bc` families without pretending the whole VM is solved.

## Metadata That Must Survive Recompilation

The compiler side will need more than pretty script text. At minimum it must preserve:

- Original class ordering and sparse class ids
- Original class-name table slotting
- Raw class header bytes not yet semantically decoded
- Raw bytes `8..11` even when a derived `code_base_minus_one` is also stored
- Raw 6-byte event records, including the unknown leading word
- Exact event order within each class
- Exact code offsets or enough relocation data to rebuild them deterministically
- Intrinsic ordinals and argument-byte counts
- Width/sign information for immediates
- Inline versus indirect payload form
- String payload encoding and terminators
- Any unknown opcode byte sequences verbatim

If any of those are dropped, a source-level editor can still be readable, but it will stop being a trustworthy recompilation format.

## Practical Naming Policy

For near-term local RE and tooling:

- Use ScummVM event names as annotation labels for event slots.
- Store intrinsic names as hints attached to ordinals.
- Keep binary-facing renames driven by raw evidence, not by ScummVM alone.
- Treat `EVENT`, `_BOOT`, and `NPCTRIG` as the strongest current active-event families.
- Treat `JELYHACK` and `JELYH2` as referent-anchor classes, not standalone event records.
- Treat `SURCAMNS` and `SURCAMEW` as callback/eventTrigger holders, not proven active-event cores.

## Conservative Parser Rule To Adopt Now

For the current owner-loaded EUSECODE and round-trip IR work, the safest reversible rule is:

- Preserve the raw four-byte header field at bytes `8..11` as authoritative.
- Derive `code_base_minus_one = raw_u32_at_8_11 - 1` for code-addressing only.
- Derive `event_count = (raw_u32_at_8_11 - 20) / 6` only when that value is non-negative, divisible by `6`, and the resulting table end stays within the class object size.
- Treat each event entry as `u16 raw_event_entry_word + u32 raw_code_offset` at `class + 20 + 6 * slot`.
- Treat the event code offset as raw/opaque unless and until the code-addressing interpretation is needed; when needed, interpret it relative to `code_base_minus_one` so that offset `1` lands on the first code byte.
- If the divisibility or bounds checks fail, keep the class opaque and preserve raw bytes rather than forcing a guessed event count.
- `tools/extract_eusecode_flx.py` now implements this rule directly for the current owner-loaded EUSECODE work and emits `class_layout_index.tsv` plus `class_event_index.tsv` so raw header/event rows can be consumed by later IR tooling without re-deriving the arithmetic from prose.

## Remaining Binary-Side Gaps

The main blockers for a real round-trip compiler are still on the binary side:

- The meaning of the first two bytes in each 6-byte Crusader event record is still unverified.
- The exact provenance of ScummVM's current `get_class_event_count()` arithmetic is still unverified; current local evidence says the owner-loaded/raw records fit `raw_u32_at_8_11 = first_code_byte_offset`, while the ScummVM count formula appears sign-shifted relative to that layout.
- The upstream writer for selector local `[BP-0x32]` in the `000d:ebe3` sequencer is still unresolved.
- The full control-flow opcode set and branch encoding are not yet recovered.
- The exact on-disk source format behind `entity_vm_runtime_owner_resource_create` is still not identified.
- No direct descriptor-family to slot-mask mapping is proven yet.
- Callback/eventTrigger descriptors still do not have a callback-specific opcode family.

## Best Current Path

The strongest present path to a usable compiler/decompiler is:

1. Parse classes/events exactly as ScummVM does.
2. Keep the class/object indexing and event-entry layout from ScummVM, but use the conservative local event-count rule above for owner-loaded/raw class parsing until a main USECODE sample proves otherwise.
3. Decompile only the proven operator families into structured IR.
4. Preserve unknown bytes verbatim in place.
5. Attach ScummVM event and intrinsic names as hints, not as truth.
6. Recompile by rebuilding the original class header and event table layout first, then re-emitting decoded and opaque ops together.

That gets to a reversible editor sooner than waiting for a full semantic VM recovery.