Crusader_Decomp/docs/regret-hidden-debugger-investigation.md

1080 lines
No EOL
51 KiB
Markdown

# No Regret Hidden Debugger Investigation
## Scope
This note records a deeper live-Ghidra pass on `REGRET.EXE` for the hidden usecode debugger, with emphasis on three questions:
- did No Regret keep the hidden debugger as a real subsystem,
- did it keep a real bootstrap path that retail No Remorse appears to have lost,
- and did it keep any launch/open path that is materially stronger than retail No Remorse?
## Short Answer
Current best read: No Regret still contains the hidden usecode debugger as a real relocated subsystem, and unlike retail No Remorse it also preserves both a recovered writer/bootstrap for the debugger-state global and a live debugger-facing vtable override that can open the debugger when break/step conditions hit.
This is the most important cross-build finding so far for the retail unlock problem. In retail `CRUSADER.EXE`, the current blocker is that fresh live recovery still finds reads but no recovered writer for `1478:659c/659e`, and the retail break-state object still points at inert vtable slots. In `REGRET.EXE`, the analogous global at `1480:712c/712e` has both many debugger-side reads and one direct write from a compact constructor/store stub at `1398:0000`, and that same bootstrap rewires the freshly created object away from the inert base vtable and onto a live frontend vtable whose slot `0` is `usecode_debugger_open_for_current_unit`.
## Verified Regret Findings
### 1. The broad hidden cheat/debug framework still exists
Live decompile of `1148:34d2 Key_CheckSecretCodeSequences` confirms that No Regret still ships a broad hidden cheat/debug family rather than only a few leftover strings.
Recovered sequence behavior in this pass:
- `jassica16` now routes to the visible taunt `Of course we changed the cheats...` rather than acting as the real cheat-enable sequence.
- `loosecannon` toggles the main cheat-active latches at `1480:0ac0` and `1480:009b`.
- Additional hidden sequences still drive `Pix`, `Memory profiler`, and `Video test` style branches.
This does not prove the usecode debugger by itself, but it does show that No Regret retained a wider debug/testing framework than a stripped retail-only reading would suggest.
### 2. The hidden usecode debugger UI wrappers still exist, relocated
The debugger-specific names were not already present in the live `REGRET.EXE` session, but the structural equivalents were recovered by pivoting from `Dispatch_ModalGump` callers.
These functions are now renamed live in the active `REGRET.EXE` database:
- `1398:0000` = `usecode_debugger_bootstrap_init`
- `1398:0086` = `usecode_debugger_open_for_current_unit`
- `1398:020d` = `usecode_debugger_open_modal`
- `1398:0291` = `usecode_debugger_format_expression_to_shared_buffer`
- `1398:19b1` = `usecode_debugger_gump_create`
- `1398:1c2c` = `usecode_debugger_translate_registered_event`
- `1398:1dc6` = `usecode_debugger_forward_child_event`
- `1398:1df3` = `usecode_debugger_handle_event`
- `13e0:0000` = `usecode_debugger_break_state_create`
- `13e0:0053` = `usecode_debugger_break_state_update_line_and_maybe_break`
- `13e0:0419` = `usecode_debugger_break_state_enable_single_step`
- `13e0:0432` = `usecode_debugger_break_state_clear_runtime_break_flags`
- `13e0:0444` = `usecode_debugger_break_state_get_current_entry`
- `13e0:046f` = `usecode_debugger_break_state_vtable_slot0_noop`
- `13e0:0474` = `usecode_debugger_break_state_vtable_slot1_return_zero`
- `13f0:038b` = `usecode_debugger_interpreter_hook`
Why `1398:0086` matches the current-unit wrapper:
- It builds a debugger gump via `1398:19b1`.
- It reads the current debugger object from `1480:712c/712e`.
- It calls `13e0:0444` to fetch the current entry/unit context from that object.
- It resolves a usecode/unit file path under the shared `s_usecode` root.
- It conditionally loads the corresponding unit file and then dispatches the gump modally.
Why `1398:020d` matches the modal-open wrapper:
- It is the smaller sibling that creates the same gump and immediately sends it through `Dispatch_ModalGump` without the extra current-unit file-load path.
### 3. The event dispatcher is the real debugger dispatcher, not a generic text UI
Live recovery of `1398:1df3` shows the same usecode-debugger-style command/state machine seen in retail No Remorse, just relocated.
Recovered dispatcher lanes in this pass include:
- file open with `FILE NOT FOUND` and `Unable to open this file`
- run / break-next / single-step state changes
- `Goto Line`
- `Watch what?`
- `Inspect what?`
- `Global name`
- symbol-not-found / range-check / `Done` flows
- search flows with `Search for`, `Nothing to find`, and `Not found`
This is much stronger than a generic console/editor interpretation. It is the same hidden debugger family in functional form.
### 4. No Regret preserves the missing bootstrap writer
This was the key result of the pass.
`get_data_uses(1480:712c)` returned many reads from the recovered debugger wrappers/dispatcher and one direct write at `1398:0064`, followed immediately by a read at `1398:0067`.
Live decompile of `usecode_debugger_bootstrap_init` now gives the fuller shape. The function:
- allocates two debugger-side support buffers (`0x19a` and `0x32`)
- allocates `0x2f2`
- calls `13e0:0000`
- overwrites the newly created object's vtable pointer
- stores the returned far pointer into `1480:712c/712e`
- immediately touches object byte `+0x74`
Relevant instruction sequence:
- `1398:0029` pushes `0x2f2` to the allocator
- `1398:0042` calls `13e0:0000`
- `1398:004d` stores `0x6972` into the object vtable slot
- `1398:0060` writes `DX` to `1480:712e`
- `1398:0064` writes `AX` to `1480:712c`
- `1398:0067` reloads the far pointer
- `1398:006b` sets byte `ES:[BX + 0x74] = 1`
There are still no normal direct callers to `usecode_debugger_bootstrap_init`, which makes it look more like a segment/bootstrap initializer than a normal gameplay-invoked helper. But regardless of how it is reached, this is exactly the kind of recovered write-side path that is still missing in current retail No Remorse analysis.
### 5. The seg13e0 helper cluster is the Regret break-state family
Live decompile of `13e0:0000` shows a real constructor-sized helper for the debugger object:
- null-check plus optional internal allocation
- vtable/root pointer store
- bulk `0xffff` initialization across the breakpoint/callstack state area
- explicit zeroing of selected control fields
- total object size matching the `0x2f2` bootstrap allocation
Other recovered matches in the same cluster:
- `13e0:0444` returns the current line/entry-derived record pointer from the debugger object, matching the role needed by `1398:0086`
- `13e0:0419` arms the single-step state by zeroing `+0x76/+0x78` and setting byte `+0x75 = 1`
- `13e0:0432` clears the two runtime break/step bytes at `+0x74/+0x75` and is called by the interpreter-side hook after it sees the debugger object
- `13e0:0053` updates the current line, resolves the current entry, checks breakpoint/step conditions, and calls through vtable slot `0` when a break should fire
The current safest read is that seg13e0 is the No Regret counterpart of the retail seg1408 break-state helper family.
### 6. The bootstrap does more than create the object: it upgrades the base vtable to a live frontend vtable
This is the most important deeper result of the second pass.
`usecode_debugger_break_state_create` seeds the object with base vtable root `1480:713b`. Live vtable analysis shows that this base table still has the retail-style inert slot pair:
- slot `0` -> `13e0:046f usecode_debugger_break_state_vtable_slot0_noop`
- slot `1` -> `13e0:0474 usecode_debugger_break_state_vtable_slot1_return_zero`
But `usecode_debugger_bootstrap_init` immediately overwrites the object's vtable pointer with root `1480:6972`.
That replacement vtable is not inert. Its first recovered slots are:
- slot `0` -> `1398:0086 usecode_debugger_open_for_current_unit`
- slot `1` -> `1398:0291 usecode_debugger_format_expression_to_shared_buffer`
This is the clearest structural reason No Regret is stronger than retail for debugger launch behavior.
Retail No Remorse still has the base break-state object and the inert callback slots. No Regret keeps those same inert base methods in the constructor, but then the bootstrap upgrades the object onto a live frontend-aware vtable.
### 7. The interpreter-side hook also still appears to be wired
`13f0:038b` is called from `13f8:10fa` and reads the debugger global at `1480:712c/712e` before calling `13e0:0432`.
Recovered disassembly around the key handoff:
- `13f0:03dd` loads `1480:712c`
- `13f0:03e0` tests `1480:712e`
- `13f0:03e6` / `03ea` push the debugger far pointer
- `13f0:03ee` calls `13e0:0432`
That is the current strongest evidence that No Regret preserves not just the UI/event layer and not just the constructor/store path, but also the interpreter-side consumer/hook path that can actually consult the debugger object during VM execution.
### 8. No Regret appears to keep an auto-open-on-break path that retail No Remorse lacks
This is the practical launch-path conclusion.
`usecode_debugger_break_state_update_line_and_maybe_break` sets the current line, resolves the current entry, evaluates breakpoint/step conditions, and when those conditions hit it calls through vtable slot `0`.
Because the Regret bootstrap overwrites the object onto vtable root `1480:6972`, that slot `0` call is no longer inert. It now lands at `usecode_debugger_open_for_current_unit`.
That means Regret still appears to preserve this path:
- debugger object exists
- interpreter-side hook sees that object
- break/step state reaches `usecode_debugger_break_state_update_line_and_maybe_break`
- slot `0` dispatch opens the current-unit debugger UI
This is exactly the kind of end-to-end `break leads to debugger UI` path that retail No Remorse does not currently show, because retail still routes the equivalent slot `0` call into a no-op stub.
### 9. But no direct user-facing launch path has been recovered yet
The stronger Regret result still has an important limit.
Current direct-caller state in the live database:
- no normal direct callers to `usecode_debugger_bootstrap_init`
- no normal direct callers to `usecode_debugger_open_for_current_unit`
- no normal direct callers to `usecode_debugger_open_modal`
- `usecode_debugger_handle_event` is reached only through debugger callback wrappers (`usecode_debugger_translate_registered_event` and `usecode_debugger_forward_child_event`)
- `usecode_debugger_interpreter_hook` is called from `13f8:10da`, which looks like a real runtime/interpreter-side consumer
So the current best distinction is:
- Regret does preserve a real debugger-opening path once break/step conditions are active
- but this pass still did not recover a clean player-facing hotkey/menu/command-line entry that intentionally opens the debugger from normal gameplay
In other words, No Regret looks better wired than retail No Remorse, but still not yet proven to expose a deliberate user-facing launcher.
## Current Cross-Build Implications
### What this changes
The earlier retail conclusion was: the hidden debugger survives as an orphaned subsystem, but the missing writer/bootstrap makes minimum-modification entry hard to justify.
No Regret materially changes that framing.
The best current cross-build model is now:
- retail No Remorse still has the debugger UI, event dispatcher, and break-state helper family, but not yet a recovered writer for the debugger-state global and still appears to route slot `0` through inert stubs
- No Regret keeps the same broad subsystem shape, keeps a compact bootstrap stub at `1398:0000` that writes the debugger-state global, and upgrades the object from the inert base vtable at `1480:713b` to the live frontend vtable at `1480:6972`
- therefore the key difference is no longer only `writer exists versus writer missing`; it is also `live debugger-opening vtable path exists versus inert retail stubs`
### Why this matters for the retail unlock problem
If retail lost only a small bootstrap/writer path while Regret preserved it, then the smallest defensible retail activation path may be much closer to:
- recovering a missing retail analogue of the Regret bootstrap,
- restoring the retail equivalent of the Regret live vtable override,
- or identifying one surviving retail callsite that used to seed the same state,
- or borrowing the exact structural delta from a sibling build,
instead of inventing another wider retail-only interpreter patch chain.
## Force Options In No Regret
The remaining practical question is not only `does Regret keep the debugger?`, but `can we force it up by hacking?`
Current best read: yes, Regret is now the best known build for a forced debugger bring-up, but the viable options split into three very different classes:
- direct executable patching
- live memory forcing / trainer-style intervention
- usecode-assisted hybrids
Pure usecode-only forcing is still not evidenced.
### 1. Direct executable patching is now the strongest practical route
This is the biggest Regret-side difference from retail No Remorse.
Retail still appears to be missing both the write-side bootstrap and the live vtable promotion. Regret already has both of those pieces. That means a Regret-specific patch does **not** need to recreate the entire subsystem. It only needs to route some existing hidden/debug input lane into already-existing debugger entry points.
#### Best direct-open patch target
The lowest-risk explicit open path is now:
- ensure `usecode_debugger_bootstrap_init` has run or call it if the debugger global is null
- then call `usecode_debugger_open_modal`
Why this is the safest direct-open target:
- `usecode_debugger_open_modal` does not require the current-unit preload path
- it still uses the real Regret debugger gump constructor and event dispatcher
- it avoids depending on the current-entry index being valid before the first open
This should produce the most reliable `force the menu up` result if the immediate goal is simply to display the debugger UI.
#### Best full-context patch target
If the goal is not just `open a shell`, but `open the debugger on the current unit with real context`, the stronger path is:
- ensure the debugger object exists
- force the break/step state
- let the next interpreter-side debugger path auto-open through slot `0`
In Regret this is now structurally plausible because:
- `usecode_debugger_break_state_update_line_and_maybe_break` dispatches through slot `0`
- the live Regret vtable promotion makes slot `0` equal `usecode_debugger_open_for_current_unit`
- `usecode_debugger_interpreter_hook` is still wired into a real runtime caller at `13f8:10da`
This is the best path if the goal is `show me the real source/current-unit debugger experience`, not merely `show me a debugger window`.
#### Best patch host families
The most practical patch hosts are not generic gameplay code. They are hidden/debug-adjacent control points that already sit near cheat or diagnostic behavior.
Best current candidates:
- `Key_HandleOptionKeys` at `1148:0a9a`
- `Key_CheckSecretCodeSequences` at `1148:34d2`
Why these are the strongest patch hosts:
- they already belong to hidden/debug functionality rather than normal gameplay
- they already execute in live gameplay input conditions
- they are much smaller intervention points than rebuilding the retail interpreter patch family
Current concrete patch shapes worth considering:
1. repurpose one hidden secret-code completion lane to call `usecode_debugger_open_modal`
2. repurpose one hidden secret-code completion lane to call `usecode_debugger_bootstrap_init` if needed and then set the break/step bytes for the next interpreter break
3. repurpose one hidden option-key branch inside `Key_HandleOptionKeys` to do the same thing behind an intentional key combo
#### Why this is smaller than retail patching
In retail No Remorse, the smallest viable patch still had to recreate missing state bootstrap and missing live callback behavior. In Regret, those pieces are already there.
So the Regret patch problem is no longer `invent a debugger from fragments`. It is `redirect an existing hidden/debug input lane into an already-live debugger subsystem`.
That is a much smaller and more defensible hack surface.
### 2. Live memory forcing is viable, but only if the debugger object is actually live
Memory-only forcing is now plausible in Regret in a way that was not cleanly plausible in retail.
But there is one major condition:
- the debugger object and support buffers must already be initialized
If `usecode_debugger_bootstrap_init` has not run, then a raw memory poke to `1480:712c/712e` is not enough by itself. The bootstrap allocates support buffers, constructs the base object, and performs the vtable override. A pure `write one pointer` memory hack would skip too much setup.
#### Best memory-only open path if the object is live
If the object already exists and `1480:712c/712e` is valid, then the simplest trainer/debugger intervention is:
- call `usecode_debugger_open_modal`
This is the safest menu-only force path because it avoids needing a valid current-entry index first.
#### Best memory-only real-debug path if the object is live
If the goal is a real current-unit break rather than just a menu shell, the stronger live-memory path is:
- set the break/step state bytes in the debugger object
- let the next interpreted usecode activity reach `usecode_debugger_interpreter_hook`
The strongest currently evidenced control bytes are:
- byte `+0x74`: likely break-next / force-break style latch
- byte `+0x75`: single-step style latch
- words `+0x76` and `+0x78`: cleared by `usecode_debugger_break_state_enable_single_step`
Evidence:
- bootstrap sets `+0x74 = 1`
- `usecode_debugger_break_state_clear_runtime_break_flags` clears `+0x74/+0x75`
- `usecode_debugger_break_state_enable_single_step` sets `+0x75 = 1` and zeros `+0x76/+0x78`
- `usecode_debugger_handle_event` contains paths that also manipulate `+0x74/+0x75`
Current safest read is therefore:
- setting single-step or break-next state in memory and then provoking the next live usecode opcode is a plausible force path in Regret
- but it depends on the debugger object already being real and on the current interpreter context being valid
#### Main limitation of memory-only forcing
Memory-only forcing is practical for a debugger/trainer workflow, but it is not the best first patch if the goal is a reusable mod or stable reproduction path. It depends on runtime timing and on the hidden object already existing.
### 3. Pure usecode-only forcing is still not evidenced
This remains the weakest route.
Current negative evidence:
- no recovered usecode-visible primitive constructs the debugger break-state object
- no recovered usecode-visible primitive writes `1480:712c/712e`
- no recovered usecode-visible primitive directly calls `usecode_debugger_open_modal` or `usecode_debugger_open_for_current_unit`
- the hidden secret-code and option-key lanes that are currently recovered are still compiled cheat/debug code, not usecode-side entry points
So the current safest conclusion is:
> pure usecode by itself is still not an evidence-backed way to bring the Regret debugger up.
### 4. A usecode-assisted hybrid is plausible and may be the cleanest experimental setup
Even though pure usecode does not currently look sufficient, usecode can still be useful as part of a hybrid forcing plan.
The strongest hybrid model is:
- use a tiny executable patch or runtime memory intervention to ensure the debugger object exists and the break/step latch is armed
- use a replacement `EUSECODE.FLX` or a known script trigger to force predictable nearby usecode execution
- let the next interpreter-side debugger break land in a controlled current-unit context
This hybrid is stronger than pure usecode because it uses usecode only for what the current evidence actually supports: providing predictable runtime context after the compiled debugger path has already been armed.
That means `-u` can still matter in Regret, but not as `open debugger from script`. Its best role is:
- generate deterministic usecode activity once the debugger break/step machinery has already been forced by code or memory.
### 5. Ranked Regret force options
If the goal is simply `make the debugger menu appear`, the current ranking is:
1. small executable patch that routes a hidden key/sequence lane into `usecode_debugger_open_modal`
2. live memory/trainer call into `usecode_debugger_open_modal` if the object is already initialized
3. small executable patch that arms break/step state and lets the next interpreter break auto-open `usecode_debugger_open_for_current_unit`
4. live memory forcing of break/step bytes plus a controlled next usecode opcode
5. usecode-assisted hybrid using `-u` only after one of the code/memory force paths is in place
6. pure usecode-only forcing
If the goal is specifically `open on the current unit with the real debugger context`, the order changes slightly:
1. executable patch that ensures bootstrap and then forces break/step state
2. live memory forcing of break/step state if the debugger object is already present
3. hybrid `code/memory force + -u` setup to land in a deterministic usecode context
4. direct `usecode_debugger_open_for_current_unit` call from a patch host if runtime current-entry state is known good
5. direct `usecode_debugger_open_modal`
### 6. Bottom line on forcing the Regret debugger
The main result of this pass is that No Regret is no longer just an evidence source for retail comparison. It is also the first build where a practical forced debugger bring-up looks realistically hackable without rebuilding half the subsystem.
Current safest forcing conclusion:
- executable patching is the strongest practical route
- live memory forcing is plausible if the object already exists
- usecode is useful only as a hybrid context-generator, not yet as a direct launcher
If the immediate goal is `bring the menu up by any reasonable hack`, the best current target is a small Regret-specific patch from a hidden cheat/input lane into `usecode_debugger_open_modal` or into the break/step auto-open path.
## Live DOSBox-X Runtime Check
After the static analysis, a first real DOSBox-X debugger pass was used to test the least-invasive runtime ideas directly.
These results materially narrow the practical options.
### 1. The debugger object is not live during ordinary gameplay
Using the DOSBox-X debugger's data view on the runtime data selector:
- `D 05FF 712C`
the first four bytes at `05FF:712C` were:
- `00 00 00 00`
That means the runtime debugger-object pointer is still null in ordinary gameplay at the tested point.
This is an important practical constraint:
- the poke-only single-step plan cannot be the first move in ordinary gameplay, because there is no live object to poke yet.
### 2. Known hidden input lanes did not initialize the object
Two runtime checks were performed after confirming the pointer was null.
#### `loosecannon`
After entering the real No Regret cheat-enable sequence and re-checking:
- `D 05FF 712C`
the pointer remained:
- `00 00 00 00`
#### F10 option-key lane
After triggering the Regret F10 cheat lane and re-checking:
- `D 05FF 712C`
the pointer still remained:
- `00 00 00 00`
Current safest conclusion from those live tests:
- neither the main hidden cheat-enable sequence nor the known F10 option-key cheat lane initializes the debugger object in ordinary gameplay.
That materially weakens the earlier hope that a purely existing hidden input path might already seed the debugger for us.
### 3. The runtime vtable selectors were still valuable to recover
Even though the first live forcing attempts did not succeed, the DOSBox-X reads still recovered the runtime code selectors for the debugger family.
Reading the vtable roots directly from the live data segment:
- `D 05FF 6972` -> `86 00 17 05 91 02 17 05`
- `D 05FF 713B` -> `6F 04 5F 05 74 04 5F 05`
This gives the runtime selector mapping:
- live frontend debugger code selector = `0517`
- live base break-state helper selector = `055F`
and confirms the runtime slot mapping:
- `0517:0086` = `usecode_debugger_open_for_current_unit`
- `0517:0291` = `usecode_debugger_format_expression_to_shared_buffer`
- `055F:046F` = inert base slot `0`
- `055F:0474` = inert base slot `1`
So the DOSBox-X pass still strengthened the runtime model even though the forcing attempt failed.
### 4. Raw DOSBox-X far-call injection is currently not robust
Two manual far-call experiments were attempted through the DOSBox-X debugger.
#### Manual bootstrap call attempt
A manual far-call frame was built on the stack and execution was redirected to:
- `0517:0000` (`usecode_debugger_bootstrap_init`)
Result:
- execution entered the function body successfully
- but after return/break handling, the debugger pointer at `05FF:712C` still read as `00 00 00 00`
Current safest read:
- the raw DOSBox-X call injection method is not yet a trustworthy way to initialize the debugger object from an arbitrary paused gameplay state.
#### Manual `open_modal` call attempt
A second manual far-call frame was built and execution was redirected to:
- `0517:020D` (`usecode_debugger_open_modal`)
Result:
- DOSBox-X reported `DYNX86: Can't run code in this page!`
- the game effectively crashed / failed to continue cleanly
Current safest read:
- raw DOSBox-X debugger call injection into the Regret debugger frontend is currently too fragile to recommend as the primary least-invasive workflow.
This does **not** prove the underlying debugger functions are broken. It only shows that this particular external debugger-based call-injection method is unstable from the tested paused context.
## Concrete Manual Hex Retarget
The best first on-disk experiment is no longer a generic "patch some hidden key path" idea. Regret now has one specific low-risk family that stays inside existing hidden-sequence control flow and preserves the original caller stack discipline.
### Recommended patch family
Use the `loosecannon` secret-sequence completion lane in `1148:34d2 Key_CheckSecretCodeSequences`.
Important NE format note:
- the code sites themselves are on-disk `CALLF` placeholders: `9A FF FF 00 00`
- the real internal far-call target lives in the segment relocation record, not in the opcode immediate bytes
- so the manual hex patch should edit the fixup records, not the `9A FF FF 00 00` code bytes
Why this is the current best manual hex target:
- it is already a hidden debug-adjacent input path reached from ordinary gameplay
- it already plays a short side-effect call and then opens a modal message path
- the call shapes line up well enough to retarget without adding a trampoline
- the patch can stay at `10` bytes for the normal case, or `15` bytes if the Christmas-only alternate message lane is also covered
### Patch level 1: 5-byte smoke test
If the goal is only `prove that a tiny retarget can bring the debugger menu up`, the smallest first test is a single call-target replacement in the normal `loosecannon` active-message lane.
Code site for reference:
- live address `1148:3702`
- raw file offset `0x7A502`
- on-disk bytes `9A FF FF 00 00 83 C4 14`
Fixup record to edit:
- file offset `0x7BB05`
- old record `03 00 03 37 6B 00 46 00`
- new record `03 00 03 37 74 00 0D 02`
That changes the internal target from:
- segment index `0x006B`, offset `0x0046` = `1350:0046`
to:
- segment index `0x0074`, offset `0x020D` = `1398:020d` (`usecode_debugger_open_modal`)
Why this site is attractive:
- the call is reached only after the hidden sequence has completed and the cheat-active lane has been selected
- immediately before the call, the original code pushes two zero words
- `usecode_debugger_open_modal` is the safest known menu-only debugger entry because it does not require the current-unit preload path
Why the stack is plausibly safe here:
- the original message builder takes many arguments, but the final two pushed words at this site are both `0`
- retargeting to `usecode_debugger_open_modal` means the debugger wrapper should see `(0, 0)` as its first two arguments
- the caller already does `ADD SP,0x14` after return, so the extra message-builder arguments are still discarded cleanly
Main limitation of the 5-byte smoke test:
- ordinary gameplay still showed `1480:712c/712e = 0`, so `open_modal` may still depend on state that is not always ready
- this is the best tiny proof-of-life patch, but not the strongest first patch if the goal is `make it work reliably`
### Patch level 2: recommended 10-byte bring-up patch
If the goal is the smallest practical patch with a real chance to work from a clean ordinary gameplay state, use a two-call retarget in the same `loosecannon` path:
1. replace the ambient-sound call with `usecode_debugger_bootstrap_init`
2. replace the modal message call with `usecode_debugger_open_modal`
#### Site A: bootstrap retarget
Code site for reference:
- live address `1148:3678`
- raw file offset `0x7A478`
- on-disk bytes `9A FF FF 00 00 83 C4 02`
Fixup record to edit:
- file offset `0x7BB25`
- old record `03 00 79 36 5C 00 D0 04`
- new record `03 00 79 36 74 00 00 00`
That changes the internal target from:
- segment index `0x005C`, offset `0x04D0` = `12d8:04d0`
to:
- segment index `0x0074`, offset `0x0000` = `1398:0000` (`usecode_debugger_bootstrap_init`)
Why this site works well:
- it executes before the `loosecannon` lane branches into the active/inactive message paths
- the original call already has exactly one pushed word argument (`PUSH 1901`), and the caller already removes it with `ADD SP,0x2`
- `usecode_debugger_bootstrap_init` is currently treated as a no-argument helper, so the extra stack word should be ignored and then cleaned by the existing caller logic
What this changes functionally:
- the patch trades the one-shot confirmation sound for a chance to seed the debugger object and support buffers before the modal-open retarget runs
#### Site B: open-modal retarget
At `0x7BB05`, replace the fixup record:
- old: `03 00 03 37 6B 00 46 00`
- new: `03 00 03 37 74 00 0D 02`
This is the same site as the 5-byte smoke test, but paired with the bootstrap retarget it becomes the best current low-byte bring-up candidate.
### Patch level 2b: 15-byte Christmas-safe variant
The `loosecannon` active-message lane has a seasonal split:
- `1148:36c0` = Christmas-only message call
- `1148:3702` = normal active-message call
If the patch should work regardless of the `Date_IsAlmostChristmas` branch, also retarget `1148:36c0` the same way:
- code site for reference: `1148:36c0` = raw file offset `0x7A4C0` with on-disk bytes `9A FF FF 00 00 83 C4 14`
- fixup record at `0x7BB15`
- old record `03 00 C1 36 6B 00 46 00`
- new record `03 00 C1 36 74 00 0D 02`
That makes the full robust manual patch set:
- `1148:3678` -> bootstrap retarget
- `1148:3702` -> normal active-message open-modal retarget
- optional `1148:36c0` -> Christmas active-message open-modal retarget
Byte-budget summary:
- `5` bytes: menu-only smoke test
- `10` bytes: recommended normal-case bring-up patch
- `15` bytes: recommended bring-up patch plus Christmas coverage
### Why this family is better than the earlier raw DOSBox-X call injection
The DOSBox-X experiments showed that arbitrary external far-call injection was too fragile from a paused gameplay context. This retarget stays inside an already-valid in-engine call path instead:
- real gameplay input reaches `Key_CheckSecretCodeSequences`
- the original function prologue, data selectors, and stack frame are already valid
- the retargeted calls return into the original control flow rather than into a manually fabricated debugger stack frame
So even though the patch is still speculative until tested, it is materially better grounded than another raw debugger-console call injection attempt.
### Expected trigger flow
With the recommended `10`-byte patch in place, the expected test flow is:
1. boot the patched `REGRET.EXE`
2. enter gameplay normally
3. type `loosecannon`
4. the original sound call is replaced by `usecode_debugger_bootstrap_init`
5. the original active cheat-message call is replaced by `usecode_debugger_open_modal`
6. if the stack and UI assumptions hold, the hidden debugger menu should appear instead of the normal `Cheats are now active.` message
### Side effects and risk profile
This patch family is attractive because it avoids trampolines, new code, and interpreter edits, but it still has visible tradeoffs:
- entering `loosecannon` still flips the normal cheat-active latches first
- the original confirmation sound is lost at the bootstrap-retarget site
- the original cheat-active dialog is replaced by the debugger modal
- if `usecode_debugger_bootstrap_init` has hidden caller assumptions that were not recovered statically, the `1148:3678` retarget could still fail or destabilize the lane
Even so, this remains the best current manual-hex candidate because every other known route either needs more bytes, needs an injected trampoline, or depends on a runtime object that is still null in ordinary gameplay.
### If the 10-byte patch fails
The next fallback should still stay in the same family rather than jumping straight to a wider rewrite.
Recommended fallback order:
1. add the Christmas-safe `1148:36c0` retarget if only the normal lane was patched
2. keep the bootstrap retarget and move the open retarget to a different hidden-sequence message site with the same `CALLF 1350:0046` argument shape
3. only after that, consider a low-teens-byte local trampoline that does `bootstrap_init` and `open_modal` back-to-back explicitly
The important point is that the first practical Regret patch should still be a call-target retarget, not an interpreter rewrite.
## Least-Invasive Short List
Given the practical constraint that larger patches tend to break startup, the current Regret options compress to a much shorter ranked list.
### Best overall: live memory call hack
If the goal is `make the debugger appear with the fewest moving parts`, the current best candidate is a live memory call sequence rather than an on-disk EXE patch.
#### Menu-only sequence
Current best minimal sequence:
1. if `1480:712c/712e` is null, call `usecode_debugger_bootstrap_init`
2. call `usecode_debugger_open_modal(0, 0)`
Why this is currently the cleanest candidate:
- it is only one or two far calls
- it does not need a persistent EXE modification
- it avoids relying on current-unit state being valid first
- it reuses the already-working Regret gump/event machinery directly
Current practical status after DOSBox-X testing:
- in theory this is still the smallest call sequence
- in practice, ordinary gameplay had a null debugger pointer, and the raw DOSBox-X manual-call method did not initialize it reliably
So this remains the smallest **theoretical** path, but not the smallest **validated DOSBox-X workflow**.
#### Current-unit sequence
If the goal is `open the real current-unit debugger rather than just show the menu shell`, the current best live-memory sequence is:
1. if `1480:712c/712e` is null, call `usecode_debugger_bootstrap_init`
2. call `usecode_debugger_break_state_enable_single_step(DAT_1480_712c)`
3. provoke the next usecode/interpreter activity
Why this is stronger than a direct call to `usecode_debugger_open_for_current_unit`:
- it uses the already-recovered Regret runtime path instead of cold-entering the wrapper
- it should land through the real interpreter-side hook and slot-`0` auto-open path
- it gives the best chance of valid current-unit/current-line state on entry
Current practical status after DOSBox-X testing:
- this is still structurally the cleanest `real debugger context` path
- but it is blocked in ordinary gameplay until some method really seeds the debugger object first
### Best poke-only candidate: 3 to 4 writes plus one usecode trigger
If the tooling can poke memory but not call functions, the current least invasive poke recipe is the single-step path.
Current best candidate writes inside the object pointed to by `1480:712c/712e`:
- byte `+0x75 = 1`
- word `+0x76 = 0`
- word `+0x78 = 0`
Optional fourth write if needed for stability testing:
- byte `+0x74 = 0`
Then provoke one nearby live usecode action.
Why this is the best poke-only candidate:
- it mirrors the verified helper `usecode_debugger_break_state_enable_single_step`
- it is only 3 required writes, 4 at most
- it tries to reuse the real Regret interpreter path rather than bypassing it
Main limitation:
- this still assumes the debugger object already exists
- if `1480:712c/712e` is null, this poke-only plan is not enough by itself
Current live status:
- ordinary gameplay checks in DOSBox-X found `1480:712c/712e` still null even after `loosecannon` and the F10 cheat lane
- so this poke-only recipe is currently a second-stage trick, not a first-stage bring-up
### Best tiny EXE patch family: retarget one hidden cheat/input lane
If an on-disk patch is acceptable only when it is very small, the current best family is still a key/sequence-lane retarget rather than any interpreter rewrite.
#### Current smallest credible patch shape
Current best concept:
- hijack one hidden secret-sequence completion lane in `Key_CheckSecretCodeSequences`
- or hijack one hidden option-key lane in `Key_HandleOptionKeys`
- route it into `usecode_debugger_open_modal`
- optionally call `usecode_debugger_bootstrap_init` first if runtime testing shows the object is not always live
Why this remains the smallest credible EXE family:
- these hosts already sit in hidden/debug control paths
- they are already reachable from normal gameplay input
- they avoid touching the interpreter core, startup path, or loader path
#### Byte-budget estimate
Current realistic byte budgets are:
- about `5` bytes if a single existing `CALLF` target can be retargeted cleanly
- about `10-16` bytes if one additional local cleanup/skip patch is needed after the redirected call
- larger than that only if runtime testing proves `usecode_debugger_bootstrap_init` must also be injected into the same path
Current safest reading of those budgets:
- a true one-call retarget is the ideal target
- a two-site patch in the low-teens of bytes is still likely acceptable under the current `few bytes only` constraint
- anything that grows into a mini-trampoline or interpreter rewrite is no longer in the preferred class
#### Best first executable candidates
Current best on-disk patch hosts remain:
- one hidden secret-code completion lane in `Key_CheckSecretCodeSequences`
- one hidden option-key lane in `Key_HandleOptionKeys`
Current best behavior targets remain:
- `usecode_debugger_open_modal` for the smallest reliable menu bring-up
- `usecode_debugger_break_state_enable_single_step` if the goal is to reuse the existing auto-open path and keep the patch logic tiny
Current practical note after DOSBox-X runtime testing:
- because the object stayed null through normal gameplay and the raw DOSBox-X far-call method was unstable, a very small executable retarget now looks *more* attractive than a complicated external live-call recipe for the first reproducible bring-up attempt.
### Weakest option under the current constraints: pure usecode soft-mod
Soft-modding through usecode is still the safest class operationally, but it is not the strongest class evidentially.
Current best read remains:
- pure usecode cannot currently bootstrap the debugger object
- pure usecode cannot currently write `1480:712c/712e`
- pure usecode cannot currently call the debugger wrappers directly
So under the current `least invasive` constraint, usecode only becomes competitive if it is paired with one tiny compiled or live-memory intervention first.
### Final least-invasive ranking
For `just make the menu appear`:
1. tiny EXE patch that routes one hidden key/sequence lane into `open_modal`
2. live memory call hack through a smarter in-process tool, not raw DOSBox-X far-call injection: `bootstrap if needed -> open_modal`
3. tiny EXE patch that routes one hidden key/sequence lane into `enable_single_step`
4. live memory poke hack: single-step bytes then trigger one usecode action, but only after some other method creates the object
5. usecode-assisted hybrid after one tiny compiled or live-memory intervention
6. pure usecode-only forcing
For `open the real current-unit debugger with context`:
1. tiny EXE patch into the single-step auto-open path
2. live memory call hack through a smarter in-process tool: `bootstrap if needed -> enable_single_step -> trigger usecode`
3. poke-only single-step recipe on an already-live object
4. direct `open_for_current_unit` call if runtime context is already known good
5. pure menu-open via `open_modal`
Under the user's current stability constraint, the strongest recommendation is therefore:
> prefer the smallest possible hidden-input EXE retarget first if it can truly stay in the single-digit or low-teens byte range. Raw DOSBox-X far-call injection is currently too fragile to be the primary workflow, and poke-only memory forcing still cannot start from a null object.
## What The `.unk` File Actually Looks Like
Now that the patched Regret build can open the hidden debugger menu, the next blocker is the file-open prompt. The immediate practical question is whether that prompt wants one of the stock `UNK*.DAT` files or some other artifact.
Current best answer: the debugger does **not** appear to want `UNKDS.DAT` directly. It wants a per-unit `.unk` file under the `USECODE` root, and that file is expected to be a plain text source-style listing, not a small binary metadata blob.
### Short answer
- the debugger builds `<current-unit-name>.unk` under `s_usecode`
- the debugger-side source loader opens that path as a normal file, reads the entire file into memory, splits it into lines, and rejects files that do not look like text
- `UNKCOFF.DAT` is a separate fixed archive-like table of 4-byte code pointers used by the older `unkcoffs` tooling
- `UNKDS.DAT` is a second tiny fixed sidecar loaded by a different code path; it is probably related to the same broader usecode/debug data family, but it is **not** the same thing as the `.unk` unit file the menu is asking for
So the current safest read is:
> `.unk` is a human-readable per-unit debugger/source listing, while `UNKCOFF.DAT` and `UNKDS.DAT` are shared support data files.
### 1. The debugger explicitly appends `.unk` to the current unit name
The retail exported source for `usecode_debugger_open_for_current_unit` still shows the clearest shape, and the Regret wrapper is the relocated counterpart.
Recovered flow:
- fetch current callstack entry unit name via `UsecodeDebuggerBreakState::CurrentEntryGetUnitName`
- call `Filespec_GetFullPath(0, s_usecode, unit_name, ".unk")`
- if that resolved path differs from the currently loaded file, call `usecode_debugger_load_unit_file(...)`
That is strong direct evidence that the debugger wants a filename of the form:
- `USECODE\<unitname>.unk`
not one of the fixed shared filenames like `UNKDS.DAT`.
### 2. The `.unk` loader expects a text file, not a tiny binary blob
The debugger-side file path goes through:
- `usecode_debugger_load_unit_file`
- `usecode_debugger_source_file_create_or_open`
- `usecode_debugger_source_file_load_path`
- `Debugump_13a0_2e0a`
What `Debugump_13a0_2e0a` does is the key format clue:
- seeks to the file end to get its size
- allocates a buffer for the whole file plus terminator space
- reads the whole file into memory
- runs a line-splitting pass that walks the buffer looking for `\n`
- replaces the byte before each newline with `0`, which is consistent with stripping CRLF text lines into C strings
- stores per-line pointers for later indexed access through `usecode_debugger_source_file_get_line_text`
- shows `This is probably not a text file` if the loaded content fails its text-like sanity check
This is not the behavior of a parser for a small binary metadata file. It is the behavior of a source/listing viewer for line-oriented text.
Current safest format read:
- `.unk` should be an ASCII or DOS-text file
- line-oriented
- newline-delimited, very likely CRLF-friendly
- small enough to fit the fixed debugger source buffer and line table
### 3. No `.unk` samples are currently present in the workspace
A workspace-wide search for `*.unk` returned no files.
That means the current evidence is asymmetric:
- the loader shape is clear
- the expected filename suffix is clear
- but no stock sample `.unk` file has been recovered yet from the current repo/game folders
So the format conclusion is based on the loader and parser behavior, not on a known-good sample file.
### 4. What `UNKCOFF.DAT` appears to be
The existing `tools/unkcoffs` utilities are the strongest clue here.
Those scripts:
- read `UNKCOFF.DAT` as raw 4-byte integers
- split each entry into `segment` and `offset`
- convert them into NE far addresses like `1000 + (seg - 1) * 8 : off`
- align the resulting entries with intrinsic name dumps such as `reg_functions.txt`, `rem_functions.txt`, `regret_ints.py`, and `remorse_ints.py`
So `UNKCOFF.DAT` is not a text source file. It is a compact offset table.
Verified current sizes:
- Remorse `USECODE\UNKCOFF.DAT` = `1244` bytes = `311` 4-byte entries
- Regret `USECODE\REGRET\UNKCOFF.DAT` = `1400` bytes = `350` 4-byte entries
Current safest interpretation:
- `UNKCOFF.DAT` is a shared intrinsic or function-offset table used to map usecode ordinals to compiled handlers
- it belongs to the broader usecode/debug symbol ecosystem
- but it is not the per-unit `.unk` text listing the debugger menu is trying to open
This also explains the long-standing `unkcoffs/` tooling and naming in the repo: that tooling is built around `UNKCOFF.DAT` as an address table.
### 5. What `UNKDS.DAT` appears to be
`UNKDS.DAT` is opened by a separate fixed-filename loader in the old retail export lane near `seg_1410`, not by the debugger's unit-file wrapper.
What is currently verified:
- the executable contains a literal `unkds.dat` string
- a dedicated startup/helper path opens that fixed file under `s_usecode`
- the loaded blob is tiny compared with `UNKCOFF.DAT`
Verified current sizes:
- Remorse `USECODE\UNKDS.DAT` = `66` bytes
- Regret `USECODE\REGRET\UNKDS.DAT` = `36` bytes
That size profile does **not** match the debugger's line-oriented unit-source loader.
Current safest interpretation:
- `UNKDS.DAT` is some small shared sidecar for the same broad usecode-debug/intrinsic ecosystem
- it may be a descriptor table, count/header block, or small symbol-side metadata blob
- but it is not a drop-in substitute for `<unit>.unk`
### 6. `UNKOFF.DAT` vs `UNKCOFF.DAT`
The files present in the repo and game folders are named:
- `UNKCOFF.DAT`
- `UNKDS.DAT`
The older tooling and docs also consistently use the `unkcoffs/` name.
So if `UNKOFF.DAT` was seen elsewhere, the safest current read is:
- either a typo
- or a shorthand reference to the same `UNKCOFF.DAT` family
There is no current workspace evidence for a separate stock file literally named `UNKOFF.DAT`.
### 7. Practical implication for the live Regret debugger
Now that the menu opens, the next practical consequence is straightforward:
- copying `UNKCOFF.DAT` or `UNKDS.DAT` into the open-file prompt is unlikely to satisfy the debugger's unit-source viewer
- the debugger most likely wants a text file named after the current usecode unit, for example something structurally like `<unitname>.unk`
Because no sample `.unk` corpus is currently present, the best next move is not to guess blindly at `UNKDS.DAT`. The better next move is:
1. capture the exact current-unit name the debugger is asking for
2. locate or reconstruct a matching text listing for that unit
3. only treat `UNKCOFF.DAT` / `UNKDS.DAT` as support data for reconstruction, not as direct replacements
### 7.5. First manufactured `.unk` corpus
That reconstruction step is now far enough along to try in practice.
A new generator was added at `tools/generate_usecode_unk.py`. It walks an extracted Crusader USECODE root, parses each class body through the existing local usecode decompiler, and writes one synthesized `<unit>.unk` file per class name.
Current generated Regret corpus:
- output root: `USECODE/REGRET`
- manifest: `USECODE/REGRET/SYNTH_UNK_MANIFEST.tsv`
- synthesized files written: `477`
Current shape of each generated `.unk` file:
- filename = uppercased eight-character unit/class name, for example `ALARMHAT.unk` or `EVENT.unk`
- when recovered debug line markers exist, the top of the file is reserved for sparse line-mapped content keyed to those original line numbers
- after that, each file appends readable reconstructed pseudocode for every decoded slot in that class
Current practical limitation:
- in the present Regret extracted corpus, the parsed class bodies examined so far appear to have `0` surviving `line_number` op markers
- that means the generated Regret `.unk` files currently behave mostly as readable reconstructed source appendices, not as truly line-accurate original debug-source replicas
That limitation matters for breakpoint fidelity, but it does **not** block the basic experiment the user actually needs next:
- the debugger's file loader only needs a line-oriented text file to open and display
- these synthesized `.unk` files now provide that text payload in the exact filename family the debugger expects
So the current live experiment surface is no longer hypothetical: the patched Regret build can now be tested directly against manufactured per-unit `.unk` files under `USECODE/REGRET`.
### 8. Best current synthesis
The current evidence fits one coherent model:
- `UNKCOFF.DAT` = shared far-pointer table for usecode/intrinsic mapping
- `UNKDS.DAT` = small shared sidecar in the same broader ecosystem
- `<unit>.unk` = per-unit text listing that the hidden debugger actually displays
That model matches all of the currently recovered evidence:
- filename construction in the debugger wrapper
- text-file parsing and line indexing in `Debugump_13a0_2e0a`
- the older `unkcoffs` tooling
- the very small size of `UNKDS.DAT`
- and the complete absence of any stock `.unk` files in the current workspace
## Open Questions After This Pass
1. Does retail No Remorse still contain a dormant analogue of the Regret vtable override, not just the writer/bootstrap?
2. Are there any Regret-side callsites outside the current xref model that still seed break/step state intentionally for the debugger?
3. Does any hidden Regret cheat/debug key or menu path manipulate the break-state bytes at `+0x74/+0x75/+0x76/+0x78` or otherwise force the slot-`0` auto-open path?
4. Is `usecode_debugger_bootstrap_init` a real segment-load initializer, and if so where exactly is that load boundary controlled?
## Recommended Next Step
Use No Regret, not JP, as the primary retail comparison anchor and move back to retail with the stronger Regret model in hand.
The concrete retail follow-up should compare:
- the missing writer of `1478:659c/659e`
- any retail analogue of `usecode_debugger_bootstrap_init`
- any retail analogue of the `1480:713b -> 1480:6972` vtable promotion
- the retail equivalent of the Regret slot-`0` auto-open path through `usecode_debugger_break_state_update_line_and_maybe_break`
That remains the highest-value cross-build step. But if the immediate practical goal is only `make the debugger appear somewhere`, Regret is now the better live hack target than retail itself.