This commit introduces a comprehensive document outlining the various executable-patching attempts aimed at revealing the hidden retail usecode debugger within the CRUSADER.EXE file. The document serves multiple purposes, including preserving negative evidence, recording patch shapes and their rationales, and ensuring that runtime outcomes are linked to specific patch generations.
Key sections include:
- Ground rules for patching and validation processes.
- A table of stable facts regarding the debugger's structure and behavior.
- A detailed attempt log documenting each patch's shape, mechanical and runtime results, and verdicts.
- Root-cause findings from failed paths, providing insights into the challenges faced during the patching process.
- Current live candidates for further testing and exploration.
This documentation is intended to streamline future patching efforts and improve the understanding of the underlying mechanics of the debugger.
- Introduced README.md for the PyGhidra Crusader Toolkit, detailing setup and usage instructions.
- Added bootstrap_env.ps1 script to create and refresh the Python virtual environment with necessary packages.
- Implemented _tmp_patch_hidden_cheat_menu.py and _tmp_patch_hidden_cheat_menu_deferred.py scripts for patching specific memory addresses in Ghidra.
- Introduced a new command 'annotate-usecode' to import USECODE IR JSON annotation hints as Ghidra comments on compiled anchors.
- Added argument parsing for multiple IR JSON files, comment type selection, and a dry-run option.
- Implemented logic to read annotation records from the provided IR files and set comments on the corresponding addresses in Ghidra.
- Enhanced JSON schema to include response structure for the new command.
- Enhance `extract_eusecode_flx.py` to derive class event rows with additional metadata including derived body windows and repeated template statuses.
- Introduce `usecode_family_compare.py` for comparing event families, analyzing commonalities in event bodies, and generating reports on identical groups and differences.
- Implement new data structures for managing class event rows and family artifact specifications.
- Update output formats to include derived body information and repeated family regression checks.
- Ensure robust validation of repeated family expectations against actual extracted data.
- Introduced new file `vm_mask_ladder.tsv` containing detailed mappings for Crusader USECODE VM masks and their associated descriptors.
- Added comprehensive documentation in `scummvm-crusader-reference.md` outlining the structure, findings, and implications for reverse-engineering the Crusader engine within ScummVM.
- Created `usecode-roundtrip-ir.md` to document the plan for converting Crusader USECODE bytes into a human-readable format, detailing the container layout, event names, and intrinsic tables.
- Implemented a PowerShell script `temp_usecode_sample.ps1` for extracting and analyzing USECODE data from the Crusader FLX files, providing insights into class and event structures.
- Implemented a Python script to extract data from the EUSECODE.FLX file format.
- Defined data structures for candidate entries and extracted chunks using dataclasses.
- Added functions to read and parse the FLX table, extract candidate data, and generate human-readable output files.
- Included functionality for analyzing extracted data, including generating summaries, descriptors, and event family reports.
- Implemented utilities for calculating printable ratios, zero ratios, and identifying text-like data.
- Added support for writing various output formats, including JSON, TSV, and Markdown.
- Updated Ghidra instructions to emphasize keeping analysis batches small.
- Added new binary files: `db.104.gbf`, `db.105.gbf`, and `db.27.gbf`.
- Expanded decompilation notes for `cheat_code_check`, detailing its internal workings and verified cheat actions.
- Revised segment coverage ledger to reflect new findings and promote segments from `Foothold` to `Partial`.
- Enhanced `plan-mid.md` with updated estimates and focus areas for ongoing analysis.
- Added new binary files for segment coverage in `Crusader.rep/idata/00/~00000006.db/`
- Updated `crusader_segment_coverage_ledger.csv` to reflect new findings and classifications, including:
- Renamed segments for clarity on allocator mechanics and dispatch entry roles.
- Adjusted coverage status for segments related to startup/display orchestration and allocator phase finalization.
- Revised `plan-mid.md` to include recent progress on segment recovery and classification, emphasizing the ongoing work on the `0x4588` callback object and related functions.
- Created `crusader_segment_coverage_ledger.csv` to track segment coverage status, types, and known functions.
- Introduced `plan-mid.md` as a mid-project tracker outlining progress, objectives, and implementation priorities for the decompilation effort.
- Added scripts in `pyghidra_plans` to assist with instruction window dumping and reference inspection for the object at `0x4588`.
- Implemented functionality to scan for instruction uses of specific targets related to the decompilation project.
This commit introduces a new script, `apply_ne_far_call_fixups.py`, which processes internal and import far call fixups from TSV files. The script includes functions to parse addresses, load fixup data, patch internal far calls, and annotate import calls. It handles conflicts in fixup mappings and provides detailed output on the applied patches and any skipped instructions. The script is designed to enhance the handling of NE format far calls in Ghidra.
- Added new commands to the CLI for dumping regions, renaming functions by address, and setting various types of comments.
- Implemented JSON output formatting for CLI commands.
- Introduced functions for decompiling and disassembling functions, as well as retrieving cross-references.
- Enhanced common utilities with functions for reading memory regions, iterating Java items, and managing function metadata.
- Added suppress_output context manager to hide process output during Ghidra startup.
- Updated existing functions to improve error handling and output formatting.
- Introduced `seg043_boundary_repair.json` to manage function boundaries in segment 043.
- Created `read_file.py` for reading and printing file content size.
- Added `resolve_bb4f.py` to resolve specific function call targets.
- Implemented `resolve_top_targets.py` to find resolved NE targets for top-called wrapper functions.
- Added `script_contents.txt` to summarize NE relocation far calls.
- Updated `tier4_ghidra.txt`, `tier4_ghidra_check.txt`, `tier4_output.txt`, and `tier4_result.txt` with function call statistics.
- Created `tier5_errors.txt` for error logging and `tier5_output.txt` for additional function call statistics.
- Established `tools` directory with helper scripts for the Ghidra project, including CLI and common functionalities.
- Implemented command-line interface in `cli.py` for various project operations.
- Added `common.py` for shared functions and configurations across tools.
- Introduced `validate_fixups.py` to validate NE relocation fixups against known addresses.
