Add new modules for Crusader map rendering and processing

- Implemented `formats.py` to define data structures and functions for handling map data, including reading and decoding shape and map items.
- Created `png.py` for generating PNG images from shape frames and pixel data.
- Developed `sorting.py` to manage the sorting and rendering order of map items based on their properties and spatial relationships.
- Introduced `render_all_maps.py` to facilitate the rendering of all maps for specified games, including command-line argument parsing and subprocess management for rendering tasks.

docs/map-rendering.md

@@ -7,6 +7,22 @@ This note starts a dedicated lane for offline Crusader map extraction and PNG re
 Current implementation entry point:
 
 - `tools/render_crusader_map.py`
+- `render_maps.bat` for whole-game batch runs into `out/remorse` and `out/regret`
+
+Current renderer diagnostics:
+
+- large maps now emit progress checkpoints during item collection, dependency sorting, paint-order resolution, and blitting
+- `collect_render_items()` only expands `FIXED.DAT` glob eggs once, instead of recursively re-expanding glob-emitted glob eggs
+- metadata now records sampled invalid shape/frame references plus a conservative map-usage hint block
+- roofs/exploration obscurers are now optional and disabled by default
+- editor/debug/marker-style map content is now enabled by default instead of being silently discarded
+
+Internal package layout:
+
+- `tools/crusader_map/formats.py` for Crusader archive and record parsing
+- `tools/crusader_map/sorting.py` for the dependency-graph overlap sorter
+- `tools/crusader_map/png.py` for PNG buffer/blit helpers
+- `tools/crusader_map/cli.py` for command-line orchestration
 
 Current supported data roots:
 
@@ -146,6 +162,11 @@ This is enough for:
 - expanding `SF_GLOBEGG`
 - documenting future work toward a better sorter
 
+Current offline rendering policy differs from the live game intentionally:
+
+- `SI_ROOF` shapes are hidden by default because they commonly act as exploration obscurers or roof covers that gameplay later removes or pops
+- editor/debug/marker-style content is shown by default so offline renders expose more of what the shipped data actually contains
+
 The current tool does not yet use the footpad values for full ItemSorter-equivalent overlap resolution.
 
 ## Current Projection And Painting Rules
@@ -197,6 +218,58 @@ Render a bounded world-space region only:
 c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 0 --world-rect 0 0 4096 4096 --output out/map0-quarter.png
 ```
 
+Render with roofs restored:
+
+```powershell
+c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 1 --include-roofs --output out/map1-with-roofs.png
+```
+
+Render without the extra hidden/editor marker content:
+
+```powershell
+c:/Users/Maddo/.PYENV/PYENV-WIN/versions/3.14.3/python.exe tools/render_crusader_map.py --game remorse --map 1 --no-include-hidden-markers --no-include-editor --output out/map1-minimal.png
+```
+
+Render every No Remorse map to `out/remorse`:
+
+```cmd
+render_maps.bat remorse
+```
+
+Render every No Regret map to `out/regret`:
+
+```cmd
+render_maps.bat regret
+```
+
+The batch runner also accepts an optional `start_map end_map` range for partial runs while validating changes:
+
+```cmd
+render_maps.bat remorse 1 3
+```
+
+You can also forward extra renderer arguments through the `RENDER_ARGS` environment variable, for example a bounded validation run:
+
+```cmd
+set RENDER_ARGS=--world-rect 0 0 16384 16384
+render_maps.bat regret 5 5
+```
+
+Batch behavior notes:
+
+- empty maps are skipped in batch mode and do not produce PNG or JSON outputs
+- there is no default max-pixel cap anymore; full-map renders are attempted unless you pass `--max-pixels`
+- batch item-count skipping is now opt-in only; set `BATCH_MAX_ITEMS` to a positive value if you want the batch runner to skip very large full maps
+- the renderer emits progress by default every 2000 items; pass `--progress-every 0` through `RENDER_ARGS` to silence it
+- batch runs now default to `include_editor=true`, `include_hidden_markers=true`, and `include_roofs=false`
+
+Metadata notes:
+
+- `invalid_items` contains a capped sample of bad `(shape, frame, x, y, z, source, reason)` records so broken `FIXED.DAT` references can be inspected without rerunning a scan
+- `usage` is conservative: it reports known reference-backed map hints when available and otherwise stays `unknown`; it does not yet prove orphan status
+- `base_item_summary` reports how many roof, editor, egg-family, invisible, and NPC-linked records were present in the raw map payload
+- `filters` records whether the render included roofs, editor shapes, and hidden marker content
+
 ## Current Deliberate Limits
 
 This tool is a start, not a complete engine clone.
@@ -210,6 +283,7 @@ Current gaps:
 5. It does not yet consume `ANIM.DAT`, `DAMAGE.FLX`, `DTABLE.FLX`, `WPNOVLAY.DAT`, or palette transforms such as `XFORMPAL.DAT`.
 6. It uses `GAMEPAL.PAL` directly and does not yet model alternate or transformed palettes.
 7. It writes a plain RGBA PNG using only the standard library; there is no zoomed viewer, tile atlas exporter, or sprite manifest yet.
+8. Some maps still contain invalid shape/frame references in `FIXED.DAT`; the renderer now skips those items instead of aborting the whole map, but that means some broken placements remain missing until the source of those references is understood.
 
 ## Immediate Follow-Ups