Refactor downloader and file manager for improved rclone integration and add healthcheck and smoke test options

- Renamed download flags in ContentDownloader for clarity.
- Enhanced FileManager with methods to build upload paths and verify existing files for rclone uploads.
- Updated StreamProcessor to return success status for stream processing.
- Added rclone smoke test and healthcheck functions to validate configuration and tool availability.
- Improved environment variable handling with a utility function.
- Updated TwitchArchive to incorporate new rclone verification and processing logic.
- Added unit tests for new functionality and refactored existing tests for clarity and coverage.

Co-authored-by: Copilot <copilot@github.com>
This commit is contained in:
MaddoScientisto 2026-04-25 11:54:03 +02:00
commit f97e0200d6
23 changed files with 1013 additions and 289 deletions

View file

@ -26,6 +26,73 @@ Notes:
Python script to check, download live stream, VOD, chat and upload them to any cloud storage supported by rclone.
## Docker
This repository now includes a Python-only container setup for the archiver. The dotnet subapp is not part of this container flow.
Files:
- `docker/python.Dockerfile`: production image for the Python archiver
- `docker-compose.yml`: deployment-oriented compose file
- `docker-compose.override.yml`: local development and testing override
- `.env.production`: production container and app environment template
- `.env.development`: development container and app environment template
- `dockerstart.bat`: Windows helper to run the container like the old batch launcher
### Container layout
- Mount your external archive folder to `/app/archive`
- Mount your external config folder to `/app/config`
- Put your `rclone.conf` file at `/app/config/rclone.conf` on the mounted host path
- The container exports `RCLONE_CONFIG=/app/config/rclone.conf`, so rclone will use that file automatically
### Production deployment
1. Edit `.env.production` with your image name, Twitch credentials, bind paths, and default arguments.
2. Place your streamer JSON files and `rclone.conf` in the mounted config folder.
3. Start the container:
```powershell
docker compose --env-file .env.production up -d
```
4. Follow logs:
```powershell
docker compose --env-file .env.production logs -f twitch-archive
```
### Development and local testing
The override compose file builds the image locally and mounts the repository for faster iteration.
Start it with:
```powershell
docker compose --env-file .env.development -f docker-compose.yml -f docker-compose.override.yml up --build
```
Run a one-off manual test for another streamer:
```powershell
docker compose --env-file .env.development -f docker-compose.yml -f docker-compose.override.yml run --rm twitch-archive python twitch-archive.py -u hackerling --verbose
```
Or use the Windows helper:
```powershell
.\dockerstart.bat vinesauce --verbose
```
That batch launcher mirrors the old pattern and expands to a compose `run` command, so you can test any streamer manually.
### Healthcheck and smoke tests
- Container healthcheck command: `python twitch-archive.py --healthcheck -u vinesauce`
- Rclone smoke test command: `python twitch-archive.py -u vinesauce --rclone-smoke-test`
The healthcheck verifies config loading plus `streamlink`, `ffmpeg`, `TwitchDownloaderCLI`, and `rclone` availability. The smoke test writes a tiny file, uploads it with the configured rclone remote, and prints the live rclone output into the container logs.
## ⚡ FFmpeg 8.0 Enhanced
Now with FFmpeg 8.0+ support featuring hardware acceleration and performance improvements!
- **5-10x faster encoding** with NVIDIA, Intel, or AMD GPUs