TwitchDownloader/UpgradePlan.md

Plan: C# .NET 10 Twitch Archive Rewrite
A complete port of the Python archiver to C# .NET 10 with Blazor Server UI, real-time process output, SQLite state tracking, full DI/service pattern, NLog logging, and a resilient recording engine. Placed under dotnet/ in the existing repo.

Project Layout
Step 1 — Solution & Project scaffolding
Create the SLN and three projects:

TwitchArchive.Core — classlib, targets net10.0
TwitchArchive.Web — Blazor Server (blazorserver), targets net10.0
TwitchArchive.Tests — xUnit, targets net10.0
NuGet packages:

Core: Microsoft.EntityFrameworkCore.Sqlite, Polly, NLog, NLog.Extensions.Logging
Web: all Core packages + Microsoft.AspNetCore.SignalR, NLog.Web.AspNetCore
Tests: xunit, Moq, coverlet.collector, Microsoft.EntityFrameworkCore.InMemory
Step 2 — Configuration models
Mirror the existing JSON schemas as C# POCOs with System.Text.Json attributes:

GlobalConfig.cs — one property per key in config/global.json.example
StreamerConfig.cs — all fields nullable (override semantics), only Username and Enabled required
EffectiveConfig.cs — computed merge of global + per-streamer; exposes resolved values
AppSettings.cs — app-level settings (password hash, tool paths, .env secrets)
IConfigurationService / ConfigurationService:

LoadGlobal() / SaveGlobal(GlobalConfig)
LoadStreamer(string username) / SaveStreamer(StreamerConfig)
GetAllStreamers(), GetEffectiveConfig(string username) (merge logic)
Reads/writes global.json and config/streamers/*.json — same files as Python
Step 3 — Infrastructure layer
TwitchApiClient (injectable, mockable):

GetOAuthTokenAsync() — POST to https://id.twitch.tv/oauth2/token, caches token, refreshes on 401
CheckStreamStatusAsync(string username) — GQL query for live stream + archiveVideo.id
GetLatestVodAsync(string username) — GQL query for most recent VOD
ValidateUsernameAsync(string username) — Helix /users endpoint
Credentials read from environment (CLIENT-ID, CLIENT-SECRET, OAUTH-PRIVATE-TOKEN)
All methods return typed result objects, never throw on network errors — return Result<T> (or OneOf)
HttpResiliencePolicy (Polly):

Wraps HttpClient for TwitchApiClient
WaitAndRetryForever with exponential backoff starting at 15 s, doubling, capped at 10 minutes
Only applies to transient errors (5xx, timeout, HttpRequestException) — not 401/404
Logged via NLog on each retry attempt
ProcessRunner (injectable + mockable for tests):

RunAsync(ProcessRunOptions options, CancellationToken ct) → int exitCode
StartAsync(ProcessRunOptions options, CancellationToken ct) → IRunningProcess handle (for long-lived processes like streamlink)
Reads stdout and stderr line by line asynchronously
Reports each line to IProcessOutputStore (streamer + job context)
Forwards to NLog
ProcessRunOptions: FileName, Arguments, WorkingDirectory, RedirectOutput

Step 4 — Core services (all behind interfaces)
IStreamMonitorService / StreamMonitorService

Wraps TwitchApiClient
CheckIsLiveAsync(string username) → LiveStreamInfo?
GetLatestVodAsync(string username) → VodInfo?
IRecorderService / RecorderService

StartRecordingAsync(string username, string quality, string outputPath, CancellationToken ct) → Task<RecordingResult>
Invokes streamlink via ProcessRunner
Passes --hls-live-restart, --stream-segment-threads, optional OAuth header
Returns when streamlink exits (either stream ended or ct cancelled)
IProcessorService / ProcessorService

ProcessRawStreamAsync(string rawPath, string outputPath, EffectiveConfig cfg, CancellationToken ct)
Builds ffmpeg args: hwaccel, thread count, error recovery flags, faststart, copy codecs
MergeVideoChatAsync(string videoPath, string chatVideoPath, string outputPath, string layout, CancellationToken ct)
IDownloaderService / DownloaderService

DownloadVodAsync(VodInfo vod, string outputPath, EffectiveConfig cfg, CancellationToken ct) → bool
Invokes TwitchDownloaderCLI videodownload
Chat download methods stubbed with NotImplementedException / commented structure; interface is defined now to keep the architecture clean
IUploadService / UploadService

UploadAsync(string localRoot, IEnumerable<string> relativeFilePaths, string rcloneDest, CancellationToken ct) → bool
Writes a temp files-from list, invokes rclone copy --files-from
Returns success/failure; preserves local files on failure
IFileManagerService / FileManagerService

InitializeDirectories(string rootPath, string username)
GetPaths(string rootPath, string username, string filenameBase) → ArchivePaths record (all expected paths)
CleanRawFile(string path, bool cleanRaw)
DeleteLocalFiles(ArchivePaths paths, EffectiveConfig cfg)
GetUniquePath(string path) → adds numeric suffix if file exists
Step 5 — Recording resilience engine
RecoveryPolicy (POCO, unit-testable, no DI deps):

Encodes a state machine with these states:

State	Meaning
Monitoring	Normal polling at refresh interval
Recording	streamlink subprocess active
FastReconnect	Stream ended; checking every 10 s for up to 2 minutes
SlowReconnect	Still not live after 2 min; checking every 60 s concurrently with post-processing
PostProcessing	Confirmed ended; ffmpeg / VOD download / upload running
NetworkFault	Twitch API unreachable; exponential back-off (30 s → capped at 10 min)
Transitions:

Recording → streamlink exits → enter FastReconnect, record phase start time
FastReconnect → live confirmed → start new Recording (new filename/segment)
FastReconnect (2 min elapsed) → enter SlowReconnect + kick off PostProcessing concurrently
SlowReconnect → live confirmed → start new Recording
SlowReconnect / Monitoring → API call throws network error → enter NetworkFault
NetworkFault → successful API response → return to previous state (Monitoring or re-enter FastReconnect if we were mid-reconnect)
NetworkFault backoff: 30s * 2^attempt, capped at 600s
RecoveryPolicy is a pure class with a Tick(DateTime now, bool? isLive, bool networkError) method → returns RecoveryDecision (what to do next + sleep duration). Fully unit-testable with no async or DI.

StreamWorker : BackgroundService

One instance per enabled streamer
Holds RecoveryPolicy instance
Main loop: evaluate policy decision → execute the corresponding service call → loop
Started/stopped by StreamWorkerManager
Writes job records to SQLite on start/complete/fail
StreamWorkerManager

StartWorker(string username), StopWorker(string username), RestartWorker(string username)
Called at app startup for all enabled streamers
Called from Web UI on enable/disable/config change
Workers stored in ConcurrentDictionary<string, (StreamWorker, CancellationTokenSource)>
Step 6 — Persistence (SQLite + EF Core)
ArchiveDbContext with three tables:

StreamSessions: Id, StreamerUsername, TwitchStreamId, Title, StartedAt, EndedAt, Status (Recording/Processing/Uploading/Complete/Failed)

ArchiveJobs: Id, SessionId, JobType (enum: RecordLive, ProcessLive, DownloadVod, ProcessVod, UploadCloud, DeleteLocal), Status, StartedAt, CompletedAt, FilePath, ErrorMessage

StreamerStates: Username, IsMonitoring, LastCheckedAt, CurrentRecoveryState

Migrations via EF Core CLI. ISessionRepository / IJobRepository interfaces for testability with in-memory EF provider in tests.

Step 7 — Process output streaming
IProcessOutputStore:

AppendLine(string streamerId, Guid jobId, string line, bool isError)
GetRecentLines(string streamerId, Guid jobId, int count = 500) → IReadOnlyList<OutputLine>
In-memory circular buffer (1000 lines per job, last 20 jobs per streamer)
ProcessOutputHub : Hub (SignalR):

Clients call SubscribeToStreamer(string username) → join group streamer:{username}
Clients call SubscribeToJob(Guid jobId) → join group job:{jobId}
Server pushes ReceiveLine(OutputLine line) from ProcessRunner via IHubContext<ProcessOutputHub>
On subscribe: server immediately sends buffered lines from IProcessOutputStore
Step 8 — Blazor Server Web UI
Authentication: Cookie-based single-password auth via ASP.NET Core minimal auth middleware. Password stored as BCrypt hash in AppSettings. Login.razor page at /login. Protected routes with [Authorize].

Pages & Components:

Dashboard.razor (/) — grid of all configured streamers showing: username, live/offline badge, current recovery state, last recorded session, quick Start/Stop monitoring toggle

StreamerDetail.razor (/streamer/{username}) — live status, current job pipeline steps (record → process → upload with progress), ProcessOutputConsole.razor showing real-time terminal output via SignalR

ProcessOutputConsole.razor — reusable Blazor component; subscribes to SignalR on mount, renders an auto-scrolling <pre> with colored output (stdout = white, stderr = orange/red), handles reconnect

Sessions.razor (/sessions) — paginated list of past archive sessions with job statuses and expandable per-job output

GlobalConfig.razor (/config/global) — EditForm bound to GlobalConfig model with data annotations validation, Save button calls IConfigurationService.SaveGlobal()

StreamerConfig.razor (/config/{username}) — similar form for per-streamer overrides; each field has a nullable toggle (inherit from global vs override)

AddStreamer.razor (/config/new) — minimal form: username + enabled; creates new config/streamers/{username}.json

AppSettings.razor (/settings) — tool paths (streamlink, ffmpeg, TwitchDownloaderCLI, rclone), change password

Step 9 — NLog configuration
nlog.config (XML): two targets:

Console (colored, with level formatting)
File rolling (logs/archive-${shortdate}.log, keep 30 days)
Log structured context: StreamerUsername, JobId, JobType as NLog ScopeContext properties. Service methods open a scope via ILogger.BeginScope(...).

Step 10 — Docker
Dockerfile (multi-stage):

Build stage: mcr.microsoft.com/dotnet/sdk:10.0
Runtime stage: mcr.microsoft.com/dotnet/aspnet:10.0 (Linux)
Install ffmpeg, streamlink (via pip), download TwitchDownloaderCLI binary for linux-x64
rclone installed via shell script or apt
Expose port 8080
ENTRYPOINT ["dotnet", "TwitchArchive.Web.dll"]
docker-compose.yml:

Volume mounts: ./config:/app/config, ./archive:/app/archive, ./logs:/app/logs
Environment variables: CLIENT-ID, CLIENT-SECRET, OAUTH-PRIVATE-TOKEN
Windows dev: run directly with dotnet run; tool paths auto-detected (Windows vs Linux) via RuntimeInformation.IsOSPlatform(OSPlatform.Windows) in ToolPathResolver

Step 11 — Unit tests
TwitchArchive.Tests covers:

RecoveryPolicyTests — state machine transitions, timing phases, network fault backoff; pure synchronous tests
ConfigurationServiceTests — JSON load/save/merge with temp files
TwitchApiClientTests — mocked HttpMessageHandler; OAuth, GQL queries, 401 refresh, network errors
FileManagerServiceTests — path generation, directory creation with temp directories
DownloaderServiceTests / RecorderServiceTests — mocked ProcessRunner; verify correct CLI arguments
UploadServiceTests — mocked ProcessRunner; verify rclone argument construction
SessionRepositoryTests — EF Core in-memory provider
EffectiveConfigTests — global + streamer override merge logic
Verification
dotnet build dotnet/TwitchArchive.sln — zero warnings, zero errors
dotnet test dotnet/TwitchArchive.Tests/ — all tests green
docker compose up --build in the dotnet/ folder → app reachable at http://localhost:8080
Manual: add a test streamer config, enable monitoring, confirm it polls and records a live stream, runs ffmpeg, uploads via rclone
Resilience manual test: kill network during recording → verify FastReconnect phase kicks in, resumes after connectivity restored
Decisions

Blazor Server chosen for real-time terminal output without a separate API; no WASM needed
streamlink for live + TwitchDownloaderCLI for VOD (same split as Python; streamlink gives better live resilience)
Simple BCrypt password auth (not full Identity — this is a single-user tool)
RecoveryPolicy as a pure POCO state machine keeps the resilience logic fully unit-testable without async/mocking
Polly WaitAndRetryForever on HttpClient handles persistent network failure independently of the application-level state machine; they are complementary — Polly handles individual HTTP call retries, RecoveryPolicy handles the overall workflow state
Chat download service interface is defined in Step 4 but methods are stubbed — adding implementation later requires only filling in DownloaderService without touching any other layer
Config files remain in the same format/location so the Python and C# versions can share the same config directory