Refactor code structure for improved readability and maintainability

UpgradePlan.md

@@ -0,0 +1,214 @@
+Plan: C# .NET 10 Twitch Archive Rewrite
+A complete port of the Python archiver to C# .NET 10 with Blazor Server UI, real-time process output, SQLite state tracking, full DI/service pattern, NLog logging, and a resilient recording engine. Placed under dotnet/ in the existing repo.
+
+Project Layout
+Step 1 — Solution & Project scaffolding
+Create the SLN and three projects:
+
+TwitchArchive.Core — classlib, targets net10.0
+TwitchArchive.Web — Blazor Server (blazorserver), targets net10.0
+TwitchArchive.Tests — xUnit, targets net10.0
+NuGet packages:
+
+Core: Microsoft.EntityFrameworkCore.Sqlite, Polly, NLog, NLog.Extensions.Logging
+Web: all Core packages + Microsoft.AspNetCore.SignalR, NLog.Web.AspNetCore
+Tests: xunit, Moq, coverlet.collector, Microsoft.EntityFrameworkCore.InMemory
+Step 2 — Configuration models
+Mirror the existing JSON schemas as C# POCOs with System.Text.Json attributes:
+
+GlobalConfig.cs — one property per key in config/global.json.example
+StreamerConfig.cs — all fields nullable (override semantics), only Username and Enabled required
+EffectiveConfig.cs — computed merge of global + per-streamer; exposes resolved values
+AppSettings.cs — app-level settings (password hash, tool paths, .env secrets)
+IConfigurationService / ConfigurationService:
+
+LoadGlobal() / SaveGlobal(GlobalConfig)
+LoadStreamer(string username) / SaveStreamer(StreamerConfig)
+GetAllStreamers(), GetEffectiveConfig(string username) (merge logic)
+Reads/writes global.json and config/streamers/*.json — same files as Python
+Step 3 — Infrastructure layer
+TwitchApiClient (injectable, mockable):
+
+GetOAuthTokenAsync() — POST to https://id.twitch.tv/oauth2/token, caches token, refreshes on 401
+CheckStreamStatusAsync(string username) — GQL query for live stream + archiveVideo.id
+GetLatestVodAsync(string username) — GQL query for most recent VOD
+ValidateUsernameAsync(string username) — Helix /users endpoint
+Credentials read from environment (CLIENT-ID, CLIENT-SECRET, OAUTH-PRIVATE-TOKEN)
+All methods return typed result objects, never throw on network errors — return Result<T> (or OneOf)
+HttpResiliencePolicy (Polly):
+
+Wraps HttpClient for TwitchApiClient
+WaitAndRetryForever with exponential backoff starting at 15 s, doubling, capped at 10 minutes
+Only applies to transient errors (5xx, timeout, HttpRequestException) — not 401/404
+Logged via NLog on each retry attempt
+ProcessRunner (injectable + mockable for tests):
+
+RunAsync(ProcessRunOptions options, CancellationToken ct) → int exitCode
+StartAsync(ProcessRunOptions options, CancellationToken ct) → IRunningProcess handle (for long-lived processes like streamlink)
+Reads stdout and stderr line by line asynchronously
+Reports each line to IProcessOutputStore (streamer + job context)
+Forwards to NLog
+ProcessRunOptions: FileName, Arguments, WorkingDirectory, RedirectOutput
+
+Step 4 — Core services (all behind interfaces)
+IStreamMonitorService / StreamMonitorService
+
+Wraps TwitchApiClient
+CheckIsLiveAsync(string username) → LiveStreamInfo?
+GetLatestVodAsync(string username) → VodInfo?
+IRecorderService / RecorderService
+
+StartRecordingAsync(string username, string quality, string outputPath, CancellationToken ct) → Task<RecordingResult>
+Invokes streamlink via ProcessRunner
+Passes --hls-live-restart, --stream-segment-threads, optional OAuth header
+Returns when streamlink exits (either stream ended or ct cancelled)
+IProcessorService / ProcessorService
+
+ProcessRawStreamAsync(string rawPath, string outputPath, EffectiveConfig cfg, CancellationToken ct)
+Builds ffmpeg args: hwaccel, thread count, error recovery flags, faststart, copy codecs
+MergeVideoChatAsync(string videoPath, string chatVideoPath, string outputPath, string layout, CancellationToken ct)
+IDownloaderService / DownloaderService
+
+DownloadVodAsync(VodInfo vod, string outputPath, EffectiveConfig cfg, CancellationToken ct) → bool
+Invokes TwitchDownloaderCLI videodownload
+Chat download methods stubbed with NotImplementedException / commented structure; interface is defined now to keep the architecture clean
+IUploadService / UploadService
+
+UploadAsync(string localRoot, IEnumerable<string> relativeFilePaths, string rcloneDest, CancellationToken ct) → bool
+Writes a temp files-from list, invokes rclone copy --files-from
+Returns success/failure; preserves local files on failure
+IFileManagerService / FileManagerService
+
+InitializeDirectories(string rootPath, string username)
+GetPaths(string rootPath, string username, string filenameBase) → ArchivePaths record (all expected paths)
+CleanRawFile(string path, bool cleanRaw)
+DeleteLocalFiles(ArchivePaths paths, EffectiveConfig cfg)
+GetUniquePath(string path) → adds numeric suffix if file exists
+Step 5 — Recording resilience engine
+RecoveryPolicy (POCO, unit-testable, no DI deps):
+
+Encodes a state machine with these states:
+
+State	Meaning
+Monitoring	Normal polling at refresh interval
+Recording	streamlink subprocess active
+FastReconnect	Stream ended; checking every 10 s for up to 2 minutes
+SlowReconnect	Still not live after 2 min; checking every 60 s concurrently with post-processing
+PostProcessing	Confirmed ended; ffmpeg / VOD download / upload running
+NetworkFault	Twitch API unreachable; exponential back-off (30 s → capped at 10 min)
+Transitions:
+
+Recording → streamlink exits → enter FastReconnect, record phase start time
+FastReconnect → live confirmed → start new Recording (new filename/segment)
+FastReconnect (2 min elapsed) → enter SlowReconnect + kick off PostProcessing concurrently
+SlowReconnect → live confirmed → start new Recording
+SlowReconnect / Monitoring → API call throws network error → enter NetworkFault
+NetworkFault → successful API response → return to previous state (Monitoring or re-enter FastReconnect if we were mid-reconnect)
+NetworkFault backoff: 30s * 2^attempt, capped at 600s
+RecoveryPolicy is a pure class with a Tick(DateTime now, bool? isLive, bool networkError) method → returns RecoveryDecision (what to do next + sleep duration). Fully unit-testable with no async or DI.
+
+StreamWorker : BackgroundService
+
+One instance per enabled streamer
+Holds RecoveryPolicy instance
+Main loop: evaluate policy decision → execute the corresponding service call → loop
+Started/stopped by StreamWorkerManager
+Writes job records to SQLite on start/complete/fail
+StreamWorkerManager
+
+StartWorker(string username), StopWorker(string username), RestartWorker(string username)
+Called at app startup for all enabled streamers
+Called from Web UI on enable/disable/config change
+Workers stored in ConcurrentDictionary<string, (StreamWorker, CancellationTokenSource)>
+Step 6 — Persistence (SQLite + EF Core)
+ArchiveDbContext with three tables:
+
+StreamSessions: Id, StreamerUsername, TwitchStreamId, Title, StartedAt, EndedAt, Status (Recording/Processing/Uploading/Complete/Failed)
+
+ArchiveJobs: Id, SessionId, JobType (enum: RecordLive, ProcessLive, DownloadVod, ProcessVod, UploadCloud, DeleteLocal), Status, StartedAt, CompletedAt, FilePath, ErrorMessage
+
+StreamerStates: Username, IsMonitoring, LastCheckedAt, CurrentRecoveryState
+
+Migrations via EF Core CLI. ISessionRepository / IJobRepository interfaces for testability with in-memory EF provider in tests.
+
+Step 7 — Process output streaming
+IProcessOutputStore:
+
+AppendLine(string streamerId, Guid jobId, string line, bool isError)
+GetRecentLines(string streamerId, Guid jobId, int count = 500) → IReadOnlyList<OutputLine>
+In-memory circular buffer (1000 lines per job, last 20 jobs per streamer)
+ProcessOutputHub : Hub (SignalR):
+
+Clients call SubscribeToStreamer(string username) → join group streamer:{username}
+Clients call SubscribeToJob(Guid jobId) → join group job:{jobId}
+Server pushes ReceiveLine(OutputLine line) from ProcessRunner via IHubContext<ProcessOutputHub>
+On subscribe: server immediately sends buffered lines from IProcessOutputStore
+Step 8 — Blazor Server Web UI
+Authentication: Cookie-based single-password auth via ASP.NET Core minimal auth middleware. Password stored as BCrypt hash in AppSettings. Login.razor page at /login. Protected routes with [Authorize].
+
+Pages & Components:
+
+Dashboard.razor (/) — grid of all configured streamers showing: username, live/offline badge, current recovery state, last recorded session, quick Start/Stop monitoring toggle
+
+StreamerDetail.razor (/streamer/{username}) — live status, current job pipeline steps (record → process → upload with progress), ProcessOutputConsole.razor showing real-time terminal output via SignalR
+
+ProcessOutputConsole.razor — reusable Blazor component; subscribes to SignalR on mount, renders an auto-scrolling <pre> with colored output (stdout = white, stderr = orange/red), handles reconnect
+
+Sessions.razor (/sessions) — paginated list of past archive sessions with job statuses and expandable per-job output
+
+GlobalConfig.razor (/config/global) — EditForm bound to GlobalConfig model with data annotations validation, Save button calls IConfigurationService.SaveGlobal()
+
+StreamerConfig.razor (/config/{username}) — similar form for per-streamer overrides; each field has a nullable toggle (inherit from global vs override)
+
+AddStreamer.razor (/config/new) — minimal form: username + enabled; creates new config/streamers/{username}.json
+
+AppSettings.razor (/settings) — tool paths (streamlink, ffmpeg, TwitchDownloaderCLI, rclone), change password
+
+Step 9 — NLog configuration
+nlog.config (XML): two targets:
+
+Console (colored, with level formatting)
+File rolling (logs/archive-${shortdate}.log, keep 30 days)
+Log structured context: StreamerUsername, JobId, JobType as NLog ScopeContext properties. Service methods open a scope via ILogger.BeginScope(...).
+
+Step 10 — Docker
+Dockerfile (multi-stage):
+
+Build stage: mcr.microsoft.com/dotnet/sdk:10.0
+Runtime stage: mcr.microsoft.com/dotnet/aspnet:10.0 (Linux)
+Install ffmpeg, streamlink (via pip), download TwitchDownloaderCLI binary for linux-x64
+rclone installed via shell script or apt
+Expose port 8080
+ENTRYPOINT ["dotnet", "TwitchArchive.Web.dll"]
+docker-compose.yml:
+
+Volume mounts: ./config:/app/config, ./archive:/app/archive, ./logs:/app/logs
+Environment variables: CLIENT-ID, CLIENT-SECRET, OAUTH-PRIVATE-TOKEN
+Windows dev: run directly with dotnet run; tool paths auto-detected (Windows vs Linux) via RuntimeInformation.IsOSPlatform(OSPlatform.Windows) in ToolPathResolver
+
+Step 11 — Unit tests
+TwitchArchive.Tests covers:
+
+RecoveryPolicyTests — state machine transitions, timing phases, network fault backoff; pure synchronous tests
+ConfigurationServiceTests — JSON load/save/merge with temp files
+TwitchApiClientTests — mocked HttpMessageHandler; OAuth, GQL queries, 401 refresh, network errors
+FileManagerServiceTests — path generation, directory creation with temp directories
+DownloaderServiceTests / RecorderServiceTests — mocked ProcessRunner; verify correct CLI arguments
+UploadServiceTests — mocked ProcessRunner; verify rclone argument construction
+SessionRepositoryTests — EF Core in-memory provider
+EffectiveConfigTests — global + streamer override merge logic
+Verification
+dotnet build dotnet/TwitchArchive.sln — zero warnings, zero errors
+dotnet test dotnet/TwitchArchive.Tests/ — all tests green
+docker compose up --build in the dotnet/ folder → app reachable at http://localhost:8080
+Manual: add a test streamer config, enable monitoring, confirm it polls and records a live stream, runs ffmpeg, uploads via rclone
+Resilience manual test: kill network during recording → verify FastReconnect phase kicks in, resumes after connectivity restored
+Decisions
+
+Blazor Server chosen for real-time terminal output without a separate API; no WASM needed
+streamlink for live + TwitchDownloaderCLI for VOD (same split as Python; streamlink gives better live resilience)
+Simple BCrypt password auth (not full Identity — this is a single-user tool)
+RecoveryPolicy as a pure POCO state machine keeps the resilience logic fully unit-testable without async/mocking
+Polly WaitAndRetryForever on HttpClient handles persistent network failure independently of the application-level state machine; they are complementary — Polly handles individual HTTP call retries, RecoveryPolicy handles the overall workflow state
+Chat download service interface is defined in Step 4 but methods are stubbed — adding implementation later requires only filling in DownloaderService without touching any other layer
+Config files remain in the same format/location so the Python and C# versions can share the same config directory