12 KiB
Plan: C# .NET 10 Twitch Archive Rewrite A complete port of the Python archiver to C# .NET 10 with Blazor Server UI, real-time process output, SQLite state tracking, full DI/service pattern, NLog logging, and a resilient recording engine. Placed under dotnet/ in the existing repo.
Project Layout Step 1 — Solution & Project scaffolding Create the SLN and three projects:
TwitchArchive.Core — classlib, targets net10.0 TwitchArchive.Web — Blazor Server (blazorserver), targets net10.0 TwitchArchive.Tests — xUnit, targets net10.0 NuGet packages:
Core: Microsoft.EntityFrameworkCore.Sqlite, Polly, NLog, NLog.Extensions.Logging Web: all Core packages + Microsoft.AspNetCore.SignalR, NLog.Web.AspNetCore Tests: xunit, Moq, coverlet.collector, Microsoft.EntityFrameworkCore.InMemory Step 2 — Configuration models Mirror the existing JSON schemas as C# POCOs with System.Text.Json attributes:
GlobalConfig.cs — one property per key in config/global.json.example StreamerConfig.cs — all fields nullable (override semantics), only Username and Enabled required EffectiveConfig.cs — computed merge of global + per-streamer; exposes resolved values AppSettings.cs — app-level settings (password hash, tool paths, .env secrets) IConfigurationService / ConfigurationService:
LoadGlobal() / SaveGlobal(GlobalConfig) LoadStreamer(string username) / SaveStreamer(StreamerConfig) GetAllStreamers(), GetEffectiveConfig(string username) (merge logic) Reads/writes global.json and config/streamers/*.json — same files as Python Step 3 — Infrastructure layer TwitchApiClient (injectable, mockable):
GetOAuthTokenAsync() — POST to https://id.twitch.tv/oauth2/token, caches token, refreshes on 401 CheckStreamStatusAsync(string username) — GQL query for live stream + archiveVideo.id GetLatestVodAsync(string username) — GQL query for most recent VOD ValidateUsernameAsync(string username) — Helix /users endpoint Credentials read from environment (CLIENT-ID, CLIENT-SECRET, OAUTH-PRIVATE-TOKEN) All methods return typed result objects, never throw on network errors — return Result (or OneOf) HttpResiliencePolicy (Polly):
Wraps HttpClient for TwitchApiClient WaitAndRetryForever with exponential backoff starting at 15 s, doubling, capped at 10 minutes Only applies to transient errors (5xx, timeout, HttpRequestException) — not 401/404 Logged via NLog on each retry attempt ProcessRunner (injectable + mockable for tests):
RunAsync(ProcessRunOptions options, CancellationToken ct) → int exitCode StartAsync(ProcessRunOptions options, CancellationToken ct) → IRunningProcess handle (for long-lived processes like streamlink) Reads stdout and stderr line by line asynchronously Reports each line to IProcessOutputStore (streamer + job context) Forwards to NLog ProcessRunOptions: FileName, Arguments, WorkingDirectory, RedirectOutput
Step 4 — Core services (all behind interfaces) IStreamMonitorService / StreamMonitorService
Wraps TwitchApiClient CheckIsLiveAsync(string username) → LiveStreamInfo? GetLatestVodAsync(string username) → VodInfo? IRecorderService / RecorderService
StartRecordingAsync(string username, string quality, string outputPath, CancellationToken ct) → Task Invokes streamlink via ProcessRunner Passes --hls-live-restart, --stream-segment-threads, optional OAuth header Returns when streamlink exits (either stream ended or ct cancelled) IProcessorService / ProcessorService
ProcessRawStreamAsync(string rawPath, string outputPath, EffectiveConfig cfg, CancellationToken ct) Builds ffmpeg args: hwaccel, thread count, error recovery flags, faststart, copy codecs MergeVideoChatAsync(string videoPath, string chatVideoPath, string outputPath, string layout, CancellationToken ct) IDownloaderService / DownloaderService
DownloadVodAsync(VodInfo vod, string outputPath, EffectiveConfig cfg, CancellationToken ct) → bool Invokes TwitchDownloaderCLI videodownload Chat download methods stubbed with NotImplementedException / commented structure; interface is defined now to keep the architecture clean IUploadService / UploadService
UploadAsync(string localRoot, IEnumerable relativeFilePaths, string rcloneDest, CancellationToken ct) → bool Writes a temp files-from list, invokes rclone copy --files-from Returns success/failure; preserves local files on failure IFileManagerService / FileManagerService
InitializeDirectories(string rootPath, string username) GetPaths(string rootPath, string username, string filenameBase) → ArchivePaths record (all expected paths) CleanRawFile(string path, bool cleanRaw) DeleteLocalFiles(ArchivePaths paths, EffectiveConfig cfg) GetUniquePath(string path) → adds numeric suffix if file exists Step 5 — Recording resilience engine RecoveryPolicy (POCO, unit-testable, no DI deps):
Encodes a state machine with these states:
State Meaning Monitoring Normal polling at refresh interval Recording streamlink subprocess active FastReconnect Stream ended; checking every 10 s for up to 2 minutes SlowReconnect Still not live after 2 min; checking every 60 s concurrently with post-processing PostProcessing Confirmed ended; ffmpeg / VOD download / upload running NetworkFault Twitch API unreachable; exponential back-off (30 s → capped at 10 min) Transitions:
Recording → streamlink exits → enter FastReconnect, record phase start time FastReconnect → live confirmed → start new Recording (new filename/segment) FastReconnect (2 min elapsed) → enter SlowReconnect + kick off PostProcessing concurrently SlowReconnect → live confirmed → start new Recording SlowReconnect / Monitoring → API call throws network error → enter NetworkFault NetworkFault → successful API response → return to previous state (Monitoring or re-enter FastReconnect if we were mid-reconnect) NetworkFault backoff: 30s * 2^attempt, capped at 600s RecoveryPolicy is a pure class with a Tick(DateTime now, bool? isLive, bool networkError) method → returns RecoveryDecision (what to do next + sleep duration). Fully unit-testable with no async or DI.
StreamWorker : BackgroundService
One instance per enabled streamer Holds RecoveryPolicy instance Main loop: evaluate policy decision → execute the corresponding service call → loop Started/stopped by StreamWorkerManager Writes job records to SQLite on start/complete/fail StreamWorkerManager
StartWorker(string username), StopWorker(string username), RestartWorker(string username) Called at app startup for all enabled streamers Called from Web UI on enable/disable/config change Workers stored in ConcurrentDictionary<string, (StreamWorker, CancellationTokenSource)> Step 6 — Persistence (SQLite + EF Core) ArchiveDbContext with three tables:
StreamSessions: Id, StreamerUsername, TwitchStreamId, Title, StartedAt, EndedAt, Status (Recording/Processing/Uploading/Complete/Failed)
ArchiveJobs: Id, SessionId, JobType (enum: RecordLive, ProcessLive, DownloadVod, ProcessVod, UploadCloud, DeleteLocal), Status, StartedAt, CompletedAt, FilePath, ErrorMessage
StreamerStates: Username, IsMonitoring, LastCheckedAt, CurrentRecoveryState
Migrations via EF Core CLI. ISessionRepository / IJobRepository interfaces for testability with in-memory EF provider in tests.
Step 7 — Process output streaming IProcessOutputStore:
AppendLine(string streamerId, Guid jobId, string line, bool isError) GetRecentLines(string streamerId, Guid jobId, int count = 500) → IReadOnlyList In-memory circular buffer (1000 lines per job, last 20 jobs per streamer) ProcessOutputHub : Hub (SignalR):
Clients call SubscribeToStreamer(string username) → join group streamer:{username} Clients call SubscribeToJob(Guid jobId) → join group job:{jobId} Server pushes ReceiveLine(OutputLine line) from ProcessRunner via IHubContext On subscribe: server immediately sends buffered lines from IProcessOutputStore Step 8 — Blazor Server Web UI Authentication: Cookie-based single-password auth via ASP.NET Core minimal auth middleware. Password stored as BCrypt hash in AppSettings. Login.razor page at /login. Protected routes with [Authorize].
Pages & Components:
Dashboard.razor (/) — grid of all configured streamers showing: username, live/offline badge, current recovery state, last recorded session, quick Start/Stop monitoring toggle
StreamerDetail.razor (/streamer/{username}) — live status, current job pipeline steps (record → process → upload with progress), ProcessOutputConsole.razor showing real-time terminal output via SignalR
ProcessOutputConsole.razor — reusable Blazor component; subscribes to SignalR on mount, renders an auto-scrolling
with colored output (stdout = white, stderr = orange/red), handles reconnectSessions.razor (/sessions) — paginated list of past archive sessions with job statuses and expandable per-job output
GlobalConfig.razor (/config/global) — EditForm bound to GlobalConfig model with data annotations validation, Save button calls IConfigurationService.SaveGlobal()
StreamerConfig.razor (/config/{username}) — similar form for per-streamer overrides; each field has a nullable toggle (inherit from global vs override)
AddStreamer.razor (/config/new) — minimal form: username + enabled; creates new config/streamers/{username}.json
AppSettings.razor (/settings) — tool paths (streamlink, ffmpeg, TwitchDownloaderCLI, rclone), change password
Step 9 — NLog configuration nlog.config (XML): two targets:
Console (colored, with level formatting) File rolling (logs/archive-${shortdate}.log, keep 30 days) Log structured context: StreamerUsername, JobId, JobType as NLog ScopeContext properties. Service methods open a scope via ILogger.BeginScope(...).
Step 10 — Docker Dockerfile (multi-stage):
Build stage: mcr.microsoft.com/dotnet/sdk:10.0 Runtime stage: mcr.microsoft.com/dotnet/aspnet:10.0 (Linux) Install ffmpeg, streamlink (via pip), download TwitchDownloaderCLI binary for linux-x64 rclone installed via shell script or apt Expose port 8080 ENTRYPOINT ["dotnet", "TwitchArchive.Web.dll"] docker-compose.yml:
Volume mounts: ./config:/app/config, ./archive:/app/archive, ./logs:/app/logs Environment variables: CLIENT-ID, CLIENT-SECRET, OAUTH-PRIVATE-TOKEN Windows dev: run directly with dotnet run; tool paths auto-detected (Windows vs Linux) via RuntimeInformation.IsOSPlatform(OSPlatform.Windows) in ToolPathResolver
Step 11 — Unit tests TwitchArchive.Tests covers:
RecoveryPolicyTests — state machine transitions, timing phases, network fault backoff; pure synchronous tests ConfigurationServiceTests — JSON load/save/merge with temp files TwitchApiClientTests — mocked HttpMessageHandler; OAuth, GQL queries, 401 refresh, network errors FileManagerServiceTests — path generation, directory creation with temp directories DownloaderServiceTests / RecorderServiceTests — mocked ProcessRunner; verify correct CLI arguments UploadServiceTests — mocked ProcessRunner; verify rclone argument construction SessionRepositoryTests — EF Core in-memory provider EffectiveConfigTests — global + streamer override merge logic Verification dotnet build dotnet/TwitchArchive.sln — zero warnings, zero errors dotnet test dotnet/TwitchArchive.Tests/ — all tests green docker compose up --build in the dotnet/ folder → app reachable at http://localhost:8080 Manual: add a test streamer config, enable monitoring, confirm it polls and records a live stream, runs ffmpeg, uploads via rclone Resilience manual test: kill network during recording → verify FastReconnect phase kicks in, resumes after connectivity restored Decisions
Blazor Server chosen for real-time terminal output without a separate API; no WASM needed streamlink for live + TwitchDownloaderCLI for VOD (same split as Python; streamlink gives better live resilience) Simple BCrypt password auth (not full Identity — this is a single-user tool) RecoveryPolicy as a pure POCO state machine keeps the resilience logic fully unit-testable without async/mocking Polly WaitAndRetryForever on HttpClient handles persistent network failure independently of the application-level state machine; they are complementary — Polly handles individual HTTP call retries, RecoveryPolicy handles the overall workflow state Chat download service interface is defined in Step 4 but methods are stubbed — adding implementation later requires only filling in DownloaderService without touching any other layer Config files remain in the same format/location so the Python and C# versions can share the same config directory