Chat Updater Rewrite + GUI frontend, QOL changes (#442)

* Move chatupdate functions to Core, prepare for additional update options, shorten chatupdate file/class names * Make chatdownload obey custom temp folder * Fixed all embeds being doubled with arg --embed-missing, fixed crash when embeddedData is null, made embed fetching async * Continue before processing the new embedded item * Implement chat crop moving * Rename an arg, add input = output warning, typo * Update readme * Prepare for fetching missing comments (needs #440), cleanup * Update readme * Make ChatUpdater task based * Add missing comment fetching, make default true bools nullable, add more progress reporters * Move chat crop above embed fetching due to dependency, cleanup * Fix comment appending not being fully thread-safe, add comment count report, ensure temp file is deleted, cleanup * Update readmes * Typo * Clarify ReportType enum, fix chatupdate progress reports, clamp chat crops, cleanup * Fix first party emotes not proprely checking for already added embedded emotes, fix emoji count check for Linux * Rename 'DownloadFormat' to 'ChatFormat', add exception for catching unsupported chatupdate formats * Fix future chat crop error when using non-json formats * Simplify switch expressions with pattern matching * Add GUI frontend for ChatUpdate, more UI consistency, cleanup * Fix clip chat updating, add enqueuing support * Add chat update to default settings * Didn't mean to stage that * Add output format conversion (input must still be json), cleanup * Add step counter to progress chatupdate reports * Add percent and universal constructor to ProgressReport class, add percent reports to chatupdate * Use async methods with text chat writer, combine ParseChatJson and ParseChatJsonInfo, cleanup * Update 'what can it do' section in readme * Cleanup PreParseArgs, fix #448 * Fix NRE caused by 4867ddb, fix catch block in TwitchHelper * Better solution than 552ba3b, add additional icon sizes * Add build instructions to README, remove last traces of .Net Framework, other README improvements * Fix StatusInfo not fully overwriting the previous status, Move json, html, txt chat operations to dedicated classes, Fix sample text height being computed every render tick as it is expensive, Rewrite SubstringToTextWidth() to use spans, Cleanup * Sealed core runmode classes, minor TwitchHelper optimizations, convert emoji check from LINQ to foreach (faster), use arrays instead of lists, stringbuilders instead of strings where possible, combine some Rtl and non-Rtl functions, cleanup * Only use DrawShapedText() with messages containing RTL
lay295 · Dec 11, 2022 · 2def0e9 · 2def0e9
1 parent 3fa7cf4
commit 2def0e9
Show file tree

Hide file tree

Showing 57 changed files with 3,099 additions and 1,813 deletions.
diff --git a/README.md b/README.md
@@ -23,7 +23,8 @@ https://user-images.githubusercontent.com/1060681/197653099-c3fd12c2-f03a-4580-8
 ## What can it do?
 - Download Twitch VODs
 - Download Twitch Clips
-- Download chat for VODS and Clips, in either a [JSON with all the information](https://pastebin.com/raw/YDgRe6X4) or a [simple text file](https://pastebin.com/raw/016azeQX)
+- Download chat for VODS and Clips, in either a [JSON with all the original information](https://pastebin.com/raw/YDgRe6X4), a browser HTML file, or a [plain text file](https://pastebin.com/raw/016azeQX)
+- Update the contents of a previously generated JSON chat file with an option to save as another format
 - Use a previously generated JSON chat file to render the chat with FFZ, BTTV and 7TV support (including GIFS)
 
 # GUI
@@ -52,61 +53,73 @@ Check twitch-downloader-gui on [github](https://github.com/mohad12211/twitch-dow
 
 ## MacOS?
 
-No GUI is avaiable for MacOS yet :(
+No GUI is available for MacOS yet :(
 
 # CLI
 
-The CLI is cross platform and performs the main functions of the program. It works on Windows, Linux, and MacOS.*
+### [See the full CLI documentation here](TwitchDownloaderCLI/README.md).
 
-*Only Intel Macs have been tested
+The CLI is cross-platform and implements the main functions of the program. It works on Windows, Linux, and MacOS<sup>*</sup>.
 
-### [CLI Documentation here](TwitchDownloaderCLI/README.md). 
+<sup>*Only Intel Macs have been tested</sup>
 
-I've never really made a command line utility before so things may change in the future. If you're on Linux, make sure `fontconfig` and `libfontconfig1` are installed `(apt-get install fontconfig libfontconfig1)`.
-
-For example, you could copy/paste this into a `.bat` file on Windows, to download a VOD, chat, and then render in a single go.  
-```
+With the CLI, it is possible to automate video processing using external scripts. For example, you could copy-paste the following code into a `.bat` file on Windows to download a VOD and its chat, and then render the chat, all from a single input.
+```bat
 @echo off
 set /p vodid="Enter VOD ID: "
 TwitchDownloaderCLI.exe videodownload --id %vodid% --ffmpeg-path "ffmpeg.exe" -o %vodid%.mp4
-TwitchDownloaderCLI.exe chatdownload --id %vodid% -o %vodid%_chat.json
+TwitchDownloaderCLI.exe chatdownload --id %vodid% -o %vodid%_chat.json -E
 TwitchDownloaderCLI.exe chatrender -i %vodid%_chat.json -h 1080 -w 422 --framerate 30 --update-rate 0 --font-size 18 -o %vodid%_chat.mp4
 ```
----
-### Linux – Getting started
 
-1. Go to [Releases](https://github.com/lay295/TwitchDownloader/releases/) and download the latest version for Linux.
-2. Extract `TwitchDownloaderCLI`
-3. Browse to where you extracted the file and give it executable rights in Terminal:
+## Windows - Getting started
+
+1. Go to [Releases](https://github.com/lay295/TwitchDownloader/releases/) and download the latest version for Windows or [build from source](#building-from-source).
+2. Extract `TwitchDownloaderCLI.exe`.
+3. Browse to where you extracted the file in the terminal.
+4. If you do not have ffmpeg, you can install it via [Chocolatey package manager](https://community.chocolatey.org/), or you can get it as a standalone file from [ffmpeg.org](https://ffmpeg.org/download.html) or by using TwitchDownloaderCLI:
+```
+TwitchDownloaderCLI.exe ffmpeg --download
+```
+5. You can now start using the downloader, for example:
+```
+TwitchDownloaderCLI.exe videodownload --id <vod-id-here> -o out.mp4
+```
+
+## Linux – Getting started
+
+1. Ensure both `fontconfig` and `libfontconfig1` are installed. `apt-get install fontconfig libfontconfig1` on Ubuntu.
+2. Go to [Releases](https://github.com/lay295/TwitchDownloader/releases/) and download the latest binary for Linux, grab the [AUR Package](https://aur.archlinux.org/packages/twitch-downloader-bin/) for Arch Linux, or [build from source](#building-from-source).
+3. Extract `TwitchDownloaderCLI`.
+4. Browse to where you extracted the file and give it executable rights in the terminal:
 ```
 sudo chmod +x TwitchDownloaderCLI
 ```
-4. If you do not have ffmpeg, you should install it via your distro package manager, however you can also get it as a standalone file from [ffmpeg.org](https://ffmpeg.org/download.html) or by using TwitchDownloaderCLI:
+5. a) If you do not have ffmpeg, you should install it via your distro package manager, however you can also get it as a standalone file from [ffmpeg.org](https://ffmpeg.org/download.html) or by using TwitchDownloaderCLI:
 ```
 ./TwitchDownloaderCLI ffmpeg --download
 ```
-If downloaded as a standalone file, you must also give it executable rights with:
+5. b) If downloaded as a standalone file, you must also give it executable rights with:
 ```
 sudo chmod +x ffmpeg
 ```
-5. You can now start using the downloader, for example:
+6. You can now start using the downloader, for example:
 ```
 ./TwitchDownloaderCLI videodownload --id <vod-id-here> -o out.mp4
 ```
-For Arch Linux, there's an [AUR Package](https://aur.archlinux.org/packages/twitch-downloader-bin/)
 
-### MacOS – Getting started
-1. Go to [Releases](https://github.com/lay295/TwitchDownloader/releases/) and download the latest version for MacOS.
-2. Extract `TwitchDownloaderCLI`
-3. Browse to where you extracted the file and give it executable rights in Terminal:
+## MacOS – Getting started
+1. Go to [Releases](https://github.com/lay295/TwitchDownloader/releases/) and download the latest binary for MacOS or [build from source](#building-from-source).
+2. Extract `TwitchDownloaderCLI`.
+3. Browse to where you extracted the file and give it executable rights in the terminal:
 ```
 chmod +x TwitchDownloaderCLI
 ```
-4. If you do not have ffmpeg, you can install it via [Homebrew package manager](https://brew.sh/), or you can get it as a standalone file from [ffmpeg.org](https://ffmpeg.org/download.html) or by using TwitchDownloaderCLI:
+4. a) If you do not have ffmpeg, you can install it via [Homebrew package manager](https://brew.sh/), or you can get it as a standalone file from [ffmpeg.org](https://ffmpeg.org/download.html) or by using TwitchDownloaderCLI:
 ```
 ./TwitchDownloaderCLI ffmpeg --download
 ```
-If downloaded as a standalone file, you must also give it executable rights with:
+4. b) If downloaded as a standalone file, you must also give it executable rights with:
 ```
 chmod +x ffmpeg
 ```
@@ -115,6 +128,44 @@ chmod +x ffmpeg
 ./TwitchDownloaderCLI videodownload --id <vod-id-here> -o out.mp4
 ```
 
+# Building from source
+
+## Requirements
+
+- [.NET 6.0.x SDK](https://dotnet.microsoft.com/en-us/download/dotnet/6.0)
+
+## Build Instructions
+
+1. Clone the repository:
+```
+git clone https://github.com/lay295/TwitchDownloader.git
+```
+2. Navigate to the solution folder:
+```
+cd TwitchDownloader
+```
+3. Restore the solution:
+```
+dotnet restore
+```
+4. a) Build the GUI:
+```
+dotnet publish TwitchDownloaderWPF -p:PublishProfile=Windows -p:DebugType=None -p:DebugSymbols=false
+```
+4. b) Build the CLI:
+```
+dotnet publish TwitchDownloaderCLI -p:PublishProfile=<Profile> -p:DebugType=None -p:DebugSymbols=false
+```
+- Applicable Profiles: `Windows`, `Linux`, `LinuxAlpine`, `LinuxArm`, `MacOS`
+5. a) Navigate to the GUI build folder:
+```
+cd TwitchDownloaderWPF/bin/Release/net6.0-windows/publish/win-x64
+```
+5. b) Navigate to the CLI build folder:
+```
+cd TwitchDownloaderCLI/bin/Release/net6.0/publish
+```
+
 # License
 
 [MIT](./LICENSE.txt)
diff --git a/TwitchDownloaderCLI/Modes/Arguments/ChatDownloadArgs.cs b/TwitchDownloaderCLI/Modes/Arguments/ChatDownloadArgs.cs
@@ -10,7 +10,7 @@ public class ChatDownloadArgs
         [Option('u', "id", Required = true, HelpText = "The ID of the VOD or clip to download that chat of.")]
         public string Id { get; set; }
 
-        [Option('o', "output", Required = true, HelpText = "Path to output file. File extension will be used to determine download type. Valid extensions are json, html, and txt.")]
+        [Option('o', "output", Required = true, HelpText = "Path to output file. File extension will be used to determine download type. Valid extensions are: json, html, and txt.")]
         public string OutputFile { get; set; }
 
         [Option('b', "beginning", HelpText = "Time in seconds to crop beginning.")]
@@ -31,13 +31,13 @@ public class ChatDownloadArgs
         [Option("stv", Default = true, HelpText = "Enable 7tv embedding in chat download. Requires -E / --embed-images!")]
         public bool? StvEmotes { get; set; }
 
-        [Option("timestamp", Default = false, HelpText = "Enable timestamps for .txt chat downloads.")]
-        public bool Timestamp { get; set; }
-
-        [Option("timestamp-format", Default = TimestampFormat.Relative, HelpText = "Sets the timestamp format for .txt chat logs. Valid values are Utc, Relative, and None")]
+        [Option("timestamp-format", Default = TimestampFormat.Relative, HelpText = "Sets the timestamp format for .txt chat logs. Valid values are: Utc, Relative, and None")]
         public TimestampFormat TimeFormat { get; set; }
 
         [Option("chat-connections", Default = 4, HelpText = "Number of downloading connections for chat")]
         public int ChatConnections { get; set; }
+
+        [Option("temp-path", Default = "", HelpText = "Path to temporary folder to use for cache.")]
+        public string TempFolder { get; set; }
     }
 }
diff --git a/TwitchDownloaderCLI/Modes/Arguments/ChatDownloadUpdaterArgs.cs b/TwitchDownloaderCLI/Modes/Arguments/ChatDownloadUpdaterArgs.cs
diff --git a/TwitchDownloaderCLI/Modes/Arguments/ChatRenderArgs.cs b/TwitchDownloaderCLI/Modes/Arguments/ChatRenderArgs.cs
@@ -69,10 +69,10 @@ public class ChatRenderArgs
         [Option("update-rate", Default = 0.2, HelpText = "Time in seconds to update chat render output.")]
         public double UpdateRate { get; set; }
 
-        [Option("input-args", Default = "-framerate {fps} -f rawvideo -analyzeduration {max_int} -probesize {max_int} -pix_fmt {pix_fmt} -video_size {width}x{height} -i -", HelpText = "Input (pass 1) arguments for ffmpeg chat render.")]
+        [Option("input-args", Default = "-framerate {fps} -f rawvideo -analyzeduration {max_int} -probesize {max_int} -pix_fmt {pix_fmt} -video_size {width}x{height} -i -", HelpText = "Input arguments for ffmpeg chat render.")]
         public string InputArgs { get; set; }
 
-        [Option("output-args", Default = "-c:v libx264 -preset veryfast -crf 18 -pix_fmt yuv420p \"{save_path}\"", HelpText = "Output (pass 2) arguments for ffmpeg chat render.")]
+        [Option("output-args", Default = "-c:v libx264 -preset veryfast -crf 18 -pix_fmt yuv420p \"{save_path}\"", HelpText = "Output arguments for ffmpeg chat render.")]
         public string OutputArgs { get; set; }
 
         [Option("ignore-users", Default = "", HelpText = "List of usernames to ignore when rendering, separated by commas.")]
@@ -81,7 +81,7 @@ public class ChatRenderArgs
         [Option("badge-filter", Default = 0, HelpText = "Bitmask of types of Chat Badges to filter out. Add the numbers of the types of badges you want to filter. For example, 6 = no broadcaster or moderator badges.\r\nKey: Other = 1, Broadcaster = 2, Moderator = 4, VIP = 8, Subscriber = 16, Predictions = 32, NoAudio/NoVideo = 64, PrimeGaming = 128")]
         public int BadgeFilterMask { get; set; }
 
-        [Option("offline", Default = false, HelpText = "Render completely offline, using only resources embedded emotes, badges, and bits in the input json.")]
+        [Option("offline", Default = false, HelpText = "Render completely offline using only embedded emotes, badges, and bits from the input json.")]
         public bool Offline { get; set; }
 
         [Option("ffmpeg-path", HelpText = "Path to ffmpeg executable.")]

diff --git a/TwitchDownloaderCLI/Modes/Arguments/ChatUpdateArgs.cs b/TwitchDownloaderCLI/Modes/Arguments/ChatUpdateArgs.cs
@@ -0,0 +1,43 @@
+using CommandLine;
+using TwitchDownloaderCore.Options;
+
+namespace TwitchDownloaderCLI.Modes.Arguments
+{
+
+    [Verb("chatupdate", HelpText = "Updates the embeded emotes, badges, bits, and crops of a chat download and/or converts a JSON chat to another format.")]
+    public class ChatUpdateArgs
+    {
+        [Option('i', "input", Required = true, HelpText = "Path to input file. Valid extensions are: json.")]
+        public string InputFile { get; set; }
+
+        [Option('o', "output", Required = true, HelpText = "Path to output file. File extension will be used to determine new chat type. Valid extensions are: json, html, and txt.")]
+        public string OutputFile { get; set; }
+
+        [Option('E', "embed-missing", Default = false, HelpText = "Embed missing emotes, badges, and cheermotes. Already embedded images will be untouched.")]
+        public bool EmbedMissing { get; set; }
+
+        [Option('R', "replace-embeds", Default = false, HelpText = "Replace all embedded emotes, badges, and cheermotes in the file. All embedded images will be overwritten!")]
+        public bool ReplaceEmbeds { get; set; }
+
+        [Option('b', "beginning", Default = -1, HelpText = "New time in seconds for chat beginning. Comments may be added but not removed. -1 = No crop.")]
+        public int CropBeginningTime { get; set; }
+
+        [Option('e', "ending", Default = -1, HelpText = "New time in seconds for chat ending. Comments may be added but not removed. -1 = No crop.")]
+        public int CropEndingTime { get; set; }
+
+        [Option("bttv", Default = true, HelpText = "Enable BTTV embedding in chat download.")]
+        public bool? BttvEmotes { get; set; }
+
+        [Option("ffz", Default = true, HelpText = "Enable FFZ embedding in chat download.")]
+        public bool? FfzEmotes { get; set; }
+
+        [Option("stv", Default = true, HelpText = "Enable 7TV embedding in chat download.")]
+        public bool? StvEmotes { get; set; }
+
+        [Option("timestamp-format", Default = TimestampFormat.Relative, HelpText = "Sets the timestamp format for .txt chat logs. Valid values are: Utc, Relative, and None")]
+        public TimestampFormat TimeFormat { get; set; }
+
+        [Option("temp-path", Default = "", HelpText = "Path to temporary folder to use for cache.")]
+        public string TempFolder { get; set; }
+    }
+}
diff --git a/TwitchDownloaderCLI/Modes/DownloadChat.cs b/TwitchDownloaderCLI/Modes/DownloadChat.cs
@@ -12,7 +12,7 @@ internal class DownloadChat
     {
         internal static void Download(ChatDownloadArgs inputOptions)
         {
-            if (inputOptions.Id == string.Empty)
+            if (string.IsNullOrWhiteSpace(inputOptions.Id))
             {
                 Console.WriteLine("[ERROR] - Invalid ID, unable to parse.");
                 Environment.Exit(1);
@@ -22,24 +22,23 @@ internal static void Download(ChatDownloadArgs inputOptions)
             {
                 DownloadFormat = Path.GetExtension(inputOptions.OutputFile)!.ToLower() switch
                 {
-                    ".json" => DownloadFormat.Json,
-                    ".html" => DownloadFormat.Html,
-                    ".htm" => DownloadFormat.Html,
-                    _ => DownloadFormat.Text
+                    ".html" or ".htm" => ChatFormat.Html,
+                    ".json" => ChatFormat.Json,
+                    _ => ChatFormat.Text
                 },
                 Id = inputOptions.Id,
                 CropBeginning = inputOptions.CropBeginningTime > 0.0,
                 CropBeginningTime = inputOptions.CropBeginningTime,
                 CropEnding = inputOptions.CropEndingTime > 0.0,
                 CropEndingTime = inputOptions.CropEndingTime,
-                Timestamp = inputOptions.Timestamp,
                 EmbedData = inputOptions.EmbedData,
                 Filename = inputOptions.OutputFile,
                 TimeFormat = inputOptions.TimeFormat,
                 ConnectionCount = inputOptions.ChatConnections,
                 BttvEmotes = (bool)inputOptions.BttvEmotes,
                 FfzEmotes = (bool)inputOptions.FfzEmotes,
-                StvEmotes = (bool)inputOptions.StvEmotes
+                StvEmotes = (bool)inputOptions.StvEmotes,
+                TempFolder = inputOptions.TempFolder
             };
 
             ChatDownloader chatDownloader = new(downloadOptions);