From 466f6109eea6084e585cfe2c1664decdeed5e7c9 Mon Sep 17 00:00:00 2001 From: Pieter-Jan Briers Date: Sun, 14 Jul 2024 20:58:19 +0200 Subject: [PATCH] Rewrite Robust.Cdn page (#259) --- .../server-hosting/setting-up-robust-cdn.md | 434 ++++++++++++++---- 1 file changed, 352 insertions(+), 82 deletions(-) diff --git a/src/en/server-hosting/setting-up-robust-cdn.md b/src/en/server-hosting/setting-up-robust-cdn.md index 9d145abde..0cd5a92d1 100644 --- a/src/en/server-hosting/setting-up-robust-cdn.md +++ b/src/en/server-hosting/setting-up-robust-cdn.md @@ -1,145 +1,415 @@ # Setting up Robust.Cdn -[Robust.Cdn](https://github.com/space-wizards/Robust.Cdn/) is the dedicated server for providing manifest-based delta updates without server ACZ. +[Robust.Cdn](https://github.com/space-wizards/Robust.Cdn/) is a dedicated server for hosting and serving game build files for Space Station 14 servers. It covers both management of game server builds as well as client delta downloads, and is recommended for any serious permanent server host. -The TL;DR of how `Robust.Cdn` operates is that it ingests directories of client zip files whenever you push a new build, then generates manifests and exposes the manifest and endpoint to do delta downloads. The games files are cached in an SQLite DB similar to how the launcher stores it. +```admonish warning +This page assumes a decent amount of pre-existing experience with various Linux sysadmin concepts. And don't blindly copy paste shit. I don't have the time or energy to baby proof this guide, so you'll need to properly read and understand everything listed here. +``` + +## Overview + +The standard publishing workflow for Space Station 14 looks like so: + +1. Builds are periodically created by a CI system like GitHub Actions. +2. Build files (client and server zips) are sent to a central server. Server builds get indexed and configuration injected for client CDN, client files are ingested so they can be downloaded by the launcher. +3. Game servers (via SS14.Watchdog) get notified of update, and automatically download new server builds from central server. +4. Injected game server configuration is used to inform clients where they can download files. + +Robust.Cdn is the glue that wires most of this together: + +* Receiving new builds directly from GitHub Actions. +* Serving game server builds for access by game servers (watchdog). +* Automatically notifying watchdogs of the new update. +* Providing client delta downloads. + +## Concepts + +Robust.Cdn currently serves two core functions: + +* Managing server manifests +* Managing client delta downloads + +The **server manifest** is effectively the list of available game server versions. This is used by the watchdog to download new server updates. + +The **client delta downloads** or "**client CDN**" is used by game clients to download new files on update, downloading only what is necessary. + +```admonish info +It is possible to run Robust.Cdn without making use of the server manifest support. In fact before 2.0, Robust.Cdn only did client CDN. See the rest of the documentation for details. +``` + +A **fork** is a single "stream of development" of the game. For example, Space Station 14 upstream (Wizard's Den) and Rouny's Marine Corps would be considered two different forks. Robust.Cdn can manage multiple forks at once, independently. + +A **version** or **build** is just a single version of the game on a fork. It is always the intent that the most recent version is used by servers, but Robust.Cdn serves older versions as well. -## Prerequisites +## Installation -You will need the ASP.NET Core 6 runtime installed. +We provide official container images of Robust.Cdn via GitHub Container Registry. Otherwise, we also have instructions for manually publishing the project via the .NET SDK. -## Building +### Container image -Clone the code, then compile it with `dotnet publish -c Release -r linux-x64 --no-self-contained` The files will be dropped in `Robust.Cdn/bin/Release/net6.0/linux-x64/publish`. `Robust.Cdn` is the executable to run, but obviously copy everything else too. +The latest stable image of Robust.Cdn is `ghcr.io/space-wizards/robust.cdn:2`. Information about the container: -## Server File Layout +* Listens on port 8080 +* Default UID/GID is 1654 +* Important volumes to mount: + * `/app/appsettings.json`: primary config file. + * `/builds`: contains server/client build zip files. + * `/manifest`: contains SQLite database for server manifest operations. + * `/database`: contains SQLite database for client content downloads. -You're going to want to put the server files somewhere on your uhhh, server. We put them at `/opt/robust_cdn`, with the following directory structure: +### Manual compilation + +If you hate containers, you can manually publish Robust.Cdn and deploy the files yourself. For this you will need Git and the .NET 8 SDK. The server that will run the build needs the matching ASP.NET Core Runtime installed, but does not need the SDK itself. + +Clone the git repo, then publish: + +``` +git clone https://github.com/space-wizards/Robust.Cdn.git +cd Robust.Cdn +dotnet publish -c Release -r linux-x64 --no-self-contained +``` + +The finished build will be dropped in `Robust.Cdn/bin/Release/net8.0/linux-x64/publish`. You can copy these into some random location you fancy like `/opt` and run `Robust.Cdn` from there. For example: ``` /opt/robust_cdn/ ├── appsettings.json -├── run.sh ├── bin │   ├── Robust.Cdn │   ├── Robust.Cdn.dll - . + . ``` -The `run.sh` looks like this by the way, yes it's just that simple: +The actual program files go in a subfolder, and we run it from the parent directory so you don't bulldoze the config files with updates or something. + +You can then run Robust.Cdn automatically with the following systemd service definition: -```bash -#!/bin/sh +```ini +# /etc/systemd/system/robust-cdn.service +[Unit] +Description=Robust.Cdn + +[Service] +Type=notify +WorkingDirectory=/opt/robust_cdn/ +ExecStart=/opt/robust_cdn/bin/Robust.Cdn +User=robust_cdn -exec bin/Robust.Cdn +[Install] +WantedBy=multi-user.target ``` -The actual program files go in a subfolder, and we run it from the parent directory so you don't bulldoze the config files with updates or something. +```admonish warning +For the love of Miku and all that is holy, do not run the CDN from directly within the build directory. Please. +``` ## Configuration -The configuration file is `appsettings.json` in the current working directory you run the server program from. You need to change at least two configuration settings: the disk path of your game versions, and the update token to trigger updates. Maybe port too. +Robust.Cdn is an ASP.NET Core app, so it supports configuration both via config file and other sources such as environment variables. You can see [ASP.NET Core's documentation](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-8.0) for a more in-depth overview. + +Most configuration of Robust.Cdn is done via the `appsettings.json` config file. Here is a complete reference of its contents: + +```admonish danger +You should go over this config file in full to understand what you are setting up. +``` ```json { + // Log level configuration, you can leave these as default. "Logging": { "LogLevel": { "Default": "Information", - "Microsoft.AspNetCore": "Warning" + "Microsoft.AspNetCore": "Warning", + "Robust": "Information" } }, - "AllowedHosts": "*", - // change port for the server to bind to here - "Urls": "http://localhost:27690/", + + // Contains configuration primarily for manifest operation, + // but is also needed for CDN operations to configure available forks. + "Manifest": { + // The location on disk where builds should be stored. + // Omit this when using official container images. + "FileDiskPath": "/var/robust-cdn/builds", + + // Database file that contains information for server manifest functionalities. + // Omit this when using official container images. + "DatabaseFileName": "/var/robust-cdn/manifest.db", + + // The set of available forks Robust.Cdn should serve. + "Forks": { + // Configuration for a single fork. The key here is the ID of the fork, which will be used in many locations. + // YOU SHOULD MAKE THE FORK ID COMPLETELY GLOBALLY UNIQUE TO AVOID ISSUES. + // DON'T JUST WRITE "TEST" HERE. + "test": { + // A token used to publish new versions to this fork. + // ***YOU SHOULD CHANGE THIS TO BE A NEW UNIQUE VALUE***. + "UpdateToken": "foobar", + + // Configuration to notify SS14.Watchdog instances of new updates. Multiple can be specified. + "NotifyWatchdogs": [ + { + // The base address of the watchdog. + "WatchdogUrl": "http://localhost:5000/", + + // The specific server instance on the watchdog to notify. + "Instance": "syndicate_mothership", + + // The ApiToken specified in the watchdog config for this instance. + "ApiToken": "Honk" + } + ], + + // Set to true to make this a "private" fork. + // Private forks limit access of server builds, which is desirable for servers with secret content. + // See below for details. + "Private": false, + + // Username and password combinations to access server files for private forks. + // Ignored if not a private fork. + "PrivateUsers": { + "foobar": "baz" + }, + + // How many days to keep around old build files. + "PruneBuildsDays": 90, + + // Nice human readable display name of this fork. + // This is displayed on locations such as the HTML builds page. + "DisplayName": "Test Fork", + + // Link destination on the HTML builds page. + "BuildsPageLink": "https://example.com", + + // Link text on the HTML builds page. + "BuildsPageLinkText": "Test Fork LINK" + } + } + }, + + // Configuration primarily for client CDN. "Cdn": { - // See https://github.com/space-wizards/Robust.Cdn/blob/master/Robust.Cdn/CdnOptions.cs for all options. - // Yes you can have comments in here by the way. - // Change this update token so people don't uh... spam update checks on your server. - "UpdateToken": "change this", + // Database file that contains information for server manifest functionalities. + // Omit this when using official container images. + "DatabaseFileName": "/var/robust-cdn/content.db", + // Increase this to reduce bandwidth usage for large downloads. Higher numbers need more CPU. "StreamCompressLevel": 5, - // The file path of the SQLite database used by the server. - // You can change this to put it on another drive or something if you want. - // Personally we put it on our server's HDD instead of the SSD, but that's up to you. - "DatabaseFileName": "content.db", - // The directory path containing the versioned client builds to load and serve. - // See below for explanation - "VersionDiskPath": "/var/lib/wizards-builds/builds" - } + + // "Fallback" fork for migration functionality from Robust.Cdn 1.x. + // This can be omitted for new installations. + "DefaultFork": "test" + }, + + // Root URL that your Robust.Cdn server is globally accessible as. + // This is necessary for correct generation of build metadata. + "BaseUrl": "https:///", + + // Valid host names for clients to use to connect to Robust.Cdn. + // You can just leave this as-is. + "AllowedHosts": "*", + + // Change port for the Robust.Cdn to bind to here. + // Omit this when using official container images. + "Urls": "http://localhost:27690/", } ``` -### Version disk +## Setting up publishing + +### GitHub Actions + +If your fork's repository is hosted on GitHub, the easiest way to automatically publish new builds to Robust.Cdn is via the GitHub Actions configuration available in the codebase. This is how official Wizard's Den builds are published. + +1. Edit `Tools/publish_github_artifact.py` to modify the "configuration parameters" at the top of the script: + * `ROBUST_CDN_URL` should be the URL at which Robust.Cdn is accessible. + * `FORK_ID` should be the ID of the fork you configured in `appsettings.json` -`Robust.Cdn` needs a directory containing builds to load, with the following layout: +2. Create an Actions secret on your GitHub repository with name `PUBLISH_TOKEN`, containing the `UpdateToken` specified for your fork in `appsettings.json`. +3. Make sure the "Publish" workflow is running, or trigger it manually. + +This should be everything you need! + +```admonish warning +Actions publishing works by uploading a temporary artifact and having Robust.Cdn download that. While this should work from a private repo there may be additional cost introduced by this, you've been warned. ``` -/var/lib/wizards-builds/builds -├── 02030cfa0ed6511ec5527c5b7d1f8bcd46fe1435 -│   ├── SS14.Client.zip -│   ├── SS14.Server_linux-arm64.zip -│   ├── SS14.Server_linux-x64.zip -│   ├── SS14.Server_osx-x64.zip -│   └── SS14.Server_win-x64.zip -├── 021d39be2876f991c5fd6e663760a921d29ac694 -│   ├── SS14.Client.zip -│   ├── SS14.Server_linux-arm64.zip + +### Custom + +For people looking to do custom publishing workflows without GitHub actions, please refer to the API reference of the "publish" endpoint. + +## Watchdog configuration + +Configuration SS14.Watchdog to use your new Robust.Cdn for getting builds is pretty easy. In the instance configuration, enter something like this: + +```yml +UpdateType: "Manifest" +Updates: + # Replace with your own Robust.Cdn URL and fork ID. + ManifestUrl: "https:///fork//manifest" ``` -It will look for `SS14.Client.zip` (configurable) in each subdirectory to the directory to specify. The subdirectory's name is the "version" of content. (Note: these are hashes because we use the Git commit hash as version "number". They are not tied to manifest or zip hashes in any way, and can be any string, really). +You will likely also want to set up `NotifyWatchdogs` in Robust.Cdn's fork configuration, so it notifies SS14.Watchdog when a new version is available. Check the reference up above. -### Automatic detection of new versions +## Builds HTML page -You can use something like the following to trigger `Robust.Cdn` to scan for new versions in the version directory: +Robust.Cdn generates a simple HTML web page to allow people to manually download the latest server builds. This page is available automatically at `/fork/`. -```bash -curl -X POST -d "" -H 'Authorization: Bearer ' "http://localhost:27690/control/update" +For example: [Wizard's Den builds](https://wizards.cdn.spacestation14.com/fork/wizards/). + +## Private forks + +```admonish failure +This feature is currently not usable as the watchdog lacks Basic auth support. +``` + +A fork can be marked as "private". This prevents Robust.Cdn from giving unauthorized people access to server builds, which is desirable for forks with secret content. + +When enabled, access to server builds is restricted behind HTTP Basic authentication. Usernames and passwords for this can be configured in the fork configuration. + +TODO: Watchdog setup guide here. + +## Builds file layout + +Robust.Cdn stores and expects build zips in the `FileDiskPath` directory (`/build` when using container image). Files in this directory have a pretty simple structure of `//.zip`. For example: + +``` +/var/robust-cdn/builds +├── wizards +│ ├── 02030cfa0ed6511ec5527c5b7d1f8bcd46fe1435 +│ │ ├── SS14.Client.zip +│ │ ├── SS14.Server_linux-arm64.zip +│ │ ├── SS14.Server_linux-x64.zip +│ │ ├── SS14.Server_osx-x64.zip +│ │ └── SS14.Server_win-x64.zip +│ ├── 021d39be2876f991c5fd6e663760a921d29ac694 +│ │ ├── SS14.Client.zip +│ │ ├── SS14.Server_linux-arm64.zip +``` + +## Migration from Robust.Cdn 1.x + +If you were hosting an existing installation of Robust.Cdn from before multi-fork/server manifest support was added (1.0), this part of the guide will help you migrate. + +As part of multi-fork and manifest support, the following changes will need to be made to your setup, at the bare minimum: + +* `Cdn.UpdateToken` in configuration has been moved to fork configuration. +* `Cdn.VersionDiskPath` has effectively been changed to `Manifest.FileDiskPath`. **Note that the file layout is different, you will need to manually move the builds one folder down to be underneath the folder (see above).** + +Robust.Cdn will automatically migrate your existing CDN content database so that all version information stored within gets assigned a fork. You must set `Cdn.DefaultFork` in the configuration so it knows what fork to assign these versions to. Existing URLs (for replays etc) will keep working after this, as configuring `Cdn.DefaultFork` will make the CDN internally map the old `/version/{version}/*` URLs to the new ones under the specified fork. + +After making the above changes, newer Robust.Cdn can still be used with the old publishing workflow (using `gen_build_info.py` and all the other scripts). Just make sure to account for the change in file structure and such. Obviously we recommend moving to the new built-in publishing system as soon as possible, however. + +### Ingesting existing server manifest contents + +You can manually ingest your existing server manifest into the manifest database with the following Python script. Note that this is only really necessary if you care about being able to easily access old server versions via the HTML page or such, skipping this step + +This is very janky and you'll need to publish at least one build normally for the JSON server manifest to be re-cached and become available. But hey, it works. If the script doesn't work because you don't have `dateutil` available, on Python 3.10+ you can remove the dateutil code by replacing `dateparser.parse` with `datetime.fromisoformat`. + +```py +#!/usr/bin/env python3 + +import sqlite3 +import json +import re + +from datetime import datetime, timezone +from dateutil import parser as dateparser + +JSON_FILE = "manifest.json" +DB_FILE = "manifest.db" +FORK_NAME = "wizards" + +def main(): + data = json.loads(open(JSON_FILE, "r").read()) + db = sqlite3.connect(DB_FILE) + cur = db.cursor() + + cur.execute("SELECT Id FROM Fork WHERE Name = ?", (FORK_NAME,)) + fork_id = cur.fetchone()[0] + + for name, build in data["builds"].items(): + time = dateparser.parse(build["time"]) + time = time.astimezone(timezone.utc) + + cur.execute( + "INSERT INTO ForkVersion (Name, ForkId, PublishedTime, ClientFileName, ClientSha256, Available, EngineVersion) VALUES (?, ?, ?, ?, ?, TRUE, '')", + (name, fork_id, time, file_name(build["client"]["url"]), bytes.fromhex(build["client"]["sha256"]))) + version_id = cur.lastrowid + + for rid, server in build["server"].items(): + cur.execute( + "INSERT INTO ForkVersionServerBuild (ForkVersionId, Platform, FileName, Sha256) VALUES (?, ?, ?, ?)", + (version_id, rid, file_name(server["url"]), bytes.fromhex(server["sha256"]))) + + db.commit() + + +def file_name(url: str) -> str: + return url.split("/")[-1] + +main() ``` -### API endpoints / build metadata +## Robust.Cdn API reference + +This part of the guide will explain all of Robust.Cdn's API endpoints that you can use and interface with. + +### Authentication + +Some API endpoints may require authentication: + +* Fork control endpoints such as publish need `Authorization: Bearer `, with the `UpdateToken` specified in the fork configuration. +* Endpoints to access server files require Basic authentication if the fork is configured as private. + +### GET `/fork/{fork}` + +Gets a nice human-readable HTML page about the last builds available. -`Robust.Cdn` exposes the `/version/{version}/manifest` and `/version/{version}/download` endpoints that do the actual updating. These need to be specified in the `build.json` in your server files, which you probably want to just specify by modifying `gen_build_info.py` or something. The new metadata required is as follows: +### POST `/fork/{fork}/control/update` + +Instructs the client CDN to re-scan for new files. You can manually run this when not using Robust.Cdn's server manifest support and only using the client CDN. You will need to put the files in the correct layout as specified below. + +This require authentication. + +### GET `/fork/{fork}/manifest` + +Gets a JSON list of every server build available for a fork. + +### POST `/fork/{fork}/publish` + +Publishes a new version to the CDN. It expects a JSON body with the following information: ```json -// (example from Wizard's Den servers) -// Robust.Cdn is available at /cdn and nginx serves static builds on /builds { - "download": "https://cdn.centcomm.spacestation14.com/builds/wizards/builds/{FORK_VERSION}/SS14.Client.zip", - "hash": "49a7f54eb7e848c0a438bcfd3a198454d862e7d58d3e11c7ce60e281ddbd205d", - "version": "e769ad27256300cfbbf10d641930d43990bff309", - "fork_id": "wizards", - "engine_version": "0.12.1.0", - // These ones are new - "manifest_url": "https://cdn.centcomm.spacestation14.com/cdn/version/{FORK_VERSION}/manifest", - "manifest_download_url": "https://cdn.centcomm.spacestation14.com/cdn/version/{FORK_VERSION}/download", - "manifest_hash": "8B175E9D944F54C1444E93E19C39EB255353392B316118B660F21DF68D56DC2D" + "version": "", + "engineVersion": "", + "archive": "" } ``` -```admonish info -Note the ability to specify `{FORK_VERSION}` in the `build.json` file. This gets replaced by `version` from the JSON at runtime. This way the URL does not need to duplicate the version string, and the URL template can be changed via CVar while the entry in `build.json` stays the same. Other possible keys are `{FORK_ID}`, `{MANIFEST_HASH}` and `{ZIP_HASH}`. -``` +The version is the new version number you are publishing. This can be anything. Engine version is the version number of the engine to use. -* `manifest_url` is the `/manifest` endpoint on `Robust.Cdn`. The version must be included in the URL for `Robust.Cdn` to work. -* `manifest_download_url` is the `/download` endpoint. Same rules as above. -* `manifest_hash` must be the [manifest hash](../other-projects/launcher/delta-updates-and-manifests.md). This hash is automatically generated by `gen_build_info.py` now and should just work. +The archive is must be URL to a zip that Robust.Cdn will download containing the build zip files (client and server). -## Systemd Unit +This require authentication. -We personally use systemd to start the server: +### POST & OPTIONS `/fork/{fork}/version/{version}/download` -```ini -[Unit] -Description=Robust.Cdn +Client CDN download endpoint for a version. See [Delta Updates](../other-projects/launcher/delta-updates-and-manifests.md) for details. -[Service] -Type=notify -WorkingDirectory=/opt/robust_cdn/ -ExecStart=/opt/robust_cdn/run.sh -User=robust_cdn -Group=robust_cdn +### GET `/fork/{fork}/version/{version}/file/{file}` -[Install] -WantedBy=multi-user.target -``` +Downloads a server or client build zip from a version. File is the file name. + +### GET `/fork/{fork}/version/{version}/manifest` + +Client CDN manifest endpoint for a version. See [Delta Updates](../other-projects/launcher/delta-updates-and-manifests.md) for details. + +### POST & OPTIONS `/version/{version}/download` + +Fallback endpoint that maps to the `DefaultFork`, if configured. This is for URL compatibility with old Robust.Cdn setups. + +### GET `/version/{version}/manifest` +Fallback endpoint that maps to the `DefaultFork`, if configured. This is for URL compatibility with old Robust.Cdn setups.