Incremental docs update (#44)

* harmonize heading levels

* docs: add exec command; couple of tweaks

* docs: rough in a new README.md

* add line about QUIC and quinn

* indicate in SUMMARY.md which links are empty since that doesn't show within the GitHub repo

* README.md tweak

* add note about transaction per request

* add more commands

* add new links to SUMMARY.md toc

* add links to more commands in cli readme

README.md

@@ -1,55 +1,58 @@
 # Corrosion
+Corrosion: Gossip-based service discovery (and more) for large distributed systems, written in Rust.
 
-## Launching on Fly
+## Why we built Corrosion
 
-```
-fly launch --no-deploy --org <org> --region <region> --copy-config --name <app-name>
-fly volumes create corro_data -y -s 10 -r <region> -a <app-name>
-fly deploy --dockerfile examples/fly/Dockerfile -a <app-name> -c examples/fly/fly.toml
-fly machine clone -a <app-name> --region <other region> --select
-fly machine clone -a <app-name> --region <yet another region> --select
-```
-
-## Reading data
+We built Corrosion specifically for service discovery across a large global network, replacing Consul’s central state database with eventually consistent state distributed across our hosts.
 
-The readable API surface for Corrosion-produced data is to use SQLite and read directly from the database.
+Our new tool needed to deliver the following:
 
-Adding indexes either through the API or schema files is the way to go to improve read performance on application-specific queries.
+### Fast reads and writes
 
-## Writing data
+Getting state (data) from a central remote source can be incredibly expensive (at least 300ms for a round-trip to something on the other side of the world), but usually takes less than 1ms from a local source.
 
-The only way to write data that propagates through the Corrosion cluster is to go through its HTTP API.
+### Fast consistency
 
-Corrosion's HTTP API is rqlite-compatible. This is both meant as a shortcut and to have ready-made clients for it.
+Strong consistency by RAFT consensus can be too slow. Eventual consistency works, if it's fast.
 
-### POST /v1/transactions
+### Flexibility
 
-See: https://rqlite.io/docs/api/api/#writing-data
+Global state for a distributed system isn't one-size-fits-all. Flexible schemas and queries are essential.
 
-Corrosion-specific limitations:
-- All requests are treated as if the `transaction` query param had been passed (everything runs in a transaction)
-- There's a limit to how many statements can be input at once to reduce changeset sizes (this shouldn't be the case for long)
+## How Corrosion works
 
-## Updating / migrating / changing the schema
+In a nutshell, Corrosion:
 
-Corrosion migrates schema changes automatically without the need for manual migration files. Each schema SQL file is expected to contain SQLite-compatible statements that create resources (e.g. `CREATE TABLE` and `CREATE INDEX`). `ALTER` or `DROP` statements are not supported. Destructive actions are ignored.
+- Hosts a SQLite database on each node, for fast local reads and writes
+- Uses [CR-SQLite](https://github.com/vlcn-io/cr-sqlite) for conflict resolution with CRDTs
+- Uses [Foca](https://github.com/caio/foca) to manage cluster membership using a SWIM protocol
+- Gossips changes from the local database throughout the cluster
+- Periodically synchronizes with a subset of other cluster nodes, to ensure consistency
 
-### File-based schemas
+## Features
 
-Configuring Corrosion with `schema_paths` lets you define filesystem directories where `.sql` files can be found. These files should, as a whole, represent the schema for your data.
+- A flexible API to read from and write to Corrosion's store using SQL statements
+- File-based schemas with on-the-fly updates
+- HTTP streaming subscriptions based on SQL queries
+- Live population of configuration files from Corrosion state with user-defined [Rhai](https://rhai.rs/) templates
+- Storage and propagation of state from locally registered Consul services, replacing the central database with Corrosion's distributed state
+- Vanilla SQLite storage for host-local data
+- Secure peer-to-peer communication with the [QUIC](https://datatracker.ietf.org/doc/html/rfc9000) transport protocol (using [Quinn](https://github.com/quinn-rs/quinn))
 
-You can add or modify schema files in known paths or even modify the `schema_paths` and then send a SIGHUP signal to Corrosion to apply schema changes, without a restart.
+## Usage overview
 
-### POST /v1/migrations
+Run the Corrosion agent on every node/host in the cluster. Other programs running on the node use [Corrosion's HTTP API](/doc/api/README.md) to query the local Corrosion SQLite database, add and update data, and subscribe to change notifications.
 
-This endpoint accepts the same type of body as the `/v1/transactions` endpoint, except it will mutate Corrosion's schema.
+The [Corrosion CLI](/doc/cli/README.md) provides commands for administration and access to database and features.
 
-## Subscribing to changes
+See the WIP [Corrosion documentation](/doc/SUMMARY.md) for more details.
 
-One of Corrosion's most powerful features is the ability to subscribe to changes as they are applied to the local node.
+## Building Corrosion
 
-This works via websockets for maximum compatibility.
+Clone [https://github.com/superfly/corrosion2.git](https://github.com/superfly/corrosion2.git).
 
-## Migrating data
+From within the repo directory:
 
-(WIP)
+```
+cargo build --release && mv target/release/corrosion ./
+```

doc/SUMMARY.md

@@ -3,39 +3,39 @@
 [Introduction](intro.md)
 
 # User guide
-- [Authorization]()
-- [Backup / Restore]()
-- [Consistency]()
+- [Authorization]() (to come)
+- [Backup / Restore]() (to come)
+- [Consistency]() (to come)
 - [CRDTs](crdts.md)
-- [Encryption]()
-- [Gossip]()
+- [Encryption]() (to come)
+- [Gossip]() (to come)
 - [Schema](schema.md)
-- [Subscriptions]()
-- [Synchronization]()
+- [Subscriptions]() (to come)
+- [Synchronization]() (to come)
 - [Telemetry](telemetry/README.md)
     - [Prometheus](telemetry/prometheus.md)
-- [Templates]()
+- [Templates]() (to come)
 
 # Reference
 - [API](api/README.md)
     - [POST /v1/transactions](api/transactions.md)
     - [POST /v1/queries](api/queries.md)
     - [POST /v1/subscriptions](api/subscriptions.md)
 - [Command-line Interface](cli/README.md)
-    - [agent]()
+    - [agent](cli/agent.md)
     - [backup](cli/backup.md)
-    - [consul]()
-    - [exec]()
-    - [query]()
-    - [reload]()
+    - [consul]() (to come)
+    - [exec](cli/exec.md)
+    - [query](cli/query.md)
+    - [reload](cli/reload.md)
     - [restore](cli/restore.md)
-    - [sync]()
-    - [template]()
+    - [sync]() (to come)
+    - [template](cli/template.md)
     - [tls](cli/tls.md)
 - [Configuration](config/README.md)
     - [db](config/db.md)
     - [gossip](config/gossip.md)
-    - [api]()
-    - [admin]()
-    - [telemetry]()
-    - [consul]()
+    - [api]() (to come)
+    - [admin]() (to come)
+    - [telemetry]() (to come)
+    - [consul]() (to come)

doc/api/README.md

@@ -1,7 +1,11 @@
-## API
+# API
 
 Each running Corrosion agent hosts a RESTful HTTP API for interacting with the cluster's synchronized database. Endpoints accept SQL statements in a JSON body, for versatility.
 
+Each request is run in a transaction (as if the  `transaction` query param had been passed).
+
+Endpoints:
+
 - [POST /v1/transactions](transactions.md) for writes
 - [POST /v1/queries](queries.md) for reads
 - [POST /v1/subscriptions](subscriptions.md) to receive streaming updates for a desired query

doc/api/queries.md

@@ -1,15 +1,15 @@
-## POST /v1/queries
+# POST /v1/queries
 
 Read from the Corrosion database. The `/v1/queries` endpoint accepts a single SQL statement in JSON format.
 
-### Sample request
+## Sample request
 ```
 curl http://localhost:8080/v1/queries \ 
  -H "content-type: application/json" \
  -d "\"SELECT sandwich FROM sandwiches\""
 ```
 
-### Sample response
+## Sample response
 ```json
 {"columns":["sandwich"]}
 {"row":[1,["burger"]]}

doc/api/subscriptions.md

@@ -3,14 +3,14 @@
 Start receiving updates for a desired SQL query. The `/v1/subscriptions` endpoint accepts a single SQL statement in JSON format.
 The Corrosion agent responds with an HTML stream that notifies of any changes to the response to this query.
 
-### Sample request
+## Sample request
 ```
 curl http://localhost:8080/v1/subscriptions \
  -H "content-type: application/json" \
- -d "\"SELECT sandwich FROM sw\""
+ -d "\"SELECT sandwich FROM sandwiches\""
 ```
 
-### Sample response
+## Sample response
 
 ```json
 {"columns":["sandwich"]}

doc/api/transactions.md

@@ -1,15 +1,15 @@
-## POST /v1/transactions
+# POST /v1/transactions
 
 Write changes to the Corrosion database for propagation through the cluster. The `/v1/transactions` endpoint accepts a JSON list of SQL statements.
 
-### Sample request
+## Sample request
 ```
 curl http://localhost:8080/v1/transactions \
  -H "content-type: application/json" \
  -d "[\"INSERT OR IGNORE INTO sandwiches (pk, sandwich) VALUES (3, 'brie and cranberry')\"]"
 ```
 
-### Sample response
+## Sample response
 ```json
 {"results":[{"rows_affected":1,"time":0.000027208}],"time":0.000300708}% 
 ```

doc/cli/README.md

@@ -1,5 +1,14 @@
 # Command-line Interface
 
-See the chapters for each subcommand:
+Corrosion has a CLI for managing the local Corrosion agent and database. It also provides commands to read from and write to the cluster's database. 
+
+The base command is `corrosion`. Run `corrosion --help` for a list of subcommands.
+
+See the pages for each subcommand:
+- [`corrosion agent`](agent.md)
 - [`corrosion backup`](backup.md)
 - [`corrosion restore`](restore.md)
+- [`corrosion exec`](exec.md)
+- [`corrosion query`](query.md)
+- [`corrosion template`](template.md)
+- [`corrosion reload`](reload.md)

doc/cli/agent.md

@@ -0,0 +1,18 @@
+# The `corrosion agent` command
+
+Starts Corrosion on the host. The `--api-addr`, `--db-path`, and `--admin-path` options override corresponding settings in the configuration file.
+
+```cmd
+$ corrosion agent --help
+Launches the agent
+
+Usage: corrosion agent [OPTIONS]
+
+Options:
+  -c, --config <CONFIG_PATH>     Set the config file path [default: /etc/corrosion/config.toml]
+      --api-addr <API_ADDR>      
+      --db-path <DB_PATH>        
+      --admin-path <ADMIN_PATH>  
+  -h, --help                     Print help
+```
+

doc/cli/exec.md

@@ -0,0 +1,25 @@
+# The `corrosion exec` command
+
+Writes to Corrosion's database, via the [`/v1/transactions/`](../api/transactions.md) endpoint hosted by the local Corrosion agent. 
+
+Use `corrosion exec` to mutate data within existing CR-SQLite-enabled Corrosion database tables, for propagation throughout the cluster.
+
+Corrosion does not sync schema changes made using this command. Use Corrosion's [schema files](/schema.md) to create and update the cluster's database schema.
+
+```
+$ corrosion exec --help
+Execute a SQL statement that mutates the state of Corrosion
+
+Usage: corrosion exec [OPTIONS] <QUERY>
+
+Arguments:
+  <QUERY>  
+
+Options:
+  -c, --config <CONFIG_PATH>     Set the config file path [default: /etc/corrosion/config.toml]
+      --timer                    
+      --api-addr <API_ADDR>      
+      --db-path <DB_PATH>        
+      --admin-path <ADMIN_PATH>  
+  -h, --help                     Print help
+```

doc/cli/query.md

@@ -0,0 +1,25 @@
+# The `corrosion query` command
+
+Reads from Corrosion's database, via the [`/v1/queries/`](../api/queries.md) endpoint hosted by the local Corrosion agent. 
+
+Use the `--columns` option to see column headings in the output.
+
+```
+$ corrosion query --help
+Query data from Corrosion w/ a SQL statement
+
+Usage: corrosion query [OPTIONS] <QUERY>
+
+Arguments:
+  <QUERY>  
+
+Options:
+  -c, --config <CONFIG_PATH>     Set the config file path [default: /etc/corrosion/config.toml]
+      --columns                  
+      --api-addr <API_ADDR>      
+      --timer                    
+      --db-path <DB_PATH>        
+      --param <PARAM>            
+      --admin-path <ADMIN_PATH>  
+  -h, --help                     Print help
+```

doc/cli/reload.md

@@ -0,0 +1,18 @@
+# The `corrosion reload` command
+
+Reloads Corrosion configuration from a file.
+
+```
+$ corrosion reload --help                             
+Reload the config
+
+Usage: corrosion reload [OPTIONS]
+
+Options:
+  -c, --config <CONFIG_PATH>     Set the config file path [default: /etc/corrosion/config.toml]
+      --api-addr <API_ADDR>      
+      --db-path <DB_PATH>        
+      --admin-path <ADMIN_PATH>  
+  -h, --help   
+```
+

doc/cli/template.md

@@ -0,0 +1,21 @@
+# The `corrosion template` command
+
+Uses Corrosion's template engine to generate and update a local file based on a [Rhai](https://rhai.rs/) script and the latest data from Corrosion.
+
+Specify the name of the `.rhai` file and the desired name for the output file. 
+
+```
+$ corrosion template --help
+Usage: corrosion template [OPTIONS] [TEMPLATE]...
+
+Arguments:
+  [TEMPLATE]...  
+
+Options:
+  -c, --config <CONFIG_PATH>     Set the config file path [default: /etc/corrosion/config.toml]
+  -o, --once                     
+      --api-addr <API_ADDR>      
+      --db-path <DB_PATH>        
+      --admin-path <ADMIN_PATH>  
+  -h, --help                     Print help
+```