Merge pull request #67 from RTradeLtd/docupdate

Documentation Updates

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 Tracks changes made in between releases
 
+# v3.2.3
+
+* gRPC based DAGService [#69](https://github.com/RTradeLtd/TxPB/pull/69)
+* Replication CLI updates [#58](https://github.com/RTradeLtd/TxPB/pull/58)
+* Documentation updates [#67](https://github.com/RTradeLtd/TxPB/pull/67)
+* Update Blockstore RPC Calls [#64](https://github.com/RTradeLtd/TxPB/pull/64)
+
 # v3.2.2
 
 * Enable retrieving IPLD object stats [#56](https://github.com/RTradeLtd/TxPB/pull/56)

doc/BENCHMARKS.md

@@ -1,5 +1,26 @@
 # Benchmarks
 
-Benchmarks for TemporalX is a constantly evolving process. With our initial release we did a comparison between processing the same workloads in the same environment on go-ipfs and with TemporalX. To read about that check out our [medium post](https://medium.com/temporal-cloud/temporalx-vs-go-ipfs-official-node-benchmarks-8457037a77cf).
+Our primary source of benchmarks for TemporalX comes from our CI builds, where with every new release, we run the same set of benchmarks. The idea behind this is that we can see performance changes in real time, and catch performance regressions as they happen, not months or weeks later. All of our CI benchmarks come from the **amazing** [gobenchdata](https://github.com/marketplace/actions/continuous-benchmarking-for-go). While our setup is a little convoluted due to a closed source repository, consisting of running the CI benchmarks, and manually uploading them to GitHub Pages, gobenchdata is an all around extremely solid benchmark recording tool. This limitation of a convoluted process is only because github actions don't work very well with private repositories.
 
-From then on we have been maintaining a set of generalized benchmarks through a public github pages site on the TemporalX repository located [here](https://rtradeltd.github.io/TemporalX/) which uses [gobenchdata](https://github.com/marketplace/actions/continuous-benchmarking-for-go) to capture new benchmarks with each release.
+# CI Benchmarks
+
+The following are benchmark samples that we capture during our CI processes to ensure consistent performance overtime, and catch performance regressions as they happen:
+
+* [Reference Counter Benchmarks](https://rtradeltd.github.io/counter/)
+  * These run benchmarks against the two types of reference counters we have
+  * Measure GC, Delete, Put times
+* [TemporalX Benchmarks](https://benchx.temporal.cloud/)
+  * These runs TemporalX's benchmarks 
+  * Measures a variety of the API calls
+
+# Published Reports
+
+Every once in awhile we will publish reports/analysis of the performance you get with TemporalX:
+
+* [TemporalX vs Go-IPFS](https://medium.com/temporal-cloud/temporalx-vs-go-ipfs-official-node-benchmarks-8457037a77cf)
+  * Initially captured when we first released TemporalX, we've gotten even faster since then. Once go-ipfs v0.5.0 is released we will do a follow-up to this benchmark
+  * You can expect 7x-10x performance gains when fully leveraging gRPC benefits
+  * You can expect 3x performance gains when leveraging gRPC like HTTP (connect to server, send request, disconnect)
+* [TemporalX Replication vs IPFS Cluster](https://medium.com/temporal-cloud/nodes-w-built-in-replication-high-performance-security-consensus-free-6657ac9e44ea)
+  * TemporalX replicated the same dataset in 128 seconds
+  * IPFS Cluster replicated the same dataset in 1020 seconds

doc/CONFIGURATION.md

@@ -378,6 +378,8 @@ counterStoreNamespace | counterstorespace  | the key namespace for the counter s
 counterStorePath      | counterstorage     | the path for storing reference counter metadata |
 counterMaxWorkers     | 10                 | the maximum number of concurrent reference counter operations (default 1) |
 
+If you are using the referene counter, you will want to make sure you don't enable blockstore caching, otherwise you will not get totally accurate reference count information, as the cached blockstore will intercept the blockstore call before our reference counter intercepts the call. By "not totally accurate" we mean that you will lmost likely get a maximum reference count of 1. For more deatils on this please consult the reference counter information page.
+
 ## Peerstore
 
 The `peerstore` section is used to define configuration options for the libp2p peerstore, which is where records of all libp2p peers we encounter are stored. We currently have two supported types:

doc/DHT.md

@@ -0,0 +1,21 @@
+# DHT
+
+TemporalX uses a forked version of `go-libp2p-kad-dht` (we call `libp2p-kad-dht`), as well as `go-libp2p-kbucket` (we call `libp2p-kbucket`). The main reason for forking these libraries is so we can have more control over what changes we bring in, and when we have to deal with breaking changes. Additionally it allows us to perform some optimizations, and codebase cleanups without having to worry about it being accepted into the upstream codebase before we use it.
+
+Recently (v0.7.0) the `go-libp2p-kad-dht` library went through some major refactors, however it's not clear how thoroughly tested these changes are, outside of in-repo unit testing, and basic testing within testground (the go-ipfs testing library). Additionally none of these changes will be thoroughly tested within go-ipfs for quite some. As such this has lead to us needing to define a consistent versioning scheme for our fork.
+
+| Repo Name | Our Version | Upstream Max Version | Notes |
+|-----------|-------------|----------------------|--------|
+| libp2p-kad-dht | v1     | v0.5.2 | current version used by TemporalX |
+| libp2p-kad-dht | v2     | v0.6.1 | next version to be used by TemporalX |
+| libp2p-kad-dht | v3     | v0.7.0 | not clear if this will be used by TemporalX |
+
+# Notes
+
+v0.7.0 of upstream represents some significant changes to the way the DHT library works. It introduced the concept of "Dual DHT" a WAN/Public DHT, and a LAN/Private DHT. Personally speaking I'm not sure this is the "right way" to fix things with the libp2p DHT. Additionally it is a pretty major change to roll out, which from what I can tell, hasn't been thoroughly tested, nor benchmarked at scale. At the time this documentation is being written, it's unclear whether or not we will move to this system. We will be moving to our v2 of `libp2p-kad-dht` eventually, as this brings us closer to a more realistic kademlia style DHT.
+
+# Notable Changes
+
+* The biggest change by far is the removal of `jbenet/goprocess` instead using `context.Context` + `context.CancelFunc` directly.
+* Removal of `ipfs/go-log`
+* Switching to `prometheus` for metric tracking

doc/GRPC_BLOCKSTORE.md

@@ -0,0 +1,5 @@
+# gRPC Based Blockstore
+
+The [`Blockstore`](https://github.com/ipfs/go-ipfs-blockstore/blob/master/blockstore.go#L35) interface is a thin wrapper around the `Datastore` interface that abstracts the bridge between IPFS CID's, and the underlying datastores which provide the key-value storage for on-disk storing of IPFS data. As with the `DAGService` interface, the `Blockstore` is widely used across any projects that want to use IPFS, and is perhaps more widely used of an interface than the `DAGService` interface is.
+
+By using the [`RemoteBlockstore`](https://github.com/RTradeLtd/go-ipfs-blockstore/pull/7) module, you can swap out internals of a variety of projects in the IPFS ecosystem to not have to run a blockstore locally, and instead deffer all blockstore functionality to a remote TemporalX node

doc/GRPC_DAG_SERVICE.md

@@ -0,0 +1,9 @@
+# gRPC Based DAGService
+
+One of the core interfaces for using IPFS is the [`DAGService`](https://github.com/ipfs/go-ipld-format/blob/master/merkledag.go#L54) interface, that specifies how you can get, and add IPLD nodes both to the network, and your local blockstore.
+
+[dag_service.go](../go/dag_service.go) showcases a working example of you can use can use the `NodeAPIClient` generated gRPC client as a way of satisfying `ipld.DAGService` using TemporalX. As such this will allow you to swap out existing `DAGService` implementations used by clients such as go-ipfs, or ipfs-lite for one that relies on a remote TemporalX server.
+
+By using this module you will not be required to run a DAGService locally, and can instead delegate all processing to a remote TemporalX server via the `Dag` RPC call. This is particular interesting for use with things like `go-ds-crdt` as well as running TemporalX in resource constrained environments while being able to fully leverage the resources of a more powerful, remote TemporalX service.
+
+For an example of how this is used, please consult [s3x](https://github.com/RTradeLtd/s3x/pull/40)

doc/MEMORY_CONSTRAINED_CONFIGURATIONS.md

@@ -0,0 +1,45 @@
+# Memory Constrained Environments
+
+In certain situations you may be running TemporalX in a memory constrained environments. When using default configurations, TemporalX is already supremely memory efficient, however there are certain situations where you may want to scale up this memory efficiency, say on a raspberry pi. Although efficient on memory, the configurations in this guide will provide a somewhat noticeable impact to performance, and as such is only recommended in memory constrained environments.
+
+# What Counts As "Memory Constrained"
+
+For the purpose of this documentation, we define memory constrained as embedded devices, ultralite laptops, and SoC boards such as the Raspbery Pi. For a guideline of what configurations you should apply based on system specifications consult the table below.
+
+| Available RAM | Recommended Configurations |
+|---------------|----------------------------|
+| 1GB           | All configurations |
+| 4GB           | Persistent DHT, Datastore Peerstore, Main Datastore |               
+| 4GB+          | Above 4GB you probably dont need to follow these recommendations |
+
+# Configuration Recommendations
+
+## Persistent DHT
+
+* By enabling persistent DHT mode, you no longer store the entire DHT subsystem in-memory, and instead store it on disk. Through our profiling and benchmarking we observed this as being a significant memory hog, and were able to vastly reduce memory consumption by running persistent DHT
+* This will provide a noticeable slow-down to DHT queries, as you will need to fetch them from disk as opposed to in-memory
+
+## Datastore Peerstore
+
+* By enabling datastore peerstore, you no longer have to store tens of thousands of peerIDs, records, and associated information in memory
+* This will provide a noticeable slow-down to peerstore queries as you will need to fetch them from disk as opposed to in-memory
+
+## Queueless Reference Counter
+
+* The queue based reference requires multiple different components that increase memory consumption
+* If you insist on using the queue based reference counter, don't use more than 1 worker
+
+## Filesystem Keystore
+
+* Use the filesystem keystore which stores keys directly on disk, as opposed to the krab keystore which uses badger, and encrypts your keys, or the the in-memory keystore
+
+## Main datastore
+
+* For the main datastore it is recommended that you use leveldb, as this is the most memory efficient datastore available
+* Alternatively you can use the badger datastore with `fileLoadingMode` set to `0`
+
+## Low Power Mode
+
+* Lower power mode is primarily used to set sensible defualts for LibP2P settings if there are none set in the configuration file
+* If you do have LibP2P settings in the configuration file, then low power mode will be ignored
+* In the future we will be phasing out usage of low power mode in preference of using the config file entirely

doc/MULTI_DATASTORE.md

@@ -0,0 +1,15 @@
+# Multi Datastore
+
+TemporalX has a novel concept of "multi-datastore" compared to the reference implementations of IPFS, which use one mega blockstore separate by key namespaces. Although in certain places we use key namespaces, the act of namespaced keys isn't free in cost. There is a slight overhead, which under most hobbyist circumstances is not that great, however at production scale workloads, overheads add up. To mitigate some of these overheads, we leverage multiple datastores for different purposes. With the exception of persistent DHT mode that namespaces around the main storage datastore, all other stores go into separate datastores.
+
+When enabled, the following subsystems use dedicated datastores:
+
+* Keystore
+* Peerstore
+* Reference counter store
+
+When enabled, the following subsystems share the same datastore using namespaced keys:
+
+* DHT (note: by default dht is stored in memory)
+* Main storage store
+* Provider queue

doc/README.md

@@ -2,15 +2,33 @@
 
 This folder contains all of our documentation including protocol buffer docs, configuration documentation, and a short getting started "tutorial". All of our protocol buffer documentation is generated using [protoc-gen-doc](https://github.com/pseudomuto/protoc-gen-doc).
 
-# Contents
+# API
+
+* [Protocol Buffer Documentation](PROTO.md)
+* [Tracing](TRACING.md)
+
+# Configuration
 
-* [Benchmarks](BENCHMARKS.md)
 * [Configuration File Reference](CONFIGURATION.md)
-* [IPFS Gateway Overview](GATEWAY.md)
-* [Getting Started](GETTING_STARTED.md)
+* [Memory Constrained Configurations](MEMORY_CONSTRAINED_CONFIGURATIONS.md)
+
+# Miscellaneous
+
+* [Benchmarks](BENCHMARKS.md)
 * [Licensing System Overview](LICENSE_OVERVIEW.md)
-* [Protocol Buffer Documentation](PROTO.md)
-* [Reference Counter Overview](REFERENCE_COUNTER.md)
-* [Replication CLI Overview](REPLICATION_CLI.md)
+
+
+# Subsystems
+
+* [DHT](DHT.md)
+* [Multi-Datastore](MULTI_DATASTORE.md)
+* [Reference Counter](REFERENCE_COUNTER.md)
 * [Replication Algorithm Overview](REPLICATION.md)
-* [Tracing Overview](TRACING.md)
+* [IPFS Gateway](GATEWAY.md)
+* [gRPC Based DAGService](GRPC_DAG_SERVICE.md)
+* [gRPC Based Blockstore](GRPC_BLOCKSTORE.md)
+
+# Tutorials
+
+* [Getting Started](GETTING_STARTED.md)
+* [Replication CLI Overview](REPLICATION_CLI.md)

doc/REFERENCE_COUNTER.md

@@ -6,4 +6,28 @@ The reference counter is implemented as a reference counter blockstore, which sa
 
 One of the really awesome features about our reference counter unlike the pinning system used by `go-ipfs` is that it is *non-blocking* unless you are doing a garbage collection run! If you want to see just how much faster our non-blocking reference counter is, check out our [benchmark blog post](https://medium.com/temporal-cloud/temporalx-vs-go-ipfs-official-node-benchmarks-8457037a77cf).
 
-In order to permanently remove blocks that have a reference count of 0, you must trigger a garbage collection using TemporalX's admin api. The admin api is only enabled when using the reference counted blockstore, and is exposed on `localhost:9999` by default.  When you trigger a garbage collection, all blockstore interaction calls are blocked, while we wait for pending reference count operations to complete. Once all pending reference count operations have finished, we look for any unferenced blocks. These blocks are then subsequently permanently removed. After removing all unreferenced blocks, we resume normal operation and unblock all blockstore interaction calls.
+In order to permanently remove blocks that have a reference count of 0, you must trigger a garbage collection using TemporalX's admin api. The admin api is only enabled when using the reference counted blockstore, and is exposed on `localhost:9999` by default.  When you trigger a garbage collection, all blockstore interaction calls are blocked, while we wait for pending reference count operations to complete. Once all pending reference count operations have finished, we look for any unferenced blocks. These blocks are then subsequently permanently removed. After removing all unreferenced blocks, we resume normal operation and unblock all blockstore interaction calls.
+
+# Why You Shouldn't Use A Cached Blockstore
+
+When using the reference counter, if you want truly accurate results, it is advisable that you do not enable blockstore caching which requires explicit configuration to be enabled. Because the cached blockstore sits ontop of the reference counter blockstore, it will intercept the calls before our reference counter does. This means that more often than not, the calls won't bubble down through to our reference counted blockstore if the caching layer has the block. The effecct of this is that unless you get a cache miss, the maximum reference count will only ever be 1. 
+
+Realistically a cached blockstore won't really reduce a lot of overhead, because the overhead of TemporalX is already so low, that you will get marginal performance gains. If you really want caching, it is advisable that you use datastore level caching through something like our badger datastore, which uses version 2 of the badger codebase, which has extensive caching features. This will ensure that you get the benefits of caching, while getting totally accurate reference count information.
+
+# How Much Blocking Is There
+
+There's only a few situations in which our blockstore wrapper is blocking across all functionality, and we try to isolate what gets blocked as much as possible.
+
+
+| Action | Mutex Lock Type | What Gets Blocked |
+|--------|-----------------|-------------------|
+| Put, Get, GetSize, PutMany, DeleteBlock | Blockstore read lock | Nothing |
+| Reference count operations | Counter write lock | Other reference count operations |
+| Garbage Collection | Blockstore write lock + Counter write lock | Everything |
+| Blockstore close | Blockstore write lock + Counter write lock | Everything
+
+Essentially the only time there is  total blockstore lock preventing any calls is during garbage collection and shutdown. The act of reference counting is non-blocking to the blockstore in general, but it is blocking to other reference counting operations. Because of our design, we're able to enable a consistent impact to performance due to reference counting operations, and during our benchmarks there was about a 2% -> 5% performance impact regardless of how much data is stored in the blockstore itself. This is a pretty negligible impact considering that the pinning system used by go-ipfs gets exponentially worse in performance the more data you store.
+
+The biggest performance sink hole in this is the garbage collection process, not the reference counting process. In and of itself garbage collection is pretty quick with our reference counter as we just need to read key->value pairs without any graph traversal, however  depending on how much data you are garbage collecting at any one momment, the biggest impact on the performance will be your disk speeds.
+
+Thankfully due to our multi-datastore configuration capabilities, if you are in an environment where performance is of extreme importance, you can place this metadata store on an NVMe SSD, or regular SSD drive and this will mitigate a large portion of the garbage collection overhead.