Merge pull request #46 from RTradeLtd/generate

Generate Missing Client Stubs & Update Documentation

.script/protoc-java.sh

@@ -3,7 +3,7 @@
 # This script installs protoc-gen-grpc-java for the given platform (defaults to linux-x86_64).
 # See: https://search.maven.org/search?q=g:io.grpc%20a:protoc-gen-grpc-java
 
-VERSION=1.19.0
+VERSION=1.27.2
 PLATFORM=$1
 if [ -z "$1" ] ; then
   PLATFORM=linux-x86_64

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 Tracks changes made in between releases
 
+# v3.1.0
+
+* Migrate replication protos out of `node` into `TxPB` [#45](https://github.com/RTradeLtd/TxPB/pull/45)
+* Change replication documentation to markdown format [#40](https://github.com/RTradeLtd/TxPB/pull/40)
+* Documetnation updates [#39](https://github.com/RTradeLtd/TxPB/pull/39)
+
 # v3.0.0
 
 * API refactor [#36](https://github.com/RTradeLtd/TxPB/pull/36)

Makefile

@@ -5,7 +5,7 @@ proto: proto-gen tidy
 
 # -I are the import paths, because we're using some plugins, we need to import the gogo protobuf helpers
 .PHONY: proto-gen
-proto-gen: gen-file gen-util gen-status gen-pubsub gen-admin gen-namesys 1gen-node gen-replication gen-docs
+proto-gen: gen-file gen-util gen-status gen-pubsub gen-admin gen-namesys gen-node gen-replication gen-docs
 
 
 # builds the static website
@@ -59,7 +59,6 @@ install:
 	npm install -g grpc-tools
 	python3 -m pip install grpcio-tools
 	bash .script/protoc-js.sh
-	bash .script/protoc-java.sh
 
 # protocol buffer generation targets
 

README.md

@@ -24,6 +24,20 @@ If you want bindings in other languages, feel free to open a github issue. Alter
 * [TemporalX HTTP Gateway](./doc/GATEWAY.md)
 * [Benchmarks](./doc/BENCHMARKS.md)
 
+# Code Generation
+
+Before generating the code, you'll need to download quite a few generators. Before downloading the generators you'll need to have a valid installation of the following, please consult appropriate installation manuals as required:
+
+* Go 1.13+
+* NodeJS
+* NPM
+* Python3
+* Make
+
+Once you have all of this installed, you can then install the needed generators which will allow you to generate the gRPC client stubs. To do this run `make install`. This has to download quite a few packages so it might take some time to complete. The only thing this doesn't install is a java generator, to do that visit https://search.maven.org/search?q=g:io.grpc%20a:protoc-gen-grpc-java and download the one for your platform.
+
+After you've downloaded all generator tooling, you can generate the client stubs, and documentation with a single `make`.
+
 # License
 
 All code in this repository is licensed under `AGPL-3.0`, and all documentation is licensed under `MIT`. Documentation is anything that is a readme, or html file. Code is pretty much anything other than documentation such as javascript, java, golang, python, and `.proto` files.

doc/PROTO.md

@@ -91,6 +91,20 @@
     - [PubSubAPI](#pb.PubSubAPI)
 
 
+- [replication.proto](#replication.proto)
+    - [AddrInfo](#pb.AddrInfo)
+    - [Replication](#pb.Replication)
+    - [ReplicationStatus](#pb.ReplicationStatus)
+    - [ServerSource](#pb.ServerSource)
+    - [SignedSubscription](#pb.SignedSubscription)
+    - [Subscription](#pb.Subscription)
+    - [SubscriptionUpdate](#pb.SubscriptionUpdate)
+
+
+
+    - [replicator](#pb.replicator)
+
+
 - [status.proto](#status.proto)
     - [StatusResponse](#pb.StatusResponse)
     - [VersionResponse](#pb.VersionResponse)
@@ -460,7 +474,7 @@ NameSysAPI provides a generic name resolution API
 | Field | Type | Label | Description |
 | ----- | ---- | ----- | ----------- |
 | cid | [string](#string) |  | cid is the identifier of the block |
-| data | [bytes](#bytes) |  | data is the actual contnets of this block |
+| data | [bytes](#bytes) |  | data is the actual contents of the block |
 
 
 
@@ -1111,6 +1125,156 @@ PubSubAPI provides a libp2p pubsub API and is equivalent to go-ipfs
 
 
 
+<a name="replication.proto"></a>
+<p align="right"><a href="#top">Top</a></p>
+
+## replication.proto
+
+
+
+<a name="pb.AddrInfo"></a>
+
+### AddrInfo
+AddrInfo can be used to ID and locate a server (see also libp2p/go-libp2p-core/peer#AddrInfo)
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| id_bytes | [bytes](#bytes) |  | id_bytes is a libp2p peer identity. It is used to verity the Peer&#39;s public key. Please use helper functions GetID and SetID for this field |
+| addrs_bytes | [bytes](#bytes) | repeated | addrBytes are bytes of Multiaddr for locating this peer. Please use helper functions GetMultiAddrs, and SetMultiAddrs for this field. |
+| grpc_port | [int32](#int32) |  | grpc port for replication protocol. will be optional once grpc is integrated into multistream. |
+
+
+
+
+
+
+<a name="pb.Replication"></a>
+
+### Replication
+Replication message is a signable data structure to represent a replication scheme
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| header | [string](#string) |  | Header must be &#34;rtrade-replication v0...&#34; for development stage, this is a required security header. - Avoid collision from other signed data. - Allow future versions to require different validation rules. Any signer must understand every header tag to sign documents. |
+| cids_bytes | [bytes](#bytes) | repeated | CIDs is the list of contents to replicate. Please use helper functions GetCIDs and AddCIDs for this field |
+| servers | [ServerSource](#pb.ServerSource) | repeated | Servers list the candidate severs to replicate to, in the order of preference. |
+| replication_factor | [int32](#int32) |  | replication_factor is the number of replications desired. |
+| refresh_interval_seconds | [int32](#int32) |  | refresh_interval_seconds is the *suggested* number of seconds to wait before checking if a remote server is up. The first check should be random from 0 to refresh_interval_seconds. Each replicator can have it&#39;s own max_interval. A sensible default value should be used if it is 0. |
+| server_down_delay_seconds | [int32](#int32) |  | server_down_delay_seconds is the number of seconds to wait after a server is down before the next reserved server is requested to be active. A sensible default value should be used if it is 0. |
+
+
+
+
+
+
+<a name="pb.ReplicationStatus"></a>
+
+### ReplicationStatus
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| ok | [bool](#bool) |  | ok report success for action submitted |
+| is_active | [bool](#bool) |  | is_active report if the replication is currently active on this server |
+| current_version | [int64](#int64) |  | current_version is the highest version this replicator/server has locally. |
+| target_version | [int64](#int64) |  | target_version is the highest version this replicator/server knows about and can verify to exist. |
+
+
+
+
+
+
+<a name="pb.ServerSource"></a>
+
+### ServerSource
+ServerSource is a list of one or more servers.
+TODO: add field to allow referring to a list of servers from another file.
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| addr_info | [AddrInfo](#pb.AddrInfo) |  |  |
+
+
+
+
+
+
+<a name="pb.SignedSubscription"></a>
+
+### SignedSubscription
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| sub_part | [Subscription](#pb.Subscription) |  |  |
+| update_part | [SubscriptionUpdate](#pb.SubscriptionUpdate) |  |  |
+
+
+
+
+
+
+<a name="pb.Subscription"></a>
+
+### Subscription
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| topic | [string](#string) |  |  |
+| author_id_bytes | [bytes](#bytes) |  |  |
+| remove | [bool](#bool) |  | if true, remove this Subscription. For replicator.Add, deactive this replication. For replicator.WaitForUpdates, stop reporting updates. |
+
+
+
+
+
+
+<a name="pb.SubscriptionUpdate"></a>
+
+### SubscriptionUpdate
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| version | [int64](#int64) |  | strictly increasing version number |
+| replication_bytes | [bytes](#bytes) |  | replication file is in bytes for signing |
+| signature | [bytes](#bytes) |  | signature signs the Subscription and above data in length delimited form in the order topic|author|version|replication |
+
+
+
+
+
+
+
+
+
+
+
+
+<a name="pb.replicator"></a>
+
+### replicator
+The replicator provides replication services.
+
+| Method Name | Request Type | Response Type | Description |
+| ----------- | ------------ | ------------- | ------------|
+| Add | [Subscription](#pb.Subscription) | [ReplicationStatus](#pb.ReplicationStatus) stream | Add is used to add a replication to this server, changing it&#39;s status from reserved to active. |
+| Status | [Subscription](#pb.Subscription) | [ReplicationStatus](#pb.ReplicationStatus) stream | Status returns an updating stream of the replication status on the server. |
+| GetSubscriptionUpdate | [Subscription](#pb.Subscription) | [SubscriptionUpdate](#pb.SubscriptionUpdate) | GetSubscriptionUpdate returns the latest version of subscribed replication |
+| SubmitReplication | [SignedSubscription](#pb.SignedSubscription) | [ReplicationStatus](#pb.ReplicationStatus) stream | SubmitReplication is used by client agents to start replications, after they have uploaded the files and retrieved the cid, and collected servers to replicate too. |
+
+
+
+
+
 <a name="status.proto"></a>
 <p align="right"><a href="#top">Top</a></p>
 

go.mod

@@ -4,8 +4,12 @@ go 1.13
 
 require (
 	github.com/gogo/protobuf v1.3.1
+	github.com/golang/protobuf v1.3.4 // indirect
 	github.com/ipfs/go-cid v0.0.4
 	github.com/libp2p/go-libp2p-core v0.3.1
 	github.com/multiformats/go-multiaddr v0.2.0
-	google.golang.org/grpc v1.26.0
+	golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d // indirect
+	golang.org/x/net v0.0.0-20191002035440-2ec189313ef0 // indirect
+	google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383 // indirect
+	google.golang.org/grpc v1.27.0
 )

go.sum

@@ -30,6 +30,9 @@ github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5y
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
+github.com/golang/protobuf v1.3.4 h1:87PNWwrRvUSnqS4dlcBU/ftvOIBep4sYuBLlh6rX2wk=
+github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
 github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
 github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY=
 github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
@@ -103,6 +106,8 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8 h1:1wopBVtVdWnn03fZelqdXTqk7U7zPQCb+T4rbU9ZEoU=
 golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d h1:1ZiEyfaQIg3Qh0EoqpwAakHVhecoE5wlSg5GjnafJGw=
+golang.org/x/crypto v0.0.0-20200221231518-2aa609cf4a9d/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
 golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
@@ -116,6 +121,8 @@ golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20191002035440-2ec189313ef0 h1:2mqDk8w/o6UmeUCu5Qiq2y7iMf6anbx+YA8d1JFoFrs=
+golang.org/x/net v0.0.0-20191002035440-2ec189313ef0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -147,11 +154,13 @@ google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoA
 google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
 google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 h1:gSJIx1SDwno+2ElGhA4+qG2zF97qiUzTM+rQ0klBOcE=
 google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
+google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383 h1:Vo0fD5w0fUKriWlZLyrim2GXbumyN0D6euW79T9PgEE=
+google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
 google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
-google.golang.org/grpc v1.26.0 h1:2dTRdpdFEEhJYQD8EMLB61nnrzSCTbG38PhqdhvOltg=
-google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
+google.golang.org/grpc v1.27.0 h1:rRYRFMVgRv6E0D70Skyfsr28tDXIuuPZyWGMPdMcnXg=
+google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=