Skip to content

Commit

Permalink
Merge pull request #52 from RTradeLtd/ex
Browse files Browse the repository at this point in the history
Examples
  • Loading branch information
bonedaddy authored Mar 24, 2020
2 parents e83d2ee + 941b5b4 commit 42a8a41
Show file tree
Hide file tree
Showing 30 changed files with 14,854 additions and 21 deletions.
73 changes: 71 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
PROTOC_GEN_TS_PATH=${HOME}/npm_modules/bin/protoc-gen-ts
PROTOC_GEN_TS_PATH=${HOME}/.npm_modules/bin/protoc-gen-ts

.PHONY: proto
proto: proto-gen tidy
Expand Down Expand Up @@ -59,6 +59,8 @@ install:
npm install -g grpc-tools
python3 -m pip install grpcio-tools
bash .script/protoc-js.sh
cargo install protobuf-codegen
cargo install grpcio-compiler

# protocol buffer generation targets

Expand Down Expand Up @@ -98,6 +100,14 @@ gen-file:
--plugin=protoc-gen-grpc-java=build/protoc-gen-grpc-java \
--grpc-java_out=java \
pb/file.proto
# generate rust bindings (file)
protoc \
-I=pb \
-I=${GOPATH}/src \
--rust_out=rs/src \
--grpc_out=rs/src \
--plugin=protoc-gen-grpc=`which grpc_rust_plugin` \
pb/file.proto

gen-util:
# generate golang bindings (util)
Expand Down Expand Up @@ -134,6 +144,15 @@ gen-util:
--plugin=protoc-gen-grpc-java=build/protoc-gen-grpc-java \
--grpc-java_out=java \
pb/util.proto
# generate rust bindings (util)
protoc \
-I=pb \
-I=${GOPATH}/src \
--rust_out=rs/src \
--grpc_out=rs/src \
--plugin=protoc-gen-grpc=`which grpc_rust_plugin` \
pb/util.proto


gen-node:
# generate golang bindings (node)
Expand All @@ -156,7 +175,7 @@ gen-node:
-I=${GOPATH}/src \
--ts_out=service=grpc-web:ts \
pb/node.proto
# generate javascript bindings (dag)
# generate javascript bindings (node)
protoc \
-I=pb \
-I=${GOPATH}/src \
Expand All @@ -171,6 +190,15 @@ gen-node:
--plugin=protoc-gen-grpc-java=build/protoc-gen-grpc-java \
--grpc-java_out=java \
pb/node.proto
# generate rust bindings (node)
protoc \
-I=pb \
-I=${GOPATH}/src \
--rust_out=rs/src \
--grpc_out=rs/src \
--plugin=protoc-gen-grpc=`which grpc_rust_plugin` \
pb/node.proto


gen-status:
# generate golang bindings (status)
Expand Down Expand Up @@ -208,6 +236,15 @@ gen-status:
--plugin=protoc-gen-grpc-java=build/protoc-gen-grpc-java \
--grpc-java_out=java \
pb/status.proto
# generate rust bindings (status)
protoc \
-I=pb \
-I=${GOPATH}/src \
--rust_out=rs/src \
--grpc_out=rs/src \
--plugin=protoc-gen-grpc=`which grpc_rust_plugin` \
pb/status.proto


gen-pubsub:
# generate golang bindings (pubsub)
Expand Down Expand Up @@ -245,6 +282,14 @@ gen-pubsub:
--plugin=protoc-gen-grpc-java=build/protoc-gen-grpc-java \
--grpc-java_out=java \
pb/pubsub.proto
# generate rust bindings (pubsub)
protoc \
-I=pb \
-I=${GOPATH}/src \
--rust_out=rs/src \
--grpc_out=rs/src \
--plugin=protoc-gen-grpc=`which grpc_rust_plugin` \
pb/pubsub.proto

gen-admin:
# generate golang bindings (admin)
Expand Down Expand Up @@ -282,6 +327,14 @@ gen-admin:
--plugin=protoc-gen-grpc-java=build/protoc-gen-grpc-java \
--grpc-java_out=java \
pb/admin.proto
# generate rust bindings (admin)
protoc \
-I=pb \
-I=${GOPATH}/src \
--rust_out=rs/src \
--grpc_out=rs/src \
--plugin=protoc-gen-grpc=`which grpc_rust_plugin` \
pb/admin.proto

gen-namesys:
# generate golang bindings (namesys)
Expand Down Expand Up @@ -318,6 +371,14 @@ gen-namesys:
--plugin=protoc-gen-grpc-java=build/protoc-gen-grpc-java \
--grpc-java_out=java \
pb/namesys.proto
# generate rust bindings (namesys)
protoc \
-I=pb \
-I=${GOPATH}/src \
--rust_out=rs/src \
--grpc_out=rs/src \
--plugin=protoc-gen-grpc=`which grpc_rust_plugin` \
pb/namesys.proto

gen-replication:
# generate golang bindings (replication)
Expand Down Expand Up @@ -355,6 +416,14 @@ gen-replication:
--plugin=protoc-gen-grpc-java=build/protoc-gen-grpc-java \
--grpc-java_out=java \
pb/replication.proto
# generate rust bindings (replication)
protoc \
-I=pb \
-I=${GOPATH}/src \
--rust_out=rs/src \
--grpc_out=rs/src \
--plugin=protoc-gen-grpc=`which grpc_rust_plugin` \
pb/replication.proto

gen-docs:
# generate documentation
Expand Down
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@
We have pre-generated gRPC bindings for the following languages:

* [golang](https://github.com/RTradeLtd/TxPB/tree/master/go)
* [javascript (grpc-web)](https://github.com/RTradeLtd/TxPB/tree/master/js)
* [javascript (nodejs)](https://github.com/RTradeLtd/TxPB/tree/master/js)
* [typescript (grpc-web)](https://github.com/RTradeLtd/TxPB/tree/master/ts)
* [java](https://github.com/RTradeLtd/TxPB/tree/master/java/pb)
* [python](https://github.com/RTradeLtd/TxPB/tree/master/py)
* [python](https://github.com/RTradeLtd/TxPB/tree/master/py)
* [rust](https://github.com/RTradeLtd/TxPB/tree/master/rs/src)

If you want bindings in other languages, feel free to open a github issue. Alternatively you can generate language bindings independently as long as the languages you are using support the appropriate protocol buffer and gRPC versions.

Expand All @@ -24,6 +26,10 @@ If you want bindings in other languages, feel free to open a github issue. Alter
* [TemporalX HTTP Gateway](./doc/GATEWAY.md)
* [Benchmarks](./doc/BENCHMARKS.md)

# Examples

We have an ever growing library of examples that will help you get started with TemporalX. For all code examples see the `examples` folder.

# Code Generation

Before generating the code, you'll need to download quite a few generators. Before downloading the generators you'll need to have a valid installation of the following, please consult appropriate installation manuals as required:
Expand Down
2 changes: 1 addition & 1 deletion doc/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ enabled | true | whether or not replication is enabl
database_location | vstorage/replication/db.sqlite | Location of the SQL database to keep track of replications. |
grpc_port | 9094 | A dedicated TCP port for replication protocol. |
white_list_location | storage/replication/publishers | Location to a collection of peer public keys that are allowed to replicate on this server. The file format is raw protobuf as defined in github.com/libp2p/go-libp2p-core/crypto |
replication|delay | 10m | ReplicationDelay is the time between two replication runs. Uses golang time.Duration string types |
replication_delay | 10m | ReplicationDelay is the time between two replication runs. Uses golang time.Duration string types |
## Storage

The `storage` section is used to configure the main storage layer of our node. It consists of a generic `datastore` with a `blockstore` on top, and is primarily used for storing our "data" (files, etc..). It also enables managing of blocks using a novel reference count system, as opposed to a pinning system.
Expand Down
34 changes: 17 additions & 17 deletions doc/PROTO.md
Original file line number Diff line number Diff line change
Expand Up @@ -1394,21 +1394,21 @@ PutResponse is a response to any data storage (put) requests

## Scalar Value Types

| .proto Type | Notes | C++ Type | Java Type | Python Type |
| ----------- | ----- | -------- | --------- | ----------- |
| <a name="double" /> double | | double | double | float |
| <a name="float" /> float | | float | float | float |
| <a name="int32" /> int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int |
| <a name="int64" /> int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long |
| <a name="uint32" /> uint32 | Uses variable-length encoding. | uint32 | int | int/long |
| <a name="uint64" /> uint64 | Uses variable-length encoding. | uint64 | long | int/long |
| <a name="sint32" /> sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int |
| <a name="sint64" /> sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long |
| <a name="fixed32" /> fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int |
| <a name="fixed64" /> fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long |
| <a name="sfixed32" /> sfixed32 | Always four bytes. | int32 | int | int |
| <a name="sfixed64" /> sfixed64 | Always eight bytes. | int64 | long | int/long |
| <a name="bool" /> bool | | bool | boolean | boolean |
| <a name="string" /> string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode |
| <a name="bytes" /> bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str |
| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby |
| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- |
| <a name="double" /> double | | double | double | float | float64 | double | float | Float |
| <a name="float" /> float | | float | float | float | float32 | float | float | Float |
| <a name="int32" /> int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| <a name="int64" /> int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| <a name="uint32" /> uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) |
| <a name="uint64" /> uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) |
| <a name="sint32" /> sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| <a name="sint64" /> sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| <a name="fixed32" /> fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) |
| <a name="fixed64" /> fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum |
| <a name="sfixed32" /> sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| <a name="sfixed64" /> sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| <a name="bool" /> bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass |
| <a name="string" /> string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) |
| <a name="bytes" /> bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) |

84 changes: 84 additions & 0 deletions examples/go/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
package main

import (
"context"
"fmt"
"io"
"io/ioutil"
"log"
"os"

pb "github.com/RTradeLtd/TxPB/v3/go"
"google.golang.org/grpc"
)

var (
xAPI = "xapi.temporal.cloud:9090"
)

func main() {
conn, err := grpc.Dial(xAPI, grpc.WithInsecure())
if err != nil {
log.Fatal(err)
}
nc := pb.NewNodeAPIClient(conn)
fc := pb.NewFileAPIClient(conn)
resp, err := nc.Dag(context.Background(), &pb.DagRequest{
RequestType: pb.DAGREQTYPE_DAG_PUT,
Data: []byte("hello world this is test data"),
})
if err != nil {
log.Fatal(err)
}
fmt.Printf("dag put hash: %s\n", resp.GetHashes()[0])
hash := uploadFile(fc)
fmt.Printf("upload file hash: %s\n", hash)
resp, err = nc.Dag(context.Background(), &pb.DagRequest{
RequestType: pb.DAGREQTYPE_DAG_GET_LINKS,
Hash: hash,
})
if err != nil {
log.Fatal(err)
}
fmt.Printf("number of links: %v\n", len(resp.GetLinks()))
}

func uploadFile(nc pb.FileAPIClient) string {
defer os.Remove("justsometestdata")
if err := ioutil.WriteFile("justsometestdata", []byte("hello world"), os.FileMode(0640)); err != nil {
log.Fatal(err)
}
stream, err := nc.UploadFile(context.Background())
if err != nil {
log.Fatal(err)
}
file, err := os.Open("justsometestdata")
if err != nil {
log.Fatal(err)
}
// declare file options
if err := stream.Send(&pb.UploadRequest{Options: &pb.UploadOptions{MultiHash: "sha2-256", Chunker: "size-1"}}); err != nil {
log.Fatal(err)
}
// upload file - chunked at 5mb each
buf := make([]byte, 4194294)
for {
n, err := file.Read(buf)
if err != nil && err == io.EOF {
// only break if we haven't read any bytes, otherwise exit
if n == 0 {
break
}
} else if err != nil && err != io.EOF {
log.Fatal(err)
}
if err := stream.Send(&pb.UploadRequest{Blob: &pb.Blob{Content: buf[:n]}}); err != nil {
log.Fatal(err)
}
}
resp, err := stream.CloseAndRecv()
if err != nil {
log.Fatal(err)
}
return resp.GetHash()
}
10 changes: 10 additions & 0 deletions examples/js/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# js

To run these examples you'll need to install the following packages:

```shell
$> npm install grpc
$> npm install google-protobuf
```

To actually run you can do `node index.js`
33 changes: 33 additions & 0 deletions examples/js/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
var services = require('../../js/node_grpc_pb');
var messages = require('../../js/node_pb');
var grpc = require('grpc');

function main() {
var client = new services.NodeAPIClient(
'xapi.temporal.cloud:9090',
grpc.credentials.createInsecure()
);
var request = new messages.DagRequest()
request.RequestType = messages.DAG_PUT;
request.Data = "hello world";
client.dag(request, function(err, response) {
if (err) {
console.error("failed to put dag", err);
return;
}
response.array.forEach(element => {
if (element === undefined) {
return
}
element.forEach(ele => {
if (ele === undefined) {
return
}
console.log("hash " + ele);
})
});
})

}

main();
10 changes: 10 additions & 0 deletions examples/py/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Python Example

To run this you will want to make sure the appropriate files are in the directory:

```shell
$> cp ../../py/util_pb2.py .
$> cp ../../py/node_pb2* .
```

After which you can run the example with `python3 main.py` (assuming you have the appropriate grpc dependencies installed)
9 changes: 9 additions & 0 deletions examples/py/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import grpc
import node_pb2_grpc
import node_pb2

channel = grpc.insecure_channel('xapi.temporal.cloud:9090')
s = node_pb2_grpc.NodeAPIStub(channel)
resp = s.Dag(node_pb2.DagRequest(requestType=node_pb2.DAG_PUT, data=bytes(0)))

print(resp)
19 changes: 19 additions & 0 deletions examples/ts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# ts

To run this you'll need to install the following dependencies

```shell
$> npm install grpc
$> npm install google-protobuf
$> npm install @improbable-eng/grpc-web
$> npm install -g typescript
```

To compile typescript to javascript run `tsc index.ts` and you can then run the example with `node index.js`

On some machines the above may cause an error and I'm not exactly sure why. To fix this you can change the `messages` and `services` import to the following:

```
var messages = require("../../js/node_pb");
var services = require("../../js/node_grpc_pb");
```
Loading

0 comments on commit 42a8a41

Please sign in to comment.