Emerald Dshackle is a Fault Tolerant Load Balancer for Blockchain API.
The goal of the Emerald Dshackle is to provide a stable routing to multiple nodes, and ensure that each request is executed on an appropriate provider. It considers nodes locations, state, current height, RPC methods it can provide and other characteristics.
It tries to recover from connection errors, faulty nodes, invalid responses, etc. If upstream lags behind others, lost peers below required, started to resync or went down, then Dshackle temporarily excludes it from requests and returns it when the upstream problem is fixed.
The upstreams may be blockchain nodes such as Bitcoind, Geth, Parity, or public providers like Infura, QuickNode, etc. It automatically verifies their availability and the current status of the network, executes commands making sure that the response is consistent and/or data successfully broadcast to the network.
Provides:
-
Standard Bitcoin and Ethereum JSON RPC API over HTTP and WebSocket
-
Enhanced gRPC-based API, with upstream selection, async execution, etc
-
Secure TLS with optional client authentication
-
Blockchain-aware edge caching, in memory and Redis
-
Routing based on data availability (peers, height, sync status)
-
Data consistency, it always gives a most actual state
-
Automatic failover and retry
-
Separate public blockchain nodes from your internal servers
Blockchains support:
-
Ethereum and Ethereum Classic
-
Kovan, Goerli, Ropsten, Rinkeby testnets
-
Bitcoin
-
Bitcoin testnet
|
Warning
|
The project is still under development, please use with caution. |
-
❏ Subscription to bitcoind notification over gRPC (instead of ZeroMQ)
-
❏ Lightweight sidecar node connector
-
❏ Configurable upstream roles
Create file dshackle.yaml with the following content:
version: v1
port: 2449
tls:
enabled: false
proxy:
host: 0.0.0.0
port: 8545
routes:
- id: eth
blockchain: ethereum
- id: kovan
blockchain: kovan
- id: btc
blockchain: bitcoin
cluster:
upstreams:
- id: infura-eth
chain: ethereum
connection:
ethereum:
rpc:
url: "https://mainnet.infura.io/v3/${INFURA_USER}"
ws:
url: "wss://mainnet.infura.io/ws/v3/${INFURA_USER}"
- id: infura-kovan
chain: kovan
connection:
ethereum:
rpc:
url: "https://kovan.infura.io/v3/${INFURA_USER}"
ws:
url: "wss://kovan.infura.io/ws/v3/${INFURA_USER}"
- id: bitcoin-main
chain: bitcoin
connection:
bitcoin:
rpc:
url: "http://localhost:8332"
basic-auth:
username: bitcoin
password: mypasswordWhich sets the following:
-
gRPC access through 0.0.0.0:2449
-
TLS security is disabled (please don’t use in production!)
-
-
JSON RPC access through 0.0.0.0:8545 (both HTTP and WebsScket)
-
proxy requests to Ethereum and Kovan upstreams
-
request path for Ethereum Mainnet is
/eth,/kovanfor Kovan Testnet, and/btcfor bitcoin -
i.e. call Ethereum Mainnet by
POST http://127.0.0.0:8545/ethwith JSON RPC payload
-
-
two upstreams, one for Ethereum Mainnet and another for Kovan Testnet (both upstreams are configured to use Infura endpoint)
-
for Ethereum Mainnet it connects using JSON RPC and WebSocket connections,
-
for Bitcoin Mainet only JSON RPC is used
-
${INFURA_USER}will be provided through environment variable
Please note that you can configure many upstreams for a single blockchains. If there is more than one upstream, then Dshackle routes requests to them as Round Robin. If one of them becomes unavailable, Dshackle continues to use only active nodes.
I.e., you can set up a node in the local network, plus Infura with role: fallback.
If anything happened to your local node, you still have access to a consistent state of the Ethereum blockchain via Infura.
Official Docker image you can find at: emeraldpay/dshackle
export INFURA_USER=...docker run -p 2449:2449 -p 8545:8545 -v $(pwd):/etc/dshackle -e "INFURA_USER=$INFURA_USER" emeraldpay/dshackle:0.11Now it listens on port 2449 at the localhost and can be connected from any gRPC compatible client. Tools such as gRPCurl can automatically parse protobuf definitions and connect to it (actual Protobuf sources are located in a separate repository which you can find at https://github.com/emeraldpay/proto)
Alternatively you can connect to port 8545 with traditional JSON RPC requests
Dshackle implements standard JSON RPC interface, providing additional caching layer, upstream readiness/liveness checks, retry and other features for building Fault Tolerant services.
curl --request POST \
--url http://localhost:8545/eth \
--header 'content-type: application/json' \
--data '{"jsonrpc":"2.0", "method":"eth_getBalance", "id":1, "params":["0x690b2bdf41f33f9f251ae0459e5898b856ed96be", "latest"]}'{"jsonrpc":"2.0","id":1,"result":"0x72fa5e0181"}Or the same Proxy URL can be accessed through WebSocket
websocat ws://localhost:8545/ethThen make RPC calls or subscriptions:
> | {"jsonrpc":"2.0", "id": 1, "method": "eth_subscribe", "params": ["newHeads"]}
< | {"jsonrpc":"2.0","id":1,"result":"1f8"}
< | {"jsonrpc":"2.0","method":"eth_subscription","params":{"result":{....},"subscription":"1f8"}}
|
Note
|
It’s not necessary to use gRPC, as Dshackle can provide standard JSON RPC proxy, but Dshackle gRPC interface improves performance and provides additional features. |
Dshackle provides a custom gRPC based API, which provides additional methods and other features such as streaming responses. Please refer to the documentation: gRPC Methods The Protobuf definitions could be found in ./proto.
grpcurl -import-path ./proto/ -proto blockchain.proto -d '{\"type\": 100}' -plaintext 127.0.0.1:2449 emerald.Blockchain/SubscribeHeadtype: 100 specifies the blockchain id, and 100 means Ethereum Mainnet. 1 is for Bitcoin Mainnet.
There we use Ethereum because it creates new blocks every 14 seconds, which works better for demo purposes, but the same request applied to Bitcoin as well.
{
"chain": "CHAIN_ETHEREUM",
"height": 8396159,
"blockId": "fc58a258adccc94466ae967b1178eea721349b0667f59d5fe1b0b436460bce75",
"timestamp": 1566423564000,
"weight": "AnMcf2VJB5kOSQ=="
}
{
"chain": "CHAIN_ETHEREUM",
"height": 8396160,
"blockId": "787899711b862b77df8d2faa69de664048598265a9f96abf178d341076e200e0",
"timestamp": 1566423574000,
"weight": "AnMch35tO6hSGg=="
}
...
...
The output above is for a streaming subscription to all new blocks on the Ethereum Mainnet.
It’s one of the services provided by Dshackle, in addition to standard methods provided by RPC JSON of underlying nodes.
grpcurl -import-path ./proto/ -proto blockchain.proto -d '{\"asset\": {\"chain\": \"100\", \"code\": \"ether\"}, \"address\": {\"address_single\": {\"address\": \"0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2\"}}}' -plaintext 127.0.0.1:2449 emerald.Blockchain/SubscribeBalance{
"asset": {
"chain": "CHAIN_ETHEREUM",
"code": "ETHER"
},
"address": {
"address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
},
"balance": "2410941696896999943701015"
}
{
"asset": {
"chain": "CHAIN_ETHEREUM",
"code": "ETHER"
},
"address": {
"address": "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2"
},
"balance": "2410930748488073834320430"
}
...
The balance subscription works with main coin (ether, bitcoin), or with tokens like ERC-20 if configured additionally. See Configuration Reference.
See other enhanced methods in the Documentation for Enhanced Methods.
For detailed documentation see docs/ directory.
Dshackle should be compatible with all standard libraries that use Ethereum JSON RPC over HTTP.
repositories {
maven { url "https://maven.emrld.io" }
}
dependencies {
implementation 'io.emeraldpay:emerald-api:0.9.2'
}
"dependencies": {
"@emeraldpay/api-node": "0.2.0-beta.1",
}See more in the documentation for Client Libraries.
|
Warning
|
The code in master branch is considered a development version, which may lack proper testing and should not be used in production.
|
gradle jib -Pdocker=gcr.io/myprojectGradle will prepare a Docker image and upload it to your custom Docker Registry at gcr.io/myproject (please change to address of your actual registry)
Want to support the project, prioritize a specific feature, or get commercial help with using Dshackle in your project? Please contact [email protected] to discuss the possibility
Copyright 2021 EmeraldPay, Inc
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.