Skip to content

Running dkv

Jump to bottom
Kinshuk Bairagi edited this page Sep 2, 2022 · 14 revisions

Launching DKV in standalone mode

DKV can be run in a single standalone mode.

Once DKV is built, the <PROJECT_ROOT>/bin folder should contain the following binaries:

  • dkvsrv - DKV server program
  • dkvctl - DKV client program

In case of running from docker, the docker image has the same set of binaries in its working path.

A single DKV instance can be launched using the following command:

$ ./bin/dkvsrv --config dkvsrv.yaml --db-folder <folder_name> --listen-addr <host:port>

All config parameters can be configured in dkvsrv.yaml, and be specifically overriden using commandline arguments.

While using the docker, the single instance can be launched by the following command:

$ docker run -it -p 8080:8080  -v `pwd`:/config dkv:latest  dkvsrv --config /config/dkvsrv.yaml

Any operations can now be done using the dkvctl cli, or using any of the the clients. An example session would look like this:

$ ./bin/dkvsrv --config dkvsrv.yaml --listen-addr 127.0.0.1:8080

$ ./bin/dkvctl --help # Prints the entire set of supported commands
$ ./bin/dkvctl -a 127.0.0.1:8080 -set foo bar
$ ./bin/dkvctl -a 127.0.0.1:8080 -get foo
bar
$ ./bin/dkvctl -a 127.0.0.1:8080 -set hello world
$ ./bin/dkvctl -a 127.0.0.1:8080 -get hello
world
$ ./bin/dkvctl -a 127.0.0.1:8080 -del foo
$ ./bin/dkvctl -a 127.0.0.1:8080 -iter "*"
hello => world

The Java Client is available via clojars

<dependency>
  <groupId>com.flipkart.dkv</groupId>
  <artifactId>dkv-client</artifactId>
  <version>${version}</version>
</dependency>

Launching DKV cluster with consensus based replication

This launch configuration allows for synchronously replicating changes to DKV keyspace on multiple instances spread across independently failing regions or availability zones. Typically such configurations are deployed over WANs so as to ensure better read & write availability in the face of individual cluster failures and disasters.

Under the hood, we use Nexus to replicate keyspace mutations across multiple DKV instances using the RAFT consensus protocol. Currently, the put API automatically replicates changes when the request is handled by given DKV instance started in a special distributed mode (see below). However, get and multiget APIs targetting such an instance serve the data from either from own local store, or does a leased-based read depending on the linearizability level required.

image

Assuming you have 3 availability zones, run the following 3 commands one in every zone in order to setup these instances for synchronous replication.

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --db-folder <folder_path> \
    --listen-addr <host:port> \
    --role master \
    --disable-discovery-client=true \     # a simplified setup flag for running on local 
    --nexus-node-url http://<host:port> \ # optional (autodetected) when running on separate IPs.
    --nexus-cluster-url <cluster_url>

All these 3 DKV instances form a database cluster each listening on separate ports for Nexus & client communications. One can now construct the value for nexusClusterUrl param in the above command using this example setup below:

NexusNodeId Hostname NexusPort
1 dkv.az1 9020
2 dkv.az2 9020
3 dkv.az3 9020

Then the value for nexus-cluster-url must be:

"http://dkv.az1:9020,http://dkv.az2:9020,http://dkv.az3:9020"

Note that same value must be used in each of the 3 commands used to launch the DKV cluster. Subsequently, dkvctl utility can be used to perform keyspace mutations against any one of the DKV instances which are then automatically replicated to the other 2 instances.

Example session on local machine:

Launch Node 1:

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --db-folder /tmp/dkvsrv/n1 \
    --listen-addr 127.0.0.1:9081 \
    --role master \
    --disable-discovery-client=true \
    --nexus-node-url http://127.0.0.1:9021 \
    --nexus-cluster-url "http://127.0.0.1:9021,http://127.0.0.1:9022,http://127.0.0.1:9023"

Launch Node 2:

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --db-folder /tmp/dkvsrv/n2 \
    --listen-addr 127.0.0.1:9082 \
    --role master \
    --disable-discovery-client=true \
    --nexus-node-url http://127.0.0.1:9022 \
    --nexus-cluster-url "http://127.0.0.1:9021,http://127.0.0.1:9022,http://127.0.0.1:9023"

Launch Node 3:

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --db-folder /tmp/dkvsrv/n3 \
    --listen-addr 127.0.0.1:9083 \
    --role master \
    --disable-discovery-client=true \
    --nexus-node-url http://127.0.0.1:9023 \
    --nexus-cluster-url "http://127.0.0.1:9021,http://127.0.0.1:9022,http://127.0.0.1:9023"

Run some operations on this cluster:

$ ./bin/dkvctl -a 127.0.0.1:9081 -set foo bar
$ ./bin/dkvctl -a 127.0.0.1:9082 -get foo
bar
$ ./bin/dkvctl -a 127.0.0.1:9083 -set hello world
$ ./bin/dkvctl -a 127.0.0.1:9081 -get hello
world

We can also add a fourth node dynamically to the above 3 node cluster:

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --db-folder /tmp/dkvsrv/n4 \
    --listen-addr 127.0.0.1:9084 \
    --role master \
    --disable-discovery-client=true \
    --nexus-node-url http://127.0.0.1:9024 \
    --nexus-cluster-url "http://127.0.0.1:9021,http://127.0.0.1:9022,http://127.0.0.1:9023" \
    --nexus-join

Add this node to the existing 3 node cluster:

$ ./bin/dkvctl -dkvAddr 127.0.0.1:9081 -addNode "http://127.0.0.1:9024"

Launching DKV cluster with asynchronous replication

This launch configuration allows for DKV instances to be started either as a master nodes (consensus) or a replica node. All mutations are permitted only on the master nodes while one or more replica nodes asynchronously replicate the changes received from master and make them available for reads. In other words, no keyspace mutations are permitted on the replica nodes, except by the replication stream received from master node.

The built-in replication mechanism guarantees sequential consistency for reads executed from the replica nodes. Moreover, all replica nodes will eventually converge to an identical state which is often referred to as strong eventual consistency.

Such a configuration is typically deployed on applications where the typical number of reads far exceed the number of writes.

image

However for this topology to be fault tolerant and continue to run even when a master node becomes unreachable, we run a quorm of master, and then allow the slaves to autodiscover the available masters and choose any of them. This is done using the discovery service which keep tracks of the entire cluster at any point in time.

auto-switchover

First lets lauch the discovery-service nodes

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --db-folder <folder_name> \
    --listen-addr <host:port> \
    --role discovery

Then launch the one or more DKV master node using the RocksDB engine with this command:

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --db-folder <folder_name> \
    --listen-addr <host:port> \
    --role master

Then launch the DKV slave node using either RocksDB or Badger engine with this command:

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --db-folder <folder_name> \
    --listen-addr <host:port> \
    --db-engine <rocksdb|badger> \
    --role slave

The slave would auto discover the masters based on the vBucket / database identifiers. Subsequently, any mutations performed on the master node's keyspace using dkvctl will be applied automatically onto the replica node's keyspace. By default, a given replica node polls for changes from its master node once every 5 seconds. This can be changed through the repl-poll-interval config parameter.

Note that only rocksdb engine is supported on the DKV master node while the replicas node can be launched with either rocksdb or badger storage engines.

InMemory mode for Badger based slaves

For slave nodes using the Badger storage engine, we also support an in-memory mode where the entire dataset is stored in RAM without any writes to disk whatsoever. This can be achieved by using the -dbDiskless option during launch as shown here.

$ ./bin/dkvsrv --config dkvsrv.yaml \
    --diskless \
    --listen-addr <host:port> \
    --db-engine badger \
    --role slave

This mode may provide better performance for reads and is also useful for deployments that are cache-like having optional durability requirements.

Table Of Contents

Clone this wiki locally