Releases: duniter/cesium-plus-pod
0.3.4
duniter4j-0.3.4 [maven-release-plugin] copy for tag duniter4j-0.3.4
0.3.2
- Add a Rest security filter
- Add a new index type: user/settings to store encrypted user content, such as settings
0.3.0
0.3.1
Fix nonce big value
0.2.1
Duniter4j allow to use ElasticSearch and a set of plugins, to store huge quantity of data from a Duniter network, in a cluster architecture.
Demo
Try it at cesium.duniter.fr
Note : Please check that extension has been enable : go inside the Settings page, then click on the last item to open specific settings fir Duniter4j.
Make sure you use a valid Duniter4j ES node, like test-net.duniter.fr:9203.
Deploying your own Elasticsearch + Duniter4j node
Installation
Duniter4j
For now, the simplest way to install Duniter4j ES plugin is to download the standalone version, that include all Elasticsearch 2.3.3 and all expected plugins.
Just download file duniter4j-elasticsearch-<version>-standalone.zip (see below) and unpack anywhere.
Java 8
Your will need to install Java 8 (or more).
After installation, just please make sure JAVA_HOME as been define.
Configuration
- Edit the configuration file:
duniter4j-elasticsearch/config/elasticsearch.yml - Then edit this properties (uncomment lines if need):
duniter.host: change to a valid Duniter node (e.g.test-net.duniter.fror your own node). Please use a remote host name.duniter.port: your node port (e.g.9201)
- Some other properties are optional:
cluster.name: choose a name for your cluster (e.g. `my-cluster') or use the default one;network.host: only if you plan to expose your node;http.port: only if you want to fix the listening port (otherwise ES will use the first available port in the range[9200-9300]);
- Save and close the configuration file.
You are now ready to start your cluster :
cd duniter4j-elasticsearch-<version>
cd bin
./elasticsearch
ES node will start, and first create all indices. Some lines (e.g Creating index [history/delete]) will only occurred during the first startup.
[2016-08-18 19:54:31,645][INFO ][node ] [Bloke] version[2.3.3], pid[24058], build[218bdf1/2016-05-17T15:40:04Z]
[2016-08-18 19:54:31,646][INFO ][node ] [Bloke] initializing ...
[2016-08-18 19:54:32,117][INFO ][plugins ] [Bloke] modules [reindex, lang-expression, lang-groovy], plugins [mapper-attachments, duniter4j-elasticsearch], sites [duniter4j-elasticsearch]
[2016-08-18 19:54:32,133][INFO ][env ] [Bloke] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [66.1gb], net total_space [101.7gb], spins? [no], types [ext4]
[2016-08-18 19:54:32,133][INFO ][env ] [Bloke] heap size [989.8mb], compressed ordinary object pointers [true]
[2016-08-18 19:54:34,078][INFO ][node ] [Bloke] initialized
[2016-08-18 19:54:34,079][INFO ][node ] [Bloke] starting ...
[2016-08-18 19:54:34,141][INFO ][org.nuiton.converter.ConverterUtil] for type : class java.util.Locale : org.nuiton.converter.LocaleConverter@626c569b
[2016-08-18 19:54:34,142][INFO ][org.duniter.elasticsearch.PluginSettings] [Bloke] Starts i18n with locale [fr] at [/home/blavenie/git/duniter4j/target/checkout/duniter4j-elasticsearch/target/duniter4j-elasticsearch-0.2.1/data/i18n]
[2016-08-18 19:54:34,171][WARN ][org.nuiton.i18n.bundle.I18nBundleUtil] fr to fr_FR [file:/home/blavenie/git/duniter4j/target/checkout/duniter4j-elasticsearch/target/duniter4j-elasticsearch-0.2.1/data/i18n/duniter4j-elasticsearch-i18n]
[2016-08-18 19:54:34,243][INFO ][transport ] [Bloke] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2016-08-18 19:54:34,247][INFO ][discovery ] [Bloke] duniter4j-elasticsearch/zS0uGIsIQRqQ4fdg4KeURQ
[2016-08-18 19:54:37,274][INFO ][cluster.service ] [Bloke] new_master {Bloke}{zS0uGIsIQRqQ4fdg4KeURQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-08-18 19:54:37,297][INFO ][http ] [Bloke] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2016-08-18 19:54:37,297][INFO ][node ] [Bloke] started
[2016-08-18 19:54:37,320][INFO ][gateway ] [Bloke] recovered [0] indices into cluster_state
[2016-08-18 19:54:44,176][INFO ][org.duniter.elasticsearch.node.DuniterNode] [Bloke] Checking Duniter indices...
[2016-08-18 19:54:44,181][INFO ][org.duniter.elasticsearch.service.RegistryService] Creating index [registry]
[2016-08-18 19:54:44,373][INFO ][cluster.metadata ] [Bloke] [registry] creating index, cause [api], templates [], shards [3]/[1], mappings [record, currency, comment, category]
[2016-08-18 19:54:44,602][INFO ][cluster.routing.allocation] [Bloke] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[registry][1], [registry][0], [registry][2]] ...]).
[2016-08-18 19:54:44,651][INFO ][cluster.metadata ] [Bloke] [registry] update_mapping [category]
[2016-08-18 19:54:44,911][INFO ][org.duniter.elasticsearch.service.MarketService] Creating index [market]
[2016-08-18 19:54:44,938][INFO ][cluster.metadata ] [Bloke] [market] creating index, cause [api], templates [], shards [2]/[1], mappings [record, comment, category]
[2016-08-18 19:54:45,027][INFO ][cluster.routing.allocation] [Bloke] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[market][0], [market][0]] ...]).
[2016-08-18 19:54:45,059][INFO ][org.duniter.elasticsearch.service.MessageService] Creating index [message/record]
[2016-08-18 19:54:45,078][INFO ][cluster.metadata ] [Bloke] [message] creating index, cause [api], templates [], shards [2]/[1], mappings [record]
[2016-08-18 19:54:45,120][INFO ][org.duniter.elasticsearch.service.UserService] Creating index [user]
[2016-08-18 19:54:45,170][INFO ][cluster.metadata ] [Bloke] [user] creating index, cause [api], templates [], shards [3]/[1], mappings [profile]
[2016-08-18 19:54:45,228][INFO ][org.duniter.elasticsearch.service.HistoryService] Creating index [history/delete]
[2016-08-18 19:54:45,304][INFO ][cluster.metadata ] [Bloke] [history] creating index, cause [api], templates [], shards [2]/[1], mappings [delete]
[2016-08-18 19:54:45,337][INFO ][org.duniter.elasticsearch.node.DuniterNode] [Bloke] Checking Duniter indices... [OK]
[2016-08-18 19:54:45,375][INFO ][cluster.routing.allocation] [Bloke] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[history][0], [history][0]] ...]).
[2016-08-18 19:54:45,483][INFO ][cluster.metadata ] [Bloke] [registry] update_mapping [currency]
[2016-08-18 19:54:45,647][INFO ][org.duniter.core.client.service.bma.BlockchainRemoteServiceImpl] Starting to listen block from [ws://192.168.0.5:9201/ws/block]...
[2016-08-18 19:54:45,825][INFO ][org.duniter.elasticsearch.service.BlockchainService] [test_net] [192.168.0.5:9201] Indexing block #31737
[2016-08-18 19:54:45,833][INFO ][cluster.metadata ] [Bloke] [test_net] creating index, cause [auto(index api)], templates [], shards [5]/[1], mappings [block]
[2016-08-18 19:54:45,966][INFO ][cluster.routing.allocation] [Bloke] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test_net][4]] ...]).
[2016-08-18 19:54:45,987][INFO ][cluster.metadata ] [Bloke] [test_net] update_mapping [block]
[2016-08-18 19:59:34,745][INFO ][org.duniter.elasticsearch.service.BlockchainService] [test_net] [192.168.0.5:9201] Indexing block #31738
Let's play !
- Open a browser to the address : http://localhost:9200 and make sure your have something like:
{
"name" : "Bloke",
"cluster_name" : "duniter4j-elasticsearch",
"version" : {
"number" : "2.3.3",
"build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
"build_timestamp" : "2016-05-17T15:40:04Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
- Now, open the integrated Cesium, with ES storage capabilities : http://localhost:9200/_plugin/duniter4j-elasticsearch
You should see new items in the left menu, like Registry, Profile, Market place.
Enjoy !
Architecture
Duniter4j ES plugin allow client (such as Cesium) to publish and index JSON documents.
Security : a full WoT usage
Documents that clients will send must follow this rules:
- have all mandatory fields :
issuer: with a valid public key (a Duniter pubkey);hash: a hash of the JSON document;signature: a valid signature for theissuerpubkey, signed without thehashfield.
To be able to verify hash and signature field successfully, a document must have been minify using JSON.stringify().
To avoid DoS attack, we plan also to only accept documents from a currency members. Maybe keep only just other pubkey to push document for Ads (Market place).
What's next ?
Blockchain indexation and WoT graphs
As you may see in logs, at startup, Duniter4j can index blocks. We plan to use this capability in clients, like Cesium, to offer some cool features:
- advanced search on the blockchain : full text search on transaction and comments, filter by date
- graphs on currency : monetary mass evolution, UD evolution, ..
- Web of Trust graphs, and certification suggestions, using Neo4j + ES.
P2P cluster
Elasticsearch architecture has been design for cluster architecture, not for a P2P network. But some plugins already exists to add HTTP or WebSocket synchronization between remote nodes.
We plan to reuse/adapt this plugins to build a full P2P architecture, to easily shared and retrieve all valid (signed) JSON documents, between individual or entreprise clusters.
A plain use of the Duniter network stack (using the peer document) could be powerful.
A lot of work has been done to release this version. The road is still long, but... do you smell ? it's smell like victory ! ;)
0.2.0
0.1
Elastic first beta release.
For installation and usage, see README.
One example to index a uCoin currency :
./ucoinj-elasticsearch.sh start index -h <host> -p <port>...and enjoy your ES node, at http://localhost:9200/