REL-273: Docker Compose of the new version of the relay (a.k.a Devo NG-Relay)

Author: davidvillarejo

.env

@@ -0,0 +1,34 @@
+################################################################
+##
+## docker-compose environment variables
+##
+################################################################
+
+################################################################
+## Devo NG-Relay
+################################################################
+
+## Version
+DEVO_NG_RELAY_VERSION = 2.0.1
+
+## Log level
+LOG_LEVEL = info
+
+## JVM Options
+# a space-separated quoted list of JVM flags
+JAVA_OPTS = "-Xmx1G -Xms1G"
+
+## Rule Ports
+# By default, user defined rules are mapped to ports in the
+# range [13003, 13020], both TCP and UDP. To change this
+# behaviour, uncomment the following two lines and set the
+# the correct range ports.
+#TCP_PORT_RANGE = 13003-13020
+#UDP_PORT_RANGE = 13003-13020
+
+################################################################
+## Devo NG-Relay Cli
+################################################################
+
+## Version
+DEVO_NG_RELAY_CLI_VERSION = 1.0.0

CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2021-08-13
+
+### Added 
+
+- Initial version of the docker-compose for the Devo NG-Relay.
+- .env file to decouple the compose file.

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (C) 2018 Devo, Inc.
+Copyright (C) 2021 Devo, Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -19,3 +19,4 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
+

README.md

@@ -1,68 +1,202 @@
-# Relay In House Docker
+# NG-Relay Docker Compose
 
-Devo Relay scoja based docker project
+This repository contains an example of how to run the Devo NG-Relay and Devo
+NG-Relay Cli together with [docker compose](https://github.com/docker/compose).
+It is not mandatory, but recommended, to run both applications.
 
-Project to deploy a Relay using docker without to be necessary install anything.
+## Requirements
 
-To connect to relay just send your event to localhost at default ports 12999/13002
+To use this docker-compose file the following requirements must be met:
 
-The compose is ready to share ports from 12999 to 13030 to cover more needs than default ports
+* docker-compose version: [v1.27+](https://github.com/docker/compose/releases)
+* the following folders **must exist at the same level as *docker-compose.yaml*
+  file**:
+    * *conf*: Stores the NG-Relay configuration. The first time the Devo NG-Relay
+      is run, a default configuration is populated to this folder. Once the default
+      configuration is populated, the subsequent times the Devo NG-Relay is run
+      nothing will be populated unless the folder is empty.
+    * *buffer*: Stores the Devo NG-Relay buffer.
+    * *logs*: Stores the Devo NG-Relay logs.
+* All the above folders must have *write* and *execute* permissions for *others*.
 
-The first time when your run the docker Relay, remember to active the relay in Devo Website
+An example of the last two points may be:
 
-Clone or [download](https://github.com/DevoInc/devo-relay/archive/master.zip) the repository.
+```bash
+# Create the folders
+mkdir conf buffer logs
 
-Be sure that [conf/docker-start](./conf/docker-start) has executable permissions after uncompress or clone the repository
+# Grant write and execution for 'others'
+sudo chmod 777 conf/ buffer/ logs/
+```
 
-## Needs to run relay: (Modify in docker-devo-relay.yml)
-1. LOGTRUST_WEB = https://eu.devo.com | https://us.devo.com | https://es.devo.com | ... (Devo web)
-2. LOGTRUST_ENDPOINT = eu.elb.relay.logtrust.net | us.elb.relay.logtrust.net | ... (Devo elb)
-3. LOGTRUST_PORT = 443 | 6514 | 877 | ... (listening port of Devo elb)
-4. RELAY_NAME = The_name_for_your_relay
-5. RELAY_APIKEY = YOUR_API_KEY from your domain when you want to send events
-6. SELFSIGNED=1 Uncomment this environtment variable in [docker-devo-relay.yml](./docker-devo-relay.yml) if you are connecting the relay to endpoint/web with selfsigned certificates
+## Quick Start
 
-## Run and Stop docker Relay
+The recommended way to launch this docker-compose is executing the following
+command:
 
-1. Run: `DCKHUB=devoinc/ docker-compose -f docker-devo-relay.yml up -d`
-2. Stop: `docker-compose -f docker-devo-relay.yml down`
+```bash
+docker-compose run -rm devo-ng-relay-cli
+```
 
-## Tips and tricks
+The command above will run the Devo NG-Relay and the Devo NG-Relay Cli and will
+present the prompt of the Devo NG-Relay Cli waiting for commands to run.
 
-1. Check relay using simple command
-    1. for i in $( seq 150 ) ; do echo "<14>$( date --iso-8601=seconds ) docker-relay test.keep.free: Docker Devo-Relay test event # $i" | nc -N localhost 13000 ; done
-    2. Check your test.keep.free table to see the events
+If you only want to launch the Devo NG-Relay, you can try one of the following
+commands:
 
-2. HowTo connect the container
-    1. docker exec -it Devo-Relay /bin/bash
+* Attached mode:
+  ```bash
+  docker-compose up devo-ng-relay
+  ```
 
-3. Run relay without compose and without to use this repository:
-    1. `docker run -it -p 12999-13030:12999-13030 -e LOGTRUST_WEB=https://eu.devo.com -e LOGTRUST_ENDPOINT=eu.elb.relay.logtrust.net -e RELAY_NAME=docker-relay-domain_name -e RELAY_APIKEY=YOUR_API_KEY --name Devo-Relay -v "$your_path_keys:/etc/logtrust/scoja/current/keys" -v "$your_path_logs:/var/log" -v "$your_path_relay-data:/opt/devo/relay-data" -v "$your_path_rules:/etc/logtrust/scoja/current/rules" -v "$your_path_unrules:/etc/logtrust/scoja/current/unrules" devoinc/devo-relay:1.1.6 /opt/devo/run/docker-start`
-    2. Set correct values for:
-        1. LOGTRUST_WEB
-        2. LOGTRUST_ENDPOINT
-        3. RELAY_NAME
-        4. RELAY_APIKEY
-        5. $your_path_xxx (absolute path required)
+* Detached mode:
+  ```bash
+  docker-compose up -d devo-ng-relay
+  ```
+If you encounter any problem, refer to the [Troubleshooting](#Troubleshooting)
+section.
 
-4. Set proxy values in [docker-devo-relay.yml](./docker-devo-relay.yml) if it's necessary for your network. Default empty values are not applied:
-    1. PROXY_HOST=
-    2. PROXY_PORT=
-    3. PROXY_USER=
-    4. PROXY_PASS=
+## Environment
 
-5. You can download the image using: `docker pull devoinc/devo-relay:1.1.6`
+To work with this docker-compose file, some environment variables need to be
+set. For this purpose, there is an environment file `.env` in which you can set
+them. There is a short description of each of them below:
 
-## Author
+| Name                  | Mandatory | Default Value | Description |
+| --------------------- | :-------: | :-----------: | ----------- |
+| NG_RELAY_VERSION      | Yes       | 2.0.1         | Version of the Devo NG-Relay. |
+| LOG_LEVEL             | No        | info          | Sets the log level for the Devo NG-Relay. |
+| JAVA_OPTS             | No        |               | space-separated quoted list in which you can activate/deactivate any of the JVM flags. |
+| TCP_PORT_RANGE        | No        | 13003-13020   | Used to open the TCP ports specified in all the user-defined rules. |
+| UDP_PORT_RANGE        | No        | 13003-13020   | Used to open the UDP ports specified in all the user-defined rules. |
+| NG_RELAY_CLI_VERSION  | Yes       | 1.0.0         | Version of the Devo NG-Relay Cli. |
 
-jordilobo | [[email protected]](mailto:[email protected])
 
-## Contact Us
+## Troubleshooting
 
-You can contact with us at [email protected].
+### Devo NG-Relay Cli cannot connect to Devo NG-Relay
 
-## License
+Often, the first time this docker-compose is launched we might see the following
+message:
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details
+```
+***************************
+APPLICATION FAILED TO START
+***************************
 
-#### Enjoy it
+Description:
+
+Error connecting to devo-ng-relay:12998
+
+Action:
+
+Consider launching the application with the following arguments:
+        --host=<HOST>
+        --port=<PORT>
+
+ERROR: 1
+
+```
+This occurs because the Devo NG-Relay is not fully started when the Devo
+NG-Relay Cli started. To solve this, simply re-run the command below:
+
+```bash
+docker-compose run -rm devo-ng-relay-cli
+```
+
+### When I modify an environment variable the changes are not reflected to the container
+
+After an environment variable is changed the Devo NG-Relay container needs to be
+re-created. To do so, just execute:
+
+```bash
+docker-compose up [-d] devo-ng-relay
+```
+
+### Error creating volume
+
+You might see an error similar to the following if the folders for the volumes
+are not created:
+
+```
+Creating network "xxxx_default" with the default driver
+Creating devo-ng-relay ... error
+Creating volume "xxxx_conf" with local driver
+ERROR: for devo-ng-relay  Cannot create container for service devo-ng-relay: failed to mount local volume: mount /<your_conf_path>:/var/lib/docker/volumes/xxxx_conf/_data, flags: 0x1000: no such file or directory
+```
+
+If this happens to you, please refer to the [Requirements](#Requirements)
+section and ensure that the folders have been created correctly.
+
+### After configuring a new relay I see exceptions thrown in the logs
+
+An example of this kind of errors is the following:
+
+```
+devo-ng-relay        | Aug  5 09:51:22 57b09a9df2e3 syslog.scoja.configuration: While (re)loading "/opt/devo/ng-relay/conf/relay/run/me.conf": org.scoja.common.ConfigurationException: Error while interpreting configuration file "/opt/devo/ng-relay/conf/relay/run/me.conf": Traceback (most recent call last):
+devo-ng-relay        |   File "/opt/devo/ng-relay/conf/relay/run/me.conf", line 10, in <module>
+devo-ng-relay        |     trans = outTransports(
+devo-ng-relay        |   File "/opt/devo/ng-relay/conf/relay/run/defs.py", line 113, in outTransports
+devo-ng-relay        |     buf = diskfall(
+devo-ng-relay        |   File "<scoja language definition>", line 550, in diskfall
+devo-ng-relay        | 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
+devo-ng-relay        | 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
+devo-ng-relay        | 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
+devo-ng-relay        | 	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
+devo-ng-relay        | 	at java.nio.file.Files.createDirectory(Files.java:674)
+devo-ng-relay        | 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
+devo-ng-relay        | 	at java.nio.file.Files.createDirectories(Files.java:767)
+devo-ng-relay        | 	at org.scoja.util.diskqueue.FilePairQueue.open(FilePairQueue.java:111)
+devo-ng-relay        | 	at org.scoja.util.diskqueue.FilePairFall.<init>(FilePairFall.java:22)
+devo-ng-relay        | 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
+devo-ng-relay        | 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
+devo-ng-relay        | 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
+devo-ng-relay        | 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
+devo-ng-relay        | 	at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:213)
+devo-ng-relay        | java.nio.file.AccessDeniedException: java.nio.file.AccessDeniedException: /var/logt/buffer/me
+```
+
+To solve this, ensure the volume folders have the right permissions. Please
+refer to the [Requirements](#Requirements) section for this matter.
+
+### Useful commands for troubleshooting
+
+Below there are some useful commands to troubleshoot your docker-compose file.
+
+#### View the Devo NG-Relay logs
+
+```bash
+docker-compose logs -f devo-ng-relay
+```
+
+#### Display information about the status and ports of containers started with this file
+
+```bash
+docker-compose ps -a
+```
+
+#### Stop and destroy all the containers
+
+```bash
+docker-compose down
+```
+
+### Ask for help
+
+For an extra guidance or to further support, contact to <[email protected]>.
+
+## Links
+
+[Devo](https://www.devo.com/)
+
+[Devo NG-Relay documentation](https://docs.devo.com/confluence/ndt/latest/sending-data-to-devo)
+
+[Devo in DockerHub](https://hub.docker.com/u/devoinc)
+
+## Follow us
+
+[Twitter](https://twitter.com/devo_inc)
+
+[Youtube](https://www.youtube.com/channel/UCC8gq9iZiHkZMcdztzqHP8Q)
+
+[LinkedIn](https://www.linkedin.com/company/devoinc/)