[WIP] Dev environment with Docker

.gitignore

@@ -67,3 +67,9 @@ build/
 
 ### Gradle Patch ###
 **/build/
+
+# Dev Environment
+/dev-env/bundles/*.jar
+/dev-env/features/*.xml
+/dev-env/report/*
+!dev-env/report/README.md

CONTRIBUTING.md

@@ -5,7 +5,15 @@ Thank you for taking the time to contribute!
 We appreciate all commits and improvements, feel free to join AET community and contribute.
 
 ## How to start
-Please refer to [AET Documentation](https://github.com/wttech/aet/wiki) to see how to setup local environment.
+To build and develop AET you will need installed:
+- JDK 8
+- Maven or Gradle
+- Docker (to run dev instance)
+- shell command line (Linux preffered)
+
+### Development
+See the instructions in the [Dev Environment](https://github.com/wttech/aet/tree/master/dev-env) to see how to setup local environment.
+
 Submit a ticket for your issue, assuming one does not already exist.
 
 - Clearly describe the issue including steps to reproduce when it is a bug.

build.gradle.kts

@@ -52,6 +52,9 @@ tasks.rat {
     excludes.add("**/gradle.properties")
     excludes.add("**/gradle-wrapper.properties")
 
+    // dev environment
+    excludes.add("**/dev-env/**")
+
     // sample-site
     excludes.add("**/integration-tests/sample-site/src/main/resources/mock/**/*.html")
     excludes.add("**/integration-tests/sample-site/src/main/webapp/assets/demo_files/accessibility/bootstrap.css")

dev-env/README.md

@@ -0,0 +1,76 @@
+# Dev AET instance with Docker Swarm
+This directory contains Docker Swarm configuration that enables running AET developer environment. It base on AET Docker images from https://github.com/malaskowski/aet-docker.
+
+Make sure you have installed:
+- Java 8 (newer JDK versions are not supported)
+- Maven or Gradle
+- Docker
+- shell command line (Linux preffered)
+
+## Running dev environment
+In order to be able to easily deploy AET artifacts on your docker instance follow these steps:
+
+1. Make sure your Docker instance meets the [prerequisites](https://github.com/malaskowski/aet-docker#prerequisites).
+2. Checkout this repository.
+3. Prepare the instance (TODO: this step can be automated and wrapped with a single Gralde task):
+  - From the root folder of this repo run `./gradlew :zip:makeZip` or `mvn clean install -Pzip` to build the whole AET project and all its artifacts. It may take some time when running for the first time.
+  - Copy AET `bundles` and `features` artifacts from `zip` directory to the corresponding dev-environment directories.
+  - Hint: Until it is automated in a better way, you may use this command to copy neccessary artifacts (run from the repo root dir) depending on build tool:
+    - Maven: 
+      - `rm dev-env/bundles/*.jar && cp zip/target/*.jar dev-env/bundles`
+      - `rm dev-env/features/*.xml && cp zip/target/aet-*.xml dev-env/features`
+    - Gradle:
+      - `rm dev-env/bundles/*.jar && cp zip/build/packages/bundles/*.jar dev-env/bundles`
+      - `rm dev-env/features/*.xml && cp zip/build/packages/features/aet-*.xml dev-env/features`
+4. From `dev-environment` directory run `docker stack deploy -c aet-swarm-dev.yml aet-dev`. It may take ~1-2 minutes until instance is ready to work.
+
+## Development
+
+### AET Core (bundles)
+This scenario shows how to update your AET Docker Swarm dev instance with changes made to any of AET bundles that runs on Karaf.
+
+1. Hack, hack, hack.
+2. Build the module you updated, e.g. `core/jobs`.
+3. Rename the `jar` file so that it does not contain the version and replace existing `jar` in `dev-environment/bundles`. Karaf should auto-discover changes and reload the bundle in a short period.
+  - If you changed configuration, don't forget to update corresponding `.cfg` file in the `configs` directory.
+  - If you added new dependencies remember to update `osgi-dependencies/aet-features.xml` and updating this file in `dev-den/features` so that Karaf can download dependencies for your bundle.
+  - Avoid duplicated bundles (hence the renaming of single module jar). All jar files are renamed by the build tools when building the distribution (`zip` module).
+
+### Report
+- ToDo
+
+## Troubleshooting
+See https://github.com/malaskowski/aet-docker#troubleshooting
+
+## Directory structure
+```
+.
+├── aet-swarm-dev.yml
+├── bundles
+│   ├── aet-module-1.jar
+│   ├── ...
+│   └── aet-module-9.jar
+├── configs
+│   ├── com.cognifide.aet.cleaner.CleanerScheduler-main.cfg
+│   └── ...
+├── features
+│   └── aet-features.xml
+├── report
+└── secrets
+    └── KARAF_EXAMPLE_SECRET
+```
+
+- `aet-swarm-dev.yml` - this file contains configuration file to run AET using Docker Swarm
+- `bundles` - directory mounted to the `/aet/core/bundles` in the Karaf service, where Karaf search for AET [OSGi bundles](https://en.wikipedia.org/wiki/OSGi#Bundles) (see [Karaf's Dockerfile](https://github.com/malaskowski/aet-docker/blob/master/karaf/Dockerfile))
+- `configs` - directory mounted to the `/aet/custom/configs` in the Karaf service, contains OSGi configs in form of `.cfg` files
+- `features` - directory mounted to the `/aet/core/features` in the Karaf service, contains [Karaf provisioning](https://karaf.apache.org/manual/latest/provisioning) configuration files - called features
+- `report` - directory that may contain AET report application, if mounted to `/usr/local/apache2/htdocs` volume in the Report service, it will override default [AET Report application](https://github.com/wttech/aet/tree/master/report)
+- `secrets` - directory contains example [Docker secret](https://docs.docker.com/engine/swarm/secrets/) files. They are scanned before Karaf starts and exported as environment variables. Read more in the [secrets configuration](https://github.com/malaskowski/aet-docker#docker-secrets).
+
+## Karaf healthcheck
+Karaf's service in this sample docker instance have [healthcheck](https://docs.docker.com/compose/compose-file/#healthcheck). It simply checks the dedicated service's endpoint `/health-check` that responses with `200` when everything is ready, with error code otherwise. If the healthcheck fails, swarm will automatically restart the service.
+Read more about this endpoint here: https://fabric8.io/guide/karaf.html#fabric8-karaf-health-checks
+
+# ToDo:
+- [ ] Improve building and deploying to dev instance experience by automating it (e.g. wrap with a single Gradle task that can deploy AET artifacts onto dev-environment).
+- [ ] Fix AET versioning with Gradle, currently the version is magically calculated by the [axion-release-plugin](https://axion-release-plugin.readthedocs.io/en/latest) and hardcoded in some manifests (e.g. [here](https://github.com/wttech/aet/blob/master/api/communication-api/build.gradle.kts#L28)).

dev-env/aet-swarm-dev.yml

@@ -0,0 +1,124 @@
+#
+# AET Docker
+#
+# Copyright (C) 2018 Maciej Laskowski
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: '3.7'
+
+networks:
+  private:
+    driver: overlay
+
+volumes:
+  mongo-data:
+
+services:
+  hub:
+    image: selenium/hub:3.14.0-arsenic
+    ports:
+      - '4444:4444'
+    deploy:
+      mode: global
+      placement:
+        constraints:
+          - node.role == manager
+    networks:
+      - private
+
+  chrome:
+    image: selenium/node-chrome:3.14.0-arsenic
+    volumes:
+      - /dev/shm:/dev/shm # Mitigates the Chromium issue described at https://code.google.com/p/chromium/issues/detail?id=519952
+    depends_on:
+      - hub
+    environment:
+      - HUB_HOST=hub
+      - HUB_PORT=4444
+      - NODE_MAX_INSTANCES=1
+      - NODE_MAX_SESSION=1
+    entrypoint: bash -c 'SE_OPTS="-host $$HOSTNAME -port 5556" /opt/bin/entry_point.sh'
+    deploy:
+      replicas: 3
+    networks:
+      - private
+
+  mongo:
+    image: mongo:3.6
+    command: mongod --enableMajorityReadConcern
+    deploy:
+      placement:
+        constraints:
+          - node.role == manager
+      restart_policy:
+        condition: on-failure
+        delay: 5s
+        max_attempts: 3
+        window: 120s
+    volumes:
+      - mongo-data:/data/db
+    ports:
+      - '27017:27017'
+    networks:
+      - private
+
+  activemq:
+    image: malaskowski/aet_activemq:1.0.0
+    ports:
+      - '8161:8161'
+    networks:
+      - private
+
+  browsermob:
+    image: malaskowski/aet_browsermob:1.0.0
+    ports:
+      - '8080:8080'
+      - '8281-8681:8281-8681'
+    networks:
+      - private
+
+  karaf:
+    image: malaskowski/aet_karaf:1.0.0
+    depends_on:
+      - hub
+      - activemq
+    volumes:
+      - ./bundles:/aet/core/bundles
+      - ./configs:/aet/custom/configs
+      - ./features:/aet/core/features
+    secrets:
+      - KARAF_EXAMPLE_SECRET
+    ports:
+      - '8181:8181'
+    #      - '5005:5005' # uncomment to be able to connect Karaf in debug mode
+    #    environment:
+    #      - KARAF_DEBUG=true # uncomment (with section above) to start Karaf in debug mode
+    networks:
+      - private
+
+  report:
+    image: malaskowski/aet_report:1.0.0
+    ports:
+      - '9999:80'
+    #    volumes:
+    #      - ./report:/usr/local/apache2/htdocs
+    #    environment:
+    #      - AET_WEB_API=http://my.karaf.com # uncomment to configure custom AET Web API endpoint
+    networks:
+      - private
+
+secrets:
+  KARAF_EXAMPLE_SECRET:
+    file: secrets/KARAF_EXAMPLE_SECRET

dev-env/bundles/README.md

@@ -0,0 +1 @@
+# AET bundles

vagrant/metadata.rb → dev-env/configs/com.cognifide.aet.cleaner.CleanerScheduler-main.cfg

@@ -1,7 +1,7 @@
 #
-# AET
+# AET Docker
 #
-# Copyright (C) 2013 Cognifide Limited
+# Copyright (C) 2018 Maciej Laskowski
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,12 +16,7 @@
 # limitations under the License.
 #
 
-name             'aet-vagrant'
-maintainer       'Karol Drazek'
-maintainer_email '[email protected]'
-license          'Apache 2.0'
-description      'Installs/Configures aet-vagrant'
-long_description 'Installs/Configures aet-vagrant'
-version          '2.1.6'
-
-depends 'aet', '~> 5.1.1'
+schedule=0 0 22 ? * *
+keepNVersions=2
+removeOlderThan=1
+dryRun=false

vagrant/Berksfile → dev-env/configs/com.cognifide.aet.proxy.RestProxyManager.cfg

@@ -1,7 +1,7 @@
 #
-# AET
+# AET Docker
 #
-# Copyright (C) 2013 Cognifide Limited
+# Copyright (C) 2018 Maciej Laskowski
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,12 +16,5 @@
 # limitations under the License.
 #
 
-source "https://supermarket.chef.io"
-
-# Uncomment one of the below lines to use aet cookbook from alternative location.
-# And update the cookbook required version in 'metadata.rb' file.
-
-cookbook 'aet', git: 'https://github.com/wttech/aet-cookbook.git', branch: 'master'
-#cookbook 'aet', path: '../../aet-cookbook'
-
-metadata
+server=browsermob
+port=8080

dev-env/configs/com.cognifide.aet.queues.DefaultJmsConnection.cfg

@@ -0,0 +1,21 @@
+#
+# AET Docker
+#
+# Copyright (C) 2018 Maciej Laskowski
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+url=failover:tcp://activemq:61616
+username=admin
+password=admin

dev-env/configs/com.cognifide.aet.rest.helpers.ReportConfigurationManager.cfg

@@ -0,0 +1,19 @@
+#
+# AET Docker
+#
+# Copyright (C) 2018 Maciej Laskowski
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+reportDomain=http://localhost:9999

dev-env/configs/com.cognifide.aet.runner.MessagesManager.cfg

@@ -0,0 +1,19 @@
+#
+# AET Docker
+#
+# Copyright (C) 2018 Maciej Laskowski
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+jmxUrl=service:jmx:rmi:///jndi/rmi://activemq:11199/jmxrmi

dev-env/configs/com.cognifide.aet.runner.RunnerConfiguration.cfg

@@ -0,0 +1,23 @@
+#
+# AET Docker
+#
+# Copyright (C) 2018 Maciej Laskowski
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+ft=120
+mttl=300
+urlPackageSize=1
+maxMessagesInCollectorQueue=3
+maxConcurrentSuitesCount=2