Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements #4

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/build/
22 changes: 22 additions & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
module.exports = {
env: {
browser: true,
es2021: true,
node: true,
},
extends: ['eslint:recommended', 'plugin:@typescript-eslint/recommended'],
parser: '@typescript-eslint/parser',
parserOptions: {
ecmaVersion: 12,
sourceType: 'module',
},
plugins: ['@typescript-eslint'],
rules: {
indent: ['error', 2],
'linebreak-style': ['error', 'unix'],
quotes: ['error', 'single'],
semi: ['error', 'always'],
'@typescript-eslint/no-non-null-assertion': 'off',
'@typescript-eslint/no-explicit-any': 'off',
},
};
9 changes: 7 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
node_modules/
build/
data/
# Build outputs, reports and cache folders
/build/
/.npm/
/.eslintcache/
/coverage/


1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/build/
12 changes: 12 additions & 0 deletions .prettierrc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"semi": true,
"singleQuote": true,
"overrides": [
{
"files": "*.md",
"options": {
"proseWrap": "always"
}
}
]
}
8 changes: 8 additions & 0 deletions .vscode/extensions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"recommendations": [
"editorconfig.editorconfig",
"esbenp.prettier-vscode",
"dbaeumer.vscode-eslint",
"firefox-devtools.vscode-firefox-debug"
]
}
17 changes: 17 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"javascript.format.enable": false,
"javascript.validate.enable": false,
"typescript.format.enable": false,
"json.format.enable": false,
"css.validate": false,

"editor.defaultFormatter": "esbenp.prettier-vscode",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.fixAll.eslint": true
},

"typescript.tsdk": "node_modules/typescript/lib",
"typescript.enablePromptUseWorkspaceTsdk": true,
"typescript.preferences.importModuleSpecifier": "relative"
}
72 changes: 56 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,45 @@
# DSW Deployment Example with PaNOSC User Office middleware

:exclamation: This example is intended for **local setup and testing**. For production use there are many more things to do such as authentication, controlling exposed ports (e.g. do not expose ports of `postgres` and `minio`), data backups, or using proxy.
:exclamation: This example is intended for **local setup and testing**. For
production use there are many more things to do such as authentication,
controlling exposed ports (e.g. do not expose ports of `postgres` and `minio`),
data backups, or using proxy.

## Usage

This is an example deployment of the [Data Stewardship Wizard](https://ds-wizard.org) using [docker-compose](https://docs.docker.com/compose/). You can clone the repository and run it with:
This is an example deployment of the
[Data Stewardship Wizard](https://ds-wizard.org) using
[docker-compose](https://docs.docker.com/compose/). You can clone the repository
and run it with:

```
$ docker-compose build

$ docker-compose up -d
```

Then visit [localhost:8080](http://localhost:8080) and login as `[email protected]` with password `password`.
The server containers needs aroud 5 minutes to be up.

Included in this repository is a PaNOSC/ExPANDS knowledge model(myorg_panosc-expands_2.0.6.km) that can be imported by login into the DMP platform and clicking "Knowledge Models" and then Import.
Then visit [localhost:8080](http://localhost:8080) and login as
`[email protected]` with password `password`.

To use the middleware one needs to simulate a user office connection to the rabbitmq by manually creating a message, this can be done by visiting the rabbitmq interface located at http://localhost:15672/ and using the credentials guest for both password and username. Once logged in visit the "Exchanges" tab and you will find a "useroffice.fanout" exchange that is going be used by your local user office to issue events that occur such as PROPOSAL_CREATED, PROPOSAL_UPDATED, by selecting "useroffice.fanout" we can simulate a user office event by publishing a message. A message has two parts we need to fill in, these are properties and payload. The properties should be set as "type=PROPOSAL_CREATED" and payload can look like:
If you want to upgrade the version, please refer to the migration
[upgrade guidelines](https://guide.ds-wizard.org/miscellaneous/self-hosted-dsw/upgrade-guidelines).

Included in this repository is a PaNOSC/ExPANDS knowledge
model(myorg_panosc-expands_2.0.6.km) that can be imported by login into the DMP
platform and clicking "Knowledge Models" and then Import.

To use the middleware one needs to simulate a user office connection to the
rabbitmq by manually creating a message, this can be done by visiting the
rabbitmq interface located at http://localhost:15672/ and using the credentials
guest for both password and username. Once logged in visit the "Exchanges" tab
and you will find a "useroffice.fanout" exchange that is going be used by your
local user office to issue events that occur such as PROPOSAL_CREATED,
PROPOSAL_UPDATED, by selecting "useroffice.fanout" we can simulate a user office
event by publishing a message. A message has two parts we need to fill in, these
are properties and payload. The properties should be set as
"type=PROPOSAL_CREATED" and payload can look like:

```
{"proposalPk": "345",
Expand All @@ -34,11 +57,14 @@ To use the middleware one needs to simulate a user office connection to the rabb

![image](https://user-images.githubusercontent.com/6403388/132503931-594b5a22-7edb-4daf-b065-13a1080ada5d.png)

This will create a DMP and a user in the data steward wizard. The user can login
with their email and the password "password". The DMP will be filled out with
the information located in the facitilyInformation.json file, to change this
information change in the json and run docker-compose build.

This will create a DMP and a user in the data steward wizard. The user can login with their email and the password "password". The DMP will be filled out with the information located in the facitilyInformation.json file, to change this information change in the json and run docker-compose build.


Then there is also the answering for individual questions as well as setting instrument specific information, this is done with the type of TOPIC_ANSWERED and takes the following as an example:
Then there is also the answering for individual questions as well as setting
instrument specific information, this is done with the type of TOPIC_ANSWERED
and takes the following as an example:

```
[
Expand All @@ -59,21 +85,35 @@ Then there is also the answering for individual questions as well as setting ins
]
```

## Important notes


- Do not expose PostgreSQL and Minio to the internet
- When you want to use DSW publicly, set up proxy (e.g. Nginx) with a
certificate for your domain and change default accounts
- Set up volume mounted to PostgreSQL and Minio containers for persistent data
- You have to create S3 bucket, either using Web UI (for Minio, you can expose
and use `http://localhost:9000`) or via client:
https://docs.min.io/docs/minio-client-complete-guide.html#mb, e.g. use
`create-bucket.sh` script.

## Event types

Currently the middleware listens to three types of events, these are;

1. PROPOSAL_CREATED - Issued when a proposal is created and has only essential information about the proposal such as title, abstract, members and proposer
2. PROPOSAL_UPDATED - Issued when core proposal information is updated, this has the same format and information as PROPOSAL_CREATED
3. TOPIC_ANSWERED - Issued when one or more questions in the user office has been answered.

1. PROPOSAL_CREATED - Issued when a proposal is created and has only essential
information about the proposal such as title, abstract, members and proposer
2. PROPOSAL_UPDATED - Issued when core proposal information is updated, this has
the same format and information as PROPOSAL_CREATED
3. TOPIC_ANSWERED - Issued when one or more questions in the user office has
been answered.

## Architecture overview

The green boxes below are for the individual facilities to change so that the information propogated to the DMP is inline with the facilities information. Currently a significant amount of the information sent to the DMP is fetched from the three json files, this could however be changed to incorporate other software repositories. The index file is listening to a rabbitmq that is sending out events concerning PROPOSAL_ACCEPTED, PROPOSAL_UPDATED and TOPIC_ANSWERED.
The green boxes below are for the individual facilities to change so that the
information propogated to the DMP is inline with the facilities information.
Currently a significant amount of the information sent to the DMP is fetched
from the three json files, this could however be changed to incorporate other
software repositories. The index file is listening to a rabbitmq that is sending
out events concerning PROPOSAL_ACCEPTED, PROPOSAL_UPDATED and TOPIC_ANSWERED.

<img width="1063" alt="Screenshot 2021-09-13 at 13 01 30" src="https://user-images.githubusercontent.com/6403388/133072843-cf45ce5b-6bc0-41bf-aece-dd470352d724.png">

15 changes: 15 additions & 0 deletions create-bucket.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/bin/sh

MINIO_NET="dsw-deployment-example_default"
MINIO_BUCKET="engine-wizard"
MINIO_USER="minio"
MINIO_PASS="minioPassword"

docker run --rm --net $MINIO_NET \
-e MINIO_BUCKET=$MINIO_BUCKET \
-e MINIO_USER=$MINIO_USER \
-e MINIO_PASS=$MINIO_PASS \
--entrypoint sh minio/mc -c "\
mc config host add dswminio http://minio:9000 minio minioPassword && \
mc mb dswminio/\$MINIO_BUCKET
"
68 changes: 44 additions & 24 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
version: '3'
services:
dsw-server:
image: datastewardshipwizard/wizard-server:3.2.2
image: datastewardshipwizard/wizard-server:3.16.2
restart: always
ports:
- 3000:3000
Expand All @@ -12,15 +12,15 @@ services:
- ./dsw.yml:/application/engine-wizard/config/application.yml:ro

dsw-client:
image: datastewardshipwizard/wizard-client:3.2.0
image: datastewardshipwizard/wizard-client:3.16.0
restart: always
ports:
- 8080:80
environment:
API_URL: http://localhost:3000

docworker:
image: datastewardshipwizard/document-worker:3.2.0
image: datastewardshipwizard/document-worker:3.16.0
restart: always
depends_on:
- postgres
Expand All @@ -29,12 +29,21 @@ services:
volumes:
- ./dsw.yml:/app/config.yml:ro

#mailer:
#image: datastewardshipwizard/mailer:3.16.0
#restart: always
#depends_on:
#- postgres
#- dsw-server
# volumes:
#- ./dsw.yml:/app/config.yml:ro

postgres:
image: postgres:13.2
image: postgres:13.7
restart: always
# (!) Expose only for debugging locally or set-up authentication
# ports:
# - 127.0.0.1:15432:5432
ports:
- 127.0.0.1:15432:5432
environment:
POSTGRES_DB: engine-wizard
POSTGRES_USER: postgres
Expand All @@ -46,11 +55,13 @@ services:
# - ./db-data/data:/var/lib/postgresql/data

minio:
image: minio/minio:RELEASE.2021-04-22T15-44-28Z
command: server /data
image: minio/minio:RELEASE.2022-08-02T23-59-16Z
restart: always
command: server /data --console-address ":9001"
# (!) Expose only for debugging locally or set-up authentication
# ports:
# - 127.0.0.1:19000:9000
ports:
- 127.0.0.1:9000:9000
- 127.0.0.1:9001:9001
# (!) Mount and backup for persistent data
environment:
MINIO_ROOT_USER: minio
Expand All @@ -60,30 +71,39 @@ services:
# - s3-data:/data
# OR
# - ./s3-data/data:/data

rabbitmq:
image: "rabbitmq:3-management"
image: 'rabbitmq:3-management'
ports:
- "15672:15672"
- "5672:5672"
- '15672:15672'
- '5672:5672'
volumes:
- ./rabbitmq-isolated.conf:/etc/rabbitmq/rabbitmq.config

middleware:
build: .
image: "middleware"
image: 'middleware'
environment:
DMP_HOST: "http://dsw-server:3000"
DMP_TAG: "302b1584-a7a2-4d9c-9770-eee990b002ec"
RABBITMQ_HOST: "rabbitmq"
RABBITMQ_USER: "guest"
RABBITMQ_PASSWORD: "guest"
PACKAGE_ID: "myorg:panosc-expands:2.0.6"
INGESTOR_EMAIL: "[email protected]"
INGESTOR_PASSWORD: "password"
DMP_HOST: 'http://dsw-server:3000'
DMP_TAG: '302b1584-a7a2-4d9c-9770-eee990b002ec'
RABBITMQ_HOST: 'rabbitmq'
RABBITMQ_USER: 'guest'
RABBITMQ_PASSWORD: 'guest'
PACKAGE_ID: 'myorg:panosc-expands:2.0.7'
INGESTOR_EMAIL: '[email protected]'
INGESTOR_PASSWORD: 'password'
depends_on:
- rabbitmq
- dsw-server
command: ["./wait-for-it.sh", "rabbitmq:15672", "--timeout=30", "--", "node", "./build/index.js"]
command:
[
'./wait-for-it.sh',
'rabbitmq:15672',
'--timeout=30',
'--',
'node',
'./build/index.js',
]
# volumes:
# db-data:
# s3-data:
# s3-data:
2 changes: 1 addition & 1 deletion dsw.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@ mail:
ssl:
authEnabled:
username:
password:
password:
Loading