diff --git a/.env b/.env index 6b5f62c1..6e558d41 100644 --- a/.env +++ b/.env @@ -24,8 +24,8 @@ REPO_CURATED=/tmp/curated REPO_CONFIG=/tmp/config # Variables for controlling external dependencies versions -POSTGRES_VERSION=13 -NGINX_VERSION=1.23 +POSTGRES_VERSION=16 +NGINX_VERSION=1.24 # Other variables DOCKER_EXECUTORS=10 diff --git a/README.md b/README.md index 148d95f1..044ccf4f 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,7 @@ Running Devspace requires access to SSH and Git configuration files used for fet Devspace code depends on the following repositories: -* [OMERO install](https://github.com/ome/omero-install/) -* [devslave-c7-docker](https://github.com/ome/devslave-c7-docker) +* [devagent-docker](https://github.com/ome/devagent-docker) # Installation @@ -25,12 +24,6 @@ The following instructions explain how to deploy a devspace on a Docker host. * Log into the Docker host using ssh -* Install the prerequisites [Docker engine](https://docs.docker.com/) and - [Docker compose](https://docs.docker.com/compose/) either globally or in - a virtual environment: - - $ pip install docker-compose - * Create a directory ``/data/username`` and change ownership: $ sudo mkdir /data/username @@ -52,7 +45,8 @@ The following instructions explain how to deploy a devspace on a Docker host. * Copy the SSH and Git configuration files used for fetching and pushing the Git repositories under `slave/.ssh` and `slave/.gitconfig`. This is usually - your own SSH and Git configuration files. + your own SSH and Git configuration files. Make sure that the permissions of the key are not + too open. If this is the case, change the permissions e.g. ``chmod 400 YOUR_KEY` You need to use a public key without a passphrase and a `.gitconfig` file containing the following sections: ``` @@ -66,10 +60,10 @@ The following instructions explain how to deploy a devspace on a Docker host. * Run `rename.py` to match your topic name. Specify the Git user corresponding to the confguration files used above. If you do not yet have - topic branches available on origin, use `develop` or one of the + topic branches available on origin, use `develop/master` or one of the main branches: - $ ./rename.py MYTOPIC --user git_user + $ ./rename.py USER MYTOPIC --user git_user * This will also replace the `USER_ID` of the various Dockerfile with the ID of the user who will run the devspace, assumed to be: `id -u`, i.e. the current user. @@ -83,27 +77,34 @@ The following instructions explain how to deploy a devspace on a Docker host. Start and configure: -* Build devspace using `docker-compose`: +* Build devspace using `docker compose`: - $ docker-compose -f docker-compose.yml build + $ docker compose -f docker-compose.yml build -* Start devspace using `docker-compose`: +* Start devspace using `docker compose`: - $ docker-compose -f docker-compose.yml up -d + $ docker compose -f docker-compose.yml up -d By default, this will use the name of the directory as the project name. In the case of a shared Docker host, it is possible to override the project name using - $ docker-compose up -p my_project -d + $ docker compose up -p my_project -d + +* Depending on the ssh key, you might have to run the following comment in the ``test-integration`` container. For example: + $ docker exec -it devspace-testintegration-1 bash + $ ssh -T git@github.com + + A message should be returned after running the command: + $ Hi snoopycrimecop! You've successfully authenticated, but GitHub does not provide shell access. * Retrieve the dynamic port of the Jenkins NGINX container. You can access the Jenkins UI from https://HOST_IP:PORT after accepting the self-signed certificate: - $ docker-compose -p my_project port nginxjenkins 443 + $ docker compose -p my_project port nginxjenkins 443 * Create the `maven-internal` Nexus repository: - $ docker-compose exec nexus /nexus-data/createRepoMavenInternal.sh + $ docker compose exec nexus /nexus-data/createRepoMavenInternal.sh * [Optional] Turn on Basic HTTP authentication for Jenkins @@ -134,7 +135,7 @@ After the script has completed you can either leave it in place so it will overr # Job configurations -* When running the OMERO-build job for the first time, select the ``PURGE_DATA`` option to create the database. +* When running the OMERO-server job for the first time, select the ``PURGE_DATA`` option to create the database. # Job workflow @@ -147,24 +148,24 @@ they are associated with and a short description of the jobs. | Job name | Name | Description | docker name | | -----------------------|-----------------| ------------------------------------------|----------------------------| | Trigger | | Runs all the following jobs in order | | -| BIOFORMATS-push | testintegration | Merges all Bio-Formats PRs | devspace_testintegration_1 | -| BIOFORMATS-build | testintegration | Builds Bio-Formats components | devspace_testintegration_1 | -| BIOFORMATS-image | testintegration | Builds a Docker image of Bio-Formats | devspace_docker_1 | -| OMERO-push | testintegration | Merges all OMERO PRs | devspace_testintegration_1 | -| OMERO-build | testintegration | Builds OMERO artifacts (server, clients) | devspace_testintegration_1 | -| OMERO-server | omero | Deploys an OMERO.server | devspace_omero_1 | -| OMERO-web | web | Deploys an OMERO.web client | devspace_web_1 | -| OMERO-test-integration | testintegration | Runs the OMERO integration tests | devspace_testintegration_1 | -| OMERO-robot | testintegration | Runs the Robot tests | devspace_testintegration_1 | -| nginx | nginx | Reloads the nginx server | devspace_nginx_1 | -| OMERO-docs | testintegration | Builds the OMERO documentation | devspace_testintegration_1 | +| BIOFORMATS-push | testintegration | Merges all Bio-Formats PRs | devspace-testintegration-1 | +| BIOFORMATS-build | testintegration | Builds Bio-Formats components | devspace-testintegration-1 | +| BIOFORMATS-image | testintegration | Builds a Docker image of Bio-Formats | devspace-docker-1 | +| OMERO-push | testintegration | Merges all OMERO PRs | devspace-testintegration-1 | +| OMERO-build | testintegration | Builds OMERO artifacts (server, clients) | devspace-testintegration-1 | +| OMERO-server | omero | Deploys an OMERO.server | devspace-omero-1 | +| OMERO-web | web | Deploys an OMERO.web client | devspace-web-1 | +| OMERO-test-integration | testintegration | Runs the OMERO integration tests | devspace-testintegration-1 | +| OMERO-robot | testintegration | Runs the Robot tests | devspace-testintegration-1 | +| nginx | nginx | Reloads the nginx server | devspace-nginx-1 | +| OMERO-docs | testintegration | Builds the OMERO documentation | devspace-testintegration-1 | This means that by default the following repositories need to be forked to your GitHub account: * [ome/openmiscrocopy](https://github.com/ome/openmicroscopy) -* [ome/ome-documentation](https://github.com/ome/ome-documentation) +* [ome/omero-documentation](https://github.com/ome/omero-documentation) * [ome/bioformats](https://github.com/ome/bioformats) If you do not have some of the repositories forked, you will need to remove the jobs from the list @@ -185,29 +186,28 @@ Alternatively create a new job in the Jenkins web-interface in the usual way. # Default packages used -| Name | Version | Optional | -| -----------|---------------| -----------------------------------| -| Java | openJDK 1.8 | openJDK 1.8 devel, oracleJDK 1.8 | -| Python | 2.7 | - | -| Ice | 3.6 | 3.5 | -| PostgreSQL | 9.4 | https://hub.docker.com/_/postgres/ | -| Nginx | 1.8 | - | -| Redis | latest | https://hub.docker.com/_/redis/ | +| Name | Version | +| -----------|-------------------| +| Java | openJDK 11-devel | +| Python | 3.9 | +| Ice | 3.6 | +| PostgreSQL | 16 | +| Nginx | 1.24 | # Troubleshooting See [Troubleshooting](Troubleshooting.md) -# ADVANCE: extend omero-install -In order to install additional components or new version of packages e.g. PostgreSQL 10, it is required to: +# Upgrade -* Modify the files in [omero-install](https://github.com/ome/omero-install) -* Create a new image of [devslave-c7-docker](https://github.com/ome/devslave-c7-docker) using the updated omero-install files -* Push the new image to [Docker Hub](https://hub.docker.com/). You will need to your own account -* Modify each Dockerfile of this repository to use the new image +See [Changelog](CHANGELOG.md) +# Run BioFormats jobs -# Upgrade +To run the BioFormats testing the various readers on sample data, you will need to activate the a private +job -See [Changelog](CHANGELOG.md) +* In the ``devspace``, create a directory ``home/jobs/DATA_REPO_CONFIG-merge/`` +* Download the job configuration from ``config.xml`` in https://github.com/openmicroscopy/management_tools/tree/master/ci/jobs/DATA_REPO_CONFIG-merge (private repository) and place it in the newly created directory +* Comment out the line ``build job: "DATA_REPO_CONFIG-merge"`` in the ``trigger`` job diff --git a/docker-compose.yml b/docker-compose.yml index 9f555734..75833203 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -12,12 +12,6 @@ services: environment: - JENKINS_OPTS= --prefix=${JENKINS_PREFIX} --argumentsRealm.passwd.${JENKINS_USERNAME}=${JENKINS_PASSWORD} --argumentsRealm.roles.${JENKINS_USERNAME}=admin - git: - build: git - volumes: - - /src - command: 'true' - pg: image: postgres:${POSTGRES_VERSION} networks: @@ -41,7 +35,7 @@ services: environment: - SLAVE_NAME=testintegration - SLAVE_EXECUTORS=2 - - SLAVE_PARAMS=-labels centos7 -labels ice36 -labels java11 -disableClientsUniqueId + - SLAVE_PARAMS=-labels rocky9 -labels ice36 -labels java11 -disableClientsUniqueId - JENKINS_MASTER=${JENKINS_BASE}${JENKINS_PREFIX} - WEB_PREFIX - REPO_CONFIG @@ -50,6 +44,10 @@ services: ports: - "${OMERO_SERVER_TEST_SSL}14064" - "${OMERO_SERVER_TEST_TCP}14063" + ulimits: + nofile: + soft: 8192 + hard: 65536 omero: build: @@ -58,16 +56,20 @@ services: networks: - omero-network volumes: - - ./server:/home/omero-server + - ./server:/home/omero environment: - SLAVE_NAME=omero - SLAVE_EXECUTORS=2 - - SLAVE_PARAMS=-labels centos7 -labels ice36 -labels java11 -disableClientsUniqueId + - SLAVE_PARAMS=-labels rocky9 -labels ice36 -labels java11 -disableClientsUniqueId - JENKINS_MASTER=${JENKINS_BASE}${JENKINS_PREFIX} - WEBHOST=http://nginx${WEB_PREFIX} ports: - "${OMERO_SERVER_SSL}4064" - "${OMERO_SERVER_TCP}4063" + ulimits: + nofile: + soft: 8192 + hard: 65536 web: build: @@ -76,12 +78,12 @@ services: networks: - omero-network volumes: - - ./web:/home/omero-web - - ./nginx/conf.d:/home/omero-web/nginx + - ./web:/home/omero + - ./nginx/conf.d:/home/omero/nginx environment: - SLAVE_NAME=web - SLAVE_EXECUTORS=2 - - SLAVE_PARAMS= -labels centos7 -labels ice36 -labels java11 -disableClientsUniqueId + - SLAVE_PARAMS= -labels rocky9 -labels ice36 -labels java11 -disableClientsUniqueId - JENKINS_MASTER=${JENKINS_BASE}${JENKINS_PREFIX} - WEB_PREFIX @@ -93,12 +95,12 @@ services: - omero-network volumes: - ./nginx/conf.d:/etc/nginx/conf.d - - ./web/static:/home/omero-web/static + - ./web/static:/home/omero/static - ./nginx/log:/var/log/nginx - ./nginx/sslcert:/etc/nginx/ssl environment: - SLAVE_NAME=nginx - - SLAVE_PARAMS=-labels centos7 -labels java11 -disableClientsUniqueId + - SLAVE_PARAMS=-labels rocky9 -labels java11 -disableClientsUniqueId - JENKINS_MASTER=${JENKINS_BASE}${JENKINS_PREFIX} ports: - "${NGINX_PORT}80" @@ -139,6 +141,9 @@ services: - SCREEN_WIDTH=1920 - SCREEN_HEIGHT=1080 - SCREEN_DEPTH=24 + - SE_EVENT_BUS_HOST=seleniumhub + - SE_EVENT_BUS_PUBLISH_PORT=4442 + - SE_EVENT_BUS_SUBSCRIBE_PORT=4443 depends_on: - seleniumhub @@ -152,6 +157,9 @@ services: - SCREEN_WIDTH=1920 - SCREEN_HEIGHT=1080 - SCREEN_DEPTH=24 + - SE_EVENT_BUS_HOST=seleniumhub + - SE_EVENT_BUS_PUBLISH_PORT=4442 + - SE_EVENT_BUS_SUBSCRIBE_PORT=4443 depends_on: - seleniumhub @@ -171,7 +179,7 @@ services: - REPO_CONFIG nexus: - image: sonatype/nexus3 + image: sonatype/nexus3:3.59.0 user: ${USER_ID} networks: - omero-network diff --git a/docker/Dockerfile b/docker/Dockerfile index 1d846dcf..4b6d82ae 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -1,22 +1,18 @@ -FROM openmicroscopy/devslave-c7:0.7.2 +FROM openmicroscopy/devagent:0.8.1 MAINTAINER OME -# manually install java11 -RUN yum -y install java-11-openjdk-devel -# Switch Java version -ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk -ENV PATH=$JAVA_HOME/bin:$PATH +USER root -RUN yum -y install -y yum-utils \ + +RUN dnf -y install -y yum-utils \ device-mapper-persistent-data \ lvm2 -RUN yum-config-manager \ - --add-repo \ - https://download.docker.com/linux/centos/docker-ce.repo -RUN yum -y install docker-ce +RUN dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo + +RUN dnf -y install docker-ce # Change user id to fix permissions issues ARG USER_ID=1000 diff --git a/git/Dockerfile b/git/Dockerfile deleted file mode 100644 index 3ad977e1..00000000 --- a/git/Dockerfile +++ /dev/null @@ -1,13 +0,0 @@ -FROM mini/base -MAINTAINER ome-devel@lists.openmicroscopy.org.uk - -ENV TAG 5.1.4 - -RUN apk-install git && \ - mkdir /src - -RUN time git clone --bare --depth=1 -b v$TAG https://github.com/ome/scripts/ /src/scripts.git -RUN time git clone --bare --depth=1 -b v$TAG https://github.com/ome/openmicroscopy /src/omero.git -RUN time git clone --bare --depth=1 -b v$TAG https://github.com/ome/bioformats /src/bio-formats.git - -WORKDIR /src diff --git a/home/jenkins.model.JenkinsLocationConfiguration.xml b/home/jenkins.model.JenkinsLocationConfiguration.xml index eb294ca4..eaad10b8 100644 --- a/home/jenkins.model.JenkinsLocationConfiguration.xml +++ b/home/jenkins.model.JenkinsLocationConfiguration.xml @@ -1,5 +1,5 @@ address not configured yet <nobody@nowhere> - https://SPACENAME-ci.openmicroscopy.org/jenkins/ + https://BRANCHCI.openmicroscopy.org/jenkins/ diff --git a/home/jobs/BIOFORMATS-build/config.xml b/home/jobs/BIOFORMATS-build/config.xml index 03294bbf..74991e48 100644 --- a/home/jobs/BIOFORMATS-build/config.xml +++ b/home/jobs/BIOFORMATS-build/config.xml @@ -48,7 +48,12 @@ false false - + + false + + + true + testintegration diff --git a/home/jobs/BIOFORMATS-push/config.xml b/home/jobs/BIOFORMATS-push/config.xml index 2794432b..67b1da5e 100644 --- a/home/jobs/BIOFORMATS-push/config.xml +++ b/home/jobs/BIOFORMATS-push/config.xml @@ -58,7 +58,9 @@ false false - + + true + testintegration diff --git a/home/jobs/OMERO-build/config.xml b/home/jobs/OMERO-build/config.xml index dc0d36c3..90df658c 100644 --- a/home/jobs/OMERO-build/config.xml +++ b/home/jobs/OMERO-build/config.xml @@ -77,7 +77,7 @@ OMERO_VEN=$WORKSPACE/omero-virtualenv source $HOME/settings.env python3 -m venv $OMERO_VEN -$OMERO_VEN/bin/pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.3.0/zeroc_ice-3.6.5-cp38-cp38-linux_x86_64.whl +$OMERO_VEN/bin/pip install https://github.com/glencoesoftware/zeroc-ice-py-rhel9-x86_64/releases/download/20230830/zeroc_ice-3.6.5-cp39-cp39-linux_x86_64.whl $OMERO_VEN/bin/pip install -U setuptools omero-py source docs/hudson/OMERO.sh @@ -85,7 +85,7 @@ source docs/hudson/OMERO.sh - src/target/*.zip,src/target/*.egg,src/target/*.log,src/target/*INFO,target/version.tsv + src/target/*.zip,src/target/*.log,src/target/*INFO false false false diff --git a/home/jobs/OMERO-robot/config.xml b/home/jobs/OMERO-robot/config.xml index b43e3e73..b564e608 100644 --- a/home/jobs/OMERO-robot/config.xml +++ b/home/jobs/OMERO-robot/config.xml @@ -58,7 +58,7 @@ python -m venv $WORKSPACE/.venv3 source $WORKSPACE/.venv3/bin/activate -pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.2.1/zeroc_ice-3.6.5-cp36-cp36m-linux_x86_64.whl +pip install https://github.com/glencoesoftware/zeroc-ice-py-rhel9-x86_64/releases/download/20230830/zeroc_ice-3.6.5-cp39-cp39-linux_x86_64.whl pip install -U pip future setuptools pip install omero-py omero-web # Latest in order to stop server. diff --git a/home/jobs/OMERO-server/config.xml b/home/jobs/OMERO-server/config.xml index 0d150248..179ccfe8 100644 --- a/home/jobs/OMERO-server/config.xml +++ b/home/jobs/OMERO-server/config.xml @@ -1,7 +1,6 @@ - - + - + false @@ -25,12 +24,11 @@ OMERO_SERVER_ZIP_URL - - false + omero @@ -42,8 +40,7 @@ false - OMERO_INSTALL=/tmp/omero-install/linux -source $OMERO_INSTALL/settings.env + source /home/settings.env OMERO_DB_NAME=$DB_NAME OMERO_DIST=$WORKSPACE/OMERO.server export OMERODIR=$OMERO_DIST @@ -56,37 +53,52 @@ if [ -e $OMERO_DIST ]; then deactivate fi +rm -rf $WORKSPACE/.venv3 +python3 -m venv $WORKSPACE/.venv3 -if [ "$PURGE_DATA" = "true" ]; then - dropdb -h $OMERO_DB_HOST -U $OMERO_DB_USER $OMERO_DB_NAME || echo "First run or already exists" +if [ "$PURGE_DATA" = "true" ]; then + dropdb -h $OMERO_DB_HOST -U $OMERO_DB_USER $OMERO_DB_NAME || echo "First run or already exists" rm -rf $OMERO_DATA_DIR else - echo "Skipping PURGE_DATA:CleanDbAndRepo" + echo "Skipping PURGE_DATA:CleanDbAndRepo" fi rm -rf $WORKSPACE/* # Should leave .venv3 + - + OMERO-build src/target/OMERO.server*zip - - + + false + true false - + OMERO-python-superbuild-build **/*.tar.gz - - + + false + + + true + false + + + OMERO-build-build + + + false + true false - OMERO_INSTALL=/tmp/omero-install/linux -source $OMERO_INSTALL/settings.env + +source /home/settings.env OMERO_DB_NAME=$DB_NAME OMERO_DATA_DIR=$DATA_DIR @@ -94,9 +106,11 @@ mkdir -p $OMERO_DATA_DIR OMERO_DIST=$WORKSPACE/OMERO.server export OMERODIR=$OMERO_DIST +export ZIP_FILE=$WORKSPACE/omero-blitz-VERSION-python.zip +export VERSION_PROPERTIES=$WORKSPACE/version.properties ZIP=$(ls OMERO.server*.zip) -if [ ! -z "$OMERO_SERVER_ZIP_URL" ] +if [ ! -z "$OMERO_SERVER_ZIP_URL" ] then rm -f $ZIP wget -q $OMERO_SERVER_ZIP_URL @@ -108,18 +122,19 @@ rm -f $ZIP mv $WORKSPACE/$DIST $OMERO_DIST source $WORKSPACE/.venv3/bin/activate -pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.3.0/zeroc_ice-3.6.5-cp38-cp38-linux_x86_64.whl + +pip install https://github.com/glencoesoftware/zeroc-ice-py-rhel9-x86_64/releases/download/20230830/zeroc_ice-3.6.5-cp39-cp39-linux_x86_64.whl pip install -U pip future pip install markdown pip install reportlab # For figure pip install omego pip install tables pip install jinja2 -pip install "urllib3<2" # to be removed if/when included in omero-py pip install omero-py pip install git+https://github.com/$SPACE_USER/omero-metadata.git@$MERGE_PUSH_BRANCH#egg=omero-metadata pip install git+https://github.com/$SPACE_USER/omero-cli-render.git@$MERGE_PUSH_BRANCH#egg=omero-cli-render +pip install git+https://github.com/$SPACE_USER/omero-cli-duplicate.git@$MERGE_PUSH_BRANCH#egg=omero-cli-duplicate pip install omero-certificates for x in *.tar.gz; do @@ -131,33 +146,38 @@ omero config set omero.db.name $OMERO_DB_NAME omero config set omero.db.host $OMERO_DB_HOST omero config set omero.db.user $OMERO_DB_USER omero config set omero.data.dir $OMERO_DATA_DIR -omero config set omero.fs.repo.path "%user%_%userId%/%thread%//%year%-%month%/%day%/%time%" +omero config set omero.fs.repo.path "%user%_%userId%/%thread%//%year%-%month%/%day%/%time%" omero config set omero.db.poolsize 25 omero config set omero.security.trustStore /etc/pki/ca-trust/extracted/java/cacerts omero config set omero.security.trustStorePassword changeit +omero config set omero.mail.config true +# omero config set omero.mail.from your_address +# omero config set omero.mail.host your_smtp_server_for_example omero certificates -if [ "$PURGE_DATA" = "true" ]; then +if [ "$PURGE_DATA" = "true" ]; then createdb -h $OMERO_DB_HOST -U $OMERO_DB_USER $OMERO_DB_NAME omego db init --serverdir $OMERO_DIST --dbname $OMERO_DB_NAME else - echo "Skipping PURGE_DATA:CleanDbAndRepo. Check for DB upgrade" + echo "Skipping PURGE_DATA:CleanDbAndRepo. Check for DB upgrade" omego db -vvv upgrade --serverdir $OMERO_DIST --dbname $OMERO_DB_NAME fi -export LANG='en_US.UTF-8' -export LANGUAGE='en_US:en' -export LC_ALL='en_US.UTF-8' +export LANG='en_US.UTF-8' +export LANGUAGE='en_US:en' +export LC_ALL='en_US.UTF-8' BUILD_ID=DONT_KILL_ME omero admin start omero admin waitup omero admin diagnostics -deactivate +deactivate + + - + \ No newline at end of file diff --git a/home/jobs/OMERO-test-integration/config.xml b/home/jobs/OMERO-test-integration/config.xml index 7f52db9a..dd480de5 100644 --- a/home/jobs/OMERO-test-integration/config.xml +++ b/home/jobs/OMERO-test-integration/config.xml @@ -15,26 +15,22 @@ rm -rf $WORKSPACE/.venv3 -python -m venv $WORKSPACE/.venv3 +python3 -m venv $WORKSPACE/.venv3 source $WORKSPACE/.venv3/bin/activate -pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.3.0/zeroc_ice-3.6.5-cp38-cp38-linux_x86_64.whl +pip install https://github.com/glencoesoftware/zeroc-ice-py-rhel9-x86_64/releases/download/20230830/zeroc_ice-3.6.5-cp39-cp39-linux_x86_64.whl pip install -U pip future setuptools pip install markdown -# Avoid installing psutil for now -# https://github.com/pytest-dev/pytest-xdist/issues/585 -pip install mox3 pytest pytest-django 'pytest-xdist<2' +pip install mox3 pytest pytest-django pytest-xdist pytest-mock pip install tables pip install jinja2 pip install PyYAML -pip install "urllib3<2" # to be removed if/when included in omero-py pip install omero-py omero-web # Latest in order to stop server. -OMERO_INSTALL=/tmp/omero-install/linux -source $OMERO_INSTALL/settings.env +source /home/settings.env OMERO_DB_NAME=$JOB_NAME SRC=$WORKSPACE/src OMERO_DIST=$SRC/dist @@ -52,7 +48,7 @@ dropdb -f -h $OMERO_DB_HOST -U $OMERO_DB_USER $OMERO_DB_NAME || echo Already exi rm -rf $WORKSPACE/* - + OMERO-build src/target/openmicroscopy*zip @@ -61,7 +57,7 @@ rm -rf $WORKSPACE/* true false - + OMERO-python-superbuild-build **/*.tar.gz @@ -70,10 +66,21 @@ rm -rf $WORKSPACE/* true false + + OMERO-build-build + + + false + + + true + false + -OMERO_INSTALL=/tmp/omero-install/linux -source $OMERO_INSTALL/settings.env +source /home/settings.env +export ZIP_FILE=$WORKSPACE/omero-blitz-VERSION-python.zip +export VERSION_PROPERTIES=$WORKSPACE/version.properties OMERO_DB_NAME=$JOB_NAME OMERO_DATA_DIR=$WORKSPACE/data diff --git a/home/jobs/OMERO-training/config.xml b/home/jobs/OMERO-training/config.xml index 9ab7987d..9e265d4c 100644 --- a/home/jobs/OMERO-training/config.xml +++ b/home/jobs/OMERO-training/config.xml @@ -15,10 +15,10 @@ rm -rf $WORKSPACE/.venv3 -python -m venv $WORKSPACE/.venv3 +python3 -m venv $WORKSPACE/.venv3 source $WORKSPACE/.venv3/bin/activate -pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.2.1/zeroc_ice-3.6.5-cp36-cp36m-linux_x86_64.whl +pip install https://github.com/glencoesoftware/zeroc-ice-py-rhel9-x86_64/releases/download/20230830/zeroc_ice-3.6.5-cp39-cp39-linux_x86_64.whl pip install -U pip future setuptools pip install markdown # Avoid installing psutil for now @@ -29,7 +29,7 @@ pip install jinja2 rm -rf $WORKSPACE/* - + OMERO-python-superbuild-build **/*.gz @@ -38,7 +38,7 @@ rm -rf $WORKSPACE/* true false - + OMERO-build src/target/OMERO.server*zip,src/target/openmicroscopy*zip @@ -47,10 +47,22 @@ rm -rf $WORKSPACE/* true false + + OMERO-build-build + + + false + + + true + false + - OMERO_INSTALL=/tmp/omero-install/linux -source $OMERO_INSTALL/settings.env + +source /home/settings.env +export ZIP_FILE=$WORKSPACE/omero-blitz-VERSION-python.zip +export VERSION_PROPERTIES=$WORKSPACE/version.properties # UNZIP openmicroscopy OMERO_EXAMPLE_DIST=$WORKSPACE/openmicroscopy diff --git a/home/jobs/OMERO-web/config.xml b/home/jobs/OMERO-web/config.xml index 8d9d3155..347233a6 100644 --- a/home/jobs/OMERO-web/config.xml +++ b/home/jobs/OMERO-web/config.xml @@ -26,13 +26,12 @@ rm -rf $WORKSPACE/.venv3 -python -m venv $WORKSPACE/.venv3 +python3 -m venv $WORKSPACE/.venv3 source $WORKSPACE/.venv3/bin/activate -pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.3.0/zeroc_ice-3.6.5-cp38-cp38-linux_x86_64.whl +pip install https://github.com/glencoesoftware/zeroc-ice-py-rhel9-x86_64/releases/download/20230830/zeroc_ice-3.6.5-cp39-cp39-linux_x86_64.whl pip install -U pip future pip install markdown -pip install "urllib3<2" # to be removed if/when included in omero-py pip install omero-py omero-web # Latest in order to stop server. NB: Re-installed from python-superbuild below @@ -53,20 +52,32 @@ fi rm -rf $HOME/static/web/ rm -rf $WORKSPACE/* - + OMERO-build src/target/OMERO.server*zip - - + + false + true false - + OMERO-python-superbuild-build - **/*.gz - - + **/*.tar.gz + + false + + + true + false + + + OMERO-build-build + + + false + true false @@ -75,7 +86,8 @@ rm -rf $WORKSPACE/* OMERO_DIST=$WORKSPACE/OMERO.web export OMERODIR=$OMERO_DIST -OMERO_INSTALL=/tmp/omero-install/linux +export ZIP_FILE=$WORKSPACE/omero-blitz-VERSION-python.zip +export VERSION_PROPERTIES=$WORKSPACE/version.properties ZIP=$(ls OMERO.server*.zip) if [ ! -z "$OMERO_PY_ZIP_URL" ] @@ -93,7 +105,7 @@ mv $WORKSPACE/$DIST $OMERO_DIST source $WORKSPACE/.venv3/bin/activate -source $OMERO_INSTALL/settings.env +source /home/settings.env # Install omero-py, omero-web etc from OMERO-python-superbuild-build job for x in *.tar.gz; do @@ -182,13 +194,16 @@ pip install git+https://github.com/$SPACE_USER/omero-webtest.git@$MERGE_PUSH_BRA omero config append omero.web.apps '"omero_webtest"' # OMERO.omero-webtagging-autotag installation -pip install omero-webtagging-autotag -omero config append omero.web.apps '"omero_webtagging_autotag"' -omero config append omero.web.ui.center_plugins '["Auto Tag", "omero_webtagging_autotag/auto_tag_init.js.html", "auto_tag_panel"]' +pip install omero-autotag +# Add autotag to webclient +omero config append omero.web.apps '"omero_autotag"' +# Add autotag to centre panel +omero config append omero.web.ui.center_plugins '["Auto Tag", "omero_autotag/auto_tag_init.js.html", "auto_tag_panel"]' + +pip install omero-tagsearch +# Add tagsearch to webclient +omero config append omero.web.apps '"omero_tagsearch"' -# OMERO.webtagging-tagsearch installation -pip install omero-webtagging-tagsearch -omero config append omero.web.apps '"omero_webtagging_tagsearch"' omero config append omero.web.ui.top_links '["Tag Search", "tagsearch"]' # omero-web-zarr diff --git a/jenkins/Dockerfile b/jenkins/Dockerfile index ba95f0e3..3964a0d7 100644 --- a/jenkins/Dockerfile +++ b/jenkins/Dockerfile @@ -1,4 +1,4 @@ -FROM jenkins/jenkins:2.375.1-lts +FROM jenkins/jenkins:2.426.3 MAINTAINER OME # Temp fix robot test results diff --git a/nginx/Dockerfile b/nginx/Dockerfile index edcc82e0..1bbbf2a5 100644 --- a/nginx/Dockerfile +++ b/nginx/Dockerfile @@ -1,18 +1,16 @@ -FROM openmicroscopy/devslave-c7:0.7.2 +FROM openmicroscopy/devagent:0.8.1 MAINTAINER OME -# manually install java11 -RUN yum -y install java-11-openjdk-devel -# Switch Java version -ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk -ENV PATH=$JAVA_HOME/bin:$PATH - -RUN yum -y localinstall http://nginx.org/packages/rhel/7/noarch/RPMS/nginx-release-rhel-7-0.el7.ngx.noarch.rpm \ - && yum clean all -RUN yum -y install nginx --disablerepo=epel \ - && yum clean all -RUN mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.disabled +COPY nginx-stable.repo /etc/yum.repos.d/ +RUN dnf -y install dnf-utils +RUN yum-config-manager --enable nginx-stable + +RUN dnf -y install nginx\ + && dnf clean all + +# comment out default server config in nginx.conf +RUN sed -i "/^\s\s\s\sserver/, /^\s\s\s\s}/ s|^|#|" /etc/nginx/nginx.conf VOLUME ["/etc/nginx/conf.d", "/var/log/nginx"] diff --git a/nginx/nginx-stable.repo b/nginx/nginx-stable.repo new file mode 100644 index 00000000..3c2d4b16 --- /dev/null +++ b/nginx/nginx-stable.repo @@ -0,0 +1,7 @@ +[nginx-stable] +name=nginx stable repo +baseurl=http://nginx.org/packages/centos/$releasever/$basearch/ +gpgcheck=1 +enabled=1 +gpgkey=https://nginx.org/keys/nginx_signing.key +module_hotfixes=true diff --git a/rename.py b/rename.py index 628022ce..aeed195e 100755 --- a/rename.py +++ b/rename.py @@ -12,6 +12,7 @@ def replace(name, branch, uid, user): cnt = 0 + branch_ci = branch + "-ci" for root, dirs, files in os.walk("."): dirs[:] = list([x for x in dirs if not x in EXCLUDE]) env = list(fnmatch.filter(files, ".env")) @@ -26,6 +27,7 @@ def replace(name, branch, uid, user): regexp = re.compile(r'(SPACE[NAME|BRANCH|USER]|1000)') if regexp.search(line) is not None: cnt += 1 + line = line.replace("BRANCHCI", branch_ci) line = line.replace("SPACENAME", name) line = line.replace("SPACEUSER", user) line = line.replace("1000", str(uid)) diff --git a/runtest.sh b/runtest.sh index 15b9f465..9873cb99 100755 --- a/runtest.sh +++ b/runtest.sh @@ -5,13 +5,13 @@ set -e -u -x source .env # start docker container -docker-compose -f docker-compose.yml up -d +docker compose -f docker-compose.yml up -d # inspect containers -service_containers=( devspace_pg_1 devspace_redis_1 ) -selenium_containers=( devspace_seleniumhub_1 devspace_seleniumfirefox_1 devspace_seleniumchrome_1 ) -omero_containers=( devspace_omero_1 devspace_web_1 devspace_nginx_1 devspace_testintegration_1 ) -jenkins_containers=( devspace_jenkins_1 devspace_nginxjenkins_1 ) +service_containers=( devspace-pg-1 devspace-redis-1 ) +selenium_containers=( devspace-seleniumhub-1 devspace-seleniumfirefox-1 devspace-seleniumchrome-1 ) +omero_containers=( devspace-omero-1 devspace-web-1 devspace-nginx-1 devspace-testintegration-1 ) +jenkins_containers=( devspace-jenkins-1 devspace-nginxjenkins-1 ) all_containers=( "${service_containers[@]}" "${selenium_containers[@]}" "${omero_containers[@]}" "${jenkins_containers[@]}") for cname in "${all_containers[@]}" @@ -22,11 +22,11 @@ done # check if Jenkins is fully up and running d=10 -while ! docker logs devspace_jenkins_1 2>&1 | grep "Jenkins is fully up and running" +while ! docker logs devspace-jenkins-1 2>&1 | grep "Jenkins is fully up and running" do sleep 10 d=$[$d -1] if [ $d -lt 0 ]; then - docker logs devspace_jenkins_1 + docker logs devspace-jenkins-1 exit 1 fi done @@ -38,11 +38,11 @@ do SLAVE_ADDR=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' $cname` echo "Checking $cname $SLAVE_ADDR is connected to jenkins" d=10 - while ! docker logs devspace_jenkins_1 2>&1 | grep "from /${SLAVE_ADDR}" + while ! docker logs devspace-jenkins-1 2>&1 | grep "from /${SLAVE_ADDR}" do sleep 10 d=$[$d -1] if [ $d -lt 0 ]; then - docker logs devspace_jenkins_1 + docker logs devspace-jenkins-1 docker logs $cname exit 1 fi @@ -50,7 +50,7 @@ do done -JENKINS_PORT=$(docker-compose port nginxjenkins 80 | cut -d: -f2) +JENKINS_PORT=$(docker compose port nginxjenkins 80 | cut -d: -f2) curl -L -k -I http://localhost:$JENKINS_PORT$JENKINS_PREFIX STATUS=$(curl -L -k --write-out %{http_code} --silent --output /dev/null http://localhost:$JENKINS_PORT$JENKINS_PREFIX) @@ -60,5 +60,5 @@ if [ ! "200" == "$STATUS" ]; then fi # CLEANUP -docker-compose -f docker-compose.yml stop -docker-compose -f docker-compose.yml rm -f +docker compose -f docker-compose.yml stop +docker compose -f docker-compose.yml rm -f diff --git a/server/Dockerfile b/server/Dockerfile index 1dd90005..01a2fb1e 100644 --- a/server/Dockerfile +++ b/server/Dockerfile @@ -1,93 +1,65 @@ -FROM openmicroscopy/devslave-c7:0.7.2 +FROM openmicroscopy/devagent:0.8.1 MAINTAINER OME -ARG JAVAVER=openjdk11-devel -ARG ICEVER=ice36 -ARG BRANCH=scl_c7 -ARG NAME=ome - -# Remove the existing omero user -RUN userdel -r omero && \ - rm -f /etc/sudoers.d/omero && \ - echo 'root:omero' | chpasswd - -# skip some omero-install -RUN echo 'export container=docker' > /etc/profile.d/docker.sh - -ENV OMERO_INSTALL_ROOT=/tmp/omero-install - -## update omero-install to the latest one -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT config --global user.email "you@example.com" -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT config --global user.name "Your Name" -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT remote add $NAME https://github.com/$NAME/omero-install.git -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT fetch $NAME -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT merge $NAME/$BRANCH - - -ADD ./settings.env $OMERO_INSTALL/settings.env -RUN chmod +x $OMERO_INSTALL/settings.env - -# Customize: run particular scripts from omero-install -RUN rm -f $OMERO_INSTALL/setup_centos_selinux.sh -RUN touch $OMERO_INSTALL/setup_centos_selinux.sh -# db will be managed by the job -RUN rm -f $OMERO_INSTALL/setup_omero_db.sh -RUN touch $OMERO_INSTALL/setup_omero_db.sh -RUN rm -f $OMERO_INSTALL/step03_all_postgres.sh -RUN touch $OMERO_INSTALL/step03_all_postgres.sh -# omeroweb will be installed by the job -RUN rm -f $OMERO_INSTALL/step04_all_omero.sh -RUN touch $OMERO_INSTALL/step04_all_omero.sh -# no systemd -RUN rm -rf $OMERO_INSTALL/step06_centos7_daemon.sh -RUN touch $OMERO_INSTALL/step06_centos7_daemon.sh -# no Pillow -RUN rm -rf $OMERO_INSTALL/requirements.txt -RUN touch $OMERO_INSTALL/requirements.txt + +ADD ./settings.env /home/settings.env +RUN chmod +x /home/settings.env WORKDIR /tmp/omero-install/linux -RUN JAVAVER=$JAVAVER ICEVER=$ICEVER PGVER=nopg bash install_centos7.sh -# install postgres tools -RUN yum -y install postgresql13 \ - && yum clean all +RUN dnf install -y python3 python3-pip +RUN pip install build + +# Ice dependencies +RUN dnf install -y 'dnf-command(config-manager)' && \ + dnf config-manager --set-enabled crb +RUN dnf install -y openssl libdb-cxx +RUN cd /tmp && \ + wget -q https://github.com/glencoesoftware/zeroc-ice-rhel9-x86_64/releases/download/20230928/Ice-3.6.5-rhel9-x86_64.tar.gz && \ + tar xf Ice-3.6.5-rhel9-x86_64.tar.gz && \ + mv Ice-3.6.5 /opt/ice-3.6.5 && \ + echo /opt/ice-3.6.5/lib64 > /etc/ld.so.conf.d/ice-x86_64.conf && \ + ldconfig +ENV ICE_HOME=/opt/ice-3.6.5 + +ENV PATH=$ICE_HOME/bin:$PATH + +# install postgres tools for pg16 (default is 13) +RUN dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm +RUN dnf -qy module disable postgresql -# Switch Java version -ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk -ENV PATH=$JAVA_HOME/bin:$PATH +RUN dnf -y install postgresql16 \ + && dnf clean all +# For OME-zarr +RUN dnf -y install epel-release +RUN dnf -y update +RUN dnf -y install blosc EXPOSE 4064 EXPOSE 4063 ADD ./run.sh /tmp/run.sh -RUN chown omero-server:omero-server /tmp/run.sh +RUN chown omero:omero /tmp/run.sh RUN chmod a+x /tmp/run.sh -ENV PATH=/opt/rh/rh-python38/root/usr/bin:$PATH -RUN python3 -m venv /py3 && /py3/bin/pip install -U pip tox build future wheel restructuredtext-lint -RUN /py3/bin/pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.3.0/zeroc_ice-3.6.5-cp38-cp38-linux_x86_64.whl -ENV VIRTUAL_ENV=/py3 -ENV PATH="$VIRTUAL_ENV/bin:$PATH" # Install mencoder # See https://github.com/ome/prod-playbooks/blob/7d8e59ced0b47a224d6be02907d913c392ef9379/ome-dundeeomero.yml#L48 -RUN yum install -y http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm -RUN yum install -y mencoder +# RUN dnf install -y http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm +# RUN dnf install -y mencoder -RUN yum install -y openssl - -RUN localedef -i en_US -f UTF-8 en_US.UTF-8 -ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8' # Change user id to fix permissions issues ARG USER_ID=1000 -RUN usermod -u $USER_ID omero-server +RUN usermod -u $USER_ID omero + + # make sure mounted volumes has correct permissions -VOLUME ["/home/omero-server"] +VOLUME ["/home/omero"] -USER omero-server +USER omero CMD ["/tmp/run.sh"] diff --git a/server/run.sh b/server/run.sh index 5c14e8dd..979e1e6e 100644 --- a/server/run.sh +++ b/server/run.sh @@ -1,12 +1,17 @@ #!/bin/bash - +sudo chmod 777 -R /home/omero +workspace=/home/omero/workspace/OMERO-server function shut_down() { - /home/omero-server/workspace/OMERO-server/OMERO.server/bin/omero admin stop + if [ -d "$workspace" ]; then + $workspace/OMERO.server/bin/omero admin stop + fi } -/tmp/jenkins-slave.sh & -/home/omero-server/workspace/OMERO-server/OMERO.server/bin/omero admin start -/home/omero-server/workspace/OMERO-server/OMERO.server/bin/omero admin diagnostics +/tmp/jenkins-slave.sh +if [ -d "$workspace" ]; then + $workspace/OMERO.server/bin/omero admin start + $workspace/OMERO.server/bin/omero admin diagnostics +fi trap "shut_down" SIGKILL SIGTERM SIGHUP SIGINT EXIT diff --git a/server/settings.env b/server/settings.env index 39484d40..5b39b7d9 100644 --- a/server/settings.env +++ b/server/settings.env @@ -5,4 +5,3 @@ OMERO_ROOT_PASS=omero OMERO_DATA_DIR=/OMERO export OMERO_DB_USER OMERO_DB_PASS OMERO_ROOT_PASS OMERO_DATA_DIR -export PATH=/opt/rh/rh-python38/root/usr/bin:$PATH diff --git a/slave/Dockerfile b/slave/Dockerfile index 7160b2af..9aa7052d 100644 --- a/slave/Dockerfile +++ b/slave/Dockerfile @@ -1,50 +1,36 @@ -FROM openmicroscopy/devslave-c7:0.7.2 +FROM openmicroscopy/devagent:0.8.1 MAINTAINER OME -ARG JAVAVER=openjdk11-devel -ARG ICEVER=ice36-devel -ARG BRANCH=scl_c7 ARG NAME=ome ARG GRADLEVER=6.9.3 -# skip some omero-install -RUN echo 'export container=docker' > /etc/profile.d/docker.sh - -ENV OMERO_INSTALL_ROOT=/tmp/omero-install -## update omero-install to the latest one -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT config --global user.email "you@example.com" -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT config --global user.name "Your Name" -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT remote add $NAME https://github.com/$NAME/omero-install.git -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT fetch $NAME -RUN git --git-dir=$OMERO_INSTALL_ROOT/.git --work-tree=$OMERO_INSTALL_ROOT merge $NAME/$BRANCH - - -ADD ./settings.env $OMERO_INSTALL/settings.env -RUN chmod +x $OMERO_INSTALL/settings.env - -# Customize: run particular scripts from omero-install -RUN rm -f $OMERO_INSTALL/setup_centos_selinux.sh -RUN touch $OMERO_INSTALL/setup_centos_selinux.sh -# db will be managed by the job -RUN rm -f $OMERO_INSTALL/setup_omero_db.sh -RUN touch $OMERO_INSTALL/setup_omero_db.sh -RUN rm -f $OMERO_INSTALL/step03_all_postgres.sh -RUN touch $OMERO_INSTALL/step03_all_postgres.sh -# omeroweb will be installed by the job -RUN rm -f $OMERO_INSTALL/step04_all_omero.sh -RUN touch $OMERO_INSTALL/step04_all_omero.sh -# no systemd -RUN rm -rf $OMERO_INSTALL/step06_centos7_daemon.sh -RUN touch $OMERO_INSTALL/step06_centos7_daemon.sh -# no Pillow -RUN rm -rf $OMERO_INSTALL/requirements.txt -RUN touch $OMERO_INSTALL/requirements.txt - -WORKDIR /tmp/omero-install/linux - -RUN JAVAVER=$JAVAVER ICEVER=$ICEVER PGVER=nopg bash install_centos7.sh +ADD ./settings.env /home/settings.env +RUN chmod +x /home/settings.env + + +# Required for the docs build +RUN dnf install -y make + +# Python job +RUN dnf install -y python3-setuptools python3-pip +RUN pip install build + +# Ice dependencies +RUN dnf install -y 'dnf-command(config-manager)' && \ + dnf config-manager --set-enabled crb +RUN dnf install -y openssl libdb-cxx +RUN cd /tmp && \ + wget -q https://github.com/glencoesoftware/zeroc-ice-rhel9-x86_64/releases/download/20230928/Ice-3.6.5-rhel9-x86_64.tar.gz && \ + tar xf Ice-3.6.5-rhel9-x86_64.tar.gz && \ + mv Ice-3.6.5 /opt/ice-3.6.5 && \ + echo /opt/ice-3.6.5/lib64 > /etc/ld.so.conf.d/ice-x86_64.conf && \ + ldconfig + +ENV ICE_HOME=/opt/ice-3.6.5 + +ENV PATH=$ICE_HOME/bin:$PATH # Install FindBugs ENV FINDBUGS_VERSION 3.0.0 @@ -55,11 +41,13 @@ RUN curl -fSLO http://downloads.sourceforge.net/project/findbugs/findbugs/$FINDB rm findbugs-$FINDBUGS_VERSION.tar.gz # Install build tools from EPEL -RUN yum -y install maven ant && yum clean all +RUN dnf -y install maven ant && dnf clean all -# install postgres tools -RUN yum -y install postgresql13-server postgresql13 \ - && yum clean all +# install postgres tools for pg16 (default is 13) +RUN dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm +RUN dnf -qy module disable postgresql +RUN dnf -y install postgresql16-server postgresql16\ + && dnf clean all # gradle RUN cd /opt && \ @@ -68,8 +56,11 @@ RUN cd /opt && \ rm gradle-$GRADLEVER-bin.zip && \ ln -s /opt/gradle-$GRADLEVER/bin/gradle /usr/local/bin/gradle -ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk -ENV PATH=$JAVA_HOME/bin:$PATH + +# For OME-zarr +RUN dnf -y install epel-release +RUN dnf -y update +RUN dnf -y install blosc EXPOSE 14064 EXPOSE 14063 @@ -78,13 +69,6 @@ ADD ./run.sh /tmp/run.sh RUN chown omero:omero /tmp/run.sh RUN chmod a+x /tmp/run.sh -ENV PATH=/opt/rh/rh-python38/root/usr/bin:$PATH -RUN python3 -m venv /py3 && /py3/bin/pip install -U pip tox build future wheel restructuredtext-lint -RUN /py3/bin/pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.3.0/zeroc_ice-3.6.5-cp38-cp38-linux_x86_64.whl -RUN ln -s /py3/bin/slice2py /usr/bin/slice2py -ENV VIRTUAL_ENV=/py3 -ENV PATH="$VIRTUAL_ENV/bin:$PATH" - # Change user id to fix permissions issues ARG USER_ID=1000 RUN usermod -u $USER_ID omero diff --git a/slave/run.sh b/slave/run.sh index 246019f5..32e76d09 100644 --- a/slave/run.sh +++ b/slave/run.sh @@ -1,12 +1,18 @@ #!/bin/bash - +sudo chmod 777 -R /home/omero +workspace=/home/omero/workspace/OMERO-test-integration function shut_down() { - /home/omero-server/workspace/OMERO-test-integration/src/dist/bin/omero admin stop + if [ -d "$workspace" ]; then + $workspace/src/dist/bin/omero admin stop + fi } -/tmp/jenkins-slave.sh & -/home/omero-server/workspace/OMERO-test-integration/src/dist/bin/omero admin start -/home/omero-server/workspace/OMERO-test-integration/src/dist/bin/omero admin diagnostics +/tmp/jenkins-slave.sh +if [ -d "$workspace" ]; then + + $workspace/src/dist/bin/omero admin start + $workspace/src/dist/bin/omero admin diagnostics +fi trap "shut_down" SIGKILL SIGTERM SIGHUP SIGINT EXIT diff --git a/slave/settings.env b/slave/settings.env index 39484d40..1c053ac7 100644 --- a/slave/settings.env +++ b/slave/settings.env @@ -4,5 +4,4 @@ OMERO_DB_PASS= OMERO_ROOT_PASS=omero OMERO_DATA_DIR=/OMERO -export OMERO_DB_USER OMERO_DB_PASS OMERO_ROOT_PASS OMERO_DATA_DIR -export PATH=/opt/rh/rh-python38/root/usr/bin:$PATH +export OMERO_DB_USER OMERO_DB_PASS OMERO_ROOT_PASS OMERO_DATA_DIR diff --git a/web/Dockerfile b/web/Dockerfile index b82df1de..daed2cd7 100644 --- a/web/Dockerfile +++ b/web/Dockerfile @@ -1,79 +1,42 @@ -FROM openmicroscopy/devslave-c7:0.7.2 +FROM openmicroscopy/devagent:0.8.1 MAINTAINER OME -ARG ICEVER=noice +ADD ./settings.env /home/settings.env +RUN chmod +x /home/settings.env -# make ICEVERSION environment variable to use in OMERO-web job -ENV ICEVERSION ${ICEVER:-noice} - - -# Remove the existing omero user -RUN userdel -r omero && \ - rm -f /etc/sudoers.d/omero && \ - echo 'root:omero' | chpasswd - -# Create the omero-web user -RUN useradd omero-web && \ - echo 'omero-web:omero-web' | chpasswd && \ - echo "omero-web ALL= (ALL) NOPASSWD: ALL" >> /etc/sudoers.d/omero-web - - -# skip some omero-install -RUN echo 'export container=docker' > /etc/profile.d/docker.sh - -ENV OMERO_INSTALL_ROOT=/tmp/omero-install - -ADD ./settings.env $OMERO_INSTALL/settings.env -RUN chmod +x $OMERO_INSTALL/settings.env - -# manually install ICEPY dependences -RUN yum -y install gcc-c++ # ant for iviewer, make for nodejs -RUN yum -y install ant make -RUN yum -y install openssl-devel bzip2-devel expat-devel -RUN curl -sL https://rpm.nodesource.com/setup_16.x | bash -RUN yum install -y nodejs +RUN dnf -y install ant make +RUN curl -sL https://rpm.nodesource.com/setup_20.x | bash +RUN dnf install -y nodejs # grunt for figure RUN npm install -g grunt -RUN yum install -y python-redis && yum clean all - -# manually install java11 -RUN yum -y install java-11-openjdk-devel -# Switch Java version -ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk -ENV PATH=$JAVA_HOME/bin:$PATH EXPOSE 4080 ADD ./run.sh /tmp/run.sh -RUN chown omero-web:omero-web /tmp/run.sh +RUN chown omero:omero /tmp/run.sh RUN chmod a+x /tmp/run.sh -RUN yum -y install centos-release-scl -RUN yum install -y rh-python38-python - -ENV PATH=/opt/rh/rh-python38/root/usr/bin:$PATH +RUN dnf install -y python3 openssl python3-pip +RUN pip install build -RUN python3 -m venv /py3 && /py3/bin/pip install -U pip tox build future wheel restructuredtext-lint -RUN /py3/bin/pip install https://github.com/ome/zeroc-ice-py-centos7/releases/download/0.3.0/zeroc_ice-3.6.5-cp38-cp38-linux_x86_64.whl -ENV VIRTUAL_ENV=/py3 -ENV PATH="$VIRTUAL_ENV/bin:$PATH" +RUN dnf clean all # Change user id to fix permissions issues ARG USER_ID=1000 -RUN usermod -u $USER_ID omero-web +RUN usermod -u $USER_ID omero -RUN chown -R omero-web:omero-web /home/omero-web/ +RUN chown -R omero:omero /home/omero/ # make sure mounted volumes has correct permissions -RUN mkdir -p /home/omero-web/nginx -RUN chown -R omero-web:omero-web /home/omero-web/nginx -RUN mkdir -p /home/omero-web/static -RUN chown -R omero-web:omero-web /home/omero-web/static +RUN mkdir -p /home/omero/nginx +RUN chown -R omero:omero /home/omero/nginx +RUN mkdir -p /home/omero/static +RUN chown -R omero:omero /home/omero/static -VOLUME ["/home/omero-web", "/home/omero-web/nginx"] +VOLUME ["/home/omero", "/home/omero/nginx"] -USER omero-web +USER omero CMD ["/tmp/run.sh"] diff --git a/web/run.sh b/web/run.sh index 76ba7a38..f9437ec0 100644 --- a/web/run.sh +++ b/web/run.sh @@ -1,5 +1,5 @@ #!/bin/bash - +sudo chmod 777 -R /home/omero workspace=/home/omero/workspace/OMERO-web djangopid="$workspace/OMERO.web/var/django.pid" if [ -f $djangopid ]; then