Sync: bring OntoPortal up-to-date with BioPortal releases 5.26.2 and …

…onward (#4) * fix get a submission metrics * Auto stash before merge of "upstream" and "upstream/master" * add the slice get endpoint * add the slices create endpoint * add the slices delete endpoint * add the slices update endpoint * Add caching for analytics for 24 hours. * Fix for #97. Check for ontology existence before brining attributes * Handle edge case for submission downloads which do not have UploadFilePath set Fixes #98 * Gemfile.lock update * Add GH workflow for capistrano deployments * Update Gemfile.lock * fix ability to run deployment manually * Fix for deprecation notice of Rack::Attack.throttled_response Update configuration to closely match rack attack documentation in order to address deprecation notice: [DEPRECATION] Rack::Attack.throttled_response is deprecated. Please use Rack::Attack.throttled_responder instead * Update version of actions/checkout to address deprecation notices * Fix: Documentation rendering (#107) * Auto stash before merge of "upstream" and "upstream/master" * fix haml gem version * Update Gemfile * Restore branch specifier to master * Update capistrano sample config to include setting which branch to deploy from * Gemfile.lock update * Gemfile.lock update * Gemfile.lock update * Gemfile.lock update * Make sure ontology is present when accessing submissions. Fixes an internal server error when accessing submission for non-existent/deleted ontology * Remove owlapi_wrapper.jar file. It is included elsewhere * Add health checks for docker services and add more config file options * Remove wait-for-it * Add health check to AG service * Add GH workflow for publishing docker images * Add missing redis port number * Restore branch specifier to develop * Set mgrep port to 55556 * add arm64 platform * bump up version of solr-ut * Fix ncbo#116 - pinned redis-store to 1.9.1 until #358 gets resolved - fixed deprecation notices "warning: calling URI.open via Kernel#open is deprecated, call URI.open directly or use URI#open" * Restore branch specifier to master * fix for wrongly replaced string * Delete fix_purls.rb fix_purls.rb moved to a rake task under ncbo_cron project * Remove google analytics depenencies since those needed only in ncbo_cron * bump up major version of oj and faraday * remove search_index.rb script * lock gem rack-cache to 1.13.0 see ncbo#118 * remove depenency on redis-activesupport rack-attack can work with redis directly so there is no need to use redis-activesupport which is no longer being actively developed * unpin redis-store solves: ncbo#105 ncbo#106 * use redis-store from forked repo containing redis 5 compat fixes this should be reverted back to original after redis 5 copatibilty issues are resolved * use patched version of agraph v7.3.1 * unpin faraday gem * Gemfile.lock update * fixed an issue with the GA4 Analytics migration * fixed an issue with the GA4 Analytics migration * reduce request limit for resource intensive api calls (#121) * Announce deployments in NewRelic (#124) * Record deployments to NewRelic https://docs.newrelic.com/docs/apm/agents/ruby-agent/features/record-deployments-ruby-agent/ * add newrelic to deployment group github actions deployment doesn't install default group so capistrano fails to find newrelic recepies unless we add it to the deployment group * add rubocop * Gemfile update, goo version including goo#138 and goo#139 * Gemfile.lock update * Gemfile.lock update * Gemfile.lock update * update slice write operation to check if user is admin * Gemfile.lock update * fixed an accidental commit of docker compose file * fixed Gemfile after merging from master * Gemfile.lock update * update Gemfile.lock * Gemfile.lock update * redis-store gem with redis 5 compatibility fix * Gemfile.lock update * make the check_access helper use filter_access if the object is a list * add test for submissions access check with two ontologies private and pubic * check access of ontologies in /ontologies/:acronym/submissions endpoint * Set gem branch specifier to develop * reset branch specifier to master * add the slice get endpoint * add the slices create endpoint * add the slices delete endpoint * add the slices update endpoint * update slice write operation to check if user is admin * add slices creation & deletion unit tests * Gemfile.lock update * Gemfile.lock update * Merged #87 from master * fixed Gemfile after merge * Gemfile.lock update * update Gemfile.lock * Add configurable option for github org where code is deployed from * Gemfile update * Gemfile update * check existance of acroym before fetching details fixes #129 * extract slice tests helper to the parent class for reusability * add a test for the creation of an admin user * enforce the security of admin user creation * enforce user deletion security to be admin only * Gemfile.lock update * update Gemfile.lock * Gemfile update * implemented the first pass at bmir-radx/radx-project#37 * set bundler version to be comptatible with ruby 2.7 * Gemfile.lock update * implemented #127 - Add API call to trigger ontology pull from remote location * Gemfile.lock * implemented a test for #127 - Add API call to trigger ontology pull from remote location * Gemfile.lock update * implemented a test for #127 - Add API call to trigger ontology pull from remote location * implemented a test for #127 - Add API call to trigger ontology pull from remote location * Gemfile.lock update * use agraph v8.0.0 * Gemfile.lock update * Gemfile.lock update --------- Co-authored-by: Alex Skrenchuk <[email protected]> Co-authored-by: mdorf <[email protected]> Co-authored-by: Jennifer Vendetti <[email protected]>
ontoportal · Jan 16, 2024 · 424079c · 424079c
1 parent 3728fab
commit 424079c
Show file tree

Hide file tree

Showing 39 changed files with 2,038 additions and 608 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -1,6 +1,7 @@
 # Git
 .git
 .gitignore
+.github
 create_permissions.log
 # Logs
 log/*
@@ -9,3 +10,8 @@ tmp/*
 # Editor temp files
 *.swp
 *.swo
+coverage
+# Ignore generated test data
+test/data/uploaded_ontologies/**/*
+test/data/ontology_files/repo/**/*
+test/log
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
@@ -0,0 +1,85 @@
+# Workflow for deploying ontologies_api to stage/prod systems via capistrano.
+# This workflow runs after a successeful execution of the unit test workflow and it
+# can also be triggered manually.
+#
+# Required github secrets:
+#
+# CONFIG_REPO - github repo containing config and customizations for the API. Format 'author/private_config_repo'
+# it is used for getting capistrano deployment configuration for stages on the github actions runner and
+# PRIVATE_CONFIG_REPO env var is constructed from it which is used by capistrano on the remote servers for pulling configs.
+#
+# GH_PAT - github Personal Access Token for accessing PRIVATE_CONFIG_REPO
+#
+# SSH_JUMPHOST - ssh jump/proxy host though which deployments have to though if app servers are hosted on private network.
+#
+# DEPLOY_ENC_KEY - key for decrypting deploymnet ssh key residing in config/deploy_id_rsa_enc (see miloserdow/capistrano-deploy)
+# this SSH key is used for accessing jump host, UI nodes, and private github repo.
+
+name: Capistrano Deployment
+# Controls when the action will run.
+on:
+  # Trigger deployment to staging after unit test action completes
+  workflow_run:
+    workflows: ["Ruby Unit Tests"]
+    types:
+      - completed
+    branches: [master, develop]
+  # Allows running this workflow manually from the Actions tab
+  workflow_dispatch:
+    branches: [master, develop]
+    inputs:
+      BRANCH:
+        description: 'Branch/tag to deploy'
+        default: develop
+        required: true
+      environment:
+        description: 'target environment to deploy to'
+        type: choice
+        options:
+          - staging
+          - production
+        default: staging
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    # run deployment only if "Ruby Unit Tests" workflow completes sucessefully or when manually triggered
+    if: ${{ (github.event.workflow_run.conclusion == 'success') || (github.event_name == 'workflow_dispatch') }}
+    env:
+      BUNDLE_WITHOUT: default #install gems required primarily for the deployment in order to speed this workflow
+      PRIVATE_CONFIG_REPO: ${{ format('[email protected]:{0}.git', secrets.CONFIG_REPO) }}
+    # Steps represent a sequence of tasks that will be executed as part of the job
+    steps:
+    - name: set branch/tag and environment to deploy from inputs
+      run: |
+        # workflow_dispatch default input doesn't get set on push so we need to set defaults
+        # via shell parameter expansion
+        # https://dev.to/mrmike/github-action-handling-input-default-value-5f2g
+        USER_INPUT_BRANCH=${{ inputs.branch }}
+        echo "BRANCH=${USER_INPUT_BRANCH:-develop}" >> $GITHUB_ENV
+        USER_INPUT_ENVIRONMENT=${{ inputs.environment }}
+        echo "TARGET=${USER_INPUT_ENVIRONMENT:-staging}" >> $GITHUB_ENV
+    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
+    - uses: actions/checkout@v3
+    - uses: ruby/setup-ruby@v1
+      with:
+        ruby-version: 2.7.6 # Not needed with a .ruby-version file
+        bundler-cache: true # runs 'bundle install' and caches installed gems automatically
+    - name: get-deployment-config
+      uses: actions/checkout@v3
+      with:
+        repository: ${{ secrets.CONFIG_REPO }} # repository containing deployment settings
+        token: ${{ secrets.GH_PAT }} # `GH_PAT` is a secret that contains your PAT
+        path:  deploy_config
+    - name: copy-deployment-config
+      run:  cp -r deploy_config/ontologies_api/* .
+    # add ssh hostkey so that capistrano doesn't complain
+    - name: Add jumphost's hostkey to Known Hosts
+      run: |
+        mkdir -p ~/.ssh
+        ssh-keyscan -H ${{ secrets.SSH_JUMPHOST }} > ~/.ssh/known_hosts
+      shell: bash
+    - uses: miloserdow/capistrano-deploy@master
+      with:
+        target: ${{ env.TARGET }} # which environment to deploy
+        deploy_key: ${{ secrets.DEPLOY_ENC_KEY }} # Name of the variable configured in Settings/Secrets of your github project
diff --git a/.github/workflows/docker-image.yml b/.github/workflows/docker-image.yml
@@ -0,0 +1,42 @@
+name: Docker Image CI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  push_to_registry:
+    name: Push Docker image to Docker Hub
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@v4
+        with:
+          images: bioportal/ontologies_api
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v4
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64
+          build-args: |
+            RUBY_VERSION=2.7
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
diff --git a/.github/workflows/ruby-unit-tests.yml b/.github/workflows/ruby-unit-tests.yml
@@ -7,19 +7,20 @@ on:
 jobs:
   test:
     strategy:
+      fail-fast: false
       matrix:
         backend: ['api', 'api-agraph'] # api runs tests with 4store backend and api-agraph runs with AllegroGraph backend
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Build docker-compose
       run: docker-compose --profile 4store build #profile flag is set in order to build all containers in this step
     - name: Run unit tests
       # unit tests are run inside a container
       # http://docs.codecov.io/docs/testing-with-docker
       run: |
         ci_env=`bash <(curl -s https://codecov.io/env)`
-        docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} wait-for-it solr-ut:8983 -- bundle exec rake test TESTOPTS='-v'
+        docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} bundle exec rake test TESTOPTS='-v'
     - name: Upload coverage reports to Codecov
       uses: codecov/codecov-action@v3
       with:

diff --git a/.gitignore b/.gitignore
@@ -68,3 +68,5 @@ test/test_run.log
 test/data/ontology_files/catalog-v001.xml
 
 create_permissions.log
+
+ontologies_api.iml
diff --git a/Capfile b/Capfile
@@ -22,6 +22,6 @@ require 'capistrano/bundler'
 # require 'capistrano/rails/assets'
 # require 'capistrano/rails/migrations'
 require 'capistrano/locally'
-
+require 'new_relic/recipes' # announce deployments in NewRelic
 # Loads custom tasks from `lib/capistrano/tasks' if you have any defined.
 Dir.glob('lib/capistrano/tasks/*.cap').each { |r| import r }
diff --git a/Dockerfile b/Dockerfile
@@ -6,7 +6,6 @@ FROM ruby:$RUBY_VERSION-$DISTRO_NAME
 RUN apt-get update -yqq && apt-get install -yqq --no-install-recommends \
   openjdk-11-jre-headless \
   raptor2-utils \
-  wait-for-it \
   && rm -rf /var/lib/apt/lists/*
 
 RUN mkdir -p /srv/ontoportal/ontologies_api
@@ -15,12 +14,16 @@ COPY Gemfile* /srv/ontoportal/ontologies_api/
 
 WORKDIR /srv/ontoportal/ontologies_api
 
-RUN gem update --system
+# set rubygem and bundler to the last version supported by ruby 2.7
+# remove version after ruby v3 upgrade
+RUN gem update --system '3.4.22'
+RUN gem install bundler -v 2.4.22
 RUN gem install bundler
 ENV BUNDLE_PATH=/srv/ontoportal/bundle
 RUN bundle install
 
 COPY . /srv/ontoportal/ontologies_api
+RUN cp /srv/ontoportal/ontologies_api/config/environments/config.rb.sample /srv/ontoportal/ontologies_api/config/environments/development.rb
 
 EXPOSE 9393
 CMD ["bundle", "exec", "rackup", "-p", "9393", "--host", "0.0.0.0"]
diff --git a/Gemfile b/Gemfile
@@ -1,12 +1,11 @@
 source 'https://rubygems.org'
 
-gem 'activesupport', '~> 3.0'
+gem 'activesupport', '~> 3.2'
 # see https://github.com/ncbo/ontologies_api/issues/69
 gem 'bigdecimal', '1.4.2'
-gem 'faraday', '~> 1.9'
 gem 'json-schema', '~> 2.0'
 gem 'multi_json', '~> 1.0'
-gem 'oj', '~> 2.0'
+gem 'oj', '~> 3.0'
 gem 'parseconfig'
 gem 'rack'
 gem 'rake', '~> 10.0'
@@ -18,7 +17,7 @@ gem 'sinatra-contrib', '~> 1.0'
 gem 'ffi'
 gem 'rack-accept', '~> 0.4'
 gem 'rack-attack', '~> 6.6.1', require: 'rack/attack'
-gem 'rack-cache', '~> 1.0'
+gem 'rack-cache', '~> 1.13.0' # see https://github.com/ncbo/ontologies_api/issues/118
 gem 'rack-cors', require: 'rack/cors'
 # GitHub dependency can be removed when https://github.com/niko/rack-post-body-to-params/pull/6 is merged and released
 gem 'rack-post-body-to-params', github: 'palexander/rack-post-body-to-params', branch: 'multipart_support'
@@ -27,18 +26,18 @@ gem 'redis-rack-cache', '~> 2.0'
 
 # Data access (caching)
 gem 'redis'
-gem 'redis-activesupport'
+gem 'redis-store', '~>1.10'
 
 # Monitoring
 gem 'cube-ruby', require: 'cube'
-gem 'newrelic_rpm'
+gem 'newrelic_rpm', group: [:default, :deployment]
 
 # HTTP server
 gem 'unicorn'
 gem 'unicorn-worker-killer'
 
 # Templating
-gem 'haml'
+gem 'haml', '~> 5.2.2' # pin see https://github.com/ncbo/ontologies_api/pull/107
 gem 'redcarpet'
 
 # NCBO
@@ -50,12 +49,19 @@ gem 'ontologies_linked_data', github: 'ncbo/ontologies_linked_data', branch: 'ma
 gem 'sparql-client', github: 'ncbo/sparql-client', branch: 'master'
 
 group :development do
+  # bcrypt_pbkdf and ed35519 is required for capistrano deployments when using ed25519 keys; see https://github.com/miloserdow/capistrano-deploy/issues/42
+  gem 'shotgun', github: 'palexander/shotgun', branch: 'ncbo'
+  gem 'rubocop'
+end
+
+group :deployment do
+  # bcrypt_pbkdf and ed35519 is required for capistrano deployments when using ed25519 keys; see https://github.com/miloserdow/capistrano-deploy/issues/42
+  gem 'bcrypt_pbkdf', '>= 1.0', '< 2.0', require: false
   gem 'capistrano', '~> 3', require: false
   gem 'capistrano-bundler', require: false
   gem 'capistrano-locally', require: false
   gem 'capistrano-rbenv', require: false
-  gem 'pry'
-  gem 'shotgun', github: 'palexander/shotgun', branch: 'ncbo'
+  gem 'ed25519', '>= 1.2', '< 2.0', require: false
 end
 
 group :profiling do
Original file line number	Diff line number	Diff line change
Expand Up		@@ -68,3 +68,5 @@ test/test_run.log
		test/data/ontology_files/catalog-v001.xml

		create_permissions.log

		ontologies_api.iml