Support local hoster (#401)

* beefing up local hoster implementation to keep merge requests on disk

* remove pytest-cov

* making e2e tests independent of a local gitlab instance

Author: defreng

.github/workflows/ci.yml

@@ -66,8 +66,8 @@ jobs:
         run: python -m mypy src tests
 
 
-  unit-tests:
-    name: unit tests on ${{ matrix.python-version }}
+  tests:
+    name: tests on ${{ matrix.python-version }}
     runs-on: ubuntu-22.04
     strategy:
       fail-fast: false
@@ -78,7 +78,7 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v4
         with:
-          python-version: ${{ env.PYTHON_LATEST }}
+          python-version: ${{ matrix.python-version }}
       - uses: snok/install-poetry@v1
         with:
           version: ${{ env.POETRY_VERSION }}
@@ -87,92 +87,15 @@ jobs:
           poetry self add "poetry-dynamic-versioning[plugin]"
           poetry install
 
-      - run: python -m pytest -m 'not e2e'
+      - run: coverage run -m pytest
         env:
           # FoxOps test configuration
-          FOXOPS_GITLAB_ADDRESS: https://nonsense.com/api/v4
-          FOXOPS_GITLAB_TOKEN: nonsense
-
-          # Test runner configuration
-          COVERAGE_FILE: .coverage.unit.${{ matrix.python-version }}
-
-      - name: Upload coverage data
-        uses: actions/upload-artifact@v3
-        with:
-          name: coverage-data
-          path: .coverage.*
-          if-no-files-found: ignore
-
-
-  e2e-tests:
-    name: e2e tests on ${{ matrix.python-version }}
-    runs-on: ubuntu-22.04
-    strategy:
-      fail-fast: false
-      matrix:
-        python-version: ["3.11"]
-
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
-        with:
-          python-version: ${{ env.PYTHON_LATEST }}
-      - uses: snok/install-poetry@v1
-        with:
-          version: ${{ env.POETRY_VERSION }}
-      - name: Install dependencies
-        run: |
-          poetry self add "poetry-dynamic-versioning[plugin]"
-          poetry install
-
-      - name: Start GitLab test instance
-        run: |
-          docker compose up -d
-          ./scripts/await-healthy.sh
-
-      - run: python -m pytest -m 'e2e'
-        env:
-          # FoxOps test configuration
-          FOXOPS_GITLAB_ADDRESS: https://nonsense.com/api/v4
-          FOXOPS_GITLAB_TOKEN: nonsense
-
-          # Test runner configuration
-          COVERAGE_FILE: .coverage.e2e.${{ matrix.python-version }}
-
-      - name: Upload coverage data
-        uses: actions/upload-artifact@v3
-        with:
-          name: coverage-data
-          path: .coverage.*
-          if-no-files-found: ignore
-
-
-  coverage:
-    name: Combine & check coverage
-    runs-on: ubuntu-latest
-    needs: [unit-tests, e2e-tests]
-
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
-        with:
-          # Use latest Python, so it understands all syntax.
-          python-version: ${{env.PYTHON_LATEST}}
-      - run: python -m pip install --upgrade coverage[toml]
-      - uses: actions/download-artifact@v3
-        with:
-          name: coverage-data
-      - name: Combine coverage & fail if it's <70%.
-        run: |
-          python -m coverage combine
-          python -m coverage html --skip-covered --skip-empty
-          python -m coverage report --fail-under=70
-      - name: Upload HTML report
-        uses: actions/upload-artifact@v3
-        with:
-          name: html-report
-          path: htmlcov
+          FOXOPS_TESTS_GITLAB_ADDRESS: https://gitlab.com
+          FOXOPS_TESTS_GITLAB_ROOT_GROUP_ID: 73622910
+          FOXOPS_TESTS_GITLAB_TOKEN: ${{ secrets.FOXOPS_TESTS_GITLAB_TOKEN }}
 
+      - name: Check test coverage
+        run: coverage report
 
   package:
     name: Build & verify package

.gitignore

@@ -1,9 +1,11 @@
 .python-version
 __pycache__/
 dist/
+run/
 .coverage*
 docs/build/
 .idea
 test.db
 **/.DS_Store
 .dmypy.json
+.env.test

CONTRIBUTING.md

@@ -4,45 +4,80 @@ This project uses [Poetry](https://python-poetry.org/)
 and [poetry-dynamic-versioning](https://pypi.org/project/poetry-dynamic-versioning/)
 for dependency management and the package building process.
 
-## Running tests
+## Run foxops locally
 
-The test suite uses `pytest` as a test runner and it's located under `tests/`.
+### With docker-compose
 
-The unit tests can be executed by excluding the `e2e` tests:
+The easiest way to get started is to run foxops locally via docker-compose. Just execute this command in the root folder of the project:
 
-```
-pytest -m 'not e2e'
+```shell
+docker compose up
 ```
 
-which doesn't require any external database nor GitLab instance.
+It will build the docker image and run foxops with an SQlite database and a local hoster configuration.
 
-To run the `e2e` tests a test GitLab instance needs to be available. It can be started using `docker compose`:
+Now foxops can be accessed at `http://localhost:8000` with `dummy` as the token.
 
-```
-docker compose up -d
+### Directly with Python
+
+First, start by making sure that your virtual Python environment is up to date by running
+
+```shell
+poetry install
 ```
 
-Be aware that an **initial startup of the Gitlab instance can take some time** (5 minutes). Check the logs with `docker-compose logs` to verify it if reached a stable state.
+Then, you can run foxops with
 
-Then, the tests can be run using `pytest`:
+```shell
+export FOXOPS_DATABASE_URL=sqlite+aiosqlite:///./foxops.db
+export FOXOPS_HOSTER_TYPE=local
+export FOXOPS_HOSTER_LOCAL_DIRECTORY=<path to a local directory where foxops can store the repositories>
+export FOXOPS_STATIC_TOKEN=dummy
 
-```
-pytest -m 'e2e'
+# initialize sqlite database in ./foxops.db
+poetry run alembic upgrade head
+
+# run foxops webserver
+poetry run uvicorn foxops.__main__:create_app --host localhost --port 5001 --reload --factory
 ```
 
-## Running foxops locally
+## Running Tests
 
-The foxops API can be run locally using `uvicorn`:
+The test suite uses `pytest` as a test runner and it's located under `tests/`.
 
-```
-uvicorn foxops.__main__:create_app --host localhost --port 5001 --reload --factory
+Simply execute the following commands to run the entire foxops test suite:
+
+```shell
+pytest
 ```
 
-For this to work foxops needs a few configuration settings to be available.
-These are at least a GitLab address and token. To use the test instance you can run the following
+Tests can also run with parallelization enabled to speed up execution. To do so, simply add the `-n` flag with the number of parallel processes to use:
 
+```shell
+pytest -n 4
 ```
-FOXOPS_STATIC_TOKEN=dummy FOXOPS_GITLAB_ADDRESS=http://localhost:5002/api/v4 FOXOPS_GITLAB_TOKEN=ACCTEST1234567890123 uvicorn foxops.__main__:create_app --host localhost --port 5001 --reload --factory
+
+Some tests require a Gitlab instance to be available and will be skipped automatically if it's not the case.
+
+### Run Tests that Require Gitlab
+
+To run tests that require a Gitlab instance, you can either [run one locally](https://docs.gitlab.com/ee/install/docker.html) or use the public [gitlab.com](https://gitlab.com) instance. The latter is typically recommended for ease of use.
+
+On that Gitlab instance, a "root group" is required, in which foxops can create temporary projects for test templates and incarnations (these will be automatically cleaned after the test execution). Also an access token is required that has access to that group.
+
+Once you have all this, add the following environment variables and rerun the tests:
+
+```shell
+# defaults to "https://gitlab.com" if not specified
+export FOXOPS_TESTS_GITLAB_ADDRESS=<address of the Gitlab instance>
+# defaults to 73622910, which is the "foxops-e2e" group on gitlab.com
+export FOXOPS_TESTS_GITLAB_ROOT_GROUP_ID=<id of the root group>
+export FOXOPS_TESTS_GITLAB_TOKEN=<access token with access to the root group>
+
+# these variables can also be set in a file called `.env.test` in the root folder of the project
+
+# execute tests (parallelization is recommended)
+pytest -n 4
 ```
 
 ### Documentation

docker-compose.yml

@@ -1,36 +1,21 @@
-# by default gitlab will run on localhost (i.e. same host where tests are run) listening on port 5002
-# this can be overriden with environment variables: GITLAB_HOST & GITLAB_PORT & GITLAB_ADDRESS
-# Eg: 
-# env GITLAB_HOST="myserver.example.com" GITLAB_PORT="8002" docker-compose up -d
-# env GITLAB_HOST="myserver.example.com" GITLAB_PORT="8002" ./script/await-healthy.sh
-# env GITLAB_ADDRESS="http://myserver.example.com:8002/api/v4" poetry run pytest -m 'e2e'
-
 version: '3'
 
 services:
-  gitlab-ce:
-    image: gitlab/gitlab-ce:${GITLAB_CE_VERSION:-15.8.3-ce.0}
-    restart: always
+  foxops:
+    build:
+      context: .
+      dockerfile: Dockerfile
     ports:
-      - ${GITLAB_PORT:-5002}:80
-    environment:
-      GITLAB_ROOT_PASSWORD: dvqMom4ruD9oqcErwtij
-      GITLAB_OMNIBUS_CONFIG: |
-        external_url "http://${GITLAB_HOST:-127.0.0.1}:${GITLAB_PORT:-5002}"
-        nginx['listen_port'] = 80
-    labels:
-      foxops-gitlab/owned: ''
+      - 8000:8000
     volumes:
-      - config-ce:/etc/gitlab
-      - logs-ce:/var/log/gitlab
-      - data-ce:/var/opt/gitlab
-      - ${PWD}/scripts/healthcheck-and-setup.sh:/healthcheck-and-setup.sh:Z
-    healthcheck:
-      test: /healthcheck-and-setup.sh
-      interval: 10s
-      timeout: 2m
+      - database:/app/database
+      - hoster:/app/hoster
+    environment:
+      FOXOPS_DATABASE_URL: sqlite+aiosqlite:////app/database/foxops.db
+      FOXOPS_HOSTER_TYPE: local
+      FOXOPS_HOSTER_LOCAL_DIRECTORY: /app/hoster
+      FOXOPS_STATIC_TOKEN: dummy
 
 volumes:
-  config-ce:
-  logs-ce:
-  data-ce:
+  database:
+  hoster:

docs/source/installation.md

@@ -15,19 +15,49 @@ docker run --rm -v "$(pwd)"/foxops_db:/database \
     alembic upgrade head
 ```
 
-With the database in place, we can now start the foxops API server:
+With the database in place, we can now start the foxops API server. Two options exist for running it locally:
+
+### Run FoxOps Connected to an Existing Gitlab Instance
+
+If you already have a running Gitlab instance where you want to host your templates and incarnations, you can run foxops connected to that instance:
+
+```bash
+export FOXOPS_HOSTER_GITLAB_ADDRESS=https://gitlab.com
+export FOXOPS_HOSTER_GITLAB_TOKEN=glpat-abcdefgh123456
+
+docker run --rm -p 8000:8000 -v "$(pwd)"/foxops_db:/database \
+    -e FOXOPS_DATABASE_URL=sqlite+aiosqlite:////database/foxops.sqlite \
+    -e FOXOPS_STATIC_TOKEN=dummy-token \
+    -e FOXOPS_HOSTER_TYPE=gitlab \
+    -e FOXOPS_HOSTER_GITLAB_ADDRESS=$FOXOPS_HOSTER_GITLAB_ADDRESS \
+    -e FOXOPS_HOSTER_GITLAB_TOKEN=$FOXOPS_HOSTER_GITLAB_TOKEN \
+    ghcr.io/roche/foxops:latest
+```
+
+### Run FoxOps Without Gitlab
+
+For the very first steps (development only), foxops can also be run without a Gitlab instance connected. This is not only useful for getting started quickly, but also when running tests in other systems that require a running foxops instance.
+
+Be aware that Gitlab in this is the replaced with a very basic implementation that creates all repositories/merge requests in a local folder. There is no UI for viewing or interacting with these repositories, except for your file browser.
 
 ```bash
-export GITLAB_TOKEN=my-gitlab-token
+# put a folder here where foxops should create the repositories
+export FOXOPS_HOSTER_LOCAL_DIRECTORY=/tmp/foxops_hoster
 
 docker run --rm -p 8000:8000 -v "$(pwd)"/foxops_db:/database \
     -e FOXOPS_DATABASE_URL=sqlite+aiosqlite:////database/foxops.sqlite \
     -e FOXOPS_STATIC_TOKEN=dummy-token \
-    -e FOXOPS_GITLAB_ADDRESS=https://gitlab.com/api/v4 \
-    -e FOXOPS_GITLAB_TOKEN=$GITLAB_TOKEN \
+    -e FOXOPS_HOSTER_TYPE=local \
+    -e FOXOPS_HOSTER_LOCAL_DIRECTORY=$FOXOPS_HOSTER_LOCAL_DIRECTORY \
     ghcr.io/roche/foxops:latest
 ```
 
+```{note}
+In this setup, your template repositories must also be placed in the `/tmp/foxops_hoster` folder. Place the git repositories of your templates in a folder called `/tmp/foxops_hoster/<template_name>/git` to make them usable by foxops.
+```
+
+### Accessing FoxOps
+
 Then, you can open your browser and go to the following URLs:
 * <http://localhost:8000/> - the foxops web UI (login with `dummy-token` when prompted for the password)
 * <http://localhost:8000/docs> - the foxops API documentation
@@ -69,8 +99,9 @@ The following is needed to run foxops:
 * Set the following environment variables:
   * `FOXOPS_STATIC_TOKEN` - Set to a (long) random and **secret** string. This secret is used to authenticate all users of the foxops UI & API
   * `FOXOPS_DATABASE_URL` - Set to the database URL
-  * `FOXOPS_GITLAB_ADDRESS` - Set to the address of your GitLab instance (e.g. `https://gitlab.com/api/v4`)
-  * `FOXOPS_GITLAB_TOKEN` - Set to a GitLab access token that has access to all repositories (incarnations & templates) that foxops should manage
+  * `FOXOPS_HOSTER_TYPE` - Set to `gitlab` for a production deployment
+  * `FOXOPS_HOSTER_GITLAB_ADDRESS` - Set to the address of your GitLab instance (e.g. `https://gitlab.com`) (if hoster type is set to `gitlab`)
+  * `FOXOPS_HOSTER_GITLAB_TOKEN` - Set to a GitLab access token that has access to all repositories (incarnations & templates) that foxops should manage (if hoster type is set to `gitlab`)
 
 #### Kubernetes Example
 
@@ -109,9 +140,11 @@ spec:
         - name: foxops
           image: ghcr.io/roche/foxops:v2.0.0
           env:
-            - name: FOXOPS_GITLAB_ADDRESS
-              value: https://gitlab.com/api/v4
-            - name: FOXOPS_GITLAB_TOKEN
+            - name: FOXOPS_HOSTER_TYPE
+              value: gitlab
+            - name: FOXOPS_HOSTER_GITLAB_ADDRESS
+              value: https://gitlab.com
+            - name: FOXOPS_HOSTER_GITLAB_TOKEN
               value: <dummy>
             - name: FOXOPS_STATIC_TOKEN
               value: <dummy>