Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/dockerfile #156

Merged
merged 41 commits into from
Feb 13, 2024
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
9b9e740
dockerfile
salamonpavel Feb 7, 2024
a919051
dockerfile
salamonpavel Feb 7, 2024
139529d
dockerfile
salamonpavel Feb 8, 2024
64c9d0b
dockerfile
salamonpavel Feb 8, 2024
8504080
ssl stuff
salamonpavel Feb 8, 2024
43096ba
ssl context, http and https servers, logging backend, configuration, …
salamonpavel Feb 9, 2024
7112c69
sbt setup
salamonpavel Feb 10, 2024
591b1b2
[email protected]
salamonpavel Feb 10, 2024
8095486
build.sbt
salamonpavel Feb 10, 2024
8b5a6d0
build.sbt
salamonpavel Feb 10, 2024
28432b7
PartitioningForDB serialized with Play
salamonpavel Feb 10, 2024
6b91971
build.sbt
salamonpavel Feb 11, 2024
6f90c77
build.sbt
salamonpavel Feb 11, 2024
1bd421d
build.sbt
salamonpavel Feb 11, 2024
84b713d
SerializationUtils
salamonpavel Feb 11, 2024
7e1549d
SerializationUtils
salamonpavel Feb 11, 2024
9df1bd6
build.yml
salamonpavel Feb 11, 2024
9b78d68
without scalac options
salamonpavel Feb 12, 2024
3b95300
clean up
salamonpavel Feb 12, 2024
e64cc23
clean up
salamonpavel Feb 12, 2024
d8ba325
tmp
salamonpavel Feb 12, 2024
cdd8486
fix stuff, added docker ci test
salamonpavel Feb 12, 2024
624eb9f
merge conficts with master
salamonpavel Feb 13, 2024
e0b4506
minors in pipeline
salamonpavel Feb 13, 2024
66d0aa8
jacoco
salamonpavel Feb 13, 2024
84e2137
docker ci
salamonpavel Feb 13, 2024
85c84b2
docker ci
salamonpavel Feb 13, 2024
60c1e67
docker ci
salamonpavel Feb 13, 2024
d49ad5c
docker ci
salamonpavel Feb 13, 2024
a9fd875
docker ci
salamonpavel Feb 13, 2024
31e7c4a
docker ci
salamonpavel Feb 13, 2024
8d4a071
docker ci
salamonpavel Feb 13, 2024
5a9ecaa
docker ci
salamonpavel Feb 13, 2024
8452f00
docker ci
salamonpavel Feb 13, 2024
de4d204
docker ci
salamonpavel Feb 13, 2024
6318154
docker ci
salamonpavel Feb 13, 2024
be48412
docker ci
salamonpavel Feb 13, 2024
0f98bbf
docker ci
salamonpavel Feb 13, 2024
d1a4ac9
minors
salamonpavel Feb 13, 2024
9dc5d1b
minors
salamonpavel Feb 13, 2024
8afce36
json serde etc
salamonpavel Feb 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 22 additions & 10 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# limitations under the License.
#

name: Build
name: Build all modules

on:
pull_request:
Expand All @@ -23,9 +23,26 @@ on:
types: [ opened, synchronize, reopened ]

jobs:
test:
build-agent-and-model:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the way you split it. However, I read, in https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#jobs, that

by default, jobs have no dependencies and run in parallel with each other

and I'm wondering how isolated it will be - i.e. if at the same time the runner would run build for scala 2.11 for the agent and also 2.13 for server - would there be a potential clash?

https://github.com/orgs/community/discussions/26769#discussioncomment-3253321

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My assumption is that this is the same runs-on: ubuntu-latest but probably it will be different instance of this OS. (But...I remember that there is some magic caching potentially happening...we'll see :D )

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to work fine :).

runs-on: ubuntu-latest
strategy:
matrix:
scala: [2.11.12, 2.12.18, 2.13.11]
steps:
- name: Checkout code
uses: actions/checkout@v2
- uses: coursier/cache-action@v5

- name: Setup Scala
uses: olafurpg/setup-scala@v10
with:
java-version: "[email protected]"

- name: Build and run tests
run: sbt "project agent" ++${{matrix.scala}} test doc "project model" ++${{matrix.scala}} test doc

build-server:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
Expand All @@ -39,14 +56,9 @@ jobs:
--health-retries 5
ports:
- 5432:5432

strategy:
fail-fast: false
lsulak marked this conversation as resolved.
Show resolved Hide resolved
matrix:
scala: [2.11.12, 2.12.18, 2.13.11]

name: Scala ${{matrix.scala}}

scala: [2.13.11]
steps:
- name: Checkout code
uses: actions/checkout@v2
Expand All @@ -55,10 +67,10 @@ jobs:
- name: Setup Scala
uses: olafurpg/setup-scala@v10
with:
java-version: "adopt@1.8"
java-version: "adopt@1.11.0-11"

- name: Prepare testing database
run: sbt flywayMigrate

- name: Build and run tests
run: sbt ++${{matrix.scala}} test doc
run: sbt "project server" ++${{matrix.scala}} test doc
66 changes: 66 additions & 0 deletions .github/workflows/docker_image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
#
# Copyright 2021 ABSA Group Limited
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: Docker Image CI
lsulak marked this conversation as resolved.
Show resolved Hide resolved

on:
pull_request:
branches: [ master ]
types: [ opened, edited, synchronize, reopened ]

jobs:
build:
runs-on: ubuntu-latest

services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: atum_db
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432

steps:
- uses: actions/checkout@v2

- name: Setup Scala environment
uses: olafurpg/setup-scala@v14
with:
java-version: "[email protected]"

- name: Prepare testing database
run: sbt flywayMigrate

- name: Run sbt assembly
run: sbt server/assembly

- name: Build the Docker image
run: |
cd server
docker buildx build --platform linux/amd64 -t absaoss/atum-service:latest --build-arg CONFIG=./src/main/resources/reference.conf --build-arg SSL=false ./

- name: Run the Docker image and execute HTTP call
run: |
docker run -d -p 8080:8080 absaoss/atum-service:latest
docker container ls
sleep 15 # Wait for the service to start
curl -Lf http://127.0.0.1:8080/docs
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that if cURL fails, this step fails?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am going to remove this file entirely for now. It turns out that you can't establish a connection between a container running in one of the steps of your pipeline and the postgres service. Basically we would have to first deploy the image some place and then fetch it in the pipeline from registry in order to be able to start it alongside the postgres service in the same network.

39 changes: 4 additions & 35 deletions .github/workflows/jacoco_check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# limitations under the License.
#

name: JaCoCo report
name: JaCoCo report agent and module

on:
pull_request:
Expand All @@ -24,8 +24,6 @@ on:
env:
scalaLong12: 2.12.18
scalaShort12: "2.12"
scalaLong13: 2.13.11
scalaShort13: "2.13"
overall: 80.0
changed: 80.0

Expand All @@ -34,45 +32,19 @@ jobs:
name: Build and test
runs-on: ubuntu-latest

services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: atum_db
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432

steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Scala
uses: olafurpg/setup-scala@v14
with:
java-version: "[email protected]"
- name: Prepare testing database
run: sbt flywayMigrate
- name: Build and run tests
continue-on-error: true
id: jacocorun
run: sbt jacoco
# server module code coverage
- name: Add coverage to PR
if: steps.jacocorun.outcome == 'success'
id: jacoco-server
uses: madrapps/[email protected]
with:
paths: ${{ github.workspace }}/server/target/jvm-${{ env.scalaShort13 }}/jacoco/report/jacoco.xml
token: ${{ secrets.GITHUB_TOKEN }}
min-coverage-overall: ${{env.overall }}
min-coverage-changed-files: ${{ env.changed }}
title: JaCoCo server module code coverage report - scala ${{ env.scalaLong13 }}
update-comment: true
run: |
sbt "project agent; jacoco"
sbt "project model; jacoco"
# agent module code coverage
- name: Add coverage to PR
if: steps.jacocorun.outcome == 'success'
Expand Down Expand Up @@ -102,8 +74,6 @@ jobs:
- name: Get the Coverage info
if: steps.jacocorun.outcome == 'success'
run: |
echo "Total sever module coverage ${{ steps.jacoco-server.outputs.coverage-overall }}"
echo "Changed Files coverage ${{ steps.jacoco-server.outputs.coverage-changed-files }}"
echo "Total agent module coverage ${{ steps.jacoco-agent.outputs.coverage-overall }}"
echo "Changed Files coverage ${{ steps.jacoco-agent.outputs.coverage-changed-files }}"
echo "Total model module coverage ${{ steps.jacoco-model.outputs.coverage-overall }}"
Expand All @@ -114,7 +84,6 @@ jobs:
with:
script: |
const coverageCheckFailed =
Number('${{ steps.jacoco-server.outputs.coverage-changed-files }}') < Number('${{ env.changed }}') ||
Number('${{ steps.jacoco-agent.outputs.coverage-changed-files }}') < Number('${{ env.changed }}') ||
Number('${{ steps.jacoco-model.outputs.coverage-changed-files }}') < Number('${{ env.changed }}');
if (coverageCheckFailed) {
Expand Down
117 changes: 117 additions & 0 deletions .github/workflows/jacoco_check_server.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
#
# Copyright 2021 ABSA Group Limited
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: JaCoCo report server

on:
pull_request:
branches: [ master ]
types: [ opened, edited, synchronize, reopened ]

env:
scalaLong13: 2.13.11
scalaShort13: "2.13"
overall: 80.0
changed: 80.0

jobs:
test:
name: Build and test
runs-on: ubuntu-latest

services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: atum_db
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432

steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Scala
uses: olafurpg/setup-scala@v14
with:
java-version: "[email protected]"
- name: Prepare testing database
run: sbt flywayMigrate
- name: Build and run tests
continue-on-error: true
id: jacocorun
run: sbt "project server; jacoco"
# server module code coverage
- name: Add coverage to PR
if: steps.jacocorun.outcome == 'success'
id: jacoco-server
uses: madrapps/[email protected]
with:
paths: ${{ github.workspace }}/server/target/jvm-${{ env.scalaShort13 }}/jacoco/report/jacoco.xml
token: ${{ secrets.GITHUB_TOKEN }}
min-coverage-overall: ${{env.overall }}
min-coverage-changed-files: ${{ env.changed }}
title: JaCoCo server module code coverage report - scala ${{ env.scalaLong13 }}
update-comment: true
- name: Get the Coverage info
if: steps.jacocorun.outcome == 'success'
run: |
echo "Total sever module coverage ${{ steps.jacoco-server.outputs.coverage-overall }}"
echo "Changed Files coverage ${{ steps.jacoco-server.outputs.coverage-changed-files }}"
- name: Fail PR if changed files coverage is less than ${{ env.changed }}%
if: steps.jacocorun.outcome == 'success'
uses: actions/github-script@v6
with:
script: |
const coverageCheckFailed =
Number('${{ steps.jacoco-server.outputs.coverage-changed-files }}') < Number('${{ env.changed }}');
if (coverageCheckFailed) {
core.setFailed('Changed files coverage is less than ${{ env.changed }}%!');
}
- name: Edit JaCoCo comments on build failure
if: steps.jacocorun.outcome != 'success'
uses: actions/github-script@v6
with:
script: |
const issue_number = context.issue.number;
const owner = context.repo.owner;
const repo = context.repo.repo;
const jacocoReportRegExp = /^### JaCoCo .* code coverage report .*/;

const comments = await github.rest.issues.listComments({
owner,
repo,
issue_number,
});

for (const comment of comments.data) {
const lines = comment.body.split('\n');
if (lines.length > 0 && jacocoReportRegExp.test(lines[0])) {
await github.rest.issues.updateComment({
owner,
repo,
comment_id: comment.id,
body: lines[0] + "\n\n### Build Failed",
});
}
}

core.setFailed('JaCoCo test coverage report generation failed, and related PR comments were updated.');
9 changes: 0 additions & 9 deletions .github/workflows/release_publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,3 @@ jobs:
PGP_SECRET: ${{ secrets.PGP_SECRET }}
SONATYPE_PASSWORD: ${{ secrets.SONATYPE_PASSWORD }}
SONATYPE_USERNAME: ${{ secrets.SONATYPE_USERNAME }}

- name: Find WAR file
id: find_war
run: echo "WAR_PATH=$(find . -name '*.war' | head -n 1)" >> $GITHUB_ENV

- name: Upload WAR file to GitHub Release
run: gh release upload ${{ github.event.release.tag_name }} ${{ env.WAR_PATH }} --repo ${{ github.repository }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
lsulak marked this conversation as resolved.
Show resolved Hide resolved
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -88,3 +88,6 @@ _site
utils/resources/*.conf

.bsp
/server/certs/
/server/selfsigned.crt
/server/selfsigned.p12
Loading
Loading