#245 Add the ability to query REST endpoints from Reader module #297

benedeki · 2024-11-01T08:29:24Z

Implemented Reader and ReaderWithPartitioningId base classes for all readers - they are to bringAtum server connectivity/querying ability
Created 'ServerConfig' case class representing values needed to connect to an Atum Server
RequestResult[R] represents an Atum server query response.
Offered implicits for MonadError type class needed for Reader and ReaderWithPartitioningId - there are Future, Cats IO and Zio Task ones (the last one available only in Scala 2.13)
AtumPartitions and AdditionalData moved from Agent to Module
ErrorResponse received a method to decode from Json based on http status code

Closes #245

Depends on #300

Release Notes:

Reader module now supports connection to Atum server.

* created new module Info * the new modul added to JaCoco and CI routines

* JaCoCo exclusion for model

* created Provider to query the data from server * support for Future, IO, and ZIO based providers * work in progress

* fixed license headers

github-actions · 2024-11-01T12:19:57Z

JaCoCo model module code coverage report - scala 2.13.11

Overall Project	58.76% `-34.41%`	🍏
Files changed	66.75%	❌

File	Coverage
JsonSyntaxExtensions.scala	94.34% `-5.66%`	🍏
ErrorResponse.scala	89.66%	❌
basic.scala	47.17%	❌

github-actions · 2024-11-01T12:19:58Z

JaCoCo agent module code coverage report - scala 2.13.11

Overall Project	78.2%	🍏
Files changed	100%	🍏

File	Coverage
AtumAgent.scala	97.13%	🍏
AtumContext.scala	91.79%	🍏

github-actions · 2024-11-01T12:19:59Z

JaCoCo reader module code coverage report - scala 2.13.11

Overall Project	90.86%	🍏
Files changed	40.31%	❌

File	Coverage
FlowReader.scala	100% `-35.71%`	❌
PartitioningReader.scala	100%	❌
Reader.scala	100% `-6.56%`	🍏
PartitioningIdProvider.scala	100%	🍏
io.scala	100%	🍏
future.scala	100%	🍏
ServerConfig.scala	88.24%	❌
RequestResult.scala	12.5%	❌

github-actions · 2024-11-01T12:20:00Z

JaCoCo server module code coverage report - scala 2.13.11

Overall Project	68.39%	🍏

There is no coverage information present for the Files changed

reader/src/main/scala/za/co/absa/atum/reader/server/GenericServerConnection.scala

benedeki · 2024-11-22T20:58:36Z

This item depends on:

~~#291: Project improvements after 0.3.0 release #300~~
By Dependent Issues.

…endpoints-from-info-module

agent/README.md

model/src/main/scala/za/co/absa/atum/model/types/basic.scala

model/src/main/scala/za/co/absa/atum/model/utils/JsonSyntaxExtensions.scala

model/src/test/scala/za/co/absa/atum/testing/implicits/StringImplicits.scala

reader/src/main/scala-2.13/za/co/absa/atum/reader/implicits/zio.scala

reader/src/main/scala/za/co/absa/atum/reader/implicits/io.scala

reader/src/test/scala-2.13/za/co/absa/atum/reader/basic/Reader_ZIOUnitTests.scala

reader/src/test/scala/za/co/absa/atum/reader/basic/Reader_CatsIOUnitTests.scala

reader/src/main/scala/za/co/absa/atum/reader/basic/Reader.scala

reader/src/main/scala/za/co/absa/atum/reader/FlowReader.scala

reader/src/main/scala/za/co/absa/atum/reader/basic/ReaderWithPartitioningId.scala

lsulak · 2024-11-25T15:47:58Z

.github/workflows/jacoco_report.yml

+          echo "Total 'model' module coverage ${{ steps.jacoco-model.outputs.coverage-overall }}"
+          echo "Changed files of 'model' module coverage ${{ steps.jacoco-model.outputs.coverage-changed-files }}"
+          echo "Total 'server' module coverage ${{ steps.jacoco-server.outputs.coverage-overall }}"
+          echo "Changed files of'server' module coverage ${{ steps.jacoco-server.outputs.coverage-changed-files }}"


Suggested change

echo "Changed files of'server' module coverage ${{ steps.jacoco-server.outputs.coverage-changed-files }}"

echo "Changed files of 'server' module coverage ${{ steps.jacoco-server.outputs.coverage-changed-files }}"

lsulak · 2024-11-25T15:48:53Z

README.md

@@ -247,6 +246,7 @@ Code coverage wil be generated on path:
 To make this project runnable via IntelliJ, do the following:
 - Make sure that your configuration in `server/src/main/resources/reference.conf` 
  is configured according to your needs
+- When building within the UI be sure to have the option `-language:higherKinds` on in the compiler options


I don't understand - what UI do you mean?

lsulak · 2024-11-25T15:55:23Z

model/src/main/scala/za/co/absa/atum/model/utils/JsonSyntaxExtensions.scala

        case Right(value) => value
-        case Left(error) => throw new RuntimeException(s"Failed to decode JSON: $error")
+        case Left(error) => throw error


why removing the prefix of the exception msg? I think it was quite useful, we would exactly know what this is coming from

ABLL526 · 2024-11-29T09:48:41Z

reader/src/test/scala/za/co/absa/atum/reader/FlowReaderUnitTests.scala

Should we not add some failure tests as well?

ABLL526 · 2024-11-29T09:48:58Z

reader/src/test/scala/za/co/absa/atum/reader/PartitioningReaderUnitTests.scala

Should we not add some failure tests as well?

salamonpavel · 2024-12-03T07:38:32Z

reader/src/test/scala/za/co/absa/atum/reader/basic/PartitioningIdProviderUnitTests.scala

@@ -82,9 +84,10 @@ class PartitioningIdProviderUnitTests extends AnyFunSuiteLike {
    }
  }

-  test("Failure to decode response body") {
+  test("Failure to decode res  " +


accidental line break maybe?

benedeki added 12 commits October 31, 2024 01:23

#244: Create the Info module

b97b603

* created new module Info * the new modul added to JaCoco and CI routines

* fixed License headers

e623974

* renamed to _Reader_

5e4eadb

* README.md update

2e1e2ea

* fix

738c904

* JaCoCO action update

df8c9bd

* added dummy code for testing coverage

5affd82

* erroneous class renamed

0f1e121

* JaCoCo exclusion for model

* Deleted wrong files

d773a93

#245 Add the ability to query REST endpoints from Reader module

0776f9c

* created Provider to query the data from server * support for Future, IO, and ZIO based providers * work in progress

* Work still in progress

38fde1c

* the first working commit

1ac2233

benedeki added the work in progress Work on this item is not yet finished (mainly intended for PRs) label Nov 1, 2024

benedeki self-assigned this Nov 1, 2024

benedeki added 3 commits November 1, 2024 09:56

* Removed temporary notes

e6dcb52

* introduced MonadError into the GenericServerConnection

6968b02

* fixed license headers

* Fixed UTs

b9bacef

benedeki added 5 commits November 4, 2024 09:55

* trying to get rid of Java 11 dependency

bbb1e7f

* Downgraded sttpClient

33e6628

* further downgrade

f7ced56

* Removed exceptions

ca2116b

* commented out parts of README.md which are not yet part of the code

e5e6f63

benedeki commented Nov 6, 2024

View reviewed changes

reader/src/main/scala/za/co/absa/atum/reader/server/GenericServerConnection.scala Outdated Show resolved Hide resolved

benedeki marked this pull request as ready for review November 6, 2024 12:12

benedeki requested review from lsulak and TebaleloS as code owners November 6, 2024 12:12

- further cleaning

0e7675e

benedeki added the dependent The item depends on some other open item (Issue or PR) label Nov 21, 2024

benedeki added 2 commits November 21, 2024 23:10

* tests progress

432716a

* several UTs added

11b0a16

benedeki removed the dependent The item depends on some other open item (Issue or PR) label Nov 22, 2024

Merge branch 'master' into feature/245-add-the-ability-to-query-rest-…

2c3f145

…endpoints-from-info-module

benedeki requested a review from ABLL526 as a code owner November 22, 2024 21:05

* last improvements before PR ready

e07dffb

benedeki removed the work in progress Work on this item is not yet finished (mainly intended for PRs) label Nov 24, 2024

benedeki added 2 commits November 25, 2024 10:57

* description to class ServerConfig

3955a50

* removed empty line

b287a66