Skip to content

Commit 5f6abb2

Browse files
tarzanektarzanek
authored
Merge pull request #111 from scylladb/actions-test
merge Actions test
2 parents c8a05d3 + c8dd1e8 commit 5f6abb2

34 files changed

+477
-27
lines changed

‎.github/workflows/tests.yml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: Tests
2+
on:
3+
push:
4+
branches:
5+
- master
6+
- actions-test
7+
pull_request:
8+
9+
jobs:
10+
test:
11+
name: Test
12+
runs-on: ubuntu-latest
13+
steps:
14+
- uses: actions/checkout@v3
15+
- name: Cache Docker images
16+
uses: ScribeMD/[email protected]
17+
with:
18+
key: docker-${{ runner.os }}-${{ hashFiles('docker-compose-tests.yml') }}
19+
- uses: actions/setup-java@v3
20+
with:
21+
distribution: temurin
22+
java-version: 8
23+
cache: sbt
24+
- name: Build migrator
25+
run: ./build.sh
26+
- name: Set up services
27+
run: docker compose -f docker-compose-tests.yml up --wait
28+
- name: Run tests
29+
run: sbt test
30+
- name: Stop services
31+
run: docker compose -f docker-compose-tests.yml down

‎CONTRIBUTING.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Contributing to scylla-migrator
2+
3+
## Testing procedure
4+
5+
Tests are implemented in the `tests` sbt submodule. They simulate the submission of a Spark job as described in the README. Therefore, before running the tests you need to build the migrator fat-jar and to set up a stack with a Spark cluster and databases to read data from and to write to.
6+
7+
1. Build the migrator fat-jar and its dependencies
8+
9+
~~~ sh
10+
./build.sh
11+
~~~
12+
13+
2. Set up the testing stack with Docker
14+
15+
~~~ sh
16+
docker compose -f docker-compose-tests.yml up
17+
~~~
18+
19+
3. Run the tests
20+
21+
~~~ sh
22+
sbt test
23+
~~~
24+
25+
Or, to run a single test:
26+
27+
~~~ sh
28+
sbt testOnly com.scylladb.migrator.BasicMigrationTest
29+
~~~
30+
31+
4. Ultimately, stop the Docker containers
32+
33+
~~~ sh
34+
docker compose -f docker-compose-tests.yml down
35+
~~~
36+
37+
Make sure to re-build the fat-jar everytime you change something in the implementation:
38+
39+
~~~ sh
40+
sbt migrator/assembly
41+
~~~
42+
43+
And then re-run the tests.

‎README.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Create a `config.yaml` for your migration using the template `config.yaml.exampl
1313

1414
The Scylla Migrator is built against Spark 2.4.4, so you'll need to run that version on your cluster.
1515

16-
After running `build.sh`, copy the jar from `./target/scala-2.11/scylla-migrator-assembly-0.0.1.jar` and the `config.yaml` you've created to the Spark master server.
16+
After running `build.sh`, copy the jar from `./migrator/target/scala-2.11/scylla-migrator-assembly-0.0.1.jar` and the `config.yaml` you've created to the Spark master server.
1717

1818
Then, run this command on the Spark master server:
1919
```shell
@@ -51,18 +51,18 @@ To run in the local Docker-based setup:
5151

5252
1. First start the environment:
5353
```shell
54-
docker-compose up -d
54+
docker compose up -d
5555
```
5656

5757
2. Launch `cqlsh` in Cassandra's container and create a keyspace and a table with some data:
5858
```shell
59-
docker-compose exec cassandra cqlsh
59+
docker compose exec cassandra cqlsh
6060
<create stuff>
6161
```
6262

6363
3. Launch `cqlsh` in Scylla's container and create the destination keyspace and table with the same schema as the source table:
6464
```shell
65-
docker-compose exec scylla cqlsh
65+
docker compose exec scylla cqlsh
6666
<create stuff>
6767
```
6868

@@ -72,11 +72,11 @@ docker-compose exec scylla cqlsh
7272

7373
6. Then, launch `spark-submit` in the master's container to run the job:
7474
```shell
75-
docker-compose exec spark-master /spark/bin/spark-submit --class com.scylladb.migrator.Migrator \
75+
docker compose exec spark-master /spark/bin/spark-submit --class com.scylladb.migrator.Migrator \
7676
--master spark://spark-master:7077 \
7777
--conf spark.driver.host=spark-master \
7878
--conf spark.scylla.config=/app/config.yaml \
7979
/jars/scylla-migrator-assembly-0.0.1.jar
8080
```
8181

82-
The `spark-master` container mounts the `./target/scala-2.11` dir on `/jars` and the repository root on `/app`. To update the jar with new code, just run `build.sh` and then run `spark-submit` again.
82+
The `spark-master` container mounts the `./migrator/target/scala-2.11` dir on `/jars` and the repository root on `/app`. To update the jar with new code, just run `build.sh` and then run `spark-submit` again.

‎build.sbt

Lines changed: 22 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
import sbt.librarymanagement.InclExclRule
22

3-
lazy val root = (project in file(".")).settings(
3+
val awsSdkVersion = "1.11.728"
4+
val sparkVersion = "2.4.4"
5+
6+
lazy val migrator = (project in file("migrator")).settings(
47
inThisBuild(
58
List(
69
organization := "com.scylladb",
@@ -20,18 +23,16 @@ lazy val root = (project in file(".")).settings(
2023
fork := true,
2124
scalafmtOnCompile := true,
2225
libraryDependencies ++= Seq(
23-
"org.apache.spark" %% "spark-streaming" % "2.4.4" % "provided",
24-
"org.apache.spark" %% "spark-sql" % "2.4.4" % "provided",
25-
"org.apache.spark" %% "spark-sql" % "2.4.4" % "provided",
26-
"com.amazonaws" % "aws-java-sdk-sts" % "1.11.728",
27-
"com.amazonaws" % "aws-java-sdk-dynamodb" % "1.11.728",
26+
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
27+
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
28+
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
29+
"com.amazonaws" % "aws-java-sdk-sts" % awsSdkVersion,
30+
"com.amazonaws" % "aws-java-sdk-dynamodb" % awsSdkVersion,
2831
("com.amazonaws" % "dynamodb-streams-kinesis-adapter" % "1.5.2")
2932
.excludeAll(InclExclRule("com.fasterxml.jackson.core")),
3033
"org.yaml" % "snakeyaml" % "1.23",
3134
"io.circe" %% "circe-yaml" % "0.9.0",
3235
"io.circe" %% "circe-generic" % "0.9.0",
33-
"org.scalatest" %% "scalatest" % "3.0.1" % "test",
34-
"org.scalacheck" %% "scalacheck" % "1.13.4" % "test"
3536
),
3637
assemblyShadeRules in assembly := Seq(
3738
ShadeRule.rename("org.yaml.snakeyaml.**" -> "com.scylladb.shaded.@1").inAll
@@ -54,12 +55,6 @@ lazy val root = (project in file(".")).settings(
5455
pomIncludeRepository := { x =>
5556
false
5657
},
57-
resolvers ++= Seq(
58-
"sonatype-releases" at "https://oss.sonatype.org/content/repositories/releases/",
59-
"Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/",
60-
"Second Typesafe repo" at "http://repo.typesafe.com/typesafe/maven-releases/",
61-
Resolver.sonatypeRepo("public")
62-
),
6358
pomIncludeRepository := { x =>
6459
false
6560
},
@@ -72,3 +67,16 @@ lazy val root = (project in file(".")).settings(
7267
Some("releases" at nexus + "service/local/staging/deploy/maven2")
7368
}
7469
)
70+
71+
lazy val tests = project.in(file("tests")).settings(
72+
libraryDependencies ++= Seq(
73+
"com.amazonaws" % "aws-java-sdk-dynamodb" % awsSdkVersion,
74+
"org.scalameta" %% "munit" % "0.7.29",
75+
"org.scala-lang.modules" %% "scala-collection-compat" % "2.11.0"
76+
),
77+
testFrameworks += new TestFramework("munit.Framework"),
78+
Test / parallelExecution := false
79+
)
80+
81+
lazy val root = project.in(file("."))
82+
.aggregate(migrator, tests)

‎build.sh

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@ pushd spark-kinesis
2121
sbt assembly
2222
popd
2323

24-
if [ ! -d "./lib" ]; then
25-
mkdir lib
24+
if [ ! -d "./migrator/lib" ]; then
25+
mkdir migrator/lib
2626
fi
2727

28-
cp ./spark-cassandra-connector/connector/target/scala-2.11/spark-cassandra-connector-assembly-*.jar ./lib
29-
cp ./spark-dynamodb/target/scala-2.11/spark-dynamodb-assembly-*.jar ./lib
30-
cp ./spark-kinesis/target/scala-2.11/spark-streaming-kinesis-asl-assembly-*.jar ./lib
28+
cp ./spark-cassandra-connector/connector/target/scala-2.11/spark-cassandra-connector-assembly-*.jar ./migrator/lib
29+
cp ./spark-dynamodb/target/scala-2.11/spark-dynamodb-assembly-*.jar ./migrator/lib
30+
cp ./spark-kinesis/target/scala-2.11/spark-streaming-kinesis-asl-assembly-*.jar ./migrator/lib
3131

32-
sbt -Djava.io.tmpdir="$TMPDIR" assembly
32+
sbt -Djava.io.tmpdir="$TMPDIR" migrator/assembly

‎docker-compose-tests.yml

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
version: '3.8'
2+
3+
services:
4+
5+
dynamodb:
6+
command: "-jar DynamoDBLocal.jar -sharedDb -inMemory"
7+
image: "amazon/dynamodb-local:latest"
8+
container_name: dynamodb
9+
networks:
10+
- scylla
11+
expose:
12+
- 8001
13+
ports:
14+
- "8001:8000"
15+
working_dir: /home/dynamodblocal
16+
17+
scylla:
18+
image: scylladb/scylla:latest
19+
networks:
20+
- scylla
21+
volumes:
22+
- "./tests/docker/scylla:/var/lib/scylla"
23+
ports:
24+
- "8000:8000"
25+
command: "--smp 2 --memory 2048M --alternator-port 8000 --alternator-write-isolation always_use_lwt"
26+
27+
spark-master:
28+
image: bde2020/spark-master:2.4.4-hadoop2.7
29+
container_name: spark-master
30+
hostname: spark-master
31+
environment:
32+
INIT_DAEMON_STEP: setup_spark
33+
SPARK_MASTER: spark://spark-master:7077
34+
SPARK_CONF_DIR: /conf
35+
SPARK_PUBLIC_DNS: spark-master
36+
networks:
37+
- scylla
38+
expose:
39+
- 7001
40+
- 7002
41+
- 7003
42+
- 7004
43+
- 7005
44+
- 7006
45+
- 7077
46+
- 6066
47+
ports:
48+
- 4040:4040
49+
- 6066:6066
50+
- 7077:7077
51+
- 8080:8080
52+
volumes:
53+
- ./migrator/target/scala-2.11:/jars
54+
- ./tests/src/test/configurations:/app/configurations
55+
56+
spark-worker:
57+
image: bde2020/spark-worker:2.4.4-hadoop2.7
58+
hostname: spark-worker
59+
container_name: spark-worker-1
60+
environment:
61+
SPARK_WORKER_CORES: 3
62+
SPARK_WORKER_MEMORY: 1024m
63+
SPARK_WORKER_WEBUI_PORT: 8081
64+
SPARK_PUBLIC_DNS: spark-worker
65+
networks:
66+
- scylla
67+
expose:
68+
- 7012
69+
- 7013
70+
- 7014
71+
- 7015
72+
- 7016
73+
- 8881
74+
ports:
75+
- 8081:8081
76+
depends_on:
77+
- spark-master
78+
79+
networks:
80+
scylla:

‎docker-compose.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ services:
4545
- 8080:8080
4646
volumes:
4747
- ./data/spark-master:/tmp/data
48-
- ./target/scala-2.11:/jars
48+
- ./migrator/target/scala-2.11:/jars
4949
- ./parquet-data:/parquet-data
5050
- ./:/app
5151

‎src/main/scala/com/scylladb/migrator/Connectors.scala renamed to ‎migrator/src/main/scala/com/scylladb/migrator/Connectors.scala

File renamed without changes.

‎src/main/scala/com/scylladb/migrator/DynamoUtils.scala renamed to ‎migrator/src/main/scala/com/scylladb/migrator/DynamoUtils.scala

File renamed without changes.

‎src/main/scala/com/scylladb/migrator/Migrator.scala renamed to ‎migrator/src/main/scala/com/scylladb/migrator/Migrator.scala

File renamed without changes.

0 commit comments

Comments
 (0)