The documentation units from the jDV are currently only available in the NeuRIS database with minimal data (document number, file number, court...). It is a provisional solution to enable references, e.g. in the legal process. For the pilot operation, this provisional solution is to be replaced by a complete (=all categories) and continuous migration of the data from the jDV.
The migration team transforms the XML data from the jDV into its own schema, which is independent of NeuRIS (so far). NeuRIS should now adopt this schema and also use it for documentation units that are created via NeuRIS.
In NeuRIS databases, the name of the "old" schema is public
and the new
is incremental_migration
.
This document will help Developers in NeuRIS to setup the new schema with data in their local environments.
- Clone ris-data-migration repository
git clone [email protected]:digitalservicebund/ris-data-migration.git cd ris-data-migration
- Checkout the repo on the commit used by staging (commit id to be
found here):
git checkout <commit-hash>
- Follow the steps here to get access to OTC buckets via command line. You can use the
AWS_
environment variables that you use forneuris-infra
: https://platform-docs.prod.ds4g.net/user-docs/how-to-guides/access-obs-via-aws-sdk/ - A running Docker.
From here you can run the import by:
chmod +x migration_schema_local_import.sh
./migration_schema_local_import.sh
To repeat files downloading, remove the import folder in the migration folder, and rerun.
-
In
ris-backend-service
restart the database to add the user and the scheme:cd ris-backend-service docker compose down postgres14 --remove-orphans docker compose up postgres14 # or use your favourite startup command, e.g. ./run.sh dev --no-backend
-
Make sure the new schema
incremental_migration
has been added -
Create a directory where you will store the xml files to import into the database in
ris-data-migration
mkdir juris-xml-data
-
Check if you can access the right bucket with
aws s3 ls --profile otc --endpoint-url https://obs.eu-de.otc.t-systems.com s3://neuris-migration-juris-data # output should look like this: # PRE daily/ # PRE monthly/
-
Download the lookup tables in
ris-data-migration
aws s3 cp --profile otc --endpoint-url https://obs.eu-de.otc.t-systems.com --recursive s3://neuris-migration-juris-data/monthly/2023/09/Tabellen ./juris-xml-data/Tabellen
-
Download BGH DocumentationUnits in
ris-data-migration
aws s3 cp --profile otc --endpoint-url https://obs.eu-de.otc.t-systems.com --recursive s3://neuris-migration-juris-data/monthly/2023/09/BGH-juris/RSP/ ./juris-xml-data/BGH-juris/RSP/2022/
-
Setup your local .env file with this command as described in Set up local env
-
Change the
.env
file inris-migration-data
with the following variables:RIS_MIGRATION_TABLES_LOCATION=juris-xml-data RIS_MIGRATION_INCLUDE_NORM_ABBREVIATIONS=true RIS_MIGRATION_CLI_MODE=true # database config RIS_MIGRATION_DB_HOST=localhost RIS_MIGRATION_DB_PORT=5432 RIS_MIGRATION_DB_NAME=neuris RIS_MIGRATION_DB_USER=migration RIS_MIGRATION_DB_PASSWORD=migration RIS_MIGRATION_DB_SCHEMA=incremental_migration
-
For console logging
export SPRING_PROFILES_ACTIVE=dev
-
Build the ris-data-migration application into a jar
./gradlew :cli:bootJar
-
Import the static lookup tables into your new schema (see Confluence "Wertetabellen" to find out what is static and dynamic)
java -jar cli/build/libs/ris-data-migration-cli.jar refdata seed
-
Import the dynamic lookup tables
java -jar cli/build/libs/ris-data-migration-cli.jar juris-table seed
-
Import the BGH DocumentationUnits
java -jar cli/build/libs/ris-data-migration-cli.jar juris-r migrate -p juris-xml-data/
-
Optional: drop the schema (manually) and restart the db
-
Optional: download the new lookup tables and Document Units (see step 5 & 6 above)
-
Checkout the
ris-data-migration
repo on the new commit used by staging (commit id to be found here):git checkout <commit-hash>
4.Repeat with steps 5 - 13 above
Delete the folder in pointed in DATA_MIGRATION_IMPORT_PATH
,and rerun the script. It will
download the necessary files again.