CultURIze-Web

This is the GitHub repository for CultURIze webservice.

About

CultURIze webservice is an automated workflow to create and activate persistent URI’s for entities, managed by cultural heritage institutions.

The cultURIze webservice has been built as a multi container docker application. It’s a python django application running inside a container, with a containerised database (PostGreSQL) and nginx as reverse proxy in front of the django application to serve static files. The django application provides multiple endpoints that are linked to different functionalities in the application and all accept JSON data as input.

To support generating data exports in the backgroud 2 celery containers and a redis container were added. Celery is a widely used python scheduling framework. It allows us to trigger a export generation with an API call and retreive the result when it's finished. This export generation is scheduled by celery beat to run every night for both records and logs. Also a cleanup job is run every night after the export to only keep the last 3 export file on the filesystem.

Getting Started

For users

More information about inspiration, governance, and howto's for setting up a Culturize webservice workflow are available on the Wiki

For developers

Culturize Web Installation

We provide a setup script to do most of the configuration work. This can be started with ./setup.sh and asks some questions before generating the different config files. This setup can also handle the SSL setup by using Let's Encrypt for certificate management.

A manual install is also possible by following the below steps:

Make sure the server is configure with SSL to ensure communication to and from the webservice is secure.

Setting up

• Clone this repo
  • This can be in a users home folder or somewhere like /opt/docker and create group permissions so that the relevant users can access it without becoming root.
• Create a environment file called .env.web in the root of the repository.
• Copy paste this inside the .env.web file. Replace "culturize" after SQL_DATABASE= and SQL_PASSWORD= with your own made up name and password. Update the DJANGO_ALLOWED_HOSTS to contain the domain name.
• important: replace the secret key in this file with a newly random generated key (it is used internally and should be as random as possible)

DEBUG=0
SECRET_KEY=ruWxpjmdysErePWRKEpckOCCefmIGp
DJANGO_ALLOWED_HOSTS=${domain_name}
SQL_ENGINE=django.db.backends.postgresql
SQL_DATABASE=culturize-web
SQL_USER=culturize
SQL_PASSWORD=culturize
SQL_HOST=db
SQL_PORT=5432
DATABASE=postgres

• Create another file at the root of the repo .env.db, make sure the username and password from SQL_DATABASE= and SQL_PASSWORD= are in sync with POSTGRES_PASSWORD= and POSTGRES_DB= from .env.web.

POSTGRES_USER=TheSameUsernameAsYouUsedAtSQL_DATABASEin.env.web
POSTGRES_PASSWORD=TheSameUsernameAsYouUsedAtSQL_DATABASEin.env.web
POSTGRES_DB=culturize-web

• The database is run in a seperate container with a docker named volume. Docker itself sets the file location on the host for the named volume. Normally in /var/lib/docker.

• Replace the default password in the "acceskey" file at culturize-web/app/culturizeweb/accesskey to something else

• if you want to handle TLS (HTTPS) encryption at this level then use the swag docker image: cp docker-compose.swag.yml docker-compose.yml You should also create the .env.nginx file with:

  VALIDATION=http
  URL=${domain_name}
  SUBDOMAINS=www
  

• if you handle TLS traffic somewhere else: cp docker-compose.nginx.yml docker-compose.yml

Installing

Run the two commands below to build and start the webservice.

• docker-compose up -d --build
• docker-compose exec web python manage.py migrate --noinput After installing running docker-compose up -f from the repo can be used to start the Culturize Webservice.

( If you get "Command 'docker-compose' not found..." you might not have installed Docker-compose or you have installed a newer version and you need to use docker compose up -d --build and docker compose exec web python manage.py migrate --noinput )

If you get unauthorized access on the last command (manage.py migrate) it can be that you first created the database container with the wrong credentials. You'll need to remove the attached docker volume to recreate the database with the correct credentials. You can list the docker volumes with 'docker volume ls' after you identified the database volume (will have db in it's name) you can remove it with 'docker volume rm '.

Upgrading

To upgrade to the latest version you can do (when using the default setup, e.g. no changes made to the docker-compose.yml files):

git pull # to fetch the latest changes
bash ./update.sh # rebuilds the images, but will use the same data set

When running a custom setup the update steps should be performed manually:

git pull
docker-compose down
diff docker-compose.yml docker-compose.nginx.yml # check for changes in the default configuration
# integrate the changes from the project
docker-compose up -d --build
docker-compose exec web python manage.py migrate # make sure the database is migrated

API documentation:

blob/master/openapi-spec.yml

https://editor.swagger.io/ can be used to view the documentation file.

The cultURIze API supports the following functions

List all records

curl -s https://culturize.web.example.com/api/record -H "Culturize-Key: meemoosecretbeerstash"

Add record

• please note that the persistent_url only contains the base domain and the path, not the https:// part.
• the resource url is the full url description

curl -X POST https://culturize.web.example.com/api/record -d '{\
  "resource_url": "https://meemoo.be/kennisbanken", \
  "persistent_url": "culturize.data/abc-123"\
}'\
-H "Content-Type: Application/JSON" -H "Culturize-Key: meemoosecretbeerstash"

When the Webservice has activated a persistent URI, it produces a respons

[{"resource_url":"https://meemoo.be/kennisbanken","persistent_url":"culturize.data/abc-123"}]l

Add multiple records

curl -X POST https://culturize.web.example.com/api/record -d '[\
  {\
    "resource_url": "https://meemoo.be/kennisbanken",\
    "persistent_url": "culturize.data/abc-123"\
  }, {\
    “resource_url”: “https://example.com”,\
    “persistent_url”: “culturize.data/123-abc”\
  }\
]' -H "Content-Type: Application/JSON" -H "Culturize-Key: meemoosecretbeerstash"`

Update record

• change disable state of a record, the persistent_url is the key identifier in the database. For obvious reasons this can not be changed, and should always be provided to identify the record. (remember: the persistent url should not contain the https:// part)

curl -X PUT https://culturize.web.example.com/api/record -d '{\
  "persistent_url": "culturize.data/abc-123",\
  "enabled": false\
}' -H "Content-Type: Application/JSON" -H "Culturize-Key: meemoosecretbeerstash"

curl -X PUT https://culturize.web.example.com/api/record -d '{\
  "persistent_url": "culturize.data/abc-123",\
  "enabled": false\
}' -H "Content-Type: Application/JSON" -H "Culturize-Key: meemoosecretbeerstash"

• the response will be the update record object as it is stored in the server database: it contains the persistent url, the resource urls, the enabled state and a database id.

• change resource URL

curl -X PUT https://culturize.web.example.com/api/record -d '{\
  "persistent_url": "culturize.data/abc-123",\
  "resource_url": "https://example.com/newpart"\
}' -H "Content-Type: Application/JSON" -H "Culturize-Key: meemoosecretbeerstash"

Print logs

curl -s https://culturize.douwe.linuxbe.com/api/logs --header "Culturize-Key: meemoosecretbeerstash"

License

CC BY-SA 4.0 meemoo

All content in this repository is released under the CC-BY-SA License.

The code of CultURIze is released under the MIT License.