FAIR MAST Data Management System

FAIR MAST is a data management system designed for fusion research, enabling efficient storage, retrieval, and management of experimental data.

📌 Development Setup

Prerequisites

• Mac Users

  If you are using Mac for development, use podman instead of docker. Follow the installation guide to set it up, then follow the below set up. Also install podman-mac-helper, which provides a compatibility layer that allows you to use most Docker commands with Podman on macOS.

  Linux/Windows Users

  If using Linux or Windows, you need to make sure you have docker and docker-compose installed on your system.

Getting Started

1. Clone the repository:

git clone [email protected]:ukaea/fair-mast.git
cd fair-mast

2. Start the development environment:

   • Mac Users

   podman compose \
   --env-file dev/docker/.env.dev  \
   -f dev/docker/docker-compose.yml \
   up \
   --build

   • Linux/Windows Users

   docker-compose \
   --env-file dev/docker/.env.dev  \
   -f dev/docker/docker-compose.yml \
   up \
   --build

   The following services will be started:

   • FastAPI REST & GraphQL Server - will start running at http://localhost:8081.
     • The REST API documentation is at http://localhost:8081/redoc.
     • The GraphQL API documentation is at http://localhost:8081/graphql.
   • Postgres Database Server - will start running at http://localhost:5432
   • Postgres Admin Server - will start running at http://localhost:5050

3. Shutting Down:

   • Mac Users

   podman compose -f dev/docker/docker-compose.yml down   
   podman volume rm --all

   • Linux/Windows Users

   docker-compose -f dev/docker/docker-compose.yml down

📦 Populating the Database

Retrieve and ingest the metadata files using s5cmd:

s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp "s3://mast/dev/mock_data*" .

Create the database and ingest data using the following command:

• Mac Users

podman exec -it mast-api python -m src.api.create /code/data/index

• Linux/Windows Users

docker exec -it mast-api python -m src.api.create /code/data/index

✅ Running Unit Tests

Verify everything is setup correctly by running the unit tests.

Follow the below instructions to set up the environment.

uv run pytest

🔧 Production Deployment

To run the production container to start the postgres database, fastapi, and minio containers. This will also start an nginx proxy and make sure https is all setup

docker compose --env-file dev/docker/.env.dev  -f dev/docker/docker-compose.yml -f dev/docker/docker-compose-prod.yml up --build --force-recreate --remove-orphans -d

To shut down the production deployment, run the following command:

docker compose --env-file dev/docker/.env.dev  -f dev/docker/docker-compose.yml -f dev/docker/docker-compose-prod.yml down

To also destory the volumes (including the metadatabase) you may add the volumes parameter:

docker compose --env-file dev/docker/.env.dev  -f dev/docker/docker-compose.yml -f dev/docker/docker-compose-prod.yml down --volumes

⚠ Note: Every time you destory volumes, the production server will mint a new certificate for HTTPS. Lets Encrypt currently limits this to 5 per week.

You'll need to download and ingest the production data like so:

mkdir -p data/mast/meta
rsync -vaP <CSD3-USERNAME>@login.hpc.cam.ac.uk:/rds/project/rds-sPGbyCAPsJI/archive/metadata data/

docker exec -it mast-api python -m src.api.create /code/data/index

Building Documentation

See the documentation guide for more details.

Citation

If you would like to reference this work, please cite the followig publication.

Samuel Jackson, Saiful Khan, Nathan Cummings, James Hodson, Shaun de Witt, Stanislas Pamela, Rob Akers, Jeyan Thiyagalingam, FAIR-MAST: A fusion device data management system, SoftwareX, Volume 27, 2024,101869,ISSN 2352-7110, https://doi.org/10.1016/j.softx.2024.101869.

In BibTex format:

@article{jackson_fair-mast_2024,
	title = {{FAIR}-{MAST}: {A} fusion device data management system},
	volume = {27},
	issn = {23527110},
	shorttitle = {{FAIR}-{MAST}},
	url = {https://linkinghub.elsevier.com/retrieve/pii/S2352711024002395},
	doi = {10.1016/j.softx.2024.101869},
	language = {en},
	urldate = {2025-01-17},
	journal = {SoftwareX},
	author = {Jackson, Samuel and Khan, Saiful and Cummings, Nathan and Hodson, James and De Witt, Shaun and Pamela, Stanislas and Akers, Rob and Thiyagalingam, Jeyan},
	month = sep,
	year = {2024},
	pages = {101869},
}