v1.3.20240909

Changelog

The format is loosely based on Keep a Changelog, and this project adheres to Semantic Versioning, except that the date of the release is used as the patch version.

Added

• Alembic service for automated database migrations

  • Added new service (containarized in docker) to update the database schema programmatically using Alembic service. (#353)

• Increased maximum length of the value attribute of the note class

  • Added a migration to extend the maximum length of the value attribute of the note class and its respective tables. (#353)

Updated

• Make host ports of docker-compose configurable through .env

  • Moved some configuration options from docker-compose to environment variables (in .env), so different deployments only have to modify the .env file and not also docker-compose file. (#345)

• Update deployment instructions in README.md

  • Added sanity check for instructions for server deployment in Readme.md file. (#337)

• Keycloak theme

  • Added AIoD style theme to keycloak. (#339)

Fixed

• Bugfix/ fix huggingface connector
  • Fixed few bugs in the HuggingFace connector that prevented it from indexing some datasets. (#335)

Release 1.3.20240619 - Bugfixes and improvements

Changelog

The format is loosely based on Keep a Changelog, and this project adheres to Semantic Versioning, except that the date of the release is used as the patch version.

Added

• Improvement/backup routine

  • Hotfix: Added --users flag to ensure user details are included in the exported realm file. (#286)

• Update docker workflow

  • Enhanced workflow to publish images to Docker Hub under aiod/metadata_catalogue.
  • Conditions for publishing:
    • On release: pushes with tags latest and version-specific tag (e.g., v1.3.20240308).
    • On push to develop branch: pushes with tag develop.
    • Via workflow dispatch through GitHub Actions: pushes with provided tag.
  • Added capability to update Docker repository description based on docker-description.md.
  • Implemented build cache to optimize workflow efficiency.
  • Note: Further validation on develop and release triggers and flow setup will be conducted on a fork. (#302)

• Enhancement/fixed version tags

  • Fixed Docker image version tags for better version control.
  • Upgraded Keycloak to version 24.0.4 to resolve deployment bug (Reference: keycloak/keycloak#25120). (#314)

Fixed

• Bugfix/remove pytest workflow

  • Removed redundant pytest workflow since pre-commit triggers pytest tests. (#284)

• Minor typos in README.md

  • Corrected minor typos in the README.md file. (#287)

• Improvement/privacy issues

  • Restricted access to sensitive metadata from the former Drupal platform (now "ai4europe_cms"):
    • Access to name, given_name, and surname fields in the persons entity and email field in contacts restricted to users with "full_view_drupal_resources" permission.
    • Email field access in contacts for other platforms now requires authentication. (#298)

• Bugfix Elastic Search

  • Bugfix ensuring the correct search for assets of specific platforms using the platforms field. (#323)

• Bugfix huggingface validator

  • Replaced the buggy validator implementation with huggingface_hub.utils.validate_repo_id as per documentation. (#330)

Updated

• Specify the references require identifiers of contact details

  • Clarified the intended usage of references, specifying the need for identifiers of contact details. (#297)

• Rename drupal to ai4europe_cms

  • Renamed all instances of "drupal" to "ai4europe_cms" for better platform representation. (#312)

Notes

• Verification of the develop and release triggers and flow setup is pending and will be tested on a fork before finalizing the merge.
• Consideration for merging build and test jobs back to potentially speed up the Docker workflow process.

Release 1.3.20240308 - Backup scripts and bugfixes

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as the patch version.

Added

• Automated Backup and Restoration System: Implemented automated incremental backup and restoration of any data related to the metadata catalogue (e.g., MySQL DB, Keycloak, connector states/logs, etc) (PR #275).

Fixed

• OpenML Connector Issue #266: Fixed the issue on PR #270 to ensure that the connector correctly detects the last asset uploaded to OpenML, preventing the offset of its state from increasing on subsequent runs. This ensures synchronization without skipping any assets.
• OpenML Connector empty distribution: Fixed on PR #270 to make sure that the connector correctly returns a distribution list when all the three fields: dependencies, installation_notes, and binary_url are filled.
• Zenodo Rate Limits Issue #267: Resolved on PR #272 to prevent skipping assets due to improper error handling. This includes:

1. Preventing connector to exceed harvesting rate limits;
2. Preventing connector to exceed the resumption token expiration time.
3. Avoiding connector to incorrectly raise an error on missing fields.

• Zenodo Uploader Error Message: Fixed truncation of error messages due to line breaks on PR #268.
• Hugging Face Connector Logging Operator: Updated the operator in connectors/huggingface/datasets.sh to prevent deletion of logs when the Docker container is restarted, fixed on PR #268.

Changed

• Connector Log Version Tags: Added version tags to the connector.log files to facilitate debugging, as implemented in PR #268.

Release 1.3.20240129 - Keycloak persistent and new features

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as patch version.

Added

• OpenML MLModels Connector: Allows fetching metadata of ML models from openml to AIoD platform (#202).
• Zenodo Uploader: This allows data files (content) to be uploaded to Zenodo with the AIoD API endpoint .../upload/datasets/{identifier}/zenodo (#214).

Fixed

• Bugfix: Make sure the API redirects the response to download the content of a dataset to the client. This way, the download will be handled on the client side instead of downloading on the server side and passing it to the client afterwards. (#237).

Changed

• Huggingface Uploader: Refactored without major functional changes (#214).
• Keycloak state persistent: Makes sure that the Keycloak container saves the state to the filesystem (#244).
• Documentation: Typos fixed on README.md (#258).

1.2.20231219 - Finegrained authorization

Added

• Instead of only a single role (edit_aiod_resources), now roles such as crud_datasets, create_datasets work as well (#232)
• Add releases url to Swagger description (#233)

Fixed

• Bugfix: make sure Contact works when it specifies an organisation (#234)
• Minor fix in documentation (#228)

Changed

• Refactoring of Pydantic Annotations, without functional changes (#230)

Hotfix 1.1.20231205

Fixed

• Bugfix: make sure HuggingFace repository identifier is created correctly for the HuggingFace Uploader (#216)
• Security fix: make the API use credentials to ask Keycloak the token information: (#215)

Changed

• You will now get an error if you try to upload a HuggingFace or OpenML item with an incorrect platform_resource_identifier (platform-specific validation) (#216)

AIoD Metadata Catalogue in production

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as patch version.

Added

• ElasticSearch + API search endpoints (#182, #207)
• Soft deletion (delete it after some time, so that logstash can synchronize for elasticsearch) (#166, #180, #184)
• Example scripts: how to authenticate / upload data (#159)
• Endpoints to get content (the actual data) (#175)
• Endpoint to get detailed count for all resources, or for single resource (#197)

Fixed

• Bugfixes for dataset connectors (#186, #191, #204)
• Bugfix for DCAT-AP conversion of datasets (#209)
• Making docker-compose work out-of the box on every machine (longer timeouts: #193, making sure all directories exist with correct permissions: #195, #196)

Changed

• Metadata changes: made contentUrl optional for MLModel distributions (#211)
• Improved the README (#174)
• Improved Swagger documentation (#208, #205, #210)
• Code cleanup (better "injection" of DbSession: #199, new SqlModel version with quite some changes: #200)

Release 1.0.20230911

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as patch version.

Added

• Connector framework for synchronization. Implemented for datasets of OpenML, HuggingFace and Zenodo (#100)
• Metadata Model v2.0: inheritance with AIoDConcept with AIoDEntry, AIResource and AIAsset. (#125)
• New resource: Experiments (#125)
• New resource: MLModels (#125)
• New resource: Services (#125)
• New resource: Teams (#136)
• New endpoints: GET enums (e.g. list all licenses or languages) (#144)
• New endpoints: GET resources by resource_identifier, agents by agent_identifier etc. (#144)
• Security improvements: Depend-a-bot. Automatic dependency updates (#86)
• Security improvements: using public client (safer) in Swagger, better token-management, and configurable (#83)
• Development improvements: docker compose, making it easier to run the code (#115, #135, #145, #154)
• Development improvements: automatic tests for Pull Requests (#92, #93, #98, #101, #102, #111)

Fixed

• Security fixes: dependency updates (e.g. #148 and many more)

Changed

• Changed metadata (metadata v2.0): Case Study (#137)
• Changed metadata (metadata v2.0): Computational Assets (previously Computational Resource) (#139)
• Changed metadata (metadata v2.0): Datasets (#125)
• Changed metadata (metadata v2.0): Educational Resources (#140)
• Changed metadata (metadata v2.0): Event (#141)
• Changed metadata (metadata v2.0): News (#142)
• Changed metadata (metadata v2.0): Organisations (#125)
• Changed metadata (metadata v2.0): Persons (#125, #136)
• Changed metadata (metadata v2.0): Projects (#142)
• Changed metadata (metadata v2.0): Publications (#125)
• Fix after metadata change: Dataset Schema converters (DCAT-AP, schema.org) (#132)
• Fix after metadata change: Huggingface Data Uploader (#133)
• Fix after metadata change: Platform and platform identifier placed on each resource (#134)
• Fix after metadata change: Examples that fill the database (#138)
• Bumping versions to v1 in the API (#130)

Release 0.4.20230627 ("Athens")

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as patch version.

Added

• New resource: Computational Resource (#75)
• New endpoint for uploading a new dataset, metadata + data, to Zenodo (#71)
• Authorization for POST / PUT (#47) / DELETE (#53) requests
• Filling the db with example items for each resource (#66)
• New endpoints to get the count of each resource (#70)
• New endpoints to add / edit / delete platforms (#78)
• Make Swagger Beautiful Again, by grouping the endpoints per resource (#76)
• Improved Swagger documentation, by distinguishing Read and Create resource as recommended by sqlmodel (#50)
• Preliminary work on a Zenodo Dataset Connector - not working yet (#49)

Fixed

• Bugfix for deletion of resources (#74)
• Better error messages & tests for adding resources without a platform (#69, #77)
• Pinning the dependency versions to make sure installation always works (#80)
• Better documentation (#51)

Changed

• Refactoring, to make it easier to add new resources. Instead of the need to change 2 files (Pydantic and ORM), now each resource is only defined once.
  • General Framework: #50
  • Dataset, Publication (#50), Case Study (#52), Presentation(#55), Project (#57), News (#58), Educational Resource (#59, #63), Events (#61)

Hotfix 0.3.20220509

Changelog

The format is loosely based on Keep a Changelog,
and this project adheres to Semantic Versioning, except that the date of the release is used as patch version.

Fixed

• Code bug: name of database table "publication" instead of "presentation" (#46)
• Fixed dependency version of urllib3, because the newest release does not play well with other packages (#46)