Leverage AiiDA ORM models in REST API #93

edan-bainglass · 2025-10-12T09:25:35Z

Based on #87

This PR discards the Pydantic models introduced in aiida-restapi in favor of those introduced in AiiDA in aiidateam/aiida-core#6255 and updated in aiidateam/aiida-core#6990.

The PR also addresses a few endpoints, but this can be separated out to another PR.

TODO

Use entry points (unique) instead of class name for Node model references
Migrate remaining entry points (e.g., links) - possibly in a separate PR?

eimrek

Thanks @edan-bainglass. I think these changes are great, and i like the overall structure. Some extra high-level comments:

Some of the top level files (e.g. aiida_db_mapping) are only used in GraphQL now. Maybe for clarity they could be moved inside that subfolder.
As discussed earlier, the /nodes endpoint should not include /attributes and /extras/ by default as those can be large. But this also applies to other endpoints - e.g. for /groups i currently get a lot of output due to extras, but i would just want the list.
(also mentioned already before, but important: the uuid should be used instead of pk when retrieving singular entities).
Bit of a conceptual question: in the POST models, the attributes are not contained in a nested dictionary any more, so e.g.
```
{
  "node_type": "data.core.int.Int",
  "value": 8
} 
```
instead of
```
{
  "node_type": "data.core.int.Int",
  "attributes": {"value": 8}
} 
```
This means that for nodes that have a lot of attributes, it's more difficult to distinguish from other types of inputs (e.g. extras that are still in a dictionary). Additionally, this doesn't reflect the corresponding GET models. thoughts?

eimrek · 2025-12-03T16:57:22Z

aiida_restapi/common/repository.py

+        node = self.entity_cls.collection.get(pk=model.pk)
+
+        total_size = 0
+        for path in repo_metadata['o']:


What does the 'o' signify?

No Idea. This is from aiida-core. Object? See aiida.repository.common.File.serialize:

def serialize(self) -> dict[str, typing.Any]: """Serialize the metadata into a JSON-serializable format. .. note:: the serialization format is optimized to reduce the size in bytes. :return: dictionary with the content metadata. """ if self.file_type == FileType.DIRECTORY: if self.objects: return {'o': {key: obj.serialize() for key, obj in self.objects.items()}} return {} return {'k': self.key}

eimrek · 2025-12-03T16:59:56Z

aiida_restapi/routers/computers.py

+        raise HTTPException(status_code=404, detail=f'Could not find any Computer with id {comp_id}')
+
+
+@router.post(


500 if i try to create a computer with a label that already exists. Would be great to check for this.

I need to update all try/except blocks, which currently (mostly) only handle Exception. Need to at least add a few obvious exceptions.

Actually, this should be changed in aiida-core (the pydantic PR). The computer model should validate this. Then we get it via FastAPI automatically. I'll update the pydantic PR.

eimrek · 2025-12-03T17:16:39Z

aiida_restapi/common/repository.py

+
+        total_size = 0
+        for path in repo_metadata['o']:
+            size = node.base.repository.get_object_size(path)


Could this be slow for many files? (e.g. for the s3 backend or so on). Either way good to keep in a more specialized endpoint.

This ends up as part of the repository metadata. which, like attributes and extras, I'm thinking of removing by default from the node GET, placing them back via dedicated endpoints, e.g., /metadata, etc.

eimrek · 2025-12-03T17:27:37Z

aiida_restapi/routers/nodes.py

+    ]
+
+
+@router.get(


This could just be achieved by querying the plain /nodes with a type filter, right?

Sure. That's what this end point does. The /nodes/types/... endpoints are for convenience. It provides pre-constructed links for much of what you'd want. I added them when I switched to keying the models/types dictionaries with node types, as node_type isn't great to type.

Maybe you can say more?

Ok, I see. I just wonder if it's better to just have a single /nodes endpoint, just to keep the API easier to grasp and the code leaner.

eimrek · 2025-12-03T17:29:19Z

aiida_restapi/routers/nodes.py

-        raise HTTPException(status_code=404, detail=f'Could no find any node with id {nodes_id}')
+        raise HTTPException(status_code=404, detail=f'Could not find any node with id {node_id}')

    if download_format is None:


Would be nice here to return the download formats for the node.

Not so sure. This endpoint is to be hit AFTER the user knows the download formats, for which we have another endpoint. See the comment there regarding adding a per-type endpoint also.

eimrek · 2025-12-03T17:30:43Z

aiida_restapi/routers/nodes.py


+    :return: A dictionary with available download formats as keys and their descriptions as values.
+    """
    return resources.get_all_download_formats()


Here the node types still have the pipe symbol at the end. I guess we can to drop this?

Sure 🙂 I left this endpoint as is. Good to know how they are used in practice. Also, I can add an endpoint for download formats per node type, and add a pre-constructed one to the response of /nodes/types. Again, for convenience, as node_type is no fun 🙂

eimrek · 2025-12-03T17:36:54Z

aiida_restapi/routers/nodes.py


-@router.post('/nodes/singlefile', response_model=models.Node)
+# TODO what about folderdata?
+@router.post(


I wonder if this endpoint should instead be unified the the /nodes POST somehow. Many nodes can have files inside them apart from FolderData and SingleFileData, and creating them might not make sense without uploading the files at the same time (e.g. BandsData, ...).

Let's discuss in person

Have a look at the /nodes/file-upload endpoint's parameters. Unlike /nodes, it does not take a model as input, but rather a parameters Form + an UploadFile. This is not my work. See #25 (comment). I believe this makes this endpoint incompatible w.r.t the /nodes endpoint

edan-bainglass · 2025-12-04T05:41:47Z

Thanks for the review @eimrek. Most of your comments I already agree with (per our in-person discussion). I provide further comments in the review, and also here below 🙂

Thanks @edan-bainglass. I think these changes are great, and i like the overall structure. Some extra high-level comments:

Some of the top level files (e.g. aiida_db_mapping) are only used in GraphQL now. Maybe for clarity they could be moved inside that subfolder.

Will do 👍

As discussed earlier, the /nodes endpoint should not include /attributes and /extras/ by default as those can be large. But this also applies to other endpoints - e.g. for /groups i currently get a lot of output due to extras, but i would just want the list.

Will do, also for repo metadata (anything that can be large) 👍

(also mentioned already before, but important: the uuid should be used instead of pk when retrieving singular entities).

This was using PKs prior to this PR, so will keep it as is here. Will open another PR to move everything to UUIDs 👍

Bit of a conceptual question: in the POST models, the attributes are not contained in a nested dictionary any more, so e.g.
{
  "node_type": "data.core.int.Int",
  "value": 8
} 
instead of
{
  "node_type": "data.core.int.Int",
  "attributes": {"value": 8}
} 
This means that for nodes that have a lot of attributes, it's more difficult to distinguish from other types of inputs (e.g. extras that are still in a dictionary). Additionally, this doesn't reflect the corresponding GET models. thoughts?

GET returns NodeModel, whereas POST works with NodeCreateModel, or rather specific ones, e.g., IntCreateModel, therefor "doesn't reflect". The idea of moving attributes outside of the attributes dict is to enable validation. If they are provided in attributes they hit Node.Model.attributes, which is a dict, so the attributes aren't actually validated. But IntCreateModel has value in its model, so providing it flat WILL validate it against IntCreateModel. As for extras, well that's REALLY just a dict and will be validated as such against the inherited Node.Model.extras field.

eimrek · 2025-12-04T09:45:17Z

GET returns NodeModel, whereas POST works with NodeCreateModel, or rather specific ones, e.g., IntCreateModel, therefor "doesn't reflect". The idea of moving attributes outside of the attributes dict is to enable validation. If they are provided in attributes they hit Node.Model.attributes, which is a dict, so the attributes aren't actually validated. But IntCreateModel has value in its model, so providing it flat WILL validate it against IntCreateModel. As for extras, well that's REALLY just a dict and will be validated as such against the inherited Node.Model.extras field.

Couldn't one in principle validate the attributes dict separately for each node type? if it's possible, it might be worth considering, e.g. for potential ideas in the future to just easily GET and POST nodes (e.g. to transfer via the web from one db to another). But of course if it makes the code very messy, this probably also is fine.

edan-bainglass · 2025-12-04T14:16:01Z

Couldn't one in principle validate the attributes dict separately for each node type? if it's possible, it might be worth considering, e.g. for potential ideas in the future to just easily GET and POST nodes (e.g. to transfer via the web from one db to another). But of course if it makes the code very messy, this probably also is fine.

Sure, but not via the built-in pydantic model system. It would be an ad-hoc system, which is potentially messier, as you say. But good to keep this in mind.

Update

Referencing here @eimrek's proposal detailed in aiidateam/aiida-core#7131

…dd a metadata endpoint

edan-bainglass changed the title ~~Use-aiida-orm-models~~ Leverage AiiDA ORM models in REST API Oct 12, 2025

edan-bainglass mentioned this pull request Oct 12, 2025

Update node serialization/deserialization and other Pydantic issues aiidateam/aiida-core#6990

Open

edan-bainglass force-pushed the use-aiida-orm-models branch from 3294194 to 96c84e8 Compare November 19, 2025 09:27

edan-bainglass marked this pull request as ready for review November 19, 2025 09:28

edan-bainglass force-pushed the use-aiida-orm-models branch from 9c2555a to 42dd5b1 Compare November 19, 2025 10:02

edan-bainglass marked this pull request as draft November 19, 2025 14:13

edan-bainglass marked this pull request as ready for review November 20, 2025 07:41

edan-bainglass requested a review from eimrek November 20, 2025 07:41

edan-bainglass added 20 commits December 2, 2025 16:13

Add common package

fcf4bae

Replace custom models with AiiDA core ORM models

6c5ef52

Rename process module to submit

9eca8fd

Fix repository and model logic

d234486

Update tests

82d24ee

Update endpoints and model logic

f494e8d

Add endpoints table route

3d44bdb

Leverage InputModel property

332e899

Update types

4ad0344

Patch file size and download url to repository metadata field

da07021

Fix computers router post model type

87e4536

Update tests

8f18421

Update REST API w.r.t recent core changes

b6ca783

Temporarily point to aiida-core fork branch

ad17534

Add eval_type_backport dependency

0b6d6eb

Fix pydantic forward reference issue

2c11b6c

Fix test

9c3c390

Extend nitpicking list

a45b216

Add __all__ in pagination.py to resolve docs issue

e78d765

Map node_type to CreateModel classes

7664ced

edan-bainglass force-pushed the use-aiida-orm-models branch from 3abb305 to 7664ced Compare December 2, 2025 15:17

edan-bainglass mentioned this pull request Dec 2, 2025

Read-only mode #95

Open

edan-bainglass mentioned this pull request Dec 3, 2025

Fix test typing and fixture usage #96

Open

eimrek requested changes Dec 3, 2025

View reviewed changes

edan-bainglass added 3 commits December 4, 2025 06:43

Discard empty comments router

6636279

Improve request error handling

10d6fa9

Clarify endpoint/method naming

5ea4ed0

edan-bainglass added 2 commits December 4, 2025 15:00

Modify metadata patching to support folders

63c06a4

Fix endpoint docs

f910639

edan-bainglass added 7 commits December 4, 2025 17:28

Move graphQL files to graphQL package

e6061ae

Refactor repository metadata patching to a separate repo method and a…

41e0a32

…dd a metadata endpoint

Refactor package for clarity

630ac7c

Exclude certain fields from certain responses

1c881c0

Add endpoints for node attributes/extras

bbeef10

Add endpoint for group extras

8b52c6b

Move entity schema fetch logic to entity repository

99d20fc

		raise HTTPException(status_code=404, detail=f'Could not find any Computer with id {comp_id}')


		@router.post(

		]


		@router.get(

Leverage AiiDA ORM models in REST API #93

Are you sure you want to change the base?

Leverage AiiDA ORM models in REST API #93

Uh oh!

Conversation

edan-bainglass commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

eimrek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edan-bainglass Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edan-bainglass Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edan-bainglass commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eimrek commented Dec 4, 2025

Uh oh!

edan-bainglass commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

edan-bainglass commented Oct 12, 2025 •

edited

Loading

edan-bainglass Dec 4, 2025 •

edited

Loading

edan-bainglass Dec 4, 2025 •

edited

Loading

edan-bainglass commented Dec 4, 2025 •

edited

Loading

edan-bainglass commented Dec 4, 2025 •

edited

Loading