Porting PyFDB to Pybind 11 #193

tbkr · 2025-11-07T18:58:06Z

Description

I'm aware that this is a somewhat mid-sized PR 😉 It's also a complete rewrite with tests and documentation.

Porting FDB to PyBind11, making the Python-API more user-friendly and introducing tests for the API layer which are testing core functionality of the FDB in an ephemeral FDB setup.

Documentation is added to the code base as well as the sphinx generated side.

The coverage was tested and is close to 100% testing all provided use-cases and functionalities.

For now I keep this PR in WIP status and I'm open to discuss Implementation details.

Notes for reviewers:

Think about the current API design and raise issues, esp. if you are aware of use-cases which aren't currently implemented
Read the documentation carefully and check whether the written functionalities are correctly mirrored from the FDB.
Feel free to raise potential issues I'm not aware of at this point in time.

Contributor Declaration

By opening this pull request, I affirm the following:

All authors agree to the Contributor License Agreement.
The code follows the project's coding standards.
I have performed self-review and added comments where needed.
I have added or updated tests to verify that my changes are effective and functional.
I have run all existing tests and confirmed they pass.

🌈🌦️📖🚧 Documentation FDB 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/dev-section/fdb/pull-requests/PR-193

codecov-commenter · 2025-11-07T19:03:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.07%. Comparing base (e1d58dd) to head (c21a48d).
⚠️ Report is 16 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #193      +/-   ##
===========================================
+ Coverage    72.93%   73.07%   +0.14%     
===========================================
  Files          362      362              
  Lines        21737    21738       +1     
  Branches      2242     2242              
===========================================
+ Hits         15853    15886      +33     
+ Misses        5884     5852      -32

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Moved from str definition implementation to repr for having debugging information

Wildcard selection is now a boolean flag in the builder

Replaced with __init__(elem, *, _internal=False) and _internal needs to be set to True manually to construct internal objects.

Works now with shutil and there is the option to zero-copy read into a buffer.

simondsmart

Some more comments for you.

Please, where you have dealt with previous comments, can you resolve the relevant comments.

simondsmart · 2026-01-15T10:15:38Z

docs/index.rst

+   z3fdb/index
+
+:ref:`FDB <FDB_Introduction>` itself is part of `ECMWF <https://www.ecmwf.int>`__’s
+high‑performance data infrastructure: it stores each GRIB message as a field,


The FDB is not restricted to GRIBs. I don't want this to be suggested in here.

I note that z3fdb does have ties to gridded data formats. That is a separate discussion.

simondsmart · 2026-01-15T10:19:56Z

docs/pyfdb/examples.rst

+
+.. code-block:: python
+
+    builder = pyfdb.SelectionBuilder()


I'm not convinced by this builder being the recommended way of doing this.

I don't understand why we can't have the typical case just being a

mars_selection = {
"class": "od",
...
"step": [x for x in range(1, 241)]
}

And hiding all of the grunge internally. That is not to say that we can't have a builder to make it easier to do more complex stuff - but I don't think it should be required.

We should probably also accept a simple string, and let metkit do its thing...

simondsmart · 2026-01-15T10:21:50Z

docs/pyfdb/examples.rst

+    fdb = pyfdb.FDB(fdb_config_path)
+    filename = data_path / "x138-300.grib"
+
+    fdb.archive(filename.read_bytes())


It would be good to have an example where the key is supplied along with the data.

simondsmart · 2026-01-15T10:23:42Z

docs/pyfdb/examples.rst

+        "time": "1800",
+    }
+
+    with fdb.retrieve(selection) as data_handle:


What is the context manager doing here? What happens when we leave the managed section.

I was expecting context managers on the archive pathway, but this is less obvious.

simondsmart · 2026-01-15T10:25:54Z

docs/pyfdb/examples.rst

+   URI[scheme=toc,name=/<path-to-db-store>/ea:0001:oper:20200101:1800:g].
+
+
+You can see that the ``ControlIdentifier`` with value ``4`` is active for the given entry of the ``FDB``.


Erm, I see the thumbs up, but not that anything has changed.

simondsmart · 2026-01-30T16:11:14Z

tests/pyfdb/integration/test_selection_mapper.py

@@ -0,0 +1,148 @@
+from pyfdb.pyfdb_type import MarsSelectionMapper
+


To be removed.

simondsmart · 2026-01-30T16:12:48Z

tests/pyfdb/integration/test_list.py

@@ -0,0 +1,181 @@
+from pyfdb import FDB
+


How about range selections? Lists where there isn't data. Lists that select a subset of what is present?

simondsmart · 2026-01-30T16:38:50Z

tests/pyfdb/integration/test_index_axis.py

+from pyfdb.pyfdb_type import WildcardMarsSelection
+
+
+def test_index_axis_string(read_only_fdb_setup):


Please add some tests where you explicitly construct the IndexAxis object, and test equality with the returned value from the fdb.axes() call.

simondsmart · 2026-01-30T16:39:44Z

tests/pyfdb/integration/test_fdb_tool_request.py

+def test_from_selection():
+    fdb_tool_request = FDBToolRequest.from_mars_selection({"key-1": "value-1"})
+
+    print(fdb_tool_request)


This doesn't test anything except for that the code doesn't crash...

simondsmart · 2026-01-30T16:40:43Z

src/pyfdb/_internal/pyfdb_internal.py

+            key_values=selection, all=isinstance(selection, WildcardMarsSelection)
+        )
+
+    def ____repr__(self) -> str:


This doesn't look right (quadruple '_')

tbkr changed the title ~~Feature/pyfdb_integration Porting PyFDB to Pybind 11~~ WIP: Feature/pyfdb_integration Porting PyFDB to Pybind 11 Nov 7, 2025

tbkr force-pushed the feature/pyfdb-integration branch 28 times, most recently from 91dfeff to 2be9698 Compare November 14, 2025 10:24

tbkr added 23 commits January 26, 2026 10:07

Minor: Fixing typos and docs

0ff956e

Minor: Rename PyFDB to FDB

afdfac7

Minor: Fix open/close data handle behavior + docs

d0b3854

Minor: Adjusting documentation

7fe87c5

Docs: Rewrote + Renaming + Python Doc Tests

fb8638d

Minor: Changes to support python 3.11

c6c94c8

Minor: Replaced str by repr

1aab445

Moved from str definition implementation to repr for having debugging information

Minor: Removed class method for wildcard selection

e539205

Wildcard selection is now a boolean flag in the builder

Minor: IndexAxis class adjustment

58a3b96

Minor: Renamed duplicates parameter to include_masked

85c631e

Docs: Added info about notes in FDB init method

d4ed30e

Minor: Adjust __init__.py to be pep8 compliant

3d3ae78

Minor: Renamed needs_flush function of FDB to dirty

042c8b8

Feature: Implemented config as python dict

411aa6d

Feature: Removed _new_ and _from_raw_ methods

3732f98

Replaced with __init__(elem, *, _internal=False) and _internal needs to be set to True manually to construct internal objects.

Minor: Fixing imports

05895a5

Docs: Unified docs

7f179b2

Minor: Added round trip for config + docs

75f348e

Docs: Switched to pydata theme

9f42ed4

Docs: Added MarsSelection and fixed remarks

5464295

Docs: Updated docs

85af47a

Minor: Fix of imports

574576f

Minor: Activated wildcard tests

21f1845

tbkr force-pushed the feature/pyfdb-integration branch from 5812380 to 21f1845 Compare January 26, 2026 10:08

tbkr added 3 commits January 29, 2026 15:08

Update index.rst

1e27927

Feature: Removed SelectionBuilder

600140d

Minor: Adjustment to data_handle read

1fea9bb

Works now with shutil and there is the option to zero-copy read into a buffer.

tbkr force-pushed the feature/pyfdb-integration branch from 51f9ba9 to 1fea9bb Compare January 30, 2026 12:54

Minor: Adjusted documentation

c21a48d

simondsmart approved these changes Jan 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Porting PyFDB to Pybind 11 #193

Porting PyFDB to Pybind 11 #193

tbkr commented Nov 7, 2025 •

edited by github-actions bot

Loading

Uh oh!

codecov-commenter commented Nov 7, 2025 •

edited

Loading

Uh oh!

simondsmart left a comment

Uh oh!

simondsmart Jan 15, 2026

Uh oh!

simondsmart Jan 15, 2026

Uh oh!

simondsmart Jan 15, 2026

Uh oh!

simondsmart Jan 15, 2026

Uh oh!

simondsmart Jan 15, 2026

Uh oh!

simondsmart Jan 30, 2026

Uh oh!

simondsmart Jan 30, 2026

Uh oh!

simondsmart Jan 30, 2026

Uh oh!

simondsmart Jan 30, 2026

Uh oh!

simondsmart Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

		URI[scheme=toc,name=/<path-to-db-store>/ea:0001:oper:20200101:1800:g].


		You can see that the ``ControlIdentifier`` with value ``4`` is active for the given entry of the ``FDB``.

		@@ -0,0 +1,148 @@
		from pyfdb.pyfdb_type import MarsSelectionMapper

		from pyfdb.pyfdb_type import WildcardMarsSelection


		def test_index_axis_string(read_only_fdb_setup):

Porting PyFDB to Pybind 11 #193

Are you sure you want to change the base?

Porting PyFDB to Pybind 11 #193

Conversation

tbkr commented Nov 7, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Notes for reviewers:

Contributor Declaration

Uh oh!

codecov-commenter commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

simondsmart left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

tbkr commented Nov 7, 2025 •

edited by github-actions bot

Loading

codecov-commenter commented Nov 7, 2025 •

edited

Loading