Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase reference count also in other load_chunk overload #1225

Merged
merged 2 commits into from
Apr 5, 2022

Conversation

franzpoeschel
Copy link
Contributor

Don't know if it is necessary, but looks like we forgot it back when we added this reference count trick.

Extracted from #1197.

Don't know if it is necessary, but looks like we forgot it earlier
Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx for catching that!

x-ref: #912 #833

@ax3l
Copy link
Member

ax3l commented Mar 14, 2022

In #912, I noted that we did not yet have an in-memory overload for load_chunk in Python.

Do we have one yet that can trigger the issue and shall we add a test similar to #912?

@franzpoeschel
Copy link
Contributor Author

franzpoeschel commented Mar 14, 2022

In #912, I noted that we did not yet have an in-memory overload for load_chunk in Python.

Do we have one yet that can trigger the issue and shall we add a test similar to #912?

This PR did not change the public Python API, just fixed a potential memory bug

I misunderstood your post, I'll need to have a look where this function is actually called

@ax3l ax3l self-assigned this Mar 14, 2022
@ax3l
Copy link
Member

ax3l commented Apr 5, 2022

Yes, I think we don't have a test in test/python/unittest/API/APITest.py for pre-allocated arrays yet, but let's add it :)

We also need to add a pybind11 signature to store into a pre-allocated python array (we can check its extends when it is passed to us in load_chunk).

@ax3l ax3l merged commit 9c4173a into openPMD:dev Apr 5, 2022
@franzpoeschel
Copy link
Contributor Author

We also need to add a pybind11 signature to store into a pre-allocated python array (we can check its extends when it is passed to us in load_chunk).

It seems that this exists already

        .def(
            "load_chunk",
            [](RecordComponent &r,
               py::buffer buffer,
               Offset const &offset_in,
               Extent const &extent_in) {
                uint8_t ndim = r.getDimensionality();

                // default arguments
                //   offset = {0u}: expand to right dim {0u, 0u, ...}
                Offset offset = offset_in;
                if (offset_in.size() == 1u && offset_in.at(0) == 0u)
                    offset = Offset(ndim, 0u);

                //   extent = {-1u}: take full size
                Extent extent(ndim, 1u);
                if (extent_in.size() == 1u && extent_in.at(0) == -1u)
                {
                    extent = r.getExtent();
                    for (uint8_t i = 0u; i < ndim; ++i)
                        extent[i] -= offset[i];
                }
                else
                    extent = extent_in;

                std::vector<bool> flatten(ndim, false);
                load_chunk(r, buffer, offset, extent);
            },
            py::arg("pre-allocated buffer"),
            py::arg_v(
                "offset", Offset(1, 0u), "np.zeros(Record_Component.shape)"),
            py::arg_v("extent", Extent(1, -1u), "Record_Component.shape"))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants