GC pile up and trouble shooting

[nicrummel]

I am writing some simple mex interfaces that allow me to pass large amounts of data between the two with as little copying as possible. Much of this is inspired by this repo. I was hoping to gain some insight into the debugging process you used with regards to memory management. I have used valgrind, but this is cumbersome. I made a very simple test script, below:

My issue is relate to needing to run the Julia code several times, and I do not want to have close and start a new Julia instance in between calls because of performance needs. I am not as knowledgeable as I need to be to understand if I am/am not allocate memory without deallocation with my methods. Any insight would be helpful.

[twadleigh]

I never used any special tools. It was all a lot of small tests and trial and error. And it took quite a lot of that at the beginning. I should also mention that I never got things working properly on linux. Only windows, and even then there were occasional issues that I never fully ironed out.

I'm not really sure what you are showing in your example and I'm not sure what your question (outside of the one about tools) is. I can say, though, that you shouldn't have to reinitialize the julia runtime between calls to your mex function.

However, if you want julia data to persist between calls, you'll need to make sure that a reference is kept for it somewhere on the Julia side. I think it would be sufficient to have a global ObjectIdDict and push stuff that you don't want GC'd onto it.

And if you want julia to be able to refer to some mxArray between calls into the julia runtime, I think that mxArray has to be marked as persistent on the matlab side (if it is possible at all). My memory on the details here are really fuzzy, though, so I'd refer to the mathworks docs on the matter.

[nicrummel]

Thank you @twadleigh for your quick response. My main concern is that there is a memory leak coming from using the julia API.

• Is jl_gc_total_bytes()' growing continuously (even with all of the popping' and other cleaning functions) problematic?
• Is there any way to wipe my GC completely between uses of Julia with out loosing precompiled code?
• Is there anything obvious I am missing in my understanding of how GC works, and where should I go to learn more?

[twadleigh]

You can always call gc() explicitly. It clears memory, but not the compile cache. However, it is a heavyweight solution, and using it is considered poor form. Still, it is something to try to see if it at least moves things in the right direction for you while you work out the "right" solution.

JL_GC_PUSH is only necessary when (1) that value has no reference to it that persists in julia-land, (2) you are going to do some non-trivial calls into the julia runtime (which might trigger a GC cycle), after which you will still want to be able to refer to that value in C/C++-land. It adds a root to the GC. JL_GC_POP, frees the paired root. Clearly, if you're not careful you will leak memory.

I did my learning about these things mostly by carefully following the documentation on embedding, occasionally chasing deeper into the sources of the API functions (though the insight-to-effort ratio there was often pretty low), gathering insights from experts when they talked about relevant things in the various fora on which I have been lurking for years, but, most of all, by good ol' black-box-interrogation/trial-and-error.