Resolved: Aspell memleak; was: Memory fragmentation #1373
Replies: 5 comments 15 replies
-
|
Gnuplot for raw data: Scroll down for the raw data, then scroll some more for the graphs. |
Beta Was this translation helpful? Give feedback.
-
|
Unlike the earlier version, this one also counts |
Beta Was this translation helpful? Give feedback.
-
|
Graphs
Note the minimum at 50 is also where the average bytes/alloc starts falling. I don't know why these should be related. |
Beta Was this translation helpful? Give feedback.
-
|
Fixed. |
Beta Was this translation helpful? Give feedback.
-
|
I cannot understand how a significant memory fragmentation can be created, as for each sentence the same data structures are mostly of the same size, so free blocks can be just reused instead of splitting bigger free blocks. Some data structures are of "random" size, and may be reallocated frequently while tokenizing/parsing (like the tokenizer alternatives and wordgraph arrays or the link arrays), but there are relatively small. Is it possible to add an allocated sizes histogram? Regarding virtual memory size, there is a potential problem in the pool allocator when it is requested to allocated power-of-2 block sizes. sent->Disjunct_pool = pool_new(__func__, "Disjunct",
/*num_elements*/2048, sizeof(Disjunct),
/*zero_out*/false, /*align*/false, /*exact*/false);
sent->Connector_pool = pool_new(__func__, "Connector",
/*num_elements*/8192, sizeof(Connector),
/*zero_out*/true, /*align*/false, /*exact*/false);If malloc adds its own info in the allocated block, and it itself allocated power-of-2 blocks, then it would need to allocated blocks that are double in size, when most of them is not used. It may be interesting to change In any case, the current memory allocation can be improved in many places to prevent many small allocations that grow on each added element. E.g. in the tokenizer. This will also increase the total speed by a few percents. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Here's are two histograms. First is number of allocation vs requested size. Second is number of allocation vs Below is a histrogram of the sizes of what malloc actually returns These are for niter=5K in |
Beta Was this translation helpful? Give feedback.
-
|
Here's the start of the histogram dataset. It shows how many mallocs there were of various sizes. Here's the tail end Everything 3648 bytes or large goes into the last bin. |
Beta Was this translation helpful? Give feedback.
-
|
Here's something near the end: Column 3 says we asked for 2496 blocks of size 3288. Malloc responded by giving us only 2493 of them, at that size, and 3 more at size 3304, so 16 bytes bigger than what we asked for, When we asked for 249 blocks of size 3328, we got 247 of the next size up, plus two more. When we asked for 39248 blocks of size 3360 .. we got most of them, and a few that were larger. |
Beta Was this translation helpful? Give feedback.
-
|
FYI Blog entry describing glibc memory fragmentation and the reasons for it. https://blog.arkey.fr/drafts/2021/01/22/native-memory-fragmentation-with-glibc/ Examines two alternatives: tcmalloc and jemalloc. Some key takeaways, quoting from the blog:
An important difference in terms of thread management.
There is also the problem that since jemalloc spin new caches to accommodate new thread ids, having a sudden spike of threads will leave you with (mostly) empty caches in the subsequent calm phase. As a result, I would recommend tcmalloc in the general case, and reserve jemalloc for very specific usages (low variation on the number of threads during the lifetime of the application). |
Beta Was this translation helpful? Give feedback.
-
|
Debian has but the persistent RSS growth remains. BTW, |
Beta Was this translation helpful? Give feedback.
-
|
With |
Beta Was this translation helpful? Give feedback.
-
|
Periodically inserting is printed just when |
Beta Was this translation helpful? Give feedback.
-
|
BTW, This blog entry suggests some reasonable malloc tuning ideas: https://developers.redhat.com/blog/2017/03/02/malloc-internals-and-you#thread_local_cache |
Beta Was this translation helpful? Give feedback.
-
|
It's the freakin spell-guesser. Setting This can't be our fault, at least not directly. The instrumented debug code, which overloads malloc and free, does not find any problems in |
Beta Was this translation helpful? Give feedback.
-
|
I've got a simple unit test. I'll stuff it into git, shortly. It leaks 64 bytes per spelling suggestion. Zooms up to a gigabyte of RSS in a few minutes. |
Beta Was this translation helpful? Give feedback.
-
|
The below leaks. It appears to be 100% correct, according to the aspell documentation. The |
Beta Was this translation helpful? Give feedback.
-
|
I posted the above code to the aspell-users mailing list. https://lists.gnu.org/archive/html/aspell-user/2023-01/msg00000.html I'm not expecting a reply, though; The mailing list is dead: 10 emails over the last 3 years, not a single one has gotten a reply. There's no way to .. report bugs. |
Beta Was this translation helpful? Give feedback.
-
|
Oh wait, I was wrong: https://github.com/GNUAspell/aspell I just now opened GNUAspell/aspell#632 ... but again ... project appears to be dead. There are 15 unmerged pull reqs, and new issues don't seem to get discussion ... |
Beta Was this translation helpful? Give feedback.
-
|
I disabled aspell by default in #1376 -- I consider this issue closed. I'm happy that the resolution was this easy. |
Beta Was this translation helpful? Give feedback.
-
Issue #1366 caused me to take a good look at memory fragmentation in link-grammar. Basically, when running for a long time, RAM usage will slowly grow. This is not a memory leak (as far as I can tell), this is memory fragmentation. Here's a report if what I've found.
Test case is the
tests/multi-threadbinary, altered to run 50000 iterations.There's no conventional memleak:
valgrind --leak-check-fullreports no leakage. i.e. all alloc'ed memory is freed at the end.It could still happen that there is some mempool that grows out of bound. However, this does not seem to happen. This size of alloced memory stays constant. (Actually, it seems to shrink, slightly!??)
Measurement: redefine
malloc,realloc,strdupandfreeand count who is mallocing freeing. This is done inmalloc-dbg2.c. It is non-trivial to get this to compile, so themalloc-debug-installedbranch inlinas/link-grammarcontains the needed hacks to getmulti-threadto compile and run. The printout will report which files are mallocig and freeing, and how much.Results:
topstarts at about 200MB and then grows to 4GB by the end.The average size of an alloc is slowly shrinking, probably because the reading of the dict uses larger allocs (e.g. the entire file) while parsing uses smaller allocs. I should probably restart counting after dict open. Hmmm.
The fragmentation rate seems to be 0.1% -- we have to malloc & free almost 1TB before 1GB RSS is lost.
While measuring this, I reorganized the code base so that most mallocs & the matching frees happen in the same file. Thus the memory report prints results, per file.
Beta Was this translation helpful? Give feedback.
All reactions