You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
using a generator for membrane.collector.collect helped a lot, but the ponder process is still killed for too much memory consumption (albeit after much longer than it used to be).
so there are still some optimizations which can happen to make this not such a ridiculous memory strain.
dug into this a for a few days and am having a hard time narrowing down what this is. for now chunking out these large tasks into smaller, distributed ones (smaller so that they have shorter process lifespans) might be a good solution, see #112
frnsys
removed this from the
v0.1.0 (Prototype) milestone
Apr 7, 2014
using a generator for
membrane.collector.collect
helped a lot, but the ponder process is still killed for too much memory consumption (albeit after much longer than it used to be).so there are still some optimizations which can happen to make this not such a ridiculous memory strain.
take a look at:
http://guillaume.segu.in/blog/code/487/optimizing-memory-usage-in-python-a-case-study/
for profiling memory usage, look at
heapy
:https://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended
potentially use
namedtuples
?https://docs.python.org/2/library/collections.html#collections.namedtuple
The text was updated successfully, but these errors were encountered: