Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement some more memory optimizations for article collection #90

Open
frnsys opened this issue Apr 2, 2014 · 1 comment
Open

implement some more memory optimizations for article collection #90

frnsys opened this issue Apr 2, 2014 · 1 comment

Comments

@frnsys
Copy link
Member

frnsys commented Apr 2, 2014

using a generator for membrane.collector.collect helped a lot, but the ponder process is still killed for too much memory consumption (albeit after much longer than it used to be).

so there are still some optimizations which can happen to make this not such a ridiculous memory strain.

take a look at:
http://guillaume.segu.in/blog/code/487/optimizing-memory-usage-in-python-a-case-study/

for profiling memory usage, look at heapy:
https://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended

potentially use namedtuples?
https://docs.python.org/2/library/collections.html#collections.namedtuple

@frnsys frnsys added this to the v0.0.1 (Prototype) milestone Apr 2, 2014
@frnsys frnsys modified the milestone: v0.0.1 (Prototype) Apr 6, 2014
@frnsys
Copy link
Member Author

frnsys commented Apr 6, 2014

dug into this a for a few days and am having a hard time narrowing down what this is. for now chunking out these large tasks into smaller, distributed ones (smaller so that they have shorter process lifespans) might be a good solution, see #112

@frnsys frnsys removed this from the v0.1.0 (Prototype) milestone Apr 7, 2014
frnsys pushed a commit that referenced this issue May 6, 2014
progress on #90 -- have get_articles use a function rather than return a list
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant