mem_top

Shows top suspects for memory leaks in your Python program.

Usage:

pip install mem_top
from mem_top import mem_top

# From time to time:
logging.debug(mem_top()) # Or just print().

# Notice which counters keep increasing over time - they are the suspects.

Counters:

• refs - number of direct references from this object to other objects, like keys and values of dict
• bytes - size of this object in bytes
• types - number of objects of this type still kept in memory after garbage collection

Real life example:

refs:
144997  <type 'collections.defaultdict'> defaultdict(<type 'collections.deque'>, {<GearmanJobRequest task='...', unique='.
144996  <type 'dict'> {'.:..............:.......': <GearmanJobRequest task='..................', unique='.................
18948   <type 'dict'> {...
1578    <type 'dict'> {...
968     <type 'dict'> {...
968     <type 'dict'> {...
968     <type 'dict'> {...
767     <type 'list'> [...
726     <type 'dict'> {...
608     <type 'dict'> {...

types:
292499  <type 'dict'>
217912  <type 'collections.deque'>
72702   <class 'gearman.job.GearmanJob'>
72702   <class 'gearman.job.GearmanJobRequest'>
12340   <type '...
3103    <type '...
1112    <type '...
855     <type '...
767     <type '...
532     <type '...

• Noticed a leak of 6GB RAM and counting.
• Added "mem_top" and let it run for a while.
• When got the result above it became absolutely clear who is leaking here:
  the Python client of Gearman kept increasing its counters over time.
• Found its known bug - Yelp/python-gearman#10
  leaking defaultdict of deques, and a dict of GearmanJobRequest-s,
  just as the "mem_top" showed.
• Replaced "python-gearman" - long story: stale 2.0.2 at PyPI, broken 2.0.X at github, etc.
• "mem_top" confirmed the leak is now completely closed.

UPDATES:

• Pass e.g. verbose_types=[dict, list] to get their values sorted by repr length in verbose_file_name.
• Added "bytes" top.

SEE ALSO:

• https://docs.python.org/2/library/gc.html#gc.garbage
• https://pypi.python.org/pypi/objgraph

Config defaults:

mem_top(
    limit=10, width=100, sep='\n',
    refs_format='{num}\t{type} {obj}', bytes_format='{num}\t {obj}', types_format='{num}\t {obj}',
    verbose_types=None, verbose_file_name='/tmp/mem_top',
)

mem_top version 0.1.7
Copyright (C) 2014-2018 by Denis Ryzhkov [email protected]
MIT License, see http://opensource.org/licenses/MIT