Is web_request.py and web_response.py supposed to grow in memory as incoming requests increase ?

[ShamariYoti]

I have a question. I've been trying to detect a memory leak because on our graph it looked like a slow increase of the memory usage which doesn't get cleaned up. I used tracemalloc trace the memory usage and the two areas that seemed to grow in memory is web_request.py and web_response.py. You can see on the image that web_request.py grew to 21.2 MiB.

I wrote a small test program to see if I could get the same behaviour and I did.

from aiohttp import web
import asyncio
import tracemalloc

from gunicorn.app.base import BaseApplication


class GunicornApplication(BaseApplication):
    def __init__(self, application, options):
        self.app = application
        self.options = options

        super(GunicornApplication, self).__init__()

    def load_config(self):
        self.cfg.set('bind', 'localhost:8080')
        self.cfg.set('workers', 1)
        self.cfg.set('worker_class', '%s' % 'aiohttp.worker.GunicornWebWorker')
        self.cfg.set('graceful_timeout', 30)

    def load(self):
        return self.app


async def index(request):
    print(f'read request')
    req = await request.json()

    return web.Response(text="Request has been receieved")

async def on_startup(app) -> None:
    asyncio.create_task(show_memory())

async def show_memory():
    print('start tracing memory')

    tracemalloc.start(25)
    start = tracemalloc.take_snapshot()

    snapshot_num = 1
    while True:
        await asyncio.sleep(20)
        current = tracemalloc.take_snapshot()
        # compare current snapshot to starting snapshot
        stats = current.compare_to(start, 'filename')

        print('top diffs since start')
        # print top diffs: current snapshot - start snapshot
        for i, stat in enumerate(stats[:15], 1):
            print(f'top diffs: {i}, {str(stat)}')

        traces = current.statistics('traceback')
        for stat in traces[:2]:
            for line in stat.traceback.format():
                print(line)

        snapshot_num = snapshot_num + 1


my_web_app = web.Application()
my_web_app.router.add_route('POST', '/image', index)
my_web_app.on_startup.append(on_startup)

g = GunicornApplication(my_web_app, 'None')
g.run()

My question is, shouldn't aiohttp be cleaning up requests once they have been served, and is this a possible leak ?

[Dreamsorcerer]

#4618

[ShamariYoti]

Hi, I think this is slightly different to my case? It seems that's for a client sending a request not a server receiving a request ? Also from looking at the code it looks like the request is being read correctly and therefore the memory should be released ?

[ShamariYoti]

Hi @Dreamsorcerer, why did you mark this as resolved ?

[Dreamsorcerer]

It seems that's for a client sending a request not a server receiving a request

Then it's probably just gunicorn in your example. Have you tried trimming the memory as shown at the end of that thread?

[ShamariYoti]

I've also ran the code without gunicorn but still get the same behaviour.