Elapsed time to download content is zero.

[BabakAmini]

As far as I know, session.get() loads headers from the destination server, and then an API like resp.text() downloads the entire content from it. I need to save elapsed time after each step for further investigation. Below is a sample code to do this task:

def do_request(self):
    self.headers_fetch_elapsed_time, self.text_fetch_elapsed_time = 0, 0
    t1 = time.perf_counter()

    async with self.session.get(self.url) as resp:
        t2 = time.perf_counter()
        self.headers_fetch_elapsed_time = round(1000 * (t2 - t1))
        self.txt = await resp.text()
        self.text_fetch_elapsed_time = round(1000 * (time.perf_counter() - t2))

I use the above function multiple times by a specific URL but most of the time self.text_fetch_elapsed_time value is zero. At first, I thought there might be a caching system in aiohttp but, I'm pretty sure that each request to the server returns a completely new response.

So, why is this happening?

[Dreamsorcerer]

As far as I know, session.get() loads headers from the destination server, and then an API like resp.text() downloads the entire content from it.

Not exactly, that's not how HTTP works. It downloads a small amount of data (maybe 4KB or something) and parses the headers from that before returning. When you call any variation of .read() it will then continue from the buffered data and fetch anything more from the server if needed. So, if your responses are small, the network request will have been completed and the connection released before your call to .text().

[Dreamsorcerer]

For your information, as the server rejects keep-alive connections, I have to use a TCPConnector with force_close=True.

Not that I'm aware of. Why would you want to access the closed (or possibly reused) connection? I can't imagine a case for general app development, but for internal aiohttp tests we've sometimes checked the connections directly in the connection pool.

[BabakAmini]

I need to save remote IPs for further investigations.

[Dreamsorcerer]

I'm not too familiar with it myself, but I think you can probably get that information through the tracing API (e.g. when DNS resolution happens or the connection is acquired): https://docs.aiohttp.org/en/stable/client_advanced.html#client-tracing

[BabakAmini]

Oh, I already implemented that part of my code but forgot about it. Maybe I should check it again and find something relative, it slows down the code though.

Thank you again for your assistance.

[Dreamsorcerer]

Well, without tracing, you could probably dig into the DNS cache and just pick out the IPs used there. I don't remember much about it, so you might need to dig into the code to figure that out.