Add `asyncio` backend #930

MarkusSintonen · 2024-06-16T18:27:20Z

Summary

Adds back native asyncio backend based on commit where it was removed here. Adjusts it to work with the new interfaces. This also adds some additional integration testing.

Doesn't make the new backend the default.

AnyIO has considerable overhead so this allows again using the native backend. Also this is going to allow removal of the AnyIO as the dependency (when the synchronization PR goes in also).

Overhead is about 1.45x here (the synchronization has a lot more overhead in the other PR).

Previously (is master):

Now:

Related discussion encode/httpx#3215 (comment).

Checklist

I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
I've updated the documentation accordingly.

MarkusSintonen · 2024-06-16T18:54:37Z

tests/conftest.py

+    return backend_name, options
+
+
+class Server(uvicorn.Server):


Seems the httpbin fixture doesnt work well for pool related testing as it always closes the connections immeaditely... So adding this for testing those aspects in integration testing file

tomchristie · 2024-06-17T09:50:58Z

In order to land this PR I'd suggest adding an asyncio native backend without any other changes.

The default backend should not change as part of the pull request, and the existing anyio backend should not be modified. I'd expect the only modified code files to be __init__.py and httpcore/_backends/asyncio.py.

The documentation for https://www.encode.io/httpcore/network-backends/#async-network-backends should also be modified to include the native asyncio backend, demo'ing it's usage with...

    network_backend = httpcore.AsyncIOBackend()
    async with httpcore.AsyncConnectionPool(network_backend=network_backend) as http:
        ...

Similar to the existing AnyIOBackend example.

(Minimal incremental changes ftw)

MarkusSintonen · 2024-06-17T12:25:14Z

The default backend should not change as part of the pull request

Yep that I didnt change as you previously suggested

MarkusSintonen · 2024-06-17T13:02:55Z

In order to land this PR I'd suggest adding an asyncio native backend without any other changes.

The default backend should not change as part of the pull request, and the existing anyio backend should not be modified. I'd expect the only modified code files to be __init__.py and httpcore/_backends/asyncio.py.

This is now done, so there is only the new backend and export. + The tests I used add the full coverage and verify it works

MarkusSintonen · 2024-06-17T13:48:48Z

The documentation for https://www.encode.io/httpcore/network-backends/#async-network-backends should also be modified to include the native asyncio backend, demo'ing it's usage with

Added this

MarkusSintonen · 2024-06-18T08:26:37Z

Asyncio-backend performs also better when all other current optimization PRs are applied (benchmark params REPEATS=100 REQUESTS=50 CONCURRENCY=20):

With AnyIO-backend:

With asyncio-backend:

tomchristie · 2024-06-18T08:52:23Z

The code & docs changes look reasonable.
Not obvious that the tests cover the functionality added?

MarkusSintonen · 2024-06-18T13:15:42Z

Not obvious that the tests cover the functionality added?

Previously the @pytest.mark.anyio runned the tests as parametrized by asyncio-anyio and trio (which are the default). But now via the conftest, we specify the test parametrization to also include the plain asyncio test mode (as documented in AnyIO fixture docs https://anyio.readthedocs.io/en/stable/testing.html#specifying-the-backends-to-run-on).

With this and the added integration testing (eg UDS, socket options) we get close to 100% coverage for the new network backend. (Actually we get even more coverage for anyio/trio backends so there is some no longer needed nocoverage)

httpcore/_backends/asyncio.py

tomchristie

Looks great, thanks so much.

Naming... should we use AsyncIOStream and AsyncIOBackend or AsyncioStream and AsyncioBackend?

bdraco · 2024-09-29T02:10:56Z

Thanks for doing this. I think this will close out the discussion I opened #893

MarkusSintonen · 2024-10-01T07:53:24Z

Looks great, thanks so much.

Naming... should we use AsyncIOStream and AsyncIOBackend or AsyncioStream and AsyncioBackend?

No problem! Its now renamed to AsyncIOBackend. Thanks

docs/network-backends.md

tomchristie · 2024-10-01T10:44:06Z

Fantastic, appreciate your time on this @MarkusSintonen.
Following SEMVER we'll want this in a 1.1.0 release. (New functionality)

We could either get #957 released first then merge this, or include this now and bump the version.

MarkusSintonen · 2024-10-01T11:03:42Z

@tomchristie no problem! Would it be possible to include also the asyncio-based synchronization changes as part of this?

tomchristie · 2024-10-01T11:40:18Z

httpcore/_backends/asyncio.py

+        self._read_lock = asyncio.Lock()
+        self._write_lock = asyncio.Lock()


We don't use locking at the stream layer, let's remove these.

(I think we had locking at this layer in the past, the locking is now handled at the HTTP11Connection/HTTP2Connection layer)

Done, they are now gone

tomchristie · 2024-10-01T11:40:31Z

httpcore/_backends/asyncio.py

+        self._read_lock = asyncio.Lock()
+        self._write_lock = asyncio.Lock()


Suggested change

self._read_lock = asyncio.Lock()

self._write_lock = asyncio.Lock()

tomchristie · 2024-10-01T11:41:24Z

httpcore/_backends/asyncio.py

+        async with self._read_lock:
+            with map_exceptions(exc_map):
+                try:
+                    return await asyncio.wait_for(
+                        self._stream_reader.read(max_bytes), timeout
+                    )
+                except AttributeError as exc:  # pragma: nocover
+                    if "resume_reading" in str(exc):
+                        # Python's asyncio has a bug that can occur when a
+                        # connection has been closed, while it is paused.
+                        # See: https://github.com/encode/httpx/issues/1213
+                        #
+                        # Returning an empty byte-string to indicate connection
+                        # close will eventually raise an httpcore.RemoteProtocolError
+                        # to the user when this goes through our HTTP parsing layer.
+                        return b""
+                    raise


Suggested change

async with self._read_lock:

with map_exceptions(exc_map):

try:

return await asyncio.wait_for(

self._stream_reader.read(max_bytes), timeout

)

except AttributeError as exc: # pragma: nocover

if "resume_reading" in str(exc):

# Python's asyncio has a bug that can occur when a

# connection has been closed, while it is paused.

# See: https://github.com/encode/httpx/issues/1213

#

# Returning an empty byte-string to indicate connection

# close will eventually raise an httpcore.RemoteProtocolError

# to the user when this goes through our HTTP parsing layer.

return b""

raise

with map_exceptions(exc_map):

try:

return await asyncio.wait_for(

self._stream_reader.read(max_bytes), timeout

)

except AttributeError as exc: # pragma: nocover

if "resume_reading" in str(exc):

# Python's asyncio has a bug that can occur when a

# connection has been closed, while it is paused.

# See: https://github.com/encode/httpx/issues/1213

#

# Returning an empty byte-string to indicate connection

# close will eventually raise an httpcore.RemoteProtocolError

# to the user when this goes through our HTTP parsing layer.

return b""

raise

tomchristie · 2024-10-01T11:41:47Z

httpcore/_backends/asyncio.py

+        async with self._write_lock:
+            with map_exceptions(exc_map):
+                self._stream_writer.write(data)
+                return await asyncio.wait_for(self._stream_writer.drain(), timeout)


Suggested change

async with self._write_lock:

with map_exceptions(exc_map):

self._stream_writer.write(data)

return await asyncio.wait_for(self._stream_writer.drain(), timeout)

with map_exceptions(exc_map):

self._stream_writer.write(data)

return await asyncio.wait_for(self._stream_writer.drain(), timeout)

httpcore/_backends/asyncio.py

tomchristie · 2024-10-01T11:46:31Z

Would it be possible to include also the asyncio-based synchronization changes as part of this?

Thanks for the ask, let's not do that. Mixing too many different concerns.
This is a self-contained PR and we can deal with it in isolation.

(Minimal incremental changes ftw)

MarkusSintonen force-pushed the asyncio-backend2 branch 2 times, most recently from 0d3a1f4 to db58896 Compare June 16, 2024 18:37

MarkusSintonen marked this pull request as ready for review June 16, 2024 18:51

MarkusSintonen mentioned this pull request Jun 16, 2024

Connection pool optimization: move socket polling from expiry checks to connection usage #928

Open

3 tasks

MarkusSintonen commented Jun 16, 2024

View reviewed changes

MarkusSintonen force-pushed the asyncio-backend2 branch 3 times, most recently from 009249a to 56447f4 Compare June 17, 2024 09:50

Add asyncio backend. Add integration testing

81b6b1f

MarkusSintonen force-pushed the asyncio-backend2 branch from 56447f4 to 81b6b1f Compare June 17, 2024 12:20

Simplify patch set

828c491

Update doc

b47b4c2

Add network backend benchmark param

3814cf4

MarkusSintonen force-pushed the asyncio-backend2 branch from 9f13ff5 to 3814cf4 Compare June 18, 2024 12:17

MarkusSintonen mentioned this pull request Jul 10, 2024

Improve async performance. encode/httpx#3215

Open

Merge branch 'master' into asyncio-backend2

ac533ed

tomchristie reviewed Sep 27, 2024

View reviewed changes

httpcore/_backends/asyncio.py Outdated Show resolved Hide resolved

tomchristie reviewed Sep 27, 2024

View reviewed changes

Rename to AsyncIOBackend

30f9c85

Merge branch 'master' into asyncio-backend2

0038a0a

tomchristie reviewed Oct 1, 2024

View reviewed changes

docs/network-backends.md Outdated Show resolved Hide resolved

tomchristie reviewed Oct 1, 2024

View reviewed changes

docs/network-backends.md Outdated Show resolved Hide resolved

tomchristie reviewed Oct 1, 2024

View reviewed changes

docs/network-backends.md Outdated Show resolved Hide resolved

Apply suggestions from code review

07772b1

tomchristie reviewed Oct 1, 2024

View reviewed changes

httpcore/_backends/asyncio.py Outdated Show resolved Hide resolved

Remove unneeded locks

ae741c0

RyanMarten mentioned this pull request Dec 4, 2024

[Bug]: Cannot get past 50 RPS BerriAI/litellm#6592

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `asyncio` backend #930

Add `asyncio` backend #930

MarkusSintonen commented Jun 16, 2024 •

edited

Loading

MarkusSintonen Jun 16, 2024 •

edited

Loading

tomchristie commented Jun 17, 2024 •

edited

Loading

MarkusSintonen commented Jun 17, 2024

MarkusSintonen commented Jun 17, 2024

MarkusSintonen commented Jun 17, 2024

MarkusSintonen commented Jun 18, 2024

tomchristie commented Jun 18, 2024

MarkusSintonen commented Jun 18, 2024 •

edited

Loading

tomchristie left a comment

bdraco commented Sep 29, 2024

MarkusSintonen commented Oct 1, 2024

tomchristie commented Oct 1, 2024

MarkusSintonen commented Oct 1, 2024

tomchristie Oct 1, 2024

MarkusSintonen Oct 1, 2024

tomchristie Oct 1, 2024

MarkusSintonen Oct 1, 2024

tomchristie Oct 1, 2024

MarkusSintonen Oct 1, 2024

tomchristie Oct 1, 2024

MarkusSintonen Oct 1, 2024

tomchristie commented Oct 1, 2024 •

edited

Loading

		self._read_lock = asyncio.Lock()
		self._write_lock = asyncio.Lock()

Add asyncio backend #930

Are you sure you want to change the base?

Add asyncio backend #930

Conversation

MarkusSintonen commented Jun 16, 2024 • edited Loading

Summary

Checklist

MarkusSintonen Jun 16, 2024 • edited Loading

Choose a reason for hiding this comment

tomchristie commented Jun 17, 2024 • edited Loading

MarkusSintonen commented Jun 17, 2024

MarkusSintonen commented Jun 17, 2024

MarkusSintonen commented Jun 17, 2024

MarkusSintonen commented Jun 18, 2024

tomchristie commented Jun 18, 2024

MarkusSintonen commented Jun 18, 2024 • edited Loading

tomchristie left a comment

Choose a reason for hiding this comment

bdraco commented Sep 29, 2024

MarkusSintonen commented Oct 1, 2024

tomchristie commented Oct 1, 2024

MarkusSintonen commented Oct 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomchristie commented Oct 1, 2024 • edited Loading

Add `asyncio` backend #930

Add `asyncio` backend #930

MarkusSintonen commented Jun 16, 2024 •

edited

Loading

MarkusSintonen Jun 16, 2024 •

edited

Loading

tomchristie commented Jun 17, 2024 •

edited

Loading

MarkusSintonen commented Jun 18, 2024 •

edited

Loading

tomchristie commented Oct 1, 2024 •

edited

Loading