Error Using 32K Fibers

[keithrbennett]

Hi, I am trying to see how many fibers I can use simultaneously without overloading my system, and it seems that it is not possible to use 32K fibers. Is there a configuration parameter I can use to set this? I tried Async::HTTP::Internet.new(connection_limit: 65536), and that changed something, but I still get these warnings:

 8.49s     warn: Async::HTTP::Client [oid=0x161c] [ec=0x1630] [pid=99608] [2024-09-02 15:10:17 +0700]
               | Waiting for Async::HTTP::Protocol::HTTPS pool to drain: #<Async::Pool::Controller(247/65536)

Here is the method I am using:

  def get_responses_using_fibers(count)
    logger.debug("Getting #{count} responses using fibers")
    responses = []
    response_count = 0
    Async do
      begin
        internet = Async::HTTP::Internet.new(connection_limit: 65536)
        count.times.with_object([]) do |_n, results|
          Async do
            begin
              response = internet.get(url)
              response_count += 1
              # logger.debug("rcount: #{response_count}. Received response: #{response.status}")
              responses << response.status
            rescue => e
              logger.error("Error during request: #{e.message}")
            ensure
              response&.finish
            end
          end
        end
      rescue => e
        logger.error("Error during outer: #{e.message}")
      ensure
        internet&.close
      end
    end.wait
    logger.debug("Finished getting #{count} responses. Response codes in JSON were (code: count): #{responses.tally.to_json}")
    responses
  end

The url I am using is "https://httpbin.org/delay/0.0001.

What is the conventional wisdom regarding the maximum number of fibers to use at a time?

[ioquatix]

Have you considered that httpbin might be dropping requests over a certain limit?

[keithrbennett]

That is certainly possible, but on further research I learned that (if I understand correctly) it is not possible to have so many connections open at once, because each connection uses a different client side port, and the number of ephemeral ports available is < 32K. I would have to devise some chunking or pooling scheme to process this many requests. Does that make sense?

[ioquatix]

Yes, although with HTTP/2 you get multiplexing so it should be possible to have 100:1 ratio of requests / TCP connections and I've certainly done more in the past, e.g. https://www.youtube.com/watch?v=Dtn9Uudw4Mo

[keithrbennett]

The author of httpbin posts his email link on his site! I have emailed him to ask about rate limiting and will post his reply here when he responds.