Skip to content
This repository has been archived by the owner on Sep 27, 2022. It is now read-only.

Error Handshake status 429 Too Many Requests running more than 10 workers that connect to node hawking. #16

Open
denis-trofimov opened this issue Sep 24, 2018 · 4 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@denis-trofimov
Copy link
Owner

denis-trofimov commented Sep 24, 2018

Upon doing read operations by means of python-bitshares client-side library for the BitShares Blockchain based on graphenelib on "low level" in the websocket module got "websocket._exceptions.WebSocketBadStatusException: Handshake status 429 Too Many Requests".

During my stress-test I launch 50 workers that connect to node": "ws://hawking.array.io:8090", each one worker performs read operation and got it result, and start over with new connection.
It looks like a server-side connection denial counter flood mechanism. As referenced 429 TOO MANY REQUESTS:

The user has sent too many requests in a given amount of time ("rate limiting").

The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.

Log

1537801622.945449 - INFO - __init__ - Start processing of scenario file "loop_scenario.json".
1537801623.317932 - WARNING - connect - Connection is already closed.
1537801623.323527 - WARNING - connect - Handshake status 429 Too Many Requests
1537801623.323527 - WARNING - connect - Handshake status 429 Too Many Requests
1537801623.324548 - WARNING - connect - Handshake status 429 Too Many Requests

Using

{
  "scenarios": [
    {
      "node": "ws://hawking.array.io:8090",
      "cycles": 1000,
      "workers": 50,
      "time_limit": 20,
      "stages": [
        {
          "method": "get_block",
          "params": {
            "block_num": 8
          }
        },
        {
          "method": "get_transaction",
          "params": {
            "block_num": 8,
            "trx_in_block": 0
          }
        },
        {
          "method": "get_chain_properties"
        },
        {
          "method": "get_global_properties"
        },
        {
          "method": "get_config"
        },
        {
          "method": "get_chain_id"
        },
        {
          "method": "get_dynamic_global_properties"
        },
        {
          "method": "get_accounts",
          "params": {
            "account_ids": [
              "1.2.0",
              "1.2.1"
            ]
          }
        }
      ]
    }
  ]
}

Got error

python3 loop-scenario.py loop_scenario.json 
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/api.py", line 65, in connect
    self.connection.connect()
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/websocket.py", line 26, in connect
    self.ws.connect(self.url)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/websocket/_core.py", line 222, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/websocket/_handshake.py", line 69, in handshake
    status, resp = _get_resp_headers(sock)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/websocket/_handshake.py", line 140, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message)
websocket._exceptions.WebSocketBadStatusException: Handshake status 429 Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/api.py", line 65, in connect
    self.connection.connect()
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/websocket.py", line 26, in connect
    self.ws.connect(self.url)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/websocket/_core.py", line 222, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/websocket/_handshake.py", line 69, in handshake
    status, resp = _get_resp_headers(sock)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/websocket/_handshake.py", line 140, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message)
websocket._exceptions.WebSocketBadStatusException: Handshake status 429 Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "loop-scenario.py", line 72, in make_call
    return NodeCall(args[0]).call_wrapper(args[1], args[2], args[3])
  File "loop-scenario.py", line 168, in call_wrapper
    result['result']: str = call(self, **kwargs)
  File "loop-scenario.py", line 183, in get_chain_properties
    self.chain = Blockchain(blockchain_instance=self.bts)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/bitshares/blockchain.py", line 28, in __init__
    BlockchainInstance.__init__(self, *args, **kwargs)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/bitshares/instance.py", line 14, in __init__
    self.blockchain = shared_blockchain_instance()
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/bitshares/instance.py", line 43, in shared_blockchain_instance
    SharedInstance.instance = bts.BitShares(**SharedInstance.config)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/bitshares/bitshares.py", line 145, in __init__
    **kwargs)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/bitshares/bitshares.py", line 174, in connect
    self.rpc = BitSharesNodeRPC(node, rpcuser, rpcpassword, **kwargs)
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/api.py", line 40, in __init__
    self.connect()
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/api.py", line 69, in connect
    self.next()
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/api.py", line 127, in next
    self.connect()
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/api.py", line 69, in connect
    self.next()
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/api.py", line 126, in next
    self.url = self.find_next()
  File "/home/denis/.virtualenvs/array/lib/python3.6/site-packages/grapheneapi/api.py", line 109, in find_next
    raise NumRetriesReached
grapheneapi.exceptions.NumRetriesReached
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "loop-scenario.py", line 238, in <module>
    Scenario(args.file)
  File "loop-scenario.py", line 61, in __init__
    NodeSequence(node_call, self.roundup).run()
  File "loop-scenario.py", line 128, in run
    for result in multiple_results.get():
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
grapheneapi.exceptions.NumRetriesReached
denis-trofimov added a commit that referenced this issue Sep 24, 2018
@denis-trofimov denis-trofimov added the bug Something isn't working label Sep 24, 2018
@denis-trofimov
Copy link
Owner Author

Can not proceed after that bug. Connection with one worker give same "Too Many Requests"

1537824124.631129 - INFO - __init__ - Start processing of scenario file "loop_scenario.json".
1537824125.316673 - WARNING - connect - Handshake status 429 Too Many Requests
1537824125.534600 - WARNING - connect - Handshake status 429 Too Many Requests

No stray python processes.

denis@denis-HP-625:~/p/bitshares-stress-test$ ps aux | grep python
root      5009  0.0  0.2 177632 17076 ?        Ssl  сен24   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
denis     8169  0.0  0.0  21536  1104 pts/2    S+   00:25   0:00 grep --color=auto python
denis    11186  0.1  0.5 758076 42252 tty1     Sl+  сен24   0:17 python3 /usr/bin/yandex-disk-indicator
denis    30147  1.9  2.9 2479708 230308 tty1   Sl+  сен24   0:58 /usr/bin/python3 /usr/lib/python3/dist-packages/eric6/eric6.py
denis    30158  0.0  0.2  48044 16912 tty1     S+   сен24   0:00 /usr/bin/python3 /usr/lib/python3/dist-packages/eric6/Utilities/BackgroundClient.py 127.0.0.1 41381 0

Running scenario

{
  "scenarios": [
    {
      "node": "ws://hawking.array.io:8090",
      "cycles": 1000,
      "workers": 1,
      "time_limit": 20,
      "stages": [
        {
          "method": "get_block",
          "params": {
            "block_num": 8
          }
        },
        {
          "method": "get_transaction",
          "params": {
            "block_num": 8,
            "trx_in_block": 0
          }
        },
        {
          "method": "get_chain_properties"
        },
        {
          "method": "get_global_properties"
        },
        {
          "method": "get_config"
        },
        {
          "method": "get_chain_id"
        },
        {
          "method": "get_dynamic_global_properties"
        },
        {
          "method": "get_accounts",
          "params": {
            "account_ids": [
              "1.2.0",
              "1.2.1"
            ]
          }
        }
      ]
    }
  ]
}

@denis-trofimov denis-trofimov changed the title Error running loop version of read operations. Error Handshake status 429 Too Many Requests running loop version of read operations. Sep 25, 2018
@denis-trofimov
Copy link
Owner Author

https://httpstatuses.com/429
429 TOO MANY REQUESTS
The user has sent too many requests in a given amount of time ("rate limiting").

The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.

@denis-trofimov denis-trofimov added the help wanted Extra attention is needed label Sep 25, 2018
@denis-trofimov
Copy link
Owner Author

Googling I got https://stackoverflow.com/questions/22786068/how-to-avoid-http-error-429-too-many-requests-python
After that test run I got same I just got a 429 on my first ever request.
Looks like I am banned on server by IP?

@denis-trofimov denis-trofimov changed the title Error Handshake status 429 Too Many Requests running loop version of read operations. Error Handshake status 429 Too Many Requests running 50 workers that connect to node hawking. Sep 25, 2018
@denis-trofimov denis-trofimov changed the title Error Handshake status 429 Too Many Requests running 50 workers that connect to node hawking. Error Handshake status 429 Too Many Requests running more than 10 workers that connect to node hawking. Sep 26, 2018
@denis-trofimov
Copy link
Owner Author

confirm http 1.1 promotion to websocket

tcpkali -T 1 -vvv -c1 --channel-lifetime=0.1 -r1 --ws -m '{"method": "call", "params": [1, "login", ["", ""]], "id": 2}' hawking.array.io:8090
WARNING: Dumb terminal, expect unglorified output.
Destination: [104.155.104.137]:8090
Interface enp0s31f6 address [192.168.0.212]:0
Using interface enp0s31f6 to connect to [104.155.104.137]:8090
Connection [192.168.0.212]:44615 -> [104.155.104.137]:8090 opened as 12
Ramped up to 1 connections.
Connection [192.168.0.212]:47565 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:38487 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:35613 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:43897 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:55587 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:59747 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:39317 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:33967 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:55549 -> [104.155.104.137]:8090 opened as 12
Connection [192.168.0.212]:47855 -> [104.155.104.137]:8090 opened as 12
Exiting worker 0
  0↓, 0↑ open connections (0 connecting)
  11 connections_counter 
  ↳ 11 connections_initiated
  ↳ 0 connections_accepted
  0 connection_failures
  ↳ 0 connection_timeouts
  1600 worker_data_sent
  1600 worker_data_rcvd
  10 worker_num_writes
  10 worker_num_reads
Last received bytes (160): [HTTP/1.1 101 Switching Protocols\r\n
	Connection: upgrade\r\n
	Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=\r\n
	Server: WebSocket++/0.8.0-dev\r\n
	Upgrade: websocket\r\n
	\r\n]
Total data sent:     1600 bytes (1600 bytes)
Total data received: 1600 bytes (1600 bytes)
Bandwidth per channel: 0.025⇅ Mbps (3.2 kBps)
Aggregate bandwidth: 0.013↓, 0.013↑ Mbps
Packet rate estimate: 10.0↓, 10.0↑ (1↓, 1↑ TCP MSS/op)
Test duration: 1.00479 s.

@denis-trofimov denis-trofimov removed this from the benchmarking read-only tool milestone Oct 1, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant