[newcomer] geoip server intermittent failures

[choksi81]

Using library geoip_client.repy, calls to geoip_record_by_addr() (and geoip_record_by_name()) get connection refused error intermittently:
111 Connection refused

The two geoip servers used are http://geoipserver.poly.edu:12679, http://geoipserver2.poly.edu:12679

However, sometimes they do respond to geoip requests.

[asm582]

I was not able to replicate this issue. can you please mention some details.

[aaaaalbert]

I think the thing to do is to create a Repy program that periodically polls the server, e.g. five times in a row, every hour of the day, for two weeks, and then see if the problem exists.

I suggest you run this on a Seattle node.

[asm582]

can this be done through cron? i think we can add this script to cron if possible

[aaaaalbert]

Just write a Repy program. We can reuse that as an integration test later.

[asm582]

I was not able to contact the servers with openconnection so i tried to write python script to connect and get response with them for around 100 times and each time i received success response. I think at present there is no issue and ticket can be closed.

[aaaaalbert]

How come openconnection did not work? Could you paste a traceback please?

[asm582]

i did try to redo all the steps where i found that geoipservers only accepts connection on port 12679 which was later added into restrictions,test. After execution i tried to print openconnection instance and i got the following traceback :-

abhishek@abhishek-VirtualBox:~/seattlelib_v2/RUNNABLE$ python repy.py restrictions.test poll_geoservers.r2py
<Namespace wrapped socket: <emulcomm.EmulatedSocket instance at 0xb6dee7ec>>Terminated

I think i can open connection successfully.

[asm582]

when i try to run the script intermittently i receive below error:-

Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is displayed last.

Full debugging traceback:
"repy.py", line 154, in execute_namespace_until_completion
"/home/abhishek/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/abhishek/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"poll_geoservers.r2py", line 18, in
"/home/abhishek/seattlelib_v2/RUNNABLE/namespace.py", line 1207, in wrapped_function
"/home/abhishek/seattlelib_v2/RUNNABLE/emulcomm.py", line 1322, in openconnection
"/home/abhishek/seattlelib_v2/RUNNABLE/emulcomm.py", line 1057, in _conn_alreadyexists_check

User traceback:
"poll_geoservers.r2py", line 18, in

I think somehow openconnection waits for some timeout may be like TCP FIN for graceful termination and then closes connection but if i run the code before graceful termination it gives error as mentioned in above trace.

[aaaaalbert]

You should be able to work around this by using a different source port for the next connection.

[asm582]

I tried to wireshark analysis on as why the server is not able to connect to any random port in openconnection and it seems that the connection just terminates after three TCP SYN's:-

[aaaaalbert]

You are using 12345 as the destport in openconnection it seems.

[asm582]

yes i am using 12345 as dest port. is that incorrect?

[aaaaalbert]

Yep. From the issue description:

The two geoip servers used are http://geoipserver.poly.edu:12679, http://geoipserver2.poly.edu:12679

Use destport 12679.

[asm582]

yes it connects on that but if i try to re-run program with same port it gives error as connection already exists..

[aaaaalbert]

Use a different source port for the next connection (see also our recent Skype conversation).

[aaaaalbert]

Any updates from you, @asm582 ?

[asm582]

tried to run sample code but i received the below exception:-

User traceback:
"dylink.r2py", line 546, in
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"poll_geoserver.r2py", line 5, in
"geoip_client.r2py", line 74, in geoip_record_by_addr
"xmlrpc_client.r2py", line 109, in send_request
"httpretrieve.r2py", line 317, in httpretrieve_get_string
"httpretrieve.r2py", line 137, in httpretrieve_open
"sockettimeout.r2py", line 301, in timeout_openconnection

Exception (with class 'exception_hierarchy.TimeoutError'): Timed-out connecting to the remote host!

[aaaaalbert]

Try with httpretrieve.r2py from my fork: https://github.com/aaaaalbert/seattlelib_v2/tree/fix-httpretrieve

[asm582]

I have used the httpretrieve.r2py file that you mentioned and implemented the below code:-

https://github.com/asm582/seattleissues/blob/master/poll_geoserver.r2py

Due to TIME_WAIT state of TCP have made a call to sleep for 65 seconds for socket to terminate successfully. while polling the servers we see that geoipserver.poly.edu fails to give response intermittently.
The trace is as below:-

abhishek@abhishek-VirtualBox:~/seattlelib_v2/RUNNABLE$ python repy.py restrictions.test dylink.r2py poll_geoservers.r2py

geoipserver.poly.edu
Cannot contact the server!
Timed-out connecting to the remote host!
geoipserver2.poly.edu
server is alive!

geoipserver.poly.edu
Cannot contact the server!
Timed-out connecting to the remote host!
geoipserver2.poly.edu
server is alive!