FTP connections can sometimes be stuck in CLOSE_WAIT

[andreleblanc11]

It seems like for some servers, FTP connections aren't able to close connections properly.

lsof -u sarra | sort -k4 | grep :ftp | tail -n3
...
python3    953350 sarra   39u     IPv4         1276152460        0t0        TCP host-server:50208->dest-server:ftp (CLOSE_WAIT)
python3    953350 sarra   40u     IPv4         1276152464        0t0        TCP host-server50222->dest-server:ftp (CLOSE_WAIT)
python3    953350 sarra   41u     IPv4         1276747610        0t0        TCP host-server:50238->dest-server:ftp (CLOSE_WAIT)

lsof -u sarra | sort -k4 | grep :ftp.*CLOSE_WAIT | wc -l
67

When I try to setup a manual FTP connection to this same server, it will fail on the ftp.quit() and the FTP connection will remain in the CLOSE_WAIT state .

I verified the same commands with another server, to confirm that the problem wasn't isolated to the sr3 code, and we didn't get the same problem on the ftp.quit().

>>> ftp = ftplib.FTP()
>>> ftp.connect('dest-server', 21, timeout=300)
'220 Connected'
>>> ftp.login('the-user', unquote('the-password'))
'230 Login (the-user)'
>>> ftp.set_pasv(True)
>>> ftp.cwd('/canada')
'250 "/canada" is the current directory.'
>>> ftp.quit()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/ftplib.py", line 659, in quit
    resp = self.voidcmd('QUIT')
  File "/usr/lib/python3.10/ftplib.py", line 286, in voidcmd
    return self.voidresp()
  File "/usr/lib/python3.10/ftplib.py", line 259, in voidresp
    resp = self.getresp()
  File "/usr/lib/python3.10/ftplib.py", line 244, in getresp
    resp = self.getmultiline()
  File "/usr/lib/python3.10/ftplib.py", line 230, in getmultiline
    line = self.getline()
  File "/usr/lib/python3.10/ftplib.py", line 218, in getline
    raise EOFError
EOFError

Looking online, it looks like sometimes FTP servers will return a EOFError when the trying to close the connection.

https://post.bytes.com/forum/topic/python/710163-ftplib-returns-eoferror
https://stackoverflow.com/questions/50914835/eoferror-uploading-csv-file-using-ftplib/50930474#50930474

Current sr3 code isn't able handle this. We'd likely need to add an attempt to ftp.close() and change the except to except EOFError

sarracenia/sarracenia/transfer/ftp.py

Lines 183 to 195 in 3234451

	def close(self):
	logger.debug("sr_ftp close")
	old_ftp = self.ftp
	self.init()
	alarm_set(self.o.timeout)
	try:
	old_ftp.quit()
	except:
	pass
	alarm_cancel()

[andreleblanc11]

When a config is affected by this, it will eventually crash and max out the available ulimit.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sarracenia/instance.py", line 276, in <module>
  File "/usr/lib/python3/dist-packages/sarracenia/instance.py", line 267, in start
  File "/usr/lib/python3/dist-packages/sarracenia/flow/__init__.py", line 608, in run
  File "/usr/lib/python3/dist-packages/sarracenia/flow/__init__.py", line 1207, in work
  File "/usr/lib/python3/dist-packages/sarracenia/flow/sender.py", line 31, in do
  File "/usr/lib/python3/dist-packages/sarracenia/flow/__init__.py", line 2965, in do_send
  File "/usr/lib/python3/dist-packages/sarracenia/flow/__init__.py", line 2420, in send
  File "/local/home/sarra/.local/lib/python3.10/site-packages/magic/__init__.py", line 179, in from_file
  File "/local/home/sarra/.local/lib/python3.10/site-packages/magic/__init__.py", line 112, in from_file
OSError: [Errno 24] Too many open files: '/path/to/my/file'

Sanity will restart it when it does crash.