Implicit FTPS support

[reidsunderland]

ESA SMOS recently stopped allowing insecure FTP access to their data and now only allows FTPS.

https://smos-diss.eo.esa.int/oads/access/

The existing FTPS code doesn't work and gives this error:

2024-06-25 14:42:07,009 [ERROR] 457523 sarracenia.flowcb.poll poll sr_poll/post_new_url: unable to connect to ftps://[email protected]@smos-diss.eo.esa.int/

FileZilla can connect:

Status:	Resolving address of smos-diss.eo.esa.int
Status:	Connecting to 131.176.196.6:990...
Status:	Connection established, initializing TLS...
Status:	TLS connection established, waiting for welcome message...
Response:	220 Service ready for new user.
Command:	USER removed
Response:	331 User name okay, need password for removed.
Command:	PASS ***********
Response:	230 User logged in, proceed.
Command:	OPTS UTF8 ON
Response:	200 Command OPTS okay.
Command:	PBSZ 0
Response:	200 Command PBSZ okay.
Command:	PROT P
Response:	200 Command PROT okay.
Command:	OPTS MLST size;modify;type;
Response:	200 Command OPTS okay.
Status:	Logged in
Status:	Retrieving directory listing...
Command:	PWD
Response:	257 "/" is current directory.
Command:	TYPE I
Response:	200 Command TYPE okay.
Command:	PASV
Response:	227 Entering Passive Mode (131,176,196,6,244,8)
Command:	MLSD
Response:	150 File status okay; about to open data connection.
Response:	226 Closing data connection.
Status:	Directory listing of "/" successful

I tried connecting using ftplib "by hand" and get various problems (ftplib.error_perm: 530 Please login with USER and PASS. in most cases).

The problem seems to be that ftplib always tries to use port 21. The ESA server does respond, but does not allow us to login (I get the same error with FileZilla when connecting to port 21).

I also tried specifying port 990 like this:

>>> ftp = FTP_TLS()
>>> ftp.connect("smos-diss.eo.esa.int", port=990)

But it just hangs.

This code works great: https://stackoverflow.com/questions/12164470/python-ftp-implicit-tls-connection-issue

So I'll implement this in the FTP transfer class.

[petersilva]

try active keyword in credentials.conf ?

[reidsunderland]

active/passive doesn't make a difference in this case :(

[reidsunderland]

I was a little bit wrong with how I was using FTPS.

I was using a URL starting with ftps:// which I realize now isn't supported and I should have been using ftp://example.com tls.

(I think ftps:// not working is its own bug, the examples in credentials.conf imply that it should work, and we're using it in v2.)

But that still doesn't work. It reveals the actual error though:

2024-06-25 15:38:05,051 [ERROR] 465081 sarracenia.transfer.ftp connect Unable to connect to smos-diss.eo.esa.int (user:[email protected])
2024-06-25 15:38:05,051 [DEBUG] 465081 sarracenia.transfer.ftp connect Exception details:
Traceback (most recent call last):
  File "/net/local/home/sunderlandr/sr3/sarracenia/transfer/ftp.py", line 222, in connect
    ftp = ftplib.FTP_TLS(self.host,
  File "/usr/lib/python3.10/ftplib.py", line 740, in __init__
    super().__init__(host, user, passwd, acct,
  File "/usr/lib/python3.10/ftplib.py", line 123, in __init__
    self.login(user, passwd, acct)
  File "/usr/lib/python3.10/ftplib.py", line 745, in login
    self.auth()
  File "/usr/lib/python3.10/ftplib.py", line 753, in auth
    resp = self.voidcmd('AUTH TLS')
  File "/usr/lib/python3.10/ftplib.py", line 286, in voidcmd
    return self.voidresp()
  File "/usr/lib/python3.10/ftplib.py", line 259, in voidresp
    resp = self.getresp()
  File "/usr/lib/python3.10/ftplib.py", line 254, in getresp
    raise error_perm(resp)
ftplib.error_perm: 530 Please login with USER and PASS.

[petersilva]

• perhaps the password encoding is off? it does "unquote"() of self.password...
• wild guess: I'm wondering if they still have an FTP server there... and you are connecting to the (disabled) ftp server, and not the FTPS server... perhaps because wrong port? is :990 in the ftp url?

[reidsunderland]

The password doesn't have any special characters that would require encoding, I even tried changing the password to be 100% sure.

---

Yes, they do still have an FTP server there and that is what we're connecting to on port 21 and getting the 530 error.

If I set port 990 in the URL and do not specify tls in the config, sr3 will try to connect on port 990 using plain FTP. It just hangs though.

sr3 does not support changing the port for explicit FTPS - in the FTP_TLS section, self.port is not used at all:

sarracenia/sarracenia/transfer/ftp.py

Lines 176 to 195 in 6b24f67

	try:
	expire = -999
	if self.o.timeout: expire = self.o.timeout
	if self.port == '' or self.port == None: self.port = 21
	if not self.tls:
	ftp = ftplib.FTP()
	ftp.encoding = 'utf-8'
	ftp.connect(self.host, self.port, timeout=expire)
	ftp.login(self.user, unquote(self.password))
	else:
	# ftplib supports FTPS with TLS
	ftp = ftplib.FTP_TLS(self.host,
	self.user,
	unquote(self.password),
	timeout=expire)
	ftp.encoding = 'utf-8'
	if self.prot_p: ftp.prot_p()
	# needed only if prot_p then set back to prot_c
	#else : ftp.prot_c()

(probably a bug we should fix?)

Even trying to specify the port with FTP_TLS outside of sr3, it still hangs/times out:

>>> import ftplib
>>> ftp = ftplib.FTP_TLS()
>>> ftp.encoding = 'utf-8'
>>> ftp.connect('smos-diss.eo.esa.int', port=990, timeout=5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/ftplib.py", line 162, in connect
    self.welcome = self.getresp()
  File "/usr/lib/python3.10/ftplib.py", line 244, in getresp
    resp = self.getmultiline()
  File "/usr/lib/python3.10/ftplib.py", line 230, in getmultiline
    line = self.getline()
  File "/usr/lib/python3.10/ftplib.py", line 212, in getline
    line = self.file.readline(self.maxline + 1)
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
TimeoutError: timed out

It does work with the modified code in the PR #1122. That changes the socket used by FTP_TLS to an instance of an ssl.SSLSocket and then the server allows the connection.

[reidsunderland]

This is implemented in #1122, but has to be configured manually. Peter mentioned on the PR that it would be nice to have the code automatically try to detect and remember which it needs to use (implicit vs explicit) so I'll leave this issue open.