FTP: non-ASCII filenames corrupted to "?" — client never negotiates UTF-8 (OPTS UTF8 ON)

[sogrbilja]

Apache NetBeans version

Apache NetBeans 31

What happened

When uploading files or folders whose names contain non-ASCII characters
(Cyrillic in my case) through a PHP project's "Remote Web Site (FTP, SFTP)"
run configuration, the names are corrupted before they reach the server.
Every non-ASCII character is replaced with a question mark.

The IDE's own FTP log shows the correct name in the outgoing command, while
the server echoes back question marks — one per character:

CWD производи
550 Can't change directory to ?????????: No such file or directory

The corruption therefore happens when the command is encoded for the control
channel, not when the name is read from disk. Names display correctly
everywhere inside the IDE.

The session contains no FEAT command and no OPTS UTF8 ON. The client never
negotiates UTF-8 (RFC 2640), so it falls back to a charset that cannot
represent these characters.

This is not only a display problem. The upload appears to succeed and no
error is shown in the IDE, but:

• MKD creates a directory literally named "?????????" on the server
• files are then uploaded into that directory under corrupted names
• the correctly named target directory is never found or used

The remote site is silently polluted with garbage directories and files
while the IDE reports success.

The server is Pure-FTPd, which supports UTF-8. WinSCP transfers the same
files to the same server with the names fully intact, so this is a
client-side issue.

Language / Project Type / NetBeans Component

No response

How to reproduce

Server requirement: any FTP server that supports UTF-8 (RFC 2640).
I used Pure-FTPd; the same behaviour should occur with vsftpd or ProFTPD
with UTF-8 enabled. Verifying with a second client (WinSCP, FileZilla with
"Force UTF-8") on the same server confirms the server side is fine.

1. Create a PHP project (File > New Project > PHP > PHP Application).
2. In Run Configuration, set "Run As" to "Remote Web Site (FTP, SFTP)".
3. Click Manage, add a new FTP connection with the server credentials,
   and confirm Test Connection succeeds.
4. In the local project's source folder, create a directory whose name
   contains non-ASCII characters, e.g. производи
5. Inside that directory, create a PHP file whose name also contains
   non-ASCII characters, e.g. весетео-офис-додатак.php
6. In the Projects window, right-click that single file and choose
   "Upload..." from the context menu.
7. Open the Output window and select the tab named after the FTP
   connection to see the protocol log.
8. Inspect the server contents with another FTP client or via the
   hosting file manager.

Result:

• The log shows 550 responses where the server echoes ????????? back
• MKD creates a directory named "?????????"
• The file is stored inside it with its own non-ASCII characters replaced
  by question marks
• The IDE reports the upload as successful, with no warning or error

Expected:

• The directory and file appear on the server with their original names

Note that the IDE itself runs under UTF-8 (see System line above), and
Cyrillic names display correctly throughout the IDE. The Windows system
ACP is 1252, but that is not the cause: the filename is intact in the
IDE's own outgoing log line and is corrupted only when encoded for the
FTP control channel. Apache Commons Net defaults to ISO-8859-1 there
regardless of platform locale, so this should reproduce on any system.

Did this work correctly in an earlier version?

No / Don't know

Operating System

Microsoft Windows 11 Pro 10.0.26200 (build 26200) 64-bit

JDK

26.0.2+10

Apache NetBeans packaging

Community provided installer

Anything else

No response

Are you willing to submit a pull request?

No

[sogrbilja]

I used Claude Opus 5 with access to the repo and it found the solution by doing this:

Root cause

FtpClient never sets a control encoding, so commons-net keeps its default of
ISO-8859-1. The control channel writer is an OutputStreamWriter over that
charset, and Java's encoder is configured with CodingErrorAction.REPLACE, so
every character outside Latin-1 is replaced with a single 0x3F — ? — before
anything reaches the socket.

That matches the log exactly: производи is 9 code points and the server
receives 9 question marks. It's a one-to-one character substitution, i.e.
charset replacement, not truncation.

Two details that make this confusing to diagnose:

• The name looks correct in the IDE's FTP log. PrintCommandListener is fed
  ProtocolCommandEvent.getMessage(), which is the Java String before the
  writer encodes it. The log is truthful, it's just observing the wrong side of
  the encoder.
• No FEAT appears in the session. commons-net only issues FEAT when UTF-8
  autodetection is enabled, and it isn't. (It never sends OPTS UTF8 ON in any
  configuration — there is no API for it.)

A tree-wide search for setControlEncoding, getControlEncoding and
setAutodetectUTF8 returns zero hits anywhere in the repository. The FTP
connection model has no encoding field either: the 16 keys in
FtpConnectionProvider (lines 40–55) contain nothing charset-related, and
FtpConfigurationPanel exposes no such control — so there is no user-side
workaround. (ENCRYPTION on line 43 is TLS/SSL selection, a false friend.)

Fix

One call in FtpClient.connect(), immediately before connect():

--- a/php/php.project/src/org/netbeans/modules/php/project/connections/ftp/FtpClient.java
+++ b/php/php.project/src/org/netbeans/modules/php/project/connections/ftp/FtpClient.java
@@ -177,6 +177,12 @@
                     ftpClient.setActivePortRange(activePortMin, activePortMax);
                 }
             }
+            // Negotiate UTF-8 on the control channel (RFC 2640) when the server
+            // advertises it in FEAT; otherwise the commons-net default of
+            // ISO-8859-1 turns every non-ASCII character into "?".
+            // Must be set before connect(): the FEAT probe and the control
+            // stream re-creation both happen inside _connectAction_().
+            ftpClient.setAutodetectUTF8(true);
             ftpClient.connect(configuration.getHost(), configuration.getPort());

Placement is not incidental. The autodetection runs inside
_connectAction_(), which the superclass invokes as the last step of
connect(). Anything below that line is too late. Worth noting for anyone
attempting a variant of this fix: calling setControlEncoding("UTF-8") after
connect() compiles, runs, changes what getControlEncoding() returns, and has
no effect on the wire — the reader and writer were already constructed with the
old charset. Autodetection is the only path in the library that rebuilds them.

One call covers every operation. FtpClient holds a single final FTPClient
field created once at line 93; store, delete, rename, listFiles,
changeWorkingDirectory, makeDirectory and SITE chmod all delegate to it.
No per-call-site changes are needed.

Why autodetect rather than forcing UTF-8

Hard-coding setControlEncoding("UTF-8") fixes this case and introduces a worse
one. ISO-8859-1 maps bytes 0x00–0xFF to U+0000–U+00FF one-to-one and
never fails in either direction, so today's client is accidentally a transparent
byte pipe: a name read from a listing on a CP1251/KOI8-R/Shift_JIS server renders
as mojibake but re-encodes to the original bytes, so CWD/DELE/RNFR against
it still succeed.

UTF-8 is not byte-transparent. Invalid sequences decode to U+FFFD, which
re-encodes to EF BF BD — different bytes — so files that are reachable today
become unreachable on those servers, and a sync could mis-target them. That's
why the bug only bites on names the IDE originates (a local file being
uploaded), and why the conditional form is the right fix.

Verification

Tested against Apache NetBeans 31, which bundles the same commons-net-3.11.1
the build declares.

Recompiling the unmodified source of FtpClient.java with JDK 21 reproduced
the shipped bytecode byte-for-byte, confirming a clean baseline. After the edit,
the only semantic delta in the class is four instructions
(aload_0 / getfield ftpClient / iconst_1 / invokevirtual setAutodetectUTF8);
the three inner classes differ solely in LineNumberTable, shifted by the six
added lines.

Driven through the same jar the IDE loads, against a mock server that advertises
UTF8 in FEAT, capturing the raw bytes of the MKD argument:

	control encoding	FEAT	bytes on the wire
before	ISO-8859-1	not sent	3F 3F 3F 3F 3F 3F 3F 3F 3F (9)
after	UTF-8	sent	D0 BF D1 80 D0 BE D0 B8 D0 B7 D0 B2 D0 BE D0 B4 D0 B8 (18)

The second row is exactly the correct UTF-8 encoding of производи; the first
reproduces the reported log. Confirmed working against the real Pure-FTPd server
from the original report.

Expected side effect

Every connection now performs a FEAT exchange, which is visible in the IDE's
FTP output window. A server that rejects FEAT is handled correctly
(hasFeature returns false, client stays on ISO-8859-1), but the negative reply
will be printed as an error line, since PrintCommandListener routes negative
replies to io.getErr(). Suppressing it isn't straightforward — the probe
happens inside connect(), so the remove/re-add listener trick used by
preventNoOperationTimeout() can't wrap it without also hiding the greeting and
login.

Not addressed here: servers that advertise UTF8 but require an explicit
OPTS UTF8 ON. commons-net has no API for it; it would need a manual
sendCommand("OPTS", "UTF8 ON") after login, and shouldn't be added
speculatively since servers that don't know the command will log a 500/502.

Two notes before you post:

• I'd trim the "Verification" section if the maintainers prefer terse issues — the bytecode-diff detail matters most in a PR, less in an issue thread.
• The diff applies to master as of this checkout. If you want, I can turn this into an actual PR branch — though this checkout isn't a git repo, so that'd mean cloning fresh first.