[Bug](fs) DFSFileSystem phantom cleanup may close Hadoop FileSystem while operations are still running

[foxtail463]

Search before asking

• I had searched in the issues and found no similar issues.

Version

branch-4.0

What's Wrong?

In branch-4.0, RemoteFSPhantomManager tracks DFSFileSystem with a PhantomReference and closes the internal org.apache.hadoop.fs.FileSystem when the DFSFileSystem object is garbage collected.

This can close the underlying Hadoop FileSystem too early. Some DFS operations may still be using the Hadoop FileSystem, RemoteIterator, FSDataInputStream, or FSDataOutputStream after the owning DFSFileSystem object is no longer strongly reachable. In that window, GC can enqueue the DFSFileSystem phantom reference, and the cleanup thread may close the Hadoop FileSystem while operations such as listStatus/listFiles, read, or write are still running.

This may cause intermittent failures in external file system access, especially when:

• DFSFileSystem is evicted from FileSystemCache by size limit
• DFSFileSystem expires after access timeout
• DFSFileSystem is created directly and not retained by the cache
• a long-running list/read/write operation only keeps the underlying Hadoop objects alive

What You Expected?

The internal Hadoop FileSystem should not be closed while any active DFS operation is still using it.

Cleanup should happen only after both conditions are true:

1. The owning DFSFileSystem is closed or garbage collected.
2. There are no active operations/streams/iterators holding the underlying Hadoop FileSystem.

How to Reproduce?

The issue is timing-sensitive and depends on GC.

General reproduction scenario:

1. Create or obtain a DFSFileSystem.
2. Start a long-running HDFS operation, for example recursive listFiles/listStatus, or create an input/output stream.
3. Drop the strong reference to the DFSFileSystem while the returned Hadoop object is still being used.
4. Trigger GC.
5. Wait for RemoteFSPhantomManager cleanup.
6. Continue using the iterator/stream.

The cleanup thread may close the internal Hadoop FileSystem, causing the still-running operation to fail with IO errors related to a closed filesystem/client.

This is less likely during normal cache hits because FileSystemCache uses a strong-value Caffeine cache, but the race is still possible after cache eviction, expiration, or direct construction outside the cache.

Anything Else?

The root cause is that phantom cleanup is tied to the lifetime of DFSFileSystem, but the actual resource users are the underlying Hadoop FileSystem and streams/iterators derived from it.

A safer design is to introduce a resource/lease layer:

• DFSFileSystemResource owns the Hadoop FileSystem.
• Each operation acquires a lease before using the resource.
• close() or phantom cleanup only marks the resource as closing.
• The Hadoop FileSystem is physically closed only after all active leases are released.

This avoids closing the Hadoop FileSystem while listFiles, reads, or writes are still in progress.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_c7a3882c3932

I checked this against current upstream/branch-4.0 (dec383f670a). The report is code-plausible and should be treated as a real branch-4.0 lifecycle bug, not only a theoretical cleanup concern.

Evidence from the branch-4.0 code:

• DFSFileSystem.nativeFileSystem() creates a Hadoop FileSystem, stores it in dfsFileSystem, creates HDFSFileOperations, and registers this with RemoteFSPhantomManager.
• RemoteFSPhantomManager maps PhantomReference<DFSFileSystem> directly to the raw Hadoop FileSystem and the cleanup thread calls fs.close() as soon as the wrapper reference is enqueued. There is no active-operation, active-stream, or active-iterator accounting before close.
• FileSystemCache uses a normal Caffeine loading cache with expireAfterAccess and maximumSize, and no removal listener that coordinates resource lifetime. Once the cache no longer strongly retains a DFSFileSystem, the phantom cleanup path is the only deferred cleanup mechanism for that wrapper.
• Several public paths obtain derived Hadoop objects from the same dfsFileSystem: listFiles() uses a RemoteIterator<LocatedFileStatus>, downloadWithFileSize() / upload() use FSDataInputStream / FSDataOutputStream, and openFile() / createFile() return streams to callers. The returned-stream paths are the clearest risk: for example HdfsInputFile.newStream() returns HdfsInputStream containing only the FSDataInputStream and path, and HdfsOutputFile.create()/createOrOverwrite() return the raw output stream; these returned objects do not visibly retain the owning DFSFileSystem.

So the failure mode described here is valid: the object used as the phantom referent is not the same object that represents active Hadoop I/O usage. If the DFSFileSystem wrapper becomes unreachable after cache eviction/expiration or direct construction while a derived stream/iterator is still active, phantom cleanup can close the underlying Hadoop FileSystem out from under that operation.

Recommended fix direction:

• Introduce a small resource owner around the Hadoop FileSystem with an active lease/reference count and a closing flag.
• Make every operation that touches the Hadoop FileSystem acquire a lease before obtaining or using Hadoop objects.
• Returned streams/iterators must hold a lease and release it on close/exhaustion/error. This is important for openFile(), createFile(), HdfsInputFile.newStream(), HdfsOutputFile.create()/createOrOverwrite(), and any RemoteIterator path.
• DFSFileSystem.close() and phantom cleanup should mark the resource closing and prevent new leases, but physically call FileSystem.close() only after the active count reaches zero.
• Add a deterministic FE unit test with a fake or mocked Hadoop FileSystem/stream/iterator that verifies phantom/close cleanup does not close the underlying FileSystem while a lease-backed stream or iterator is still open. Avoid relying only on real GC timing in the test.

Missing information that would help validate an existing production symptom:

• The exact exception and stack trace from the failed read/write/list operation.
• Whether the failure happened through Hive/HMS listing, Iceberg FileIO, backup/restore, storage vault checking, or another direct FileSystemFactory.get() caller.
• The values of max_remote_file_system_cache_num and external_cache_expire_time_seconds_after_access, plus whether the workload creates many distinct HDFS identities/properties.
• A small reproducer that keeps only a returned stream/iterator alive after dropping the DFSFileSystem/FileIO object, then forces cache eviction or GC.

I would scope the PR to branch-4.0's current filesystem implementation. Current master has a newer fe-filesystem layout and I did not find this exact RemoteFSPhantomManager path there.