[cheeeee]

Follow-up to #23445, under the same ticket (resolved, fix version
4.5.0).

Since #23445, a failed send of fetched records is passed to a handler
that re-reads the region on the scheduler, and the log directory is
offlined only if the re-read fails too. The re-read is needed because
over PLAINTEXT sendfile reads the file and writes the socket in one
call, so a disk error and a socket error are the same bare
IOException. SslTransportLayer.transferFrom does not have that
problem: it reads the file with FileChannel.read, then encrypts and
writes, so it knows which of the two failed. As discussed on #23445,
this passes that on instead of finding it out again:

• SslTransportLayer.transferFrom throws a new FileReadException, an
  IOException with the position of the read, when FileChannel.read
  or FileChannel.size fails. A ClosedChannelException, from a
  deletion or a shutdown, is rethrown as is.
• TransferableChannel.reportsFileReadFailures() says whether a
  transport does that: true for TLS, false, the default, for
  PLAINTEXT.
• FileRecords.writeTo passes a FileReadException to the handler,
  and any other failure only if the transport cannot tell the two
  apart. A failure of its own channel.size() call before the transfer
  becomes a FileReadException as well.
• LocalLog reports a FileReadException as a failure of the log
  directory without re-reading the region, on the scheduler and at most
  one report at a time per log directory, as for the re-read. Reports
  are tracked apart from re-reads, so that a re-read running in the
  same directory does not drop a known failure. Other failures are
  re-read as before.

Consequences worth stating:

• On TLS listeners a disconnect no longer re-reads the region that was
  being sent. In the test notes, consumers killed mid-fetch over TLS
  for 5 minutes made the scheduler re-read 151 MB without this change
  and nothing with it.
• On TLS listeners a read error now offlines the log directory even if
  the region could be read again by the time of a re-read, e.g. after a
  transient media error. Before, such an error was attributed to the
  socket. This is the criterion LocalLog applies to any other
  IOException.
• A failed FileChannel.size() call on the segment, over either
  transport, is now reported as a read failure without a re-read.
• PLAINTEXT listeners still re-read: sendfile cannot tell the two
  apart.
• KRaft snapshots, sent through UnalignedFileRecords, are still not
  covered.

Generated-by: Claude (Anthropic)

[cheeeee]

Test notes for 73fc46a, on trunk d18f93f.

Unit tests:

• SslTransportLayerTest, FileRecordsTest and LocalLogTest pass,
  as do :clients:test (13757 tests), :storage:test (1355 tests),
  checkstyle and spotless. 2 of 5 runs of :storage:test failed in
  TransactionsWithTieredStoreTest (a timeout, and the test JVM
  exiting), which passes alone 3 times out of 3 here and on trunk. On
  trunk 2 of 3 runs failed in tiered storage integration tests too
  (TransactionsTest, RemoteTopicCrudTest); see KAFKA-15772.
• Each new or changed test fails when the part of the change it covers
  is undone. 15 such mutations, e.g. transferFrom wrapping a closed
  file or a failed socket write, reportsFileReadFailures() returning
  false for TLS, a report sharing the re-read's per-directory slot, or
  a failed task not releasing it: each makes the test named for it
  fail.

Cluster: 3 brokers and a controller (KRaft) in Docker on a Linux 6.8
VM. Broker 1 has two log directories, one of them on a dm-dust
device. Partition t1-0 has replicas 1, 2 and 3, broker 1 leads it, and
its replica on broker 1 is in the dm-dust directory. 20000 records of
512 bytes are produced, then a console consumer reads the partition
from the beginning with an 8 MiB fetch size, over PLAINTEXT or TLS.
strace -yy on broker 1 shows which thread reads which file at which
position.

1. Persistent media error: dm-dust makes reads of the tail of the
   segment fail on the device. In every run the first read to fail is
   at position 2867200.
   • TLS, trunk: the network thread fails to read position 2867200.
     The scheduler re-reads the region from position 0, 2867200 bytes,
     and fails at 2867200. The directory goes offline.
   • TLS, this change: the same failed read on the network thread, and
     no read of the segment on the scheduler. The directory goes
     offline, with "Failed to read .../00000000000000000000.log at
     position 2867200 while writing 8388608 bytes from position 0 to a
     network channel".
   • PLAINTEXT, both: sendfile fails, the re-read on the scheduler
     fails at the same position, and the directory goes offline.
   • In all of them the leader moves to broker 2 and the consumer reads
     all 20000 records.
2. Error gone on the next read: strace makes the first read of the
   segment on each network thread of the listener fail with EIO, and
   the following reads succeed.
   • TLS, trunk: 3 failed reads, one on each network thread as the
     consumer retried, each followed by a successful re-read of 8 MiB
     on the scheduler. The directory stays online.
   • TLS, this change: the first failed read offlines the directory,
     without a re-read. The consumer reads all 20000 records from
     broker 2.
   • PLAINTEXT, both: as TLS on trunk, the directory stays online.
3. Disconnects, with a healthy disk: 3 TLS consumers read 4 partitions
   led by broker 1, about 200 MB, and are killed after 2 to 8 seconds
   and restarted, for 5 minutes, about 180 times in all.
   • trunk: 4629 reads of segment files on the scheduler, 151 MB.
   • this change: none.
   • No directory went offline in either.

With this change the failed send is still logged as a disconnect, at
DEBUG, now with the FileReadException, and the ERROR comes from
LogDirFailureChannel. The timings depend on the host, a VM under
load, so I have left them out.