Skip to content

config: add O_DIRECT hint to config read failures#31053

Open
nvartolomei wants to merge 1 commit into
redpanda-data:devfrom
nvartolomei:nv/config-odirect-hint
Open

config: add O_DIRECT hint to config read failures#31053
nvartolomei wants to merge 1 commit into
redpanda-data:devfrom
nvartolomei:nv/config-odirect-hint

Conversation

@nvartolomei

Copy link
Copy Markdown
Contributor

Config files are read with O_DIRECT (via Seastar's open_file_dma). On a
filesystem that doesn't support direct I/O the read fails with EINVAL, which
previously surfaced as an opaque Failure during startup: system_error (error system:22, Invalid argument) with no indication of the cause. This happens in
some environments that bind-mount the bootstrap config from an overlayfs whose
lower layer is squashfs.

This wraps the node- and bootstrap-config reads to report the file path and, on
EINVAL, a hint that the filesystem may not support O_DIRECT. The hint is scoped
to config files (often bind-mounted onto arbitrary filesystems); data
directories are validated separately by syschecks::disk, which does a
round-trip O_DIRECT check.

It also stops config_manager::load_bootstrap from swallowing non-ENOENT
errors: previously any filesystem_error (including the O_DIRECT EINVAL) was
treated as "bootstrap file absent", so the node started silently without
applying the bootstrap config. Now only a genuinely missing file (ENOENT) is
ignored; any other error fails startup.

See:

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v26.1.x
  • v25.3.x
  • v25.2.x

Release Notes

Improvements

  • Startup failures caused by reading config files on filesystems without
    O_DIRECT support (e.g. an overlayfs over squashfs, common when the config is
    bind-mounted) now report the file path and an O_DIRECT hint instead of an
    opaque "Invalid argument".

    Behavior change: if the bootstrap config file (.bootstrap.yaml) is
    present but unreadable, the node now fails to start instead of silently
    starting as though no bootstrap config existed. A missing bootstrap file is
    still ignored, as before.

Copilot AI review requested due to automatic review settings July 8, 2026 20:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves startup diagnostics when reading Redpanda config files via Seastar open_file_dma fails (notably with EINVAL on filesystems that don’t support O_DIRECT), and it hardens bootstrap config handling so unreadable-but-present bootstrap config fails startup instead of being silently ignored.

Changes:

  • Add utils::format_file_io_error(...) to include file path context and an O_DIRECT hint on EINVAL.
  • Wrap node config (redpanda.yaml) reads to throw clearer errors on read failures.
  • Update config_manager::load_bootstrap to ignore only ENOENT (missing file) and to fail startup on other read errors.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/v/utils/file_io.h Declares new helper to format file I/O errors with an O_DIRECT hint on EINVAL.
src/v/utils/file_io.cc Implements the formatted error helper using fmt.
src/v/utils/BUILD Adds @fmt dependency to the file_io library.
src/v/redpanda/application.cc Wraps node-config read failures to throw enriched error messages (including path + O_DIRECT hint on EINVAL).
src/v/cluster/config_manager.cc Stops swallowing non-ENOENT bootstrap read errors; improves error reporting for bootstrap read/parse failures.

Comment thread src/v/utils/file_io.cc
Comment thread src/v/cluster/config_manager.cc
Wrap failures reading the node and bootstrap config files with the file
path and, when the underlying syscall returns EINVAL, a hint that the
filesystem may not support O_DIRECT (the read helpers use open_file_dma).
Eases diagnosing startup failures on filesystems without O_DIRECT support.

This surfaces in some environments that bind-mount the bootstrap config
from an overlayfs whose lower layer is squashfs: reads through such an
overlay fail the O_DIRECT open with EINVAL, and the bare "Invalid
argument" gave no indication of the cause.

Behavior change: on some overlayfs-based configs the failure instead
arrives as a std::filesystem::filesystem_error, which load_bootstrap
previously treated as "bootstrap file absent", silently starting without
applying the bootstrap config. It now ignores only a genuinely missing
file (ENOENT) and surfaces any other error, so a node that previously
started despite an unreadable bootstrap file will now fail startup.

See:
  scylladb/seastar#3518
  redpanda-data#14473
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants